# Simulator — implementation specification Working spec for the simulator described in [`NORTH_STAR.md`](./NORTH_STAR.md). Read that first. The observable surface this spec commits the sim to reproduce lives in [`OBSERVABILITY.md`](./OBSERVABILITY.md); every obligation below cites the subsection it discharges. ## Table of contents 1. Scope and non-goals 2. Engine 3. Runtime facade 4. Network model 5. Hosted entities 6. Recording format 7. Replay 8. Observability 9. Calibration loop 10. Glossary --- ## 1. Scope and non-goals **In scope.** - The network the node-under-test touches: arbitrary topology, per- link physics (bandwidth, delay, loss, MTU), NAT and middlebox behavior, reachability, partitions, healing. - The transports that run over it (iroh today; others as added). Transports are first-class: their internal state — connection cache, NAT bindings, congestion control — must be observable on the same schema as production. See OBSERVABILITY §3.6, §4.3, §4.6. - The hosted peers: any node our code can run is sim-native; any node we cannot rebuild (third-party binary, foreign version) is accommodated via the opaque-binary escape hatch (§5.2). - The shared infrastructure those peers depend on: relays, signaling, DNS, vast.ai-style host metadata (§4.4, §4.7, OBSERVABILITY §3.10). - A deterministic engine (§2) so calibration has a stable baseline. - A recording format identical to the production recording format (§6, OBSERVABILITY §3). **Out of scope.** - The algorithms under test. SWIM and whatever swactor hosts run as the same code paths in sim and prod. The sim does not stub them. - The CPU / memory cost of running each peer's logic. We assume peers fit; the host-budget model (§4.8) only accounts for what is visible to the peer from the host's perspective. - Anything from OBSERVABILITY §5 ("Out of scope"). The sim is not obligated to model kernel system calls, GPU compute internals, allocator behavior, or CPU performance counters. If that list shrinks, this scope grows accordingly. - One-time configuration tooling. Topology specs are inputs; how they are authored is outside this spec. **Non-goal: faster than wall-clock.** Speed is a nice property of discrete-event simulation, not a requirement. The bar is *fidelity*, measured against the calibration loop (§9). A run that completes in half the time but diverges on a tier-2 observation is worse than a run that takes twice as long but matches every distribution. ## 2. Engine ### 2.1 Determinism contract Same input, same output, byte for byte. "Input" means: the topology spec, the link policies, the peer set, the workload, and the seed. "Output" means: every record emitted to the recording (§6), in the same order, with the same values. Determinism extends to: - Per-node `monotonic_seq` counters (OBSERVABILITY §3.3). - `snapshot_id` strings (§3.4 there). - Trace IDs once §4.2 lands (the deterministic RNG is the source). - Any timing field driven by virtual time (§2.2). Non-determinism that would cross this boundary — `SystemTime::now()`, `tokio::spawn` ordering, `HashMap` iteration order seeded from the process — is removed at the runtime-facade layer (§3) before it can leak into peer code. The list of sources we close off lives in §2.4. The contract is enforced by a divergence detector (§2.5). It is *also* a calibration assumption: when a sim run and a prod run diverge in distribution, we need to know that the sim run, at least, is reproducible from its seed. If the sim is also non-deterministic, calibration loses its baseline. ### 2.2 Virtual time and the event loop The engine drives a discrete-event loop over a virtual-time priority queue. Every action that takes time — a packet in flight, a timer firing, a sleep — yields to the engine, which advances virtual time to the next scheduled event and resumes the relevant fiber. Virtual time is the basis of every `wall_ms` field in the recording (OBSERVABILITY §3.1, §3.3, §3.4). Sim-run `wall_ms` values are exactly virtual-time millis; the §3.5 clock-alignment math sees zero drift, which is the correct answer (and the parity check should see zero drift on sim runs and bounded drift on prod runs). Peer code does not observe virtual time directly. It calls the runtime facade (§3), whose sim implementation hooks into the event loop. From peer code's perspective, it ran in time — the time was just controlled by the engine rather than the kernel. ### 2.3 Scheduler and executor The engine owns a single-threaded executor that drives every fiber in the run. Fibers correspond to peers, transport tasks, SWIM ticks, snapshot timers, etc. The executor's pick order is deterministic (sorted by virtual-time priority, then by fiber-id tiebreaker). When two events are scheduled at the same virtual tick, the tiebreaker is **(node_id, fiber_id, event_seq)**. Same input ⇒ same tiebreaker resolution. No part of the executor ever falls back to wall-clock ordering or thread-local randomness. The OBSERVABILITY §4.5 actor-mailbox surface flows out of the executor: per-actor mailbox depth and per-task age are the executor's own state, exposed to the recording on every snapshot. ### 2.4 Sources of non-determinism Each source below has a defined sim-side handling. Anything not on this list that leaks non-determinism is a bug. | Source | Sim handling | |---|---| | `SystemTime::now` | Facade-routed to virtual time. | | `std::time::Instant` | Facade-routed to virtual time. | | `tokio::spawn` ordering | Executor (§2.3) imposes deterministic order. | | `HashMap` / `HashSet` iteration | Replaced with deterministic-iteration containers behind the facade. | | Channel select races | Engine resolves via fiber-id tiebreaker. | | OS UDP / TCP socket APIs | Routed to the simulated network (§4). | | DNS resolution | Routed to the simulated DNS (§4.4). | | `getrandom` / `rand::thread_rng` | Per-node deterministic RNG seeded from `(run_seed, node_id, stream_label)`. | | File-system access | Per-node sandbox rooted at `{run_dir}/{node_id}/`; reads of system files (`/proc/net/udp`, `/etc/resolv.conf`) routed to the simulated host model (§4). | | Process spawn | Sim-native: a new fiber, not a new OS process. Opaque-binary case is §5.2. | | `std::env::var` | Routed to per-node env table from the topology spec. | | Iterator orderings that depend on `Box` vtable addresses | Avoided at the facade boundary; any helper that returns trait objects has a defined-order wrapper. | ### 2.5 Replay and divergence detection Running the engine with the same `(spec, seed)` twice produces identical records (§2.1). Running it with the same `(spec, seed)` and a *different* peer-code version produces records that diverge at the first observable difference; the engine detects this by hashing each emitted record's bytes and comparing against the stored hash chain (if a baseline run is supplied) or storing one (if a baseline is not). Divergence is a structured `Error` event plus a non-zero exit; the bundle is preserved for inspection. Divergence detection is the engine-level mechanism. The *calibration*-level mechanism (sim vs. prod) is §9; the two are distinct because sim-vs-prod is statistical, while sim-vs-sim is exact. ## 3. Runtime facade ### 3.1 Surface The facade is the narrow trait family that peer code calls instead of the raw `std::*` / `tokio::*` / OS APIs. Its surface is *exactly* what the peer code needs and nothing more. Concretely: - A clock (`now`, `sleep_until`, `interval`). - An UDP socket (`bind`, `send_to`, `recv_from`). - A QUIC endpoint (iroh's `Endpoint` shape; not the OS UDP). - A DNS resolver (`resolve_a`, `resolve_aaaa`). - A spawn primitive (`spawn`, `spawn_local`). - A randomness source (`get_rng()` returning a streamed RNG). - A file-system handle (`open_read`, `open_write`) scoped to the per-node sandbox. - An env-var read. - A process metadata read (`hostname`, `pid`). Anything the recording observes (OBSERVABILITY §3) is observable on the facade — either because the facade *emits* the corresponding event itself (e.g., `MessageSent`), or because the subsystem calling the facade emits it. ### 3.2 Prod implementation Each method delegates to the obvious real backend: tokio runtime, std net stack, libc DNS, etc. The implementation lives in the existing crates (`crates/distribution` and friends). Production code paths never see a sim-aware branch — the swap is at the facade-trait boundary, not at every call site. ### 3.3 Sim implementation Each method delegates to the engine. `now` reads virtual time; `sleep_until` schedules a wake-up event; UDP send hands a packet to the network model; DNS reads the simulated zone; `spawn` registers a fiber with the executor; `get_rng` returns a stream keyed by `(node_id, stream_label)`. The sim implementation owns the runtime facade for *all* sim-native hosts in the run, in one OS process. Opaque-binary hosts (§5.2) get their own runtime facade implementation that virtualizes the boundary at the syscall level instead. ### 3.4 Build-time switching The active implementation is chosen by a single Cargo feature. No runtime branching, no `if cfg!(sim)` sprinkled through peer code. Two binaries are built from the same source: a prod binary linked against the prod facade, and a sim driver that links the sim facade and exposes the topology-spec entry point. Peer code does not need a `Cargo.toml` change to be used in the sim — being written against the facade trait is sufficient. ## 4. Network model The network is what peer code talks through. Its job is to be *indistinguishable* from a real network on every channel the recording observes (OBSERVABILITY §3.2, §3.3, §3.6, §3.7, §4.1). ### 4.1 Topology A topology is a directed graph of nodes and links plus a set of shared-infrastructure entities (relays, DNS zones, vast.ai-style host metadata sources). Topologies are declarative — they are an input to the engine, not assembled imperatively at run time. Graphs are arbitrary. The sim does not enumerate named topologies; it accepts whatever the spec describes. The spec lists each node's role, its peer set, and the link to each peer (or to the shared infrastructure). Asymmetric links are first-class: `bandwidth(A→B) ≠ bandwidth(B→A)`, and `loss(A→B) ≠ loss(B→A)`, without special-casing. Partition and heal are topology mutations applied at scheduled virtual times. Each mutation is a record on the recording stream (emitted as a sim-only `Custom` event, OBSERVABILITY §6 rule 4) so the post-processor can correlate it with peer-observed events. ### 4.2 Link physics Each directed link carries: - `bandwidth_bps` — capacity. Excess packets queue (per the queue policy) or drop (per the drop policy). - `one_way_delay_ms` — base latency. Distributions (lognormal, etc.) are allowed; the spec declares which. - `jitter_ms` — per-packet delta on top of the base. - `loss_rate` — independent or correlated; the spec declares which. - `mtu` — packets above MTU are fragmented per the L3 / L4 rules. - `queue_policy` — FIFO / SFQ / FQ-CoDel; spec-declared. These are the *applied* values; OBSERVABILITY §4.8 commits the sim to also emit them on every snapshot in the `links_applied` block (sim-only field) so a replay knows what conditions it ran under. The *observed* values flow out of the same passive-estimation code prod uses, on the `links` block (sim and prod). Per-link parameters can vary over time per the spec. Time-varying parameters are also part of the topology declaration; they are applied at scheduled virtual times, the same way partitions are. ### 4.3 L3 — IP The sim addresses every host with one or more IPv4 and IPv6 addresses. The address assignment is in the topology spec. Per-host interface metadata (name, MTU, up/down) is read by the host scrape (OBSERVABILITY §3.8); the sim populates it from the spec. IP-level behavior modelled: - Per-packet TTL decrement at every hop. Packets with TTL=0 are dropped and the drop is emitted as a `WirePacket` (§4.1 in OBSERVABILITY) with outcome `dropped_ttl`. - ECN bits passed through (no AQM model unless the spec adds one). - Fragmentation when MTU is exceeded. Reassembly at the destination, with the standard timeout. Reassembly failures observable. - Source-address validation. Unspoofable per default; the spec can override per-link to allow spoofing for adversarial scenarios. IP-level behavior *not* modelled: ICMP error generation beyond the two cases needed for path-MTU discovery (`Fragmentation Needed`) and dead-peer detection (`Port Unreachable`). The sim emits both when the corresponding condition occurs. ### 4.4 NAT and middleboxes NATs are first-class entities in the topology, with type drawn from {Full-Cone, Restricted-Cone, Port-Restricted-Cone, Symmetric}. Each NAT maintains a binding table; bindings expire per the spec's keepalive-timeout setting and are refreshed by outbound packets. OBSERVABILITY §4.6 commits the sim to expose the binding table on every snapshot (`nat_bindings` per peer). The sim's authoritative table is the source; the recording shows the binding from the hosted side. NAT-binding refresh interval and refresh count match real-world dynamics under each modelled NAT class. Beyond NAT, the spec can place middleboxes that drop packets by DPI signature (used for adversarial tests of relay traffic). Middlebox drops are observable as `WirePacket` outcomes. ### 4.5 L4 — UDP, TCP, QUIC **UDP** is a thin wrapper over the L3 model: send-to / recv-from, no flow control, drops surfaced as silent loss. Probes (OBSERVABILITY §3.9) ride on this layer. **TCP** is reserved for shared-infrastructure protocols (e.g., the collector HTTP sink). It is modelled as a faithful loss-recovery implementation, but its connection state is *not* exposed in tier-2 because production iroh does not use it for peer traffic. **QUIC** is the load-bearing L4. The sim runs the same QUIC implementation iroh runs in prod (quinn), linked against the sim UDP. The QUIC handshake, congestion control, and stream multiplexing behave exactly as they do in prod because the *implementation* is the same; only the underlying UDP is the sim's. This is what makes the OBSERVABILITY §4.3 (congestion-control state) and §4.7 (handshake records) parity bars achievable: the data structures reporting on each are the real ones. ### 4.6 Endpoint stack Each sim host runs the full iroh stack (or its equivalent for non-iroh transports): MagicSock, discovery, NodeMap, connection cache. These are *not* re-implemented; they are the production code linked against the sim runtime facade. That is the only way their internal state — the state surfaced in OBSERVABILITY §3.6 (`Tier2IrohState`) — matches prod by construction. Per-node configuration (relay set, discovery providers) is from the topology spec. The DNS resolver used by iroh's relay-URL lookup is the simulated one (§4.4 in this spec — *not* the OS resolver — the simulated host model). ### 4.7 Clocks Each host's clock is virtual time plus a per-host skew and drift configured by the spec. Skew is a constant offset; drift is a rate (ppm). Clock samples (OBSERVABILITY §3.5) capture the relationship between each host's clock and the collector's ("virtual UTC"), so the post-processor's alignment code does the same work on sim and prod bundles. A host can experience a clock jump (e.g., NTP step) at a scheduled virtual time. The jump is a topology mutation; it is observable because every event after it carries the new `wall_ms` while `monotonic_seq` (OBSERVABILITY §3.3) keeps going up. ### 4.8 Host budget Each host has a modelled budget: CPU shares, memory bytes, open-fd count. These map to the OBSERVABILITY §3.11 process-stats fields. The sim does not enforce the budget against peer code (peer code runs as fibers); instead, the budget is tracked indirectly: - CPU time per fiber is approximated from the number of times the fiber yields per virtual-time unit, weighted by the per-host CPU share. The result feeds the `cpu_ms` field on the process snapshot. - Memory is tracked from the peer code's own allocations against the per-node allocator wrapper. RSS is reported as the high-water mark seen since the previous snapshot. VmSize is the current live byte count. - Open-fd count is the number of sim-side sockets currently bound by the node (UDP, QUIC, TCP, plus simulated files). This is the imperfect-but-honest implementation NORTH_STAR §"What it models" calls for: the recording surface is populated; the underlying numbers are approximations, marked as such in the post-processor's documentation; calibration against prod will tell us when the approximations are too loose. ## 5. Hosted entities ### 5.1 Sim-native hosts A sim-native host is a peer whose code we own and rebuild against the sim runtime facade. It is the same binary in spirit as the prod binary — same crates, same code paths — linked against a different runtime crate. A run hosts arbitrarily many sim-native hosts. Each has its own address space, its own runtime facade instance, its own diagnostics aggregator (OBSERVABILITY §3). The engine routes I/O between them through the network model. Sim-native hosts cover all of OBSERVABILITY §3's parity bar because they are running the production diagnostics stack — every event and snapshot is emitted by the same code that emits them in prod. ### 5.2 Opaque-binary hosts via virtualized I/O When we need to host a peer whose code we cannot rebuild against the facade — a third-party node, a peer running an old version, a kernel that handles a packet a specific way — the sim drives the real binary through a virtualized I/O boundary. The binary runs as a real OS process; its socket calls go through a shim (LD_PRELOAD-style on Linux, or a network-namespace + tun device on platforms where syscall interception is brittle); the shim routes packets to the engine the same way sim-native UDP does. From the binary's perspective, it sees a real kernel, real sockets, real time. The engine controls what crosses the wire. Records emitted by the opaque binary (if any) are read off its own log output and translated into the recording schema as best as the adapter can manage; gaps appear as `None` fields, marked. Opaque-binary hosts are an escape hatch, not the default. The parity bar applies fully to sim-native hosts; opaque-binary hosts have inherently incomplete recording (we cannot fully instrument a binary we did not build). ### 5.3 Lifecycle Every host has a defined lifecycle in the topology spec: - `start_at_ms` — virtual time at which the host's runtime is brought up. Boot order is deterministic. - `restart_at_ms[]` — scheduled restarts. `boot_sequence` (OBSERVABILITY §3.1) increments per restart. - `stop_at_ms` — clean shutdown. - `crash_at_ms` — uncontrolled exit. Distinct from `stop` because no draining occurs. Each lifecycle event is a topology mutation; each is observable as a `Custom` event on the recording (the host that experienced the event reports it through its own diagnostics, if reachable; the engine reports it as a sim-only record otherwise). ## 6. Recording format ### 6.1 Wire-level trace The wire-level trace is the per-packet stream described in OBSERVABILITY §4.1. The sim emits one record per packet in either direction at every modelled hop. Records are written to `{run_dir}/wire/{node_id}.ndjson` and tarred into the bundle. Wire-level capture is gated by a per-run setting. Off by default because the data volume is high; on for calibration runs and for any run where the §4.1 OBSERVABILITY parity bar is being checked. When a payload-bytes capture is requested, the bytes live next to the trace as a separate file (`wire-payloads/...`) so the trace itself stays small enough to grep through. ### 6.2 Internal-observation trace This is the production recording surface itself — OBSERVABILITY §3 in its entirety. The sim writes the same files to the same paths the production collector does, under `{collector_root}/{run_id}/{node_id}/`: ``` boot.json snapshots/{snapshot_id}.json events/{batch_seq}.json finalize.json ``` A sim run produces a bundle byte-for-byte indistinguishable from a prod bundle of the same workload (modulo the sim-only fields flagged in OBSERVABILITY §3.11 / §6.5 below). That isomorphism is the parity bar. ### 6.3 Schema versioning The schema is versioned. Every record carries a `schema_version` field at the top of its envelope. The post-processor accepts any version it knows about. Schema upgrades are documented in the diagnostics crate's CHANGELOG; the sim and prod sides upgrade together (NORTH_STAR §"Calibration is ongoing"). Adding a field to a record is a minor version bump; removing or renaming is major. Old bundles remain readable forever; the post-processor handles back-compat. The sim's emitted version matches whatever version of the diagnostics crate it is linked against, the same as prod. ### 6.4 Storage layout Bundles are tarballs (`.tar.gz`) at `{collector_root}/bundles/{run_id}.tar.gz`. The internal layout matches `DIAGNOSTICS_PLAN.md` §"Bundle Format": ``` {run_id}/ MANIFEST.json orchestrator/{boot, snapshots/, events/, finalize}.json stage-N/{boot, snapshots/, events/, finalize}.json collector.log summary.md # post-processor output, optional ``` Sim runs additionally write: ``` {run_id}/ sim/ spec.toml # the topology spec used seed # the run seed wire/{node_id}.ndjson (if wire-level capture is on) links_applied.json # ground-truth link params over time mutations.log # partition/heal/restart events ``` The `sim/` subtree is sim-only and is ignored by the parity diff. It is what enables a sim-vs-sim replay (§2.5) and what gives a calibration run the ground truth to compare against. ### 6.5 Sim-only fields Sim runs emit a small number of fields prod cannot: ground-truth applied link parameters (`links_applied`, OBSERVABILITY §4.8), sim-internal scheduler decisions (OBSERVABILITY §6 rule 4), RNG-state traces. These are written to `sim/` (§6.4) and are never read by code that also runs in prod. The node-under-test cannot observe any of them. ## 7. Replay ### 7.1 Environment restoration Replay takes a recording (produced by a prod run, a sim run, or either one with a partial bundle) and plays back the *environment* the recorded run was in. Concretely: - Topology is reconstructed from the recording. NORTH_STAR §"Two modes, one engine" demands this be feasible from a real-world bundle, so the per-link parameters used by the sim must be derivable from what the recording exposes. Where the recording exposes only `links` (observed), the sim seeds its `links_applied` with the observed values and lets the parity diff quantify any drift. Where the recording exposes `links_applied` (sim-origin), the sim adopts them exactly. - Peer set is reconstructed from the boot identity blocks across the bundle. - Mutation schedule (partition, heal, restart) is reconstructed from the `Custom` events the original run emitted. - Shared infrastructure (relays, DNS, vast.ai metadata) is reconstructed from whatever the bundle's host-scrape and identity blocks recorded. ### 7.2 Peer re-execution Per NORTH_STAR §"Two modes, one engine": peers in a replay are **re-executed peer code**, not stubs driven by recorded outputs. The recording carries the *environment*; the peer logic that runs against that environment is whatever version is under test in the replay. That is how a replay tests a candidate change — by exposing the same world to new code. When the recording lacks information needed to drive a peer faithfully (a third-party peer whose code we do not have), that peer is hosted via the opaque-binary path (§5.2). If the original recording was prod and the peer was iroh-stack swactor, the sim-native path applies. ### 7.3 Divergence handling A replay diverges when a re-executed peer makes a different choice from what the recording shows the original peer made. Divergence is *expected*: that is the point of replay — to see whether the new code behaves differently against the same conditions. The replay records the new behavior at the same level of detail (§6) and the post-processor's diff mode is what surfaces the delta. Divergence is not an error; only when the divergence is *outside the parity envelope* (§9.3) is it a sim bug rather than an algorithm-change observation. ## 8. Observability ### 8.1 Shared sinks with prod The sim's diagnostics aggregator (the one running inside each sim-native host) is the *production* diagnostics aggregator from `crates/distribution/src/diagnostics`. It is linked against an HTTP sink whose target is the sim's collector — itself a real binary (also from `crates/distribution`), brought up by the engine on a virtual host at the start of every sim run. The collector writes the same bundle layout (§6.4). The post-processor reads it the same way. The "shared" in "shared sinks" means literal: the same code runs on both sides of the sim/prod boundary, so the parity bar is enforced by construction for every record kind enumerated in OBSERVABILITY §3. ### 8.2 Sim-only diagnostics The sim emits a small set of additional records the node-under-test cannot observe (OBSERVABILITY §6 rule 4): scheduler decision logs, exact virtual-time tick streams, RNG-state traces, per-link applied parameters (§6.5). They are written to the `sim/` subtree of the bundle and are read only by the engine's own debugging tooling, never by code that also runs in prod. The discipline: any time we are tempted to add a sim-only field that the node-under-test *can* observe, we have to either (a) commit to making prod observe it too and put it in OBSERVABILITY §3 or §4, or (b) keep it strictly under §8.2. There is no third option. ## 9. Calibration loop ### 9.1 Procedure Calibration is the act of measuring sim-vs-prod observable equivalence. The loop is concrete: 1. Run a workload in prod. Collect the bundle. 2. Extract the topology + mutation schedule from the bundle (§7.1). 3. Run the *same* peer-code version in the sim against that reconstructed environment. Collect the bundle. 4. Diff the two bundles, record kind by record kind, against the parity bars defined in OBSERVABILITY §3 / §4. 5. Anywhere the distributions diverge beyond the noise floor (§9.2), file a sim bug. Anywhere prod is missing a record the sim emits or vice versa, file a recording bug or a sim bug (whichever direction the asymmetry runs — see OBSERVABILITY §6 rule 3). The loop runs continuously, not as a one-time validation. Every real deployment is evidence; every divergence is an issue. ### 9.2 Noise-floor estimation Prod runs are inherently noisy: real link latencies jitter, real kernels make scheduling decisions we cannot reproduce, real clocks drift. Calibration cannot demand sim records to be byte-identical with prod records — it demands them to be *statistically indistinguishable* (NORTH_STAR §"The parity bar"). The noise floor is estimated from prod-vs-prod variance: two prod runs of the same workload, under as-matched-as-possible conditions, diff each other. The variance of that diff defines the noise floor for each numeric field; a sim-vs-prod divergence within that envelope is not a bug. A sim-vs-prod divergence beyond it is. Establishing the noise floor is an ongoing project — every new metric needs its own estimate. The post-processor's calibration report ships per-metric noise-floor numbers alongside the sim-vs-prod diff. ### 9.3 Parity metrics Per OBSERVABILITY's call for distributional parity, the calibration report tracks for each metric: - **Presence parity** — does the sim emit a record whenever prod does (and vice versa)? Boolean per record kind. - **Mean parity** — is `mean_sim ≈ mean_prod` within the noise floor of the means? - **Tail parity** — is `p99_sim ≈ p99_prod` within the noise floor of the tail? This is the bar OBSERVABILITY calls out explicitly; mean parity alone is not enough. - **Causal parity** — does the sim emit events in the same order prod does, conditioned on the same upstream causes? Tied to the causal-trace-IDs surface (OBSERVABILITY §4.2). A pass is "every metric within noise floor on every parity dimension." A fail names the metric and dimension. The post-processor's calibration report is structured exactly that way; "did calibration pass" reduces to "did any line in the report come back red." ## 10. Glossary - **Aggregator** — per-process recording state owner. Owns the reachability log, event ring, snapshot assembler, sink. Same type in sim and prod. - **Bundle** — the tarball at end-of-run containing every record emitted during the run. See §6.4. - **Calibration** — measurement of sim-vs-prod observable equivalence. See §9. - **Collector** — the HTTP server every aggregator POSTs to. Centralized per-run target; writes the bundle. - **Engine** — the discrete-event simulator core. Owns virtual time, the executor, the network model, the topology graph. See §2. - **Facade** — the trait surface peer code uses for all potentially-non-deterministic operations. Two implementations (prod, sim). See §3. - **Fiber** — an executor-managed unit of concurrency. One fiber per peer task; per-fiber scheduling is deterministic. See §2.3. - **Mutation** — a scheduled change to the topology during a run (partition, heal, restart, link-parameter change). Recorded as a `Custom` event. See §4.1. - **Noise floor** — the variance between two prod-vs-prod runs of the same workload. Sets the bar that sim-vs-prod divergence is measured against. See §9.2. - **Opaque binary** — a hosted entity whose code we do not own and cannot rebuild against the facade. Driven via syscall interception. See §5.2. - **Parity bar** — the requirement, per OBSERVABILITY, that every record kind on the observable surface match between sim and prod. See OBSERVABILITY §6. - **Recording** — the on-disk artifact of a run's observable surface. Same format in sim and prod. See §6. - **Replay** — re-execution of peer code against an environment reconstructed from a recording. See §7. - **Run** — one execution of the engine from boot to finalize. Identified by `run_id` (in the bundle) and `run_seed` (input). - **Sim-native host** — a peer whose code is linked against the sim runtime facade. See §5.1. - **Topology spec** — the input declaration of nodes, links, shared infrastructure, mutations, and run parameters. See §4.1. - **Virtual time** — the engine's authoritative clock. Wall-clock `wall_ms` fields in the recording are populated from it on sim runs. See §2.2.