swactor/crates/simulation/SPEC.md

# Simulator — implementation specification

Working spec for the simulator described in [`NORTH_STAR.md`](./NORTH_STAR.md).
Read that first. The observable surface this spec commits the sim to
reproduce lives in [`OBSERVABILITY.md`](./OBSERVABILITY.md); every
obligation below cites the subsection it discharges.

## Table of contents

1. Scope and non-goals
2. Engine
3. Runtime facade
4. Network model
5. Hosted entities
6. Recording format
7. Replay
8. Observability
9. Calibration loop
10. Glossary

---

## 1. Scope and non-goals

**In scope.**

- The network the node-under-test touches: arbitrary topology, per-
  link physics (bandwidth, delay, loss, MTU), NAT and middlebox
  behavior, reachability, partitions, healing.
- The transports that run over it (iroh today; others as added).
  Transports are first-class: their internal state — connection
  cache, NAT bindings, congestion control — must be observable on
  the same schema as production. See OBSERVABILITY §3.6, §4.3, §4.6.
- The hosted peers: any node our code can run is sim-native; any
  node we cannot rebuild (third-party binary, foreign version) is
  accommodated via the opaque-binary escape hatch (§5.2).
- The shared infrastructure those peers depend on: relays,
  signaling, DNS, vast.ai-style host metadata (§4.4, §4.7,
  OBSERVABILITY §3.10).
- A deterministic engine (§2) so calibration has a stable baseline.
- A recording format identical to the production recording format
  (§6, OBSERVABILITY §3).

**Out of scope.**

- The algorithms under test. SWIM and whatever swactor hosts run as
  the same code paths in sim and prod. The sim does not stub them.
- The CPU / memory cost of running each peer's logic. We assume
  peers fit; the host-budget model (§4.8) only accounts for what is
  visible to the peer from the host's perspective.
- Anything from OBSERVABILITY §5 ("Out of scope"). The sim is not
  obligated to model kernel system calls, GPU compute internals,
  allocator behavior, or CPU performance counters. If that list
  shrinks, this scope grows accordingly.
- One-time configuration tooling. Topology specs are inputs; how
  they are authored is outside this spec.

**Non-goal: faster than wall-clock.** Speed is a nice property of
discrete-event simulation, not a requirement. The bar is *fidelity*,
measured against the calibration loop (§9). A run that completes in
half the time but diverges on a tier-2 observation is worse than a
run that takes twice as long but matches every distribution.

## 2. Engine

### 2.1 Determinism contract

Same input, same output, byte for byte. "Input" means: the topology
spec, the link policies, the peer set, the workload, and the seed.
"Output" means: every record emitted to the recording (§6), in the
same order, with the same values. Determinism extends to:

- Per-node `monotonic_seq` counters (OBSERVABILITY §3.3).
- `snapshot_id` strings (§3.4 there).
- Trace IDs once §4.2 lands (the deterministic RNG is the source).
- Any timing field driven by virtual time (§2.2).

Non-determinism that would cross this boundary — `SystemTime::now()`,
`tokio::spawn` ordering, `HashMap` iteration order seeded from the
process — is removed at the runtime-facade layer (§3) before it can
leak into peer code. The list of sources we close off lives in §2.4.

The contract is enforced by a divergence detector (§2.5). It is
*also* a calibration assumption: when a sim run and a prod run
diverge in distribution, we need to know that the sim run, at
least, is reproducible from its seed. If the sim is also
non-deterministic, calibration loses its baseline.

### 2.2 Virtual time and the event loop

The engine drives a discrete-event loop over a virtual-time
priority queue. Every action that takes time — a packet in flight,
a timer firing, a sleep — yields to the engine, which advances
virtual time to the next scheduled event and resumes the relevant
fiber.

Virtual time is the basis of every `wall_ms` field in the recording
(OBSERVABILITY §3.1, §3.3, §3.4). Sim-run `wall_ms` values are
exactly virtual-time millis; the §3.5 clock-alignment math sees
zero drift, which is the correct answer (and the parity check
should see zero drift on sim runs and bounded drift on prod runs).

Peer code does not observe virtual time directly. It calls the
runtime facade (§3), whose sim implementation hooks into the event
loop. From peer code's perspective, it ran in time — the time was
just controlled by the engine rather than the kernel.

### 2.3 Scheduler and executor

The engine owns a single-threaded executor that drives every fiber
in the run. Fibers correspond to peers, transport tasks, SWIM
ticks, snapshot timers, etc. The executor's pick order is
deterministic (sorted by virtual-time priority, then by fiber-id
tiebreaker).

When two events are scheduled at the same virtual tick, the
tiebreaker is **(node_id, fiber_id, event_seq)**. Same input ⇒
same tiebreaker resolution. No part of the executor ever falls
back to wall-clock ordering or thread-local randomness.

The OBSERVABILITY §4.5 actor-mailbox surface flows out of the
executor: per-actor mailbox depth and per-task age are the
executor's own state, exposed to the recording on every snapshot.

### 2.4 Sources of non-determinism

Each source below has a defined sim-side handling. Anything not on
this list that leaks non-determinism is a bug.

| Source | Sim handling |
|---|---|
| `SystemTime::now` | Facade-routed to virtual time. |
| `std::time::Instant` | Facade-routed to virtual time. |
| `tokio::spawn` ordering | Executor (§2.3) imposes deterministic order. |
| `HashMap` / `HashSet` iteration | Replaced with deterministic-iteration containers behind the facade. |
| Channel select races | Engine resolves via fiber-id tiebreaker. |
| OS UDP / TCP socket APIs | Routed to the simulated network (§4). |
| DNS resolution | Routed to the simulated DNS (§4.4). |
| `getrandom` / `rand::thread_rng` | Per-node deterministic RNG seeded from `(run_seed, node_id, stream_label)`. |
| File-system access | Per-node sandbox rooted at `{run_dir}/{node_id}/`; reads of system files (`/proc/net/udp`, `/etc/resolv.conf`) routed to the simulated host model (§4). |
| Process spawn | Sim-native: a new fiber, not a new OS process. Opaque-binary case is §5.2. |
| `std::env::var` | Routed to per-node env table from the topology spec. |
| Iterator orderings that depend on `Box<dyn Trait>` vtable addresses | Avoided at the facade boundary; any helper that returns trait objects has a defined-order wrapper. |

### 2.5 Replay and divergence detection

Running the engine with the same `(spec, seed)` twice produces
identical records (§2.1). Running it with the same `(spec, seed)`
and a *different* peer-code version produces records that diverge
at the first observable difference; the engine detects this by
hashing each emitted record's bytes and comparing against the
stored hash chain (if a baseline run is supplied) or storing one
(if a baseline is not). Divergence is a structured `Error` event
plus a non-zero exit; the bundle is preserved for inspection.

Divergence detection is the engine-level mechanism. The
*calibration*-level mechanism (sim vs. prod) is §9; the two are
distinct because sim-vs-prod is statistical, while sim-vs-sim is
exact.

## 3. Runtime facade

### 3.1 Surface

The facade is the narrow trait family that peer code calls instead
of the raw `std::*` / `tokio::*` / OS APIs. Its surface is
*exactly* what the peer code needs and nothing more. Concretely:

- A clock (`now`, `sleep_until`, `interval`).
- An UDP socket (`bind`, `send_to`, `recv_from`).
- A QUIC endpoint (iroh's `Endpoint` shape; not the OS UDP).
- A DNS resolver (`resolve_a`, `resolve_aaaa`).
- A spawn primitive (`spawn`, `spawn_local`).
- A randomness source (`get_rng()` returning a streamed RNG).
- A file-system handle (`open_read`, `open_write`) scoped to the
  per-node sandbox.
- An env-var read.
- A process metadata read (`hostname`, `pid`).

Anything the recording observes (OBSERVABILITY §3) is observable on
the facade — either because the facade *emits* the corresponding
event itself (e.g., `MessageSent`), or because the subsystem
calling the facade emits it.

### 3.2 Prod implementation

Each method delegates to the obvious real backend: tokio runtime,
std net stack, libc DNS, etc. The implementation lives in the
existing crates (`crates/distribution` and friends). Production
code paths never see a sim-aware branch — the swap is at the
facade-trait boundary, not at every call site.

### 3.3 Sim implementation

Each method delegates to the engine. `now` reads virtual time;
`sleep_until` schedules a wake-up event; UDP send hands a packet
to the network model; DNS reads the simulated zone; `spawn`
registers a fiber with the executor; `get_rng` returns a stream
keyed by `(node_id, stream_label)`.

The sim implementation owns the runtime facade for *all* sim-native
hosts in the run, in one OS process. Opaque-binary hosts (§5.2)
get their own runtime facade implementation that virtualizes the
boundary at the syscall level instead.

### 3.4 Build-time switching

The active implementation is chosen by a single Cargo feature. No
runtime branching, no `if cfg!(sim)` sprinkled through peer code.
Two binaries are built from the same source: a prod binary linked
against the prod facade, and a sim driver that links the sim
facade and exposes the topology-spec entry point. Peer code does
not need a `Cargo.toml` change to be used in the sim — being
written against the facade trait is sufficient.

## 4. Network model

The network is what peer code talks through. Its job is to be
*indistinguishable* from a real network on every channel the
recording observes (OBSERVABILITY §3.2, §3.3, §3.6, §3.7, §4.1).

### 4.1 Topology

A topology is a directed graph of nodes and links plus a set of
shared-infrastructure entities (relays, DNS zones, vast.ai-style
host metadata sources). Topologies are declarative — they are an
input to the engine, not assembled imperatively at run time.

Graphs are arbitrary. The sim does not enumerate named topologies;
it accepts whatever the spec describes. The spec lists each node's
role, its peer set, and the link to each peer (or to the shared
infrastructure). Asymmetric links are first-class:
`bandwidth(A→B) ≠ bandwidth(B→A)`, and `loss(A→B) ≠ loss(B→A)`,
without special-casing.

Partition and heal are topology mutations applied at scheduled
virtual times. Each mutation is a record on the recording stream
(emitted as a sim-only `Custom` event, OBSERVABILITY §6 rule 4) so
the post-processor can correlate it with peer-observed events.

### 4.2 Link physics

Each directed link carries:

- `bandwidth_bps` — capacity. Excess packets queue (per the queue
  policy) or drop (per the drop policy).
- `one_way_delay_ms` — base latency. Distributions (lognormal, etc.)
  are allowed; the spec declares which.
- `jitter_ms` — per-packet delta on top of the base.
- `loss_rate` — independent or correlated; the spec declares which.
- `mtu` — packets above MTU are fragmented per the L3 / L4 rules.
- `queue_policy` — FIFO / SFQ / FQ-CoDel; spec-declared.

These are the *applied* values; OBSERVABILITY §4.8 commits the sim
to also emit them on every snapshot in the `links_applied` block
(sim-only field) so a replay knows what conditions it ran under.
The *observed* values flow out of the same passive-estimation code
prod uses, on the `links` block (sim and prod).

Per-link parameters can vary over time per the spec. Time-varying
parameters are also part of the topology declaration; they are
applied at scheduled virtual times, the same way partitions are.

### 4.3 L3 — IP

The sim addresses every host with one or more IPv4 and IPv6
addresses. The address assignment is in the topology spec.
Per-host interface metadata (name, MTU, up/down) is read by the
host scrape (OBSERVABILITY §3.8); the sim populates it from the
spec.

IP-level behavior modelled:

- Per-packet TTL decrement at every hop. Packets with TTL=0 are
  dropped and the drop is emitted as a `WirePacket` (§4.1 in
  OBSERVABILITY) with outcome `dropped_ttl`.
- ECN bits passed through (no AQM model unless the spec adds one).
- Fragmentation when MTU is exceeded. Reassembly at the destination,
  with the standard timeout. Reassembly failures observable.
- Source-address validation. Unspoofable per default; the spec can
  override per-link to allow spoofing for adversarial scenarios.

IP-level behavior *not* modelled: ICMP error generation beyond the
two cases needed for path-MTU discovery (`Fragmentation Needed`)
and dead-peer detection (`Port Unreachable`). The sim emits both
when the corresponding condition occurs.

### 4.4 NAT and middleboxes

NATs are first-class entities in the topology, with type drawn from
{Full-Cone, Restricted-Cone, Port-Restricted-Cone, Symmetric}. Each
NAT maintains a binding table; bindings expire per the spec's
keepalive-timeout setting and are refreshed by outbound packets.

OBSERVABILITY §4.6 commits the sim to expose the binding table on
every snapshot (`nat_bindings` per peer). The sim's authoritative
table is the source; the recording shows the binding from the
hosted side. NAT-binding refresh interval and refresh count match
real-world dynamics under each modelled NAT class.

Beyond NAT, the spec can place middleboxes that drop packets by
DPI signature (used for adversarial tests of relay traffic).
Middlebox drops are observable as `WirePacket` outcomes.

### 4.5 L4 — UDP, TCP, QUIC

**UDP** is a thin wrapper over the L3 model: send-to / recv-from,
no flow control, drops surfaced as silent loss. Probes
(OBSERVABILITY §3.9) ride on this layer.

**TCP** is reserved for shared-infrastructure protocols (e.g., the
collector HTTP sink). It is modelled as a faithful loss-recovery
implementation, but its connection state is *not* exposed in
tier-2 because production iroh does not use it for peer traffic.

**QUIC** is the load-bearing L4. The sim runs the same QUIC
implementation iroh runs in prod (quinn), linked against the sim
UDP. The QUIC handshake, congestion control, and stream multiplexing
behave exactly as they do in prod because the *implementation* is
the same; only the underlying UDP is the sim's. This is what makes
the OBSERVABILITY §4.3 (congestion-control state) and §4.7
(handshake records) parity bars achievable: the data structures
reporting on each are the real ones.

### 4.6 Endpoint stack

Each sim host runs the full iroh stack (or its equivalent for
non-iroh transports): MagicSock, discovery, NodeMap, connection
cache. These are *not* re-implemented; they are the production
code linked against the sim runtime facade. That is the only way
their internal state — the state surfaced in OBSERVABILITY §3.6
(`Tier2IrohState`) — matches prod by construction.

Per-node configuration (relay set, discovery providers) is from
the topology spec. The DNS resolver used by iroh's relay-URL
lookup is the simulated one (§4.4 in this spec — *not* the OS
resolver — the simulated host model).

### 4.7 Clocks

Each host's clock is virtual time plus a per-host skew and drift
configured by the spec. Skew is a constant offset; drift is a
rate (ppm). Clock samples (OBSERVABILITY §3.5) capture the
relationship between each host's clock and the collector's
("virtual UTC"), so the post-processor's alignment code does the
same work on sim and prod bundles.

A host can experience a clock jump (e.g., NTP step) at a scheduled
virtual time. The jump is a topology mutation; it is observable
because every event after it carries the new `wall_ms` while
`monotonic_seq` (OBSERVABILITY §3.3) keeps going up.

### 4.8 Host budget

Each host has a modelled budget: CPU shares, memory bytes,
open-fd count. These map to the OBSERVABILITY §3.11 process-stats
fields. The sim does not enforce the budget against peer code
(peer code runs as fibers); instead, the budget is tracked
indirectly:

- CPU time per fiber is approximated from the number of times the
  fiber yields per virtual-time unit, weighted by the per-host CPU
  share. The result feeds the `cpu_ms` field on the process snapshot.
- Memory is tracked from the peer code's own allocations against
  the per-node allocator wrapper. RSS is reported as the
  high-water mark seen since the previous snapshot. VmSize is the
  current live byte count.
- Open-fd count is the number of sim-side sockets currently bound
  by the node (UDP, QUIC, TCP, plus simulated files).

This is the imperfect-but-honest implementation NORTH_STAR §"What
it models" calls for: the recording surface is populated; the
underlying numbers are approximations, marked as such in the
post-processor's documentation; calibration against prod will tell
us when the approximations are too loose.

## 5. Hosted entities

### 5.1 Sim-native hosts

A sim-native host is a peer whose code we own and rebuild against
the sim runtime facade. It is the same binary in spirit as the
prod binary — same crates, same code paths — linked against a
different runtime crate.

A run hosts arbitrarily many sim-native hosts. Each has its own
address space, its own runtime facade instance, its own
diagnostics aggregator (OBSERVABILITY §3). The engine routes I/O
between them through the network model.

Sim-native hosts cover all of OBSERVABILITY §3's parity bar
because they are running the production diagnostics stack — every
event and snapshot is emitted by the same code that emits them in
prod.

### 5.2 Opaque-binary hosts via virtualized I/O

When we need to host a peer whose code we cannot rebuild against
the facade — a third-party node, a peer running an old version,
a kernel that handles a packet a specific way — the sim drives
the real binary through a virtualized I/O boundary. The binary
runs as a real OS process; its socket calls go through a shim
(LD_PRELOAD-style on Linux, or a network-namespace + tun device
on platforms where syscall interception is brittle); the shim
routes packets to the engine the same way sim-native UDP does.

From the binary's perspective, it sees a real kernel, real sockets,
real time. The engine controls what crosses the wire. Records
emitted by the opaque binary (if any) are read off its own log
output and translated into the recording schema as best as the
adapter can manage; gaps appear as `None` fields, marked.

Opaque-binary hosts are an escape hatch, not the default. The
parity bar applies fully to sim-native hosts; opaque-binary hosts
have inherently incomplete recording (we cannot fully instrument
a binary we did not build).

### 5.3 Lifecycle

Every host has a defined lifecycle in the topology spec:

- `start_at_ms` — virtual time at which the host's runtime is
  brought up. Boot order is deterministic.
- `restart_at_ms[]` — scheduled restarts. `boot_sequence`
  (OBSERVABILITY §3.1) increments per restart.
- `stop_at_ms` — clean shutdown.
- `crash_at_ms` — uncontrolled exit. Distinct from `stop` because
  no draining occurs.

Each lifecycle event is a topology mutation; each is observable as
a `Custom` event on the recording (the host that experienced the
event reports it through its own diagnostics, if reachable; the
engine reports it as a sim-only record otherwise).

## 6. Recording format

### 6.1 Wire-level trace

The wire-level trace is the per-packet stream described in
OBSERVABILITY §4.1. The sim emits one record per packet in either
direction at every modelled hop. Records are written to
`{run_dir}/wire/{node_id}.ndjson` and tarred into the bundle.

Wire-level capture is gated by a per-run setting. Off by default
because the data volume is high; on for calibration runs and for
any run where the §4.1 OBSERVABILITY parity bar is being checked.

When a payload-bytes capture is requested, the bytes live next to
the trace as a separate file (`wire-payloads/...`) so the trace
itself stays small enough to grep through.

### 6.2 Internal-observation trace

This is the production recording surface itself — OBSERVABILITY §3
in its entirety. The sim writes the same files to the same paths
the production collector does, under
`{collector_root}/{run_id}/{node_id}/`:

```
boot.json
snapshots/{snapshot_id}.json
events/{batch_seq}.json
finalize.json
```

A sim run produces a bundle byte-for-byte indistinguishable from a
prod bundle of the same workload (modulo the sim-only fields
flagged in OBSERVABILITY §3.11 / §6.5 below). That isomorphism is
the parity bar.

### 6.3 Schema versioning

The schema is versioned. Every record carries a `schema_version`
field at the top of its envelope. The post-processor accepts any
version it knows about. Schema upgrades are documented in the
diagnostics crate's CHANGELOG; the sim and prod sides upgrade
together (NORTH_STAR §"Calibration is ongoing").

Adding a field to a record is a minor version bump; removing or
renaming is major. Old bundles remain readable forever; the
post-processor handles back-compat. The sim's emitted version
matches whatever version of the diagnostics crate it is linked
against, the same as prod.

### 6.4 Storage layout

Bundles are tarballs (`.tar.gz`) at
`{collector_root}/bundles/{run_id}.tar.gz`. The internal layout
matches `DIAGNOSTICS_PLAN.md` §"Bundle Format":

```
{run_id}/
  MANIFEST.json
  orchestrator/{boot, snapshots/, events/, finalize}.json
  stage-N/{boot, snapshots/, events/, finalize}.json
  collector.log
  summary.md            # post-processor output, optional
```

Sim runs additionally write:

```
{run_id}/
  sim/
    spec.toml            # the topology spec used
    seed                 # the run seed
    wire/{node_id}.ndjson  (if wire-level capture is on)
    links_applied.json   # ground-truth link params over time
    mutations.log        # partition/heal/restart events
```

The `sim/` subtree is sim-only and is ignored by the parity diff.
It is what enables a sim-vs-sim replay (§2.5) and what gives a
calibration run the ground truth to compare against.

### 6.5 Sim-only fields

Sim runs emit a small number of fields prod cannot: ground-truth
applied link parameters (`links_applied`, OBSERVABILITY §4.8),
sim-internal scheduler decisions (OBSERVABILITY §6 rule 4),
RNG-state traces. These are written to `sim/` (§6.4) and are never
read by code that also runs in prod. The node-under-test cannot
observe any of them.

## 7. Replay

### 7.1 Environment restoration

Replay takes a recording (produced by a prod run, a sim run, or
either one with a partial bundle) and plays back the *environment*
the recorded run was in. Concretely:

- Topology is reconstructed from the recording. NORTH_STAR §"Two
  modes, one engine" demands this be feasible from a real-world
  bundle, so the per-link parameters used by the sim must be
  derivable from what the recording exposes. Where the recording
  exposes only `links` (observed), the sim seeds its `links_applied`
  with the observed values and lets the parity diff quantify any
  drift. Where the recording exposes `links_applied` (sim-origin),
  the sim adopts them exactly.
- Peer set is reconstructed from the boot identity blocks across
  the bundle.
- Mutation schedule (partition, heal, restart) is reconstructed
  from the `Custom` events the original run emitted.
- Shared infrastructure (relays, DNS, vast.ai metadata) is
  reconstructed from whatever the bundle's host-scrape and
  identity blocks recorded.

### 7.2 Peer re-execution

Per NORTH_STAR §"Two modes, one engine": peers in a replay are
**re-executed peer code**, not stubs driven by recorded outputs.
The recording carries the *environment*; the peer logic that
runs against that environment is whatever version is under test
in the replay. That is how a replay tests a candidate change — by
exposing the same world to new code.

When the recording lacks information needed to drive a peer
faithfully (a third-party peer whose code we do not have), that
peer is hosted via the opaque-binary path (§5.2). If the original
recording was prod and the peer was iroh-stack swactor, the
sim-native path applies.

### 7.3 Divergence handling

A replay diverges when a re-executed peer makes a different choice
from what the recording shows the original peer made. Divergence
is *expected*: that is the point of replay — to see whether the new
code behaves differently against the same conditions.

The replay records the new behavior at the same level of detail
(§6) and the post-processor's diff mode is what surfaces the
delta. Divergence is not an error; only when the divergence is
*outside the parity envelope* (§9.3) is it a sim bug rather than
an algorithm-change observation.

## 8. Observability

### 8.1 Shared sinks with prod

The sim's diagnostics aggregator (the one running inside each
sim-native host) is the *production* diagnostics aggregator from
`crates/distribution/src/diagnostics`. It is linked against an
HTTP sink whose target is the sim's collector — itself a real
binary (also from `crates/distribution`), brought up by the engine
on a virtual host at the start of every sim run.

The collector writes the same bundle layout (§6.4). The
post-processor reads it the same way. The "shared" in "shared
sinks" means literal: the same code runs on both sides of the
sim/prod boundary, so the parity bar is enforced by construction
for every record kind enumerated in OBSERVABILITY §3.

### 8.2 Sim-only diagnostics

The sim emits a small set of additional records the node-under-test
cannot observe (OBSERVABILITY §6 rule 4): scheduler decision logs,
exact virtual-time tick streams, RNG-state traces, per-link
applied parameters (§6.5). They are written to the `sim/` subtree
of the bundle and are read only by the engine's own debugging
tooling, never by code that also runs in prod.

The discipline: any time we are tempted to add a sim-only field
that the node-under-test *can* observe, we have to either
(a) commit to making prod observe it too and put it in
OBSERVABILITY §3 or §4, or (b) keep it strictly under §8.2. There
is no third option.

## 9. Calibration loop

### 9.1 Procedure

Calibration is the act of measuring sim-vs-prod observable
equivalence. The loop is concrete:

1. Run a workload in prod. Collect the bundle.
2. Extract the topology + mutation schedule from the bundle (§7.1).
3. Run the *same* peer-code version in the sim against that
   reconstructed environment. Collect the bundle.
4. Diff the two bundles, record kind by record kind, against the
   parity bars defined in OBSERVABILITY §3 / §4.
5. Anywhere the distributions diverge beyond the noise floor
   (§9.2), file a sim bug. Anywhere prod is missing a record the
   sim emits or vice versa, file a recording bug or a sim bug
   (whichever direction the asymmetry runs — see OBSERVABILITY
   §6 rule 3).

The loop runs continuously, not as a one-time validation. Every
real deployment is evidence; every divergence is an issue.

### 9.2 Noise-floor estimation

Prod runs are inherently noisy: real link latencies jitter, real
kernels make scheduling decisions we cannot reproduce, real clocks
drift. Calibration cannot demand sim records to be byte-identical
with prod records — it demands them to be *statistically
indistinguishable* (NORTH_STAR §"The parity bar").

The noise floor is estimated from prod-vs-prod variance: two prod
runs of the same workload, under as-matched-as-possible
conditions, diff each other. The variance of that diff defines
the noise floor for each numeric field; a sim-vs-prod divergence
within that envelope is not a bug. A sim-vs-prod divergence beyond
it is.

Establishing the noise floor is an ongoing project — every new
metric needs its own estimate. The post-processor's calibration
report ships per-metric noise-floor numbers alongside the
sim-vs-prod diff.

### 9.3 Parity metrics

Per OBSERVABILITY's call for distributional parity, the
calibration report tracks for each metric:

- **Presence parity** — does the sim emit a record whenever prod
  does (and vice versa)? Boolean per record kind.
- **Mean parity** — is `mean_sim ≈ mean_prod` within the noise
  floor of the means?
- **Tail parity** — is `p99_sim ≈ p99_prod` within the noise floor
  of the tail? This is the bar OBSERVABILITY calls out
  explicitly; mean parity alone is not enough.
- **Causal parity** — does the sim emit events in the same order
  prod does, conditioned on the same upstream causes? Tied to the
  causal-trace-IDs surface (OBSERVABILITY §4.2).

A pass is "every metric within noise floor on every parity
dimension." A fail names the metric and dimension. The
post-processor's calibration report is structured exactly that
way; "did calibration pass" reduces to "did any line in the
report come back red."

## 10. Glossary

- **Aggregator** — per-process recording state owner. Owns the
  reachability log, event ring, snapshot assembler, sink. Same
  type in sim and prod.
- **Bundle** — the tarball at end-of-run containing every record
  emitted during the run. See §6.4.
- **Calibration** — measurement of sim-vs-prod observable
  equivalence. See §9.
- **Collector** — the HTTP server every aggregator POSTs to.
  Centralized per-run target; writes the bundle.
- **Engine** — the discrete-event simulator core. Owns virtual
  time, the executor, the network model, the topology graph. See
  §2.
- **Facade** — the trait surface peer code uses for all
  potentially-non-deterministic operations. Two implementations
  (prod, sim). See §3.
- **Fiber** — an executor-managed unit of concurrency. One fiber
  per peer task; per-fiber scheduling is deterministic. See §2.3.
- **Mutation** — a scheduled change to the topology during a run
  (partition, heal, restart, link-parameter change). Recorded as
  a `Custom` event. See §4.1.
- **Noise floor** — the variance between two prod-vs-prod runs of
  the same workload. Sets the bar that sim-vs-prod divergence is
  measured against. See §9.2.
- **Opaque binary** — a hosted entity whose code we do not own and
  cannot rebuild against the facade. Driven via syscall
  interception. See §5.2.
- **Parity bar** — the requirement, per OBSERVABILITY, that every
  record kind on the observable surface match between sim and
  prod. See OBSERVABILITY §6.
- **Recording** — the on-disk artifact of a run's observable
  surface. Same format in sim and prod. See §6.
- **Replay** — re-execution of peer code against an environment
  reconstructed from a recording. See §7.
- **Run** — one execution of the engine from boot to finalize.
  Identified by `run_id` (in the bundle) and `run_seed` (input).
- **Sim-native host** — a peer whose code is linked against the
  sim runtime facade. See §5.1.
- **Topology spec** — the input declaration of nodes, links,
  shared infrastructure, mutations, and run parameters. See §4.1.
- **Virtual time** — the engine's authoritative clock.
  Wall-clock `wall_ms` fields in the recording are populated from
  it on sim runs. See §2.2.