swactor/crates/simulation/TESTING_SPEC.md

# Deterministic Testing Spec for the Simulator

> Companion to [`NORTH_STAR.md`](./NORTH_STAR.md), [`SPEC.md`](./SPEC.md),
> and [`OBSERVABILITY.md`](./OBSERVABILITY.md). Read those first.
>
> This document is the closed, enumerated contract of binary pass/fail
> checks the simulator must satisfy before it is considered built. It
> exists because the simulator's quality bar — observable equivalence
> with production, per NORTH_STAR — is statistical, and statistical
> bars are asymptotic by nature: there is always a metric just over
> the line. An asymptote cannot be the agent's done-criterion. The
> agent's done-criterion is "every check in §2–§12 of this document is
> green." Statistical parity (the calibration loop, SPEC §9) lives
> outside this spec and is gated by humans.
>
> The simulator is being built by a code agent operating without
> direct feedback. This spec, the three docs above, and the test
> suite it mandates are the only signals the agent has. Sections of
> this document must therefore be read as binding constraints, not
> guidelines.

## Table of contents

0. Reading instructions
1. Test corpus
2. Determinism oracle
3. Replay isomorphism
4. Facade integrity
5. Same-binary invariant
6. Schema floor coverage
7. Schema round-trip
8. Causality and time invariants
9. Equivariance
10. Adversarial sim-detector
11. Lifecycle observability
12. Test infrastructure integrity
13. Definition of done
14. Failure protocol
15. Out of scope
16. Glossary

---

## 0. Reading instructions

**Each section below names tests that must exist and pass.** The
checks are binary: zero tolerance, no thresholds, no "approximately."
Wherever a section calls a check binary, that is the literal
expectation — same SHA-256, identical byte ranges, identical record
counts, identical sorted symbol sets.

**The check list is closed.** New checks may be *added* (extensions
to §10 in particular are expected as new detection techniques are
discovered) but no check in this document may be removed, relaxed,
weakened, marked `#[ignore]`, gated behind an env var, or skipped
on any platform. Any change to this file or to the parity-bar test
directory requires a separate commit that is gated by §12.1.

**Tests are addressed to the code, not to themselves.** Per the
project conventions in `CLAUDE.md`, white-box / structural tests
("does this call this function") are not acceptable substitutes for
the checks listed here. Where a check is a scenario test, write the
scenario. Where it is a property, use property machinery. Where it
is an invariant on real output, assert against the real output.

**Where a check needs a fixture that does not yet exist, the agent
is to create the fixture inside the parity-bar tree, document its
provenance in `tests/parity-bar/fixtures/PROVENANCE.md`, and proceed.**
Fixtures are not optional. Tests that depend on fixtures are not
optional. There is no "future work" exit.

---

## 1. Test corpus

The schema authority for this version of the simulator is a single
production bundle from the `ds-inference` branch's N=3 vastai
investigation (see `examples/pipeline-parallel-inference/N3_DEPLOYMENT_REPORT.md`).
That bundle is the floor: every record kind present in it must be
emittable by the simulator; every field present in it must be
populated or explicitly `None` per OBSERVABILITY §6 rule 1; every
schema version in it must be readable by the post-processor against
which the simulator is checked.

### 1.1 Location

```
crates/simulation/tests/parity-bar/fixtures/
  vastai-n3-1/                   # the reference bundle (unpacked tar)
    MANIFEST.json
    orchestrator/{boot,snapshots/,events/,finalize}.json
    stage-0/{...}
    stage-1/{...}
    stage-2/{...}
    collector.log
  PROVENANCE.md                  # how this bundle was produced; do not touch
```

### 1.2 Corpus scope

This is the *only* prod-origin corpus the simulator is checked
against in v1. Schema-floor coverage (§6) is bounded by what this
bundle exposes; the simulator is not obligated to emit record kinds
that do not appear here, and is *forbidden* from emitting record
kinds that do not appear in either this bundle or OBSERVABILITY §3.

When additional prod bundles are checked in, the corresponding
sections of §6 expand. The expansion is a separate, human-gated
commit (§12.1).

### 1.3 Bundle modifications

The fixture bundle is immutable. The agent does not touch any file
under `tests/parity-bar/fixtures/`. Repairs to the corpus (e.g., a
field renamed in the schema) happen in a separate commit that also
updates the lock hash in §12.1.

---

## 2. Determinism oracle

Discharges SPEC §2.1, §2.4, §2.5.

### 2.1 Two-run byte equality

```
crates/simulation/tests/parity-bar/t_determinism.rs::two_run_byte_equality
```

Two invocations of the engine with the same `(topology spec, seed)`
produce two bundles whose every regular file has the same SHA-256.
Includes NDJSON byte order, tar header timestamps if present, and
the contents of `sim/` (§6.4 in SPEC).

Binary: zero file-level hash differences. Failure must name every
diverging path.

### 2.2 Two-process concurrent equality

```
crates/simulation/tests/parity-bar/t_determinism.rs::two_process_equality
```

The same `(spec, seed)` run in two separate OS processes concurrently
(spawned by the test harness, `std::process::Command` or equivalent)
produces byte-identical bundles. Catches non-determinism from
process-global state: allocator addresses, vtable layout under
ASLR, environment variable order, thread-local randomness.

Binary: zero file-level hash differences. The two processes must be
the same binary invoked twice; not two compilations.

### 2.3 Debug/release equality

```
crates/simulation/tests/parity-bar/t_determinism.rs::debug_release_equality
```

The same `(spec, seed)` run with `cargo run --release` and
`cargo run` (debug) produces byte-identical bundles. If this fails,
the cause is almost always floating-point ordering; the engine
must not use floating-point arithmetic on any code path whose
output reaches the recording.

Binary: zero file-level hash differences across profiles.

### 2.4 No wall-clock taint

```
crates/simulation/tests/parity-bar/t_determinism.rs::no_wall_clock_taint
```

Set `FAKETIME` (or equivalent libfaketime injection) to advance the
host clock by 1 hour. Run the engine. The bundle must be identical
to a run without the injection. Proves no kernel-time leak past the
facade (SPEC §2.4, `SystemTime`/`Instant` row).

Binary: zero file-level hash differences with vs. without injection.

### 2.5 Divergence detection

```
crates/simulation/tests/parity-bar/t_determinism.rs::divergence_detected
```

Run with `(spec, seed)`, store the bundle hash chain. Modify a
single byte in the engine's RNG output (via a `#[cfg(test)]`
poisoned-RNG facade switch — the *only* such test-only switch
permitted, and it must live in the sim facade crate). The divergence
detector (SPEC §2.5) must produce a structured `Error` event naming
the first divergent record and exit non-zero.

Binary: divergent run exits non-zero; clean run exits zero.

---

## 3. Replay isomorphism

Discharges SPEC §7.

### 3.1 Self-replay

```
crates/simulation/tests/parity-bar/t_replay.rs::self_replay_identical
```

Run the engine to produce bundle B₁. Pass B₁ to the engine in
replay mode (`--replay B₁/`). Produce B₂. B₂ must be byte-identical
to B₁.

This is the load-bearing test of recording self-sufficiency: any
piece of state needed to drive a replay must be in the recording.
If B₂ differs from B₁, the recording is missing information SPEC
§7.1 demands it carry.

Binary: zero file-level hash differences.

### 3.2 Mutation fidelity

```
crates/simulation/tests/parity-bar/t_replay.rs::mutations_preserved
```

A topology spec declaring `partition(at_ms=5000)`, `heal(at_ms=8000)`,
`restart(node=stage-1, at_ms=10000)` is run. The resulting bundle
must contain `Custom` mutation events for each, with virtual-time
`wall_ms` exactly matching the spec, in spec order, on the
recording streams SPEC §4.1 and §5.3 designate.

Binary: exact count, exact ordering, exact `wall_ms` per mutation.

### 3.3 Recording self-sufficiency under prod shape

```
crates/simulation/tests/parity-bar/t_replay.rs::replay_from_prod_shape_only
```

Run the engine to produce bundle B. Delete the `sim/` subtree from
B (SPEC §6.4 — the sim-only fields). Pass the stripped bundle to
replay. The replay must complete; the resulting bundle's non-`sim/`
files must be byte-identical to B's non-`sim/` files.

This proves replay does not depend on sim-only data leaking into
the prod-shape part of the recording. A prod bundle, which never
has `sim/`, must replay just as well.

Binary: zero file-level hash differences on non-`sim/` files;
replay exits zero.

### 3.4 Prod-bundle replay (corpus-anchored)

```
crates/simulation/tests/parity-bar/t_replay.rs::vastai_n3_replays
```

Pass `tests/parity-bar/fixtures/vastai-n3-1/` to the engine in
replay mode. The engine must:

- Reconstruct topology and peer set from the bundle's boot blocks
  (SPEC §7.1).
- Reconstruct mutation schedule from `Custom` events in the bundle.
- Complete the replay without panicking.
- Emit a bundle in the same layout (§6.4, §7.1).

This test does *not* require the emitted bundle to be byte-identical
to the input bundle. It is sufficient that the replay completes and
produces a structurally valid output. Statistical comparison
between input and output is a calibration concern (SPEC §9) and is
out of scope for this spec (§15).

Binary: replay exits zero; output bundle passes §7 checks.

---

## 4. Facade integrity

Discharges SPEC §3, §2.4.

### 4.1 Banned-API lint

```
crates/lint-deterministic/                    # new crate
crates/lint-deterministic/banned.toml          # the closed list
crates/lint-deterministic/tests/banned_apis_rejected.rs
```

A custom `cargo xtask lint-deterministic` step runs in CI before any
test. It rejects the workspace build if any module outside the
sim-facade implementation references the banned-API set. The set
is closed and version-locked in `banned.toml`. The current set:

```
std::time::SystemTime         → Facade::clock().now()
std::time::Instant            → Facade::clock().now()
std::thread::sleep            → Facade::clock().sleep_until(...)
std::thread::spawn            → Facade::spawn(...)
tokio::spawn                  → Facade::spawn(...)
tokio::time::sleep            → Facade::clock().sleep_until(...)
tokio::time::Instant          → Facade::clock().now()
std::collections::HashMap     → indexmap::IndexMap or BTreeMap
std::collections::HashSet     → indexmap::IndexSet or BTreeSet
rand::thread_rng              → Facade::rng(stream_label)
getrandom::getrandom          → Facade::rng(stream_label)
std::env::var                 → Facade::env(name)
std::env::vars                → Facade::env_iter()
std::fs                       → Facade::fs() (sandboxed per node)
std::net                      → Facade::udp() / Facade::tcp()
std::process::Command         → Facade::process(...) (errors in sim)
```

Allowlist (paths exempt from the lint):
```
crates/simulation/src/facade/sim/         # the sim facade itself
crates/runtime-facade/src/prod/           # the prod facade itself
crates/lint-deterministic/                # the lint itself
```

Binary: zero violations across the workspace.

### 4.2 Feature exclusivity

```
crates/simulation/tests/parity-bar/t_facade.rs::feature_exclusivity
```

A test in the workspace `xtask` confirms that `cargo build` with
features `facade-prod facade-sim` fails (compile error from a
`compile_error!` macro in the facade crate). The same test confirms
that `cargo build` with neither feature also fails.

Binary: both invocations exit non-zero with the expected error string.

### 4.3 No `cfg(sim)` in peer code

```
crates/lint-deterministic/tests/no_cfg_sim_in_peer.rs
```

```
grep -r 'cfg(\s*sim\s*)' \
  crates/distribution \
  crates/swactor \
  crates/node \
  crates/process \
  examples/
# must return zero matches
```

Peer code never branches on whether it is in the sim. The facade
is the only swap point (SPEC §3.4). Code that compiles
conditionally on sim-vs-prod inside a peer crate is a violation.

Binary: zero matches.

### 4.4 Facade trait surface is closed

```
crates/runtime-facade/tests/surface_locked.rs
```

The trait family exposed by `runtime-facade` is fingerprinted (each
trait's method signatures hashed in declaration order, hashes
concatenated). The fingerprint is stored in
`crates/runtime-facade/surface.lock`. Changes to the trait surface
require updating this lock in a separate commit gated by §12.1.

Binary: live fingerprint equals locked fingerprint.

---

## 5. Same-binary invariant

Discharges SPEC §3.4, §4.6, §5.1.

### 5.1 Shared load-bearing dependencies

```
crates/simulation/tests/parity-bar/t_same_binary.rs::shared_load_bearing
```

`cargo metadata` for the prod binary and the sim driver must list
the following crates, at *identical resolved versions*, in both
dependency graphs:

```
iroh
iroh-relay
quinn
quinn-proto
swactor                     # the SWIM implementation
distribution::diagnostics
postcard                    # codec
```

The set is closed in this spec. Adding to it requires a §12.1
commit.

Binary: exact set equality at exact version equality on both sides.

### 5.2 No sim-only forks of transport code

```
crates/simulation/tests/parity-bar/t_same_binary.rs::no_transport_forks
```

The sim has zero source files matching any of:
- `**/sim_swim/**`
- `**/sim_iroh/**`
- `**/swim_sim.rs`
- `**/iroh_sim.rs`
- `**/quinn_sim.rs`

The transports under sim are the production transports linked
against the sim facade (SPEC §4.6). Replacement implementations
defeat the purpose of the sim.

Binary: zero matches.

### 5.3 Symbol overlap floor

```
crates/simulation/tests/parity-bar/t_same_binary.rs::symbol_overlap
```

`nm` (or `cargo bloat`) on the prod binary and the sim driver
produces two symbol sets. Their intersection must contain every
public symbol from the §5.1 crates. The check is exact: any
listed crate's public surface missing from either binary fails.

Binary: full inclusion in both directions.

---

## 6. Schema floor coverage

Discharges OBSERVABILITY §3, NORTH_STAR §"Parity bar". Scope bounded
by the v1 corpus (§1).

### 6.1 Corpus record-kind census

```
crates/simulation/tests/parity-bar/t_schema_coverage.rs::corpus_kinds
```

The reference bundle (§1.1) is parsed at test start. The set of
record kinds present is computed and stored as the required set.
A reference sim scenario (`tests/parity-bar/fixtures/scenarios/reference.toml`)
is run. Every record kind in the corpus set must be present in the
sim output.

This is the *floor*: the sim must emit at least what prod emits.
The agent does not need to invent record kinds the corpus does not
exhibit.

Binary: corpus kinds ⊆ sim-output kinds.

### 6.2 No phantom records

```
crates/simulation/tests/parity-bar/t_schema_coverage.rs::no_phantom_records
```

The sim must not emit record kinds that are not in either:
- The reference bundle (§1.1), or
- The schemas enumerated in OBSERVABILITY §3 / §4.

This prevents the sim drifting toward sim-specific schemas that
the prod side cannot consume. Sim-only fields (SPEC §6.5) live
under `sim/` and are excluded from this check.

Binary: sim-output kinds ⊆ (corpus kinds ∪ OBSERVABILITY-documented kinds).

### 6.3 Field presence audit

```
crates/simulation/tests/parity-bar/t_schema_coverage.rs::no_silent_none
```

For every record emitted by the reference scenario, every field
documented in OBSERVABILITY §3 must be either:
- Populated (non-null), or
- Explicitly `None` with at least one `Error` event somewhere in
  the bundle whose `component` matches the introspector responsible
  and whose `message` notes the gap.

A silent `None` (no accompanying `Error`) is a failure (OBSERVABILITY
§6 rule 1).

Binary: zero silent-None fields.

### 6.4 Variant exhaustiveness

```
crates/simulation/tests/parity-bar/t_schema_coverage.rs::all_event_variants_fire
```

The reference scenario (which is a multi-mode scenario combining
boot, steady-state, partition, heal, restart, and shutdown) must
fire every `Event` variant the corpus exhibits. Variants present
in OBSERVABILITY §3.3 but absent from the corpus are *not* required
in v1.

Binary: corpus variants ⊆ sim-emitted variants.

### 6.5 Reference scenario contents

```
crates/simulation/tests/parity-bar/fixtures/scenarios/reference.toml
```

A 60-second scenario carrying:
- 3 sim-native peers in a chain (mirrors the N=3 corpus layout)
- 1 orchestrator
- 1 collector
- Boot, steady-state SWIM ticks for 20s
- Partition at t=20s isolating stage-2
- Heal at t=35s
- Restart of stage-1 at t=45s
- Clean shutdown at t=60s

This scenario is the floor on which §6.1–§6.4 run. It is the only
scenario whose contents §6 binds to. New scenarios can be added in
`tests/exploratory/`; they do not extend the §6 contract.

---

## 7. Schema round-trip

Discharges SPEC §6, OBSERVABILITY §3.

### 7.1 Prod parser eats sim bundle

```
crates/simulation/tests/parity-bar/t_round_trip.rs::prod_parser_reads_sim_bundle
```

The reference-scenario sim bundle is passed to the production
post-processor (`crates/distribution/src/diagnostics/postproc`). The
post-processor must return `Ok(_)` and its `warnings` array must
have zero `unknown_field` and zero `unknown_record_kind` entries.

Binary: post-processor exit zero, zero unknown-* warnings.

### 7.2 Sim parser eats prod bundle

```
crates/simulation/tests/parity-bar/t_round_trip.rs::sim_replay_parser_reads_prod_bundle
```

The corpus bundle (§1.1) is passed to the sim's replay loader.
The loader must return `Ok(_)` and emit zero `unknown_field` /
`unknown_record_kind` warnings. The sim parser and the prod parser
are the *same code* (per SPEC §8.1) — this test exists to confirm
that fact, not to test a separate codepath.

Binary: replay-loader exit zero, zero unknown-* warnings.

### 7.3 Bundle layout exact

```
crates/simulation/tests/parity-bar/t_round_trip.rs::bundle_layout_matches_spec
```

`tar tf` of the reference sim bundle is compared against the
expected file list:

```
{run_id}/
  MANIFEST.json
  orchestrator/boot.json
  orchestrator/snapshots/*.json
  orchestrator/events/*.json
  orchestrator/finalize.json
  stage-N/boot.json                   # for each peer
  stage-N/snapshots/*.json
  stage-N/events/*.json
  stage-N/finalize.json
  sim/spec.toml
  sim/seed
  sim/links_applied.json
  sim/mutations.log
  sim/wire/*.ndjson                   # if wire capture is enabled
```

Files that are not listed above must not be present. Files that are
listed but conditional (e.g., `wire/`) are present iff the run
config enabled them.

Note: the prod `collector.log` (SPEC §6.4) is not yet a binding
artifact in v1 — the collector binary itself lands with the
calibration loop (§15). The file's fixed bundle-root path conflicts
with §9.1's substring rename map (the engine writing a fixed
filename in both base and renamed runs forces a key mismatch under
`apply_path_rename`). v2 reintroduces the requirement once the
collector binary's diagnostics output names itself from the
collector host's spec name.

Binary: exact match modulo conditional sections.

### 7.4 Schema version pin

```
crates/simulation/tests/parity-bar/t_round_trip.rs::schema_version_pinned
```

Every record envelope's `schema_version` field equals the version
exported as `distribution::diagnostics::SCHEMA_VERSION`. The
constant is single-source.

Binary: zero records with a divergent `schema_version`.

---

## 8. Causality and time invariants

Discharges SPEC §2.2, §2.3, OBSERVABILITY §3.3.

### 8.1 `monotonic_seq` strictly increasing

```
crates/simulation/tests/parity-bar/t_causality.rs::monotonic_seq_strictly_increasing
```

For each `(node_id, boot_sequence)` in the reference bundle, the
sequence of `monotonic_seq` values across all records (events and
snapshots) is strictly increasing and contiguous (no gaps that
would imply lost records).

Binary: zero violations.

### 8.2 `wall_ms` non-decreasing modulo declared jumps

```
crates/simulation/tests/parity-bar/t_causality.rs::wall_ms_non_decreasing
```

For each node, the sequence of `wall_ms` values is non-decreasing,
except across instants marked by a `Custom { kind: "clock_jump" }`
event (which the sim emits when a topology-declared clock jump
fires per SPEC §4.7). Across each clock_jump, the delta in
`wall_ms` is allowed; before and after, monotonicity is required.

Binary: zero unexplained non-monotonic transitions.

### 8.3 Send precedes receive

```
crates/simulation/tests/parity-bar/t_causality.rs::send_precedes_receive
```

For every `MessageReceived { peer = P, kind = K, ... }` on node A,
there exists a `MessageSent { peer = A, kind = K, ... }` on node P
whose `wall_ms` is at least `link(P→A).one_way_delay_ms − link.jitter_ms`
earlier (in virtual time). The match is by `(peer, kind, size,
trace_id?)` — once causal trace IDs land (OBSERVABILITY §4.2),
the match becomes exact.

Binary: every received-message record has a matching causal sender.

### 8.4 `snapshot_id` uniqueness

```
crates/simulation/tests/parity-bar/t_causality.rs::snapshot_id_unique
```

Across the entire bundle, no two snapshots share a `snapshot_id`.

Binary: set size equals list length.

### 8.5 Tiebreaker order

```
crates/simulation/tests/parity-bar/t_causality.rs::executor_tiebreaker_deterministic
```

Construct a scenario in which three events fire at the same
virtual tick on the same node. The executor's order of resolution
must follow `(node_id, fiber_id, event_seq)` per SPEC §2.3. The
test runs the scenario twice and asserts the per-tick resolution
order is identical.

Binary: identical order across two runs.

---

## 9. Equivariance

Discharges SPEC §4.1.

### 9.1 Node-id rename invariance

```
crates/simulation/tests/parity-bar/t_equivariance.rs::rename_invariance
```

Spec₁ defines nodes `[alpha, beta, gamma]` with a given topology
graph. Spec₂ is identical but renames `[alpha→x, beta→y, gamma→z]`,
applying the rename consistently to every peer-set reference.
Both run with the same seed.

The bundles must be byte-identical after applying the rename map
to the output (file paths under `{node_id}/` get renamed, and
every occurrence of the old `node_id` strings in record contents
gets substituted).

Binary: zero diffs after rename-map application.

### 9.2 Spec ordering invariance

```
crates/simulation/tests/parity-bar/t_equivariance.rs::spec_ordering_invariance
```

Spec₁ lists nodes in `[alpha, beta, gamma]` order; spec₂ lists the
same nodes in `[gamma, alpha, beta]` order; same seed. The
canonical ordering (sorted by `node_id`) used by the engine means
the bundles must be byte-identical.

Binary: zero diffs.

### 9.3 Same-tick mutation reordering

```
crates/simulation/tests/parity-bar/t_equivariance.rs::same_tick_mutation_reorder
```

Two mutations scheduled at the same virtual tick (e.g.,
`partition(at_ms=5000, edges=[a-b])` and `partition(at_ms=5000, edges=[b-c])`)
are specified in two different orders across two spec files; same
seed. The bundles must be byte-identical.

Binary: zero diffs.

---

## 10. Adversarial sim-detector

Discharges NORTH_STAR §"Parity bar" ("must have no means, statistical
or otherwise, of detecting that it is in a sim").

### 10.1 Detector binary

```
crates/sim-detector/                      # new crate
crates/sim-detector/src/lib.rs            # enumerated detection techniques
crates/sim-detector/src/main.rs           # peer binary
```

The detector is a peer binary linked through the runtime facade.
It compiles in both `facade-prod` and `facade-sim` configurations.
Its job is to report, for each technique, one of:

```
enum Verdict {
    Indistinguishable,
    DetectedSim(&'static str),
    DetectedProd(&'static str),
}
```

A passing sim run is one where every technique returns
`Indistinguishable` or `DetectedProd`. A passing prod run is one
where every technique returns `Indistinguishable` or `DetectedSim`.
The detector exists to be wrong about *which* environment it is in
— it must never correctly identify the sim.

### 10.2 Closed enumerated technique set

The v1 set (closed; extensions require §12.1):

| ID  | Technique                                                  |
|-----|------------------------------------------------------------|
| D01 | `SystemTime::now()` drift across `sleep(1s)`               |
| D02 | `Instant::now()` vs facade clock alignment                 |
| D03 | `HashMap` iteration order across two identical inserts     |
| D04 | `Box<dyn Trait>` vtable address stability across runs      |
| D05 | `getrandom` entropy distribution test (chi-squared shape)  |
| D06 | Spawn ordering on 1000 tasks in a tight loop               |
| D07 | UDP loopback timing vs declared link delay                 |
| D08 | DNS resolution latency vs declared resolver latency        |
| D09 | Hostname / process-pid uniqueness across two peers         |
| D10 | Allocator address-range stability across runs              |
| D11 | TCP loopback handshake count (sim TCP per SPEC §4.5)       |
| D12 | `Thread::id()` stability across two facade `spawn`s        |

The detector binary, run inside the reference scenario as an
additional peer, must produce verdicts that satisfy the §10.1 rule.

```
crates/simulation/tests/parity-bar/t_detector.rs::sim_indistinguishable
crates/simulation/tests/parity-bar/t_detector.rs::prod_baseline
```

Binary: zero `DetectedSim` verdicts in sim runs; zero `DetectedProd`
verdicts in prod runs. `prod_baseline` is a sanity check that runs
the detector against the prod facade and confirms it does *not*
falsely report sim there.

### 10.3 Technique provenance

Every entry in §10.2 is documented in
`crates/sim-detector/src/techniques.md` with:

- The exact mechanism (which API, what value range constitutes detection).
- The reason it would betray a naive sim.
- The facade obligation that prevents it.

The agent must update this file when adding a technique (and must
not add techniques without a §12.1 commit).

---

## 11. Lifecycle observability

Discharges SPEC §5.3.

### 11.1 Boot-record presence

```
crates/simulation/tests/parity-bar/t_lifecycle.rs::start_emits_boot
```

Every `start_at_ms` in the spec produces exactly one `boot.json`
under that node's directory, with `boot_sequence = 0`.

Binary: one boot record per start declaration; `boot_sequence` correct.

### 11.2 Restart sequence integrity

```
crates/simulation/tests/parity-bar/t_lifecycle.rs::restart_increments_boot_sequence
```

For a node with `start_at_ms=0, restart_at_ms=[10000, 20000]`, the
bundle contains:

- `boot.json` with `boot_sequence=0` and a `finalize.json` ending the first epoch.
- A second epoch with `boot.json` `boot_sequence=1` and a `finalize.json`.
- A third epoch with `boot.json` `boot_sequence=2` (still running at end-of-run
  so finalize may be absent if the run ends mid-epoch — see §11.5).

Binary: exact count, exact `boot_sequence` values, exact order.

### 11.3 Crash distinguishability

```
crates/simulation/tests/parity-bar/t_lifecycle.rs::crash_omits_finalize
```

A node with `crash_at_ms=5000` and no further restart produces a
`boot.json` and *no* `finalize.json`. The engine emits a sim-only
`Custom { kind: "crash" }` record on the recording so the
post-processor can distinguish crash from incomplete capture.

Binary: missing `finalize.json` exactly when crashed; `Custom`
record present.

### 11.4 Clean shutdown

```
crates/simulation/tests/parity-bar/t_lifecycle.rs::stop_emits_clean_finalize
```

A node with `stop_at_ms=5000` produces a `finalize.json` whose
`shutdown_reason = "clean"` (or whatever the prod schema names
clean shutdown).

Binary: `finalize.json` present with correct reason.

### 11.5 Mid-run finalization

```
crates/simulation/tests/parity-bar/t_lifecycle.rs::end_of_run_finalizes_all_living
```

When the engine reaches its terminal virtual time with nodes still
running, each living node receives a synthetic clean shutdown and a
`finalize.json` is written. The synthetic shutdown is marked
(`shutdown_reason = "end_of_run"`).

Binary: every node has either an organic finalize, a crash mark, or
an `end_of_run` finalize. No node ends a run with neither.

---

## 12. Test infrastructure integrity

Discharges this document itself.

### 12.1 Parity directory hash lock

```
scripts/check-parity-lock.sh
tests/parity-bar/.locked-hashes
```

CI's first step computes the recursive SHA-256 of every file under
`crates/simulation/tests/parity-bar/` (excluding `.locked-hashes`
itself) and compares against the value committed in `.locked-hashes`.
Mismatch fails the build with an instruction to run
`scripts/update-parity-lock.sh` in a *separate* commit titled
`parity-bar: update lock`.

This prevents the agent from quietly weakening tests. The same
guard applies to this file: `crates/simulation/TESTING_SPEC.md` is
inside the lock.

Binary: computed hash equals committed hash.

### 12.2 No probabilistic primitives in parity tests

```
crates/lint-deterministic/tests/no_probabilistic_primitives.rs
```

Within `crates/simulation/tests/parity-bar/`, the following are
banned (compile-error if present):

```
proptest::*
quickcheck::*
rand::random
rand::thread_rng
fn fuzz_*
```

Parity-bar tests use seeded, committed inputs only. Probabilistic
testing belongs in `crates/simulation/tests/exploratory/`, which is
not part of this spec's binary checks.

Binary: zero matches.

### 12.3 No conditional skips in parity tests

```
crates/lint-deterministic/tests/no_skips_in_parity.rs
```

Within `tests/parity-bar/`, the following are banned:

```
#[ignore]
#[cfg(not(...))]                          # any cfg gate
if std::env::var(...).is_ok() { return; } # env-driven skip
#[cfg_attr(..., ignore)]
```

Binary: zero matches.

### 12.4 No `unwrap()` masking in parity tests

```
crates/lint-deterministic/tests/parity_tests_use_explicit_assert.rs
```

Within `tests/parity-bar/`, `.unwrap()` on `Result` returned by
engine APIs must be `.expect("<reason>")` with a reason string. This
is a readability lint, not a correctness one, but parity-test
failures must be debuggable from CI output alone — the agent will
not be there to add prints when something fails.

Binary: zero `Result::unwrap()` calls without messages.

### 12.5 Reference scenario locked

The reference scenario (§6.5) and the fixture bundle (§1.1) are
inside the parity directory and therefore are §12.1-protected.

---

## 13. Definition of done

The simulator is considered built when:

- The full check list in §2–§12 runs in CI via `cargo xtask parity-bar`.
- Every check is green.
- The `parity-bar` job is required on the `ds-inference` branch
  before merge.
- `tests/parity-bar/PROVENANCE.md` documents the corpus and its
  source run.
- `crates/sim-detector/src/techniques.md` documents each detection
  technique in §10.2.
- `crates/lint-deterministic/banned.toml` enumerates §4.1's set.
- The reference scenario (§6.5) runs end-to-end and produces a
  bundle passing §6, §7, §8, §11.
- The corpus replay (§3.4) succeeds.

The agent does *not* mark the simulator built when:

- Any check is `#[ignore]`d, gated, or partially-skipped.
- Any check has been replaced with a weaker version.
- Any check passes only on debug or only on release.
- Any check passes only with a specific `RUSTFLAGS` setting.
- Any check requires manual setup not encoded in `cargo xtask`.

Calibration (SPEC §9) is explicitly out of scope for v1's done
criterion. The deterministic substrate this spec defines is what
makes calibration meaningful later.

---

## 14. Failure protocol

When a check defined in this spec fails, the agent's response is
exactly one of:

1. **Fix the simulator** to satisfy the check. This is the default.
2. **Fix the check** by raising a defect in this document, *if and
   only if* the check is wrong in a way that an independent reader
   can verify against NORTH_STAR / SPEC / OBSERVABILITY. The fix is
   a separate commit titled `TESTING_SPEC: correct §N.M`, including
   a justification quoting the upstream doc.

The agent's response is *never*:

- `#[ignore]` the test.
- Replace the test with a weaker one.
- Add a `// TODO: re-enable when …` marker.
- Move the test to `exploratory/`.
- Edit `.locked-hashes` directly.
- Loosen a banned-API list.
- Skip the test on a specific platform or feature.
- Mark the check "blocked on external work" and proceed.

If the agent finds itself unable to fix the simulator and unable to
write a justified spec correction, the work stops. The artifact
checked in is the partial work plus a `BLOCKED.md` in the simulation
crate root naming the check, the symptom, and the obstacle. This is
the only acceptable exit short of green.

---

## 15. Out of scope (v1)

The following are *not* obligations under this spec. The simulator
is not required to satisfy any check below for v1 to be considered
built. Each will become a v2 obligation as the corpus and the
calibration loop mature.

- **Statistical / distributional parity.** KS tests, mean / p99
  parity, tail-shape parity. These are calibration (SPEC §9), gated
  by the noise-floor estimation work that needs a multi-bundle
  corpus.
- **Multi-bundle corpus.** v1's schema floor (§6) is bounded by the
  single vastai-n3 bundle. Adding bundles widens the floor.
- **Wire-level packet trace** (OBSERVABILITY §4.1). Optional in v1;
  the sim implements §4.1 only if a check in §6 binds to it (it does
  not, in v1).
- **Causal trace IDs** (OBSERVABILITY §4.2). Future surface. §8.3's
  send-precedes-receive check uses heuristic matching; once trace
  IDs land, the check becomes exact and §8 expands.
- **Opaque-binary hosts** (SPEC §5.2). v1 ships with sim-native
  peers only. The opaque-binary escape hatch is a v2 obligation.
- **Cross-architecture determinism.** v1 commits to determinism on
  one architecture (linux-x86_64); same-architecture / different-machine
  determinism is the §2 obligation. Cross-arch is a v2 obligation
  if and when a target requires it.
- **Statistical detector techniques.** §10.2's set is structural /
  behavioral. Statistical detectors (e.g., "the variance of inter-message
  intervals matches a sim-implausible distribution") are a v2
  expansion.

Anything listed above that the agent finds an unforced opportunity
to satisfy is welcome to satisfy. None is required.

---

## 16. Glossary

Specific to this spec; for engine / sim terms see SPEC §10.

- **Binary check.** A test whose outcome is pass or fail with no
  intermediate states, thresholds, or knobs. The unit of contract in
  this document.
- **Closed list.** A set defined explicitly in this document, locked
  by §12.1, modifiable only by a separate commit that itself updates
  the lock. New items can be added; existing items cannot be removed
  or weakened.
- **Corpus.** The set of prod-origin bundles the simulator is checked
  against. v1 corpus is §1.1.
- **Detector.** A peer binary (§10.1) attempting to identify whether
  it runs in sim or prod. Used to enforce NORTH_STAR's
  "indistinguishable" requirement as an executable test.
- **Facade.** Per SPEC §3 / §10. Banned-API allowlist scopes
  reference this term.
- **Lock.** The SHA-256 hash committed in
  `tests/parity-bar/.locked-hashes` covering every file in the
  parity-bar directory, this document included.
- **Parity directory.** `crates/simulation/tests/parity-bar/`. The
  only directory whose contents are bound by this spec.
- **Reference scenario.** §6.5. The single scenario whose execution
  is observed by the schema-floor (§6), schema-round-trip (§7),
  causality (§8), and lifecycle (§11) checks.