1614 lines
70 KiB
Markdown
1614 lines
70 KiB
Markdown
# SIM_SPEC — simulator MVP
|
||
|
||
Status: draft, pre-implementation. Lives in `examples/pipeline-parallel-inference/`
|
||
so it is not touched by the ongoing simplification of `crates/simulation`. It
|
||
moves alongside the implementation once that cleanup lands.
|
||
|
||
This spec describes an architecture and an implementation strategy concretely
|
||
enough that two independent implementers, working from this document alone,
|
||
produce code that meshes. To that end it names components, the data that
|
||
crosses every boundary between them, and the behaviour each component owes
|
||
the others. Internal data structures, module layout, and helper types are the
|
||
implementer's call; the named boundaries are not.
|
||
|
||
The MVP's first consumer is SWIM, because SWIM is the algorithm whose
|
||
production failures motivated the simulator. The architecture is not SWIM-
|
||
specific: the engine knows about *hosts*, not about SWIM. A host is a piece
|
||
of code that consumes ticks and inbound messages and emits actions. SWIM is
|
||
the first such host kind; the second is whatever we need next.
|
||
|
||
---
|
||
|
||
## 0. Motivation
|
||
|
||
The pipeline-parallel-inference example failed eight live N≥3 vast.ai deploys.
|
||
The session report (`N3_DEPLOYMENT_REPORT.md`, next to this file) decomposes
|
||
the failure into three independent bugs stacked:
|
||
|
||
- **Layer A — relay-mediated head-of-line blocking.** iroh 0.96's
|
||
`RelayMode::Default` routed gossip through n0's canary relay, which
|
||
buffered SWIM traffic for 187 seconds. A shared queue servicing
|
||
multiple peers couples otherwise-independent traffic: a 9.8 KB Ack
|
||
from one peer delays every probe behind it on the same egress.
|
||
- **Layer B — SWIM gossip-flap (B1).** In a seven-minute run, the
|
||
orchestrator refuted Suspect claims against itself 228 times —
|
||
roughly once every 1.8 seconds — and one peer ended the run marked
|
||
Dead despite probes succeeding in both directions on both sides of
|
||
the link. The bug only appears with three or more peers, multi-
|
||
region latency, and enough cumulative gossip state for piggybacked
|
||
membership updates to grow into the multi-kilobyte range.
|
||
- **Layer C — internal-cause peer death.** Pipeline stages died
|
||
51–191 seconds into run #2 from worker-process exits, not from
|
||
network. The orchestrator recorded "connect timeout" because the
|
||
peer was gone. The failure mode is internal: the host emits a
|
||
diagnostic record and then halts of its own accord.
|
||
|
||
Catching any of these in production costs about two dollars of GPU rental
|
||
per attempt, forty-five to ninety minutes of engineer time per iteration,
|
||
and produces one non-reproducible bundle of evidence per run. The same
|
||
source, run twice, produces different outcomes.
|
||
|
||
The simulator exists to make the iteration loop sub-second and the
|
||
outcomes byte-identical for a fixed seed. It is not a complete model of
|
||
production; it is the smallest model that lets us reproduce A, B, and C
|
||
deterministically and tune SWIM (and the relay, and the stage lifecycle)
|
||
without deploying. Future algorithms layer onto the same engine without
|
||
changing the behaviour this MVP guarantees.
|
||
|
||
---
|
||
|
||
## 1. What "done" means
|
||
|
||
The MVP ships when these are simultaneously true.
|
||
|
||
A property test reproduces the gossip-flap bug (Layer B) deterministically
|
||
against the current SWIM source. The same scenario with the same seed
|
||
produces byte-identical output across runs and across the architectures we
|
||
claim to support.
|
||
|
||
A property test reproduces the relay-HOL flap (Layer A) deterministically
|
||
against the current SWIM source: a multi-peer scenario routed through one
|
||
relay with bounded egress, with cumulative gossip state growing into the
|
||
multi-kilobyte range, exhibits probe timeouts that disappear when the
|
||
relay's egress capacity is widened or the message-size cap from §10.1 is
|
||
enforced.
|
||
|
||
A scenario reproduces the run-#2 stage-death failure mode (Layer C): a
|
||
stage declared with a scheduled internal exit dies on schedule, emits a
|
||
`worker_exited` record with the declared reason, halts, and is observed by
|
||
the rest of the cluster through the same diagnostic channel production
|
||
uses.
|
||
|
||
The fix workflow does not deploy. A developer writes a property, runs the
|
||
sim, sees it fail, edits the SWIM source (or the relay policy, or the
|
||
stage-supervisor lifecycle), re-runs, sees it pass — all locally, in under
|
||
a second per iteration.
|
||
|
||
The simulator is calibration-grounded against the three N3 bundles we have
|
||
(`vastai-N3-1` canary relay + real worker; `vastai-N3-2` own relay + real
|
||
worker; `vastai-N3-stub` own relay + stub worker). Distributions emitted
|
||
by the sim, when configured to mirror a given N3 run, are within declared
|
||
tolerances of the corresponding live bundle.
|
||
|
||
Each known live failure mode has at least one scenario in the library, with
|
||
a prose comment naming what it reproduces.
|
||
|
||
---
|
||
|
||
## 3. Architecture
|
||
|
||
### 3.1 Components
|
||
|
||
The simulator is one process holding six named components. Each component
|
||
has one responsibility and one set of inbound and outbound message types.
|
||
The boundaries between them are the contract two independent implementers
|
||
must agree on; nothing inside a component is.
|
||
|
||
```
|
||
scenario.toml
|
||
│
|
||
▼
|
||
┌──────────────┐
|
||
│ scenario │ parses and validates the input
|
||
│ loader │
|
||
└──────┬───────┘
|
||
│ parsed scenario
|
||
▼
|
||
┌──────────────┐ send queries ┌──────────────┐
|
||
│ ├────────────────────►│ network │
|
||
│ engine │◄────────────────────┤ │
|
||
│ │ arrival times, └──────────────┘
|
||
│ │ drop reasons
|
||
│ │
|
||
│ │ ticks, recv ┌──────────────┐
|
||
│ ├────────────────────► │ hosts │
|
||
│ │◄──────────────────── │ (per peer) │
|
||
│ │ actions └──────────────┘
|
||
│ │
|
||
│ │ events, snapshots ┌──────────────┐
|
||
│ ├────────────────────► │ bundle │
|
||
│ │ │ writer │
|
||
└──────────────┘ └──────┬───────┘
|
||
│
|
||
▼
|
||
bundle on disk
|
||
│
|
||
▼
|
||
┌──────────────┐
|
||
│ assertion │
|
||
│ evaluator │
|
||
└──────┬───────┘
|
||
│
|
||
▼
|
||
verdicts on disk
|
||
```
|
||
|
||
**Scenario loader.** Reads a TOML scenario, validates it, returns a parsed
|
||
scenario value (§8). No state; pure function from path to validated
|
||
scenario.
|
||
|
||
**Engine.** Owns the virtual clock, the scheduling queue, the table of
|
||
hosts, references to the network and the bundle writer. Single entry point:
|
||
given a parsed scenario, run to completion. §4 specifies behaviour.
|
||
|
||
**Network.** A directed-graph link model. Answers send queries
|
||
deterministically and accepts mutations on a timeline. Holds no schedule of
|
||
its own; the engine pops events, the network answers questions. §5
|
||
specifies behaviour. The network also owns the **relay vertices** (§5A): a
|
||
relay is a first-class vertex of the network graph that is not a host. It
|
||
has one ingress queue and one egress queue per outbound link, drains them
|
||
at policy-declared capacities, and applies head-of-line ordering. Hosts do
|
||
not see whether their traffic was direct or relayed; the relay is opaque
|
||
to the host layer.
|
||
|
||
**Host.** An instance of some host kind, one per peer in the scenario. The
|
||
host kinds for the MVP are two: the production SWIM state machine wrapped
|
||
in a thin adapter (§6.2) and the **pipeline-stage host kind** (§6A) that
|
||
wraps the production stage supervisor lifecycle and emits the diagnostic
|
||
records the N3 deployment report's C.1–C.3 findings require. The host
|
||
trait — what the engine calls and what the host returns — is §6.
|
||
|
||
**Bundle writer.** The only filesystem-touching component. Receives event
|
||
and snapshot records from the engine, serializes them to the production
|
||
diagnostics schema, writes them to a bundle directory. §9 specifies output.
|
||
|
||
**Assertion evaluator.** Post-run reader of the bundle. Evaluates each
|
||
declared assertion against the event stream and snapshot directory. Emits a
|
||
verdict file. §10 specifies behaviour.
|
||
|
||
### 3.2 Data flow
|
||
|
||
Each boundary is named below with the data that crosses it.
|
||
|
||
**scenario.toml → scenario loader.** A TOML file. §8 names the schema.
|
||
|
||
**scenario loader → engine.** A validated parsed scenario value. The fields
|
||
are exactly those §8 names; the engine consumes nothing else from outside.
|
||
|
||
**engine ⇄ network.** Two query methods, no others.
|
||
|
||
- `send(from, to, byte_len, sent_at_ns) → SendOutcome`. `SendOutcome` is
|
||
either `Arrive { at_ns }` or `Drop { reason }`. The network mutates its
|
||
per-link state (last-send-time, warm/cold) as it answers.
|
||
- `apply_mutation(mutation, at_ns) → Vec<InvalidatedDelivery>`. The network
|
||
updates its internal state and returns a list of currently-scheduled
|
||
deliveries the mutation invalidates. The engine removes those from its
|
||
queue.
|
||
|
||
**engine ⇄ host.** Three call sites, no others. The host responds to each
|
||
with `Vec<Action>`; the engine processes actions in returned order.
|
||
|
||
- `tick(now_ns)` — fired at the host's tick instants.
|
||
- `recv(message, now_ns)` — fired when a delivery event for this host pops.
|
||
- `snapshot() → SnapshotBytes` — fired on snapshot dispatch.
|
||
|
||
`Action` is a closed enum the engine handles exhaustively:
|
||
|
||
- `Send { to: HostId, message: HostMessage }` — engine asks the codec for
|
||
the message's byte length, asks the network for arrival time, schedules a
|
||
`Deliver` event or notifies the sender of `SendFailed` (next bullet).
|
||
- `RecordEvent { event: EventBytes }` — engine forwards to the bundle
|
||
writer with the current virtual time.
|
||
- `ScheduleTimer { at_ns, token: TimerToken }` — engine schedules a
|
||
`TimerFired` event delivered through `recv` at `at_ns`.
|
||
- `Halt` — engine stops issuing further ticks to this host. Inbound recv
|
||
still flows (so the host can observe drains) until the host's `recv`
|
||
itself returns `Halt`.
|
||
|
||
The engine never invents actions; the host produces them. The engine never
|
||
silently drops actions; an unknown variant aborts the run.
|
||
|
||
**engine → bundle writer.** A single method, `write_record(record)`. The
|
||
record is one of:
|
||
|
||
- `EventRecord { virtual_time_ns, host_id, kind_tag, event_bytes }` —
|
||
produced by host `RecordEvent` actions and by engine-synthesized events
|
||
(drops, cold-dial penalty firings, cache-invalidate firings, send-failure
|
||
notifications).
|
||
- `SnapshotRecord { virtual_time_ns, host_id, snapshot_bytes }` — produced
|
||
by snapshot dispatch.
|
||
- `MutationRecord { virtual_time_ns, mutation }` — produced when a mutation
|
||
pops.
|
||
|
||
The bundle writer is append-only and accepts records in arbitrary order.
|
||
It is the writer's responsibility to organize records into the bundle
|
||
layout in §9.
|
||
|
||
**bundle → assertion evaluator.** The evaluator reads the finished bundle
|
||
from disk. Its input is the bundle path; its output is a `verdicts.json`
|
||
written into the same bundle. §10 names the verdict format.
|
||
|
||
### 3.3 Host kinds and the codec contract
|
||
|
||
A host kind is a `(host_trait_impl, codec)` pair. The codec exists because
|
||
the bandwidth model needs to know how many bytes a host's outgoing message
|
||
will occupy on the wire, and that number must equal what the production
|
||
transport would put on the wire for the same message, or the simulator's
|
||
bandwidth-driven failure modes diverge from the live ones.
|
||
|
||
For each host kind, the codec exposes:
|
||
|
||
- `encode(message) → bytes`.
|
||
- `decode(bytes) → message`.
|
||
- `kind_tag() → str`.
|
||
|
||
A contract test asserts byte-equality between the sim's encode path and the
|
||
production transport's encode path for a representative message set;
|
||
divergence breaks the build.
|
||
|
||
For the MVP, the SWIM host kind reuses the production transport's encoding
|
||
logic directly. If production code is too tangled with iroh to import
|
||
cleanly, the encoding is factored into a small shared module that both the
|
||
production transport and the sim host call; that refactor is part of the
|
||
MVP, not deferred.
|
||
|
||
### 3.4 Independent buildability
|
||
|
||
Each component in §3.1 can be built by an independent agent against the
|
||
contracts in §3.2 and §3.3 alone. Specifically:
|
||
|
||
- The scenario loader is built against §8.
|
||
- The network is built against §5 and the substream rule in §7.
|
||
- The engine is built against §4 plus the network and host call signatures.
|
||
- The SWIM host is built against §6 and the production SWIM API.
|
||
- The bundle writer is built against §9.
|
||
- The assertion evaluator is built against §10.
|
||
|
||
A change to a component's *internal* structure is invisible to the others.
|
||
A change to a contract in §3.2 / §3.3 is a spec amendment.
|
||
|
||
---
|
||
|
||
## 4. The engine
|
||
|
||
The engine owns the virtual clock, the scheduling queue, the host table,
|
||
the network reference, and the bundle-writer reference.
|
||
|
||
### 4.1 The scheduling queue
|
||
|
||
The queue is a priority queue over `(virtual_time_ns, sequence_number)`
|
||
keys. The sequence number is assigned monotonically on enqueue; it is the
|
||
only tie-break mechanism the engine permits. Two events at the same virtual
|
||
time pop in enqueue order.
|
||
|
||
Each entry carries one of:
|
||
|
||
- `Tick { host_id }`.
|
||
- `Deliver { host_id, message_bytes }`.
|
||
- `TimerFired { host_id, token }`.
|
||
- `Mutation { mutation }`.
|
||
- `Snapshot`.
|
||
- `Terminate`.
|
||
|
||
Pre-population: at engine start, one `Tick` is enqueued per host at the
|
||
host's first-tick virtual time, one `Mutation` per scenario mutation, one
|
||
`Snapshot` per scenario snapshot request, and one `Terminate` at
|
||
`duration_ns`.
|
||
|
||
### 4.2 The main loop
|
||
|
||
Pop the smallest entry. Advance the virtual clock to its time. Dispatch by
|
||
kind (§4.3 through §4.7). Repeat until `Terminate` pops or the
|
||
early-termination condition fires (§4.8). After termination, finalize the
|
||
bundle and run the assertion evaluator.
|
||
|
||
The clock advances *only* on pop. Nothing in the engine reads any other
|
||
clock, virtual or real.
|
||
|
||
### 4.3 Tick dispatch
|
||
|
||
For `Tick { host_id }`:
|
||
|
||
1. Call `host[host_id].tick(now_ns)`. If the host has `Halt`-ed, skip the
|
||
call but still schedule the next tick (the host may un-halt only via a
|
||
`PeerResurrect` mutation).
|
||
2. Process the returned action list in order (§4.6).
|
||
3. Enqueue the next `Tick { host_id }` at
|
||
`now_ns + tick_period_ns[host_id]`.
|
||
|
||
The first tick for each host is offset by a stable, seed-derived per-host
|
||
offset. The offset is drawn from the host's RNG substream (§7) so peers do
|
||
not tick on the same virtual instants and silent symmetry artefacts do not
|
||
mask real timing bugs.
|
||
|
||
### 4.4 Delivery dispatch
|
||
|
||
For `Deliver { host_id, message_bytes }`:
|
||
|
||
1. If `host[host_id]` is halted or killed, drop the delivery and emit a
|
||
`DropOnDelivery` engine-synthesized event.
|
||
2. Otherwise, decode the bytes via the host kind's codec.
|
||
3. Call `host[host_id].recv(message, now_ns)`.
|
||
4. Process the returned action list in order (§4.6).
|
||
|
||
### 4.5 Other dispatches
|
||
|
||
- `TimerFired { host_id, token }` is delivered through `recv` with a
|
||
`TimerFired(token)` envelope and processed identically to a network
|
||
delivery.
|
||
- `Mutation { mutation }` is forwarded to `network.apply_mutation`. The
|
||
returned list of invalidated deliveries is removed from the queue (or
|
||
tombstoned — the visible behaviour is identical). The engine emits a
|
||
`MutationRecord` to the bundle writer.
|
||
- `Snapshot` asks every live host for its `snapshot()` and forwards each
|
||
result to the bundle writer as a `SnapshotRecord`.
|
||
- `Terminate` ends the main loop.
|
||
|
||
### 4.6 Action processing
|
||
|
||
For each action returned by a host:
|
||
|
||
- `Send { to, message }` — encode via the codec, ask the network with the
|
||
resulting byte length. On `Arrive`, enqueue a `Deliver` at the returned
|
||
time. On `Drop`, emit a `DropOnSend` engine-synthesized event *and*
|
||
deliver a `SendFailed` envelope to the sender via the same recv path the
|
||
production transport would.
|
||
- `RecordEvent { event }` — forward to the bundle writer with current
|
||
virtual time.
|
||
- `ScheduleTimer { at_ns, token }` — enqueue `TimerFired` at `at_ns`.
|
||
- `Halt` — mark the host halted (§4.3 covers re-entry).
|
||
|
||
### 4.7 Engine-synthesized events
|
||
|
||
The engine emits records for behaviours hosts do not see directly:
|
||
|
||
- `DropOnSend { from, to, reason }` — a send the network refused.
|
||
- `DropOnDelivery { to, reason }` — a delivery the engine refused at
|
||
arrival time (host killed mid-flight).
|
||
- `CacheStateChange { from, to, transition }` — the link warmed, the link
|
||
went cold by idle, the link was invalidated by a mutation. The network
|
||
surfaces these to the engine through a side-channel on `send` and
|
||
`apply_mutation`.
|
||
- `DialStart` / `DialOutcome` — emitted whenever the cold-dial penalty
|
||
fires on a send.
|
||
|
||
These records exist so the bundle's observable surface matches what
|
||
production diagnostics emit for the same activity. The engine never
|
||
suppresses them and never emits them for activity that did not happen.
|
||
|
||
### 4.8 Early termination
|
||
|
||
A scenario may set `early_terminate_on_all_assertions_resolved = true`. If
|
||
set, after each event dispatch the engine polls the assertion evaluator's
|
||
streaming side (§10.4); if every declared assertion has a resolved verdict,
|
||
the engine fast-forwards to `Terminate`. The bundle still records every
|
||
event that fired up to that point.
|
||
|
||
### 4.9 Time unit
|
||
|
||
The engine's virtual clock is in integer nanoseconds. The manifest records
|
||
the unit so post-processors render times consistently.
|
||
|
||
### 4.10 Behavioral tests
|
||
|
||
The engine's contract is the dispatch and action-processing behaviour of
|
||
§4. Its tests assert that it has the properties below; how each is
|
||
verified is the test author's call.
|
||
|
||
**Tick cadence.** Each host receives ticks at its declared period,
|
||
starting at a stable seed-derived offset. The offset is identical
|
||
across runs and differs across hosts in the same scenario.
|
||
|
||
**Action ordering.** A host's emitted actions are processed in returned
|
||
order. The effects of action N are fully observable before action N+1's
|
||
effects begin.
|
||
|
||
**Send semantics.** A `Send` whose network query returns `Arrive` results
|
||
in the recipient's `recv` being called at the returned arrival time with
|
||
the codec-produced bytes. A `Send` the network drops results in the
|
||
sender's `recv` being called with `SendFailed`, a `DropOnSend` record in
|
||
the bundle, and no recipient call.
|
||
|
||
**Timer fidelity.** A `ScheduleTimer { at_ns, token }` causes a
|
||
`TimerFired(token)` envelope to reach the host's `recv` at exactly
|
||
`at_ns`.
|
||
|
||
**Halt.** A halted host receives no further ticks; it continues to
|
||
receive deliveries.
|
||
|
||
**Closed action set.** An action outside the closed set §4.6 names
|
||
aborts the run with a structured error. The engine never silently
|
||
ignores or invents an action.
|
||
|
||
**Tie-break.** Events scheduled at the same virtual time pop in enqueue
|
||
order. The order is identical across runs and architectures.
|
||
|
||
**Mutation propagation.** Deliveries the network invalidates are removed
|
||
from the engine's queue; each emits a `DropOnDelivery` record at the
|
||
mutation's virtual time. No invalidated delivery reaches a host's `recv`.
|
||
|
||
**Snapshot fanout.** A scheduled snapshot produces exactly one record
|
||
per live host at the scheduled virtual time.
|
||
|
||
**Early termination is clean.** Every record emitted before the
|
||
termination time is preserved; no record carries a later virtual time.
|
||
|
||
**Determinism.** Same scenario, same seed ⇒ byte-identical
|
||
`events.ndjson`.
|
||
|
||
---
|
||
|
||
## 5. The network
|
||
|
||
### 5.1 Topology
|
||
|
||
A directed graph. Vertices are the host IDs declared in the scenario.
|
||
Edges carry link policies. An ordered pair with no declared edge is
|
||
permanently partitioned; the network returns `Drop(NoRoute)` for any send
|
||
on it. This is distinct from a temporary partition mutation, which can heal.
|
||
|
||
Asymmetry is allowed and intended: `policy(A→B)` and `policy(B→A)` are
|
||
independent.
|
||
|
||
### 5.2 Link policy
|
||
|
||
Each edge carries the following integer-valued fields:
|
||
|
||
- `latency_ns` — base one-way delivery time.
|
||
- `jitter_stddev_ns` — symmetric jitter; samples are drawn from a
|
||
precomputed integer lookup table approximating a standard-normal
|
||
distribution scaled by this stddev (§7 forbids float math in decisions).
|
||
- `loss_prob_ppm` — independent drop probability per send, parts-per-
|
||
million.
|
||
- `reorder_prob_ppm` — probability of inserting extra delay sufficient to
|
||
swap delivery order with the next message on the same edge.
|
||
- `bandwidth_bps` — bytes per second. A message of N bytes occupies the
|
||
link for `(N * 1_000_000_000) / bandwidth_bps` ns.
|
||
- `cold_dial_penalty_ns` — extra latency added when the link is cold.
|
||
- `cache_warm_after_ns` — wall of warm-time after first contact before
|
||
subsequent sends are warm.
|
||
- `cache_invalidate_after_idle_ns` — idle duration after which the link
|
||
returns to cold.
|
||
|
||
### 5.3 Per-link state
|
||
|
||
Each edge tracks:
|
||
|
||
- `last_send_ns` — virtual time of the last `send` that returned `Arrive`.
|
||
- `last_arrive_ns` — virtual time of the latest scheduled arrival; used by
|
||
the bandwidth model for the next message's serialization start.
|
||
- `cache_state` — `Cold`, `Warming(since_ns)`, or `Warm`.
|
||
|
||
State transitions happen inside `send` and `apply_mutation`.
|
||
|
||
### 5.4 The send algorithm
|
||
|
||
`send(from, to, byte_len, sent_at_ns) → SendOutcome`. Steps:
|
||
|
||
1. If the edge does not exist, return `Drop(NoRoute)`.
|
||
2. If the active partition set cuts `(from, to)`, return `Drop(Partitioned)`.
|
||
3. If a `LossBurst` mutation is active for this edge, use its override
|
||
probability; otherwise use the edge's `loss_prob_ppm`. Draw a u32 from
|
||
the edge's RNG substream; if `draw % 1_000_000 < prob_ppm`, return
|
||
`Drop(Lossy)`.
|
||
4. Compute `serialization_start = max(sent_at_ns, last_arrive_ns)`. Compute
|
||
`serialization_end = serialization_start + (byte_len * 1e9 / bandwidth_bps)`.
|
||
5. Compute `arrival = serialization_end + latency_ns + jitter_sample`,
|
||
where `jitter_sample` is one draw from the per-link substream into the
|
||
integer-Gaussian table, scaled by `jitter_stddev_ns`.
|
||
6. If a `LatencySpike` mutation is active, multiply the additive latency
|
||
contribution (latency + jitter) by the spike factor before adding.
|
||
7. If a `RelayBuffer` mutation is active, take `arrival =
|
||
max(arrival, sent_at_ns + floor_ns)`.
|
||
8. If `cache_state` is `Cold`, add `cold_dial_penalty_ns` to `arrival` and
|
||
transition `cache_state` to `Warming(now)`. Emit a `DialStart` side-
|
||
channel notification to the engine and a `DialOutcome` at the arrival
|
||
time.
|
||
9. If the reorder draw fires, add enough delay so this message arrives
|
||
after the next message scheduled on this edge.
|
||
10. Update `last_send_ns = sent_at_ns`, `last_arrive_ns = arrival`. If
|
||
`cache_state` is `Warming(since)` and `now - since >= cache_warm_after_ns`,
|
||
transition to `Warm` and emit a `CacheStateChange`.
|
||
11. Return `Arrive(arrival)`.
|
||
|
||
If `last_send_ns - now > cache_invalidate_after_idle_ns` at the start of a
|
||
send, the link returns to `Cold` and emits a `CacheStateChange` before
|
||
proceeding.
|
||
|
||
### 5.5 Mutations
|
||
|
||
Supported kinds:
|
||
|
||
- `Partition { peers_a, peers_b }` — set the active partition to cut every
|
||
edge between the two groups in both directions. In-flight messages on
|
||
cut edges are returned in the invalidated-deliveries list.
|
||
- `Heal` — clear the active partition set. No in-flight invalidations.
|
||
- `LatencySpike { links, factor_x100, duration_ns }` — multiply additive
|
||
latency on named links by `factor_x100 / 100` for a duration. In-flight
|
||
messages are not retroactively delayed.
|
||
- `LossBurst { links, prob_ppm, duration_ns }` — override loss probability
|
||
on named links for a duration.
|
||
- `RelayBuffer { links, floor_ns, duration_ns }` — impose a minimum
|
||
delivery delay on named links for a duration. (Legacy per-edge floor;
|
||
the relay vertex of §5A models shared-queue HOL more faithfully.)
|
||
- `PeerKill { peer }` — drop the peer's inbox. In-flight deliveries to the
|
||
peer are invalidated. The peer's ticks are stopped by the engine.
|
||
- `PeerResurrect { peer, preserve_state }` — restart the peer. If
|
||
`preserve_state`, the engine reuses the host instance; otherwise a fresh
|
||
host of the same kind is instantiated from the scenario's peer
|
||
declaration.
|
||
- `WorkerExit { peer, reason, status_code?, signal? }` — delivers a
|
||
`WorkerExit` envelope to the named peer at the mutation's time via the
|
||
same recv path that `TimerFired` and `SendFailed` use today. The host
|
||
kind decides what to do with it; the stage host (§6A) emits a
|
||
`worker_exited` event and returns `Halt`. Targeting a host kind that
|
||
does not accept the envelope (e.g. the SWIM kind) aborts the run with a
|
||
structured error — silent fallback is the bug class the simulator
|
||
exists to prevent.
|
||
- `RelayKill { relay }` — the relay stops forwarding. All currently
|
||
queued messages are returned in the invalidated-deliveries list.
|
||
Subsequent sends through the relay drop with reason `RelayDown`.
|
||
- `RelayBoot { relay }` — the relay returns to service. Its queues are
|
||
empty; the next forwarded message pays the `cold_start_penalty_ns`.
|
||
- `RelayCapacityChange { relay, ingress_capacity_bps?,
|
||
egress_capacity_bps_per_link?, queue_depth_bytes? }` — at the named
|
||
time, replace any subset of the three relay policy fields (§5A.2).
|
||
In-flight messages already past ingress complete at their previously-
|
||
computed arrival times; messages that arrive after the mutation use
|
||
the new policy.
|
||
|
||
### 5.6 Determinism within the network
|
||
|
||
Every random draw the network makes comes from a substream keyed by
|
||
`("link", from_id, to_id)` (§7). Editing one link's policy must not
|
||
perturb the draws on any other link.
|
||
|
||
### 5.7 Behavioral tests
|
||
|
||
The network is a pure function of (state, query). Its tests assert that
|
||
it has the properties §5 names; how each property is verified is the
|
||
test author's call.
|
||
|
||
**Reachability.** A `send` over an ordered pair returns `Arrive` iff
|
||
that pair is declared as an edge and the active partition set does not
|
||
cut it. A `send` the network refuses leaves the network's state
|
||
unchanged.
|
||
|
||
**Partition heals to identity.** A `Partition` followed by a `Heal` at
|
||
later virtual times leaves the network indistinguishable on subsequent
|
||
sends from one that experienced neither.
|
||
|
||
**Mutation invalidation is exact.** Every delivery a mutation renders
|
||
impossible appears in the mutation's invalidated-deliveries return. No
|
||
delivery the mutation does not invalidate appears in that return.
|
||
|
||
**Loss is Bernoulli.** Drops on a link are independent draws with the
|
||
link's declared probability. A `LossBurst` substitutes its override
|
||
probability for the duration it names and only the duration it names.
|
||
|
||
**Bandwidth serializes.** A link with finite bandwidth never overlaps
|
||
two messages' wire-occupancy intervals: each message's arrival is
|
||
delayed at least until the previous message's arrival plus that
|
||
message's transmission time.
|
||
|
||
**Latency is additive.** A send's arrival decomposes into base latency,
|
||
serialization delay, jitter, cold-dial penalty when applicable, and the
|
||
contributions of active mutations. The terms are independent in the
|
||
policy and combine without interaction beyond what §5.4 specifies.
|
||
|
||
**Jitter is symmetric and integer-valued.** Jitter samples come from
|
||
the precomputed table of §7, are symmetric around zero, and are never
|
||
non-integer.
|
||
|
||
**Cache state follows traffic.** The link's `cache_state` reflects
|
||
recent traffic: warm after sufficient activity, cold again after
|
||
sufficient idleness, with the thresholds the policy names. The
|
||
cold-dial penalty is paid by exactly the sends the network classifies
|
||
cold.
|
||
|
||
**Cache transitions are observable.** Every cold↔warm transition emits
|
||
exactly one `CacheStateChange` notification at the transition's virtual
|
||
time. No transition is silent and no notification fires without a
|
||
transition.
|
||
|
||
**Mutation scoping.** A mutation affects exactly the links its `links`
|
||
field names and exactly the duration it declares. Sends on other links,
|
||
or on the named links outside the duration, are unaffected.
|
||
|
||
**Substream isolation.** Editing one link's policy does not change any
|
||
draw the network makes on any other link. This is the property that
|
||
makes bisecting scenario edits possible.
|
||
|
||
**Determinism.** Same topology, same seed, same query sequence ⇒
|
||
identical `SendOutcome` sequence and identical invalidated-deliveries
|
||
returns.
|
||
|
||
---
|
||
|
||
## 5A. The relay vertex
|
||
|
||
A relay is a vertex of the network's directed graph that is not a host.
|
||
Relay IDs share the ID namespace with host IDs (§5.1 and §8 of this
|
||
spec together require uniqueness across both sets). An edge's `from`
|
||
and `to` may name either.
|
||
|
||
A route between two hosts is either direct — exactly one edge from `A`
|
||
to `B` — or relayed — an edge `A→R`, the relay `R`, and an edge
|
||
`R→B`. Multi-hop relayed routes (`A→R₁→R₂→B`) are out of scope; the
|
||
scenario loader rejects them in §8.2.
|
||
|
||
If a scenario declares both a direct `A→B` edge and a relayed `A→R→B`
|
||
route, the scenario loader rejects the ambiguity. Each ordered host
|
||
pair has at most one route.
|
||
|
||
### 5A.2 Relay policy
|
||
|
||
Each relay declares the following integer-valued fields:
|
||
|
||
- `ingress_capacity_bps` — maximum bytes per second the relay accepts
|
||
*across all inbound links combined*.
|
||
- `egress_capacity_bps_per_link` — maximum bytes per second the relay
|
||
serves *per outbound link*.
|
||
- `queue_depth_bytes` — maximum bytes buffered across all egress
|
||
queues combined. A message that would push the total beyond this
|
||
limit at enqueue time is dropped.
|
||
- `queue_discipline` — `Fifo` is the only value the MVP accepts.
|
||
- `cold_start_penalty_ns` — extra latency added to the first message
|
||
the relay forwards after a `RelayBoot` mutation.
|
||
|
||
A relay has no jitter, loss, or cache-state fields of its own; the
|
||
edges feeding it carry their own such fields per §5.2. A relay's drop
|
||
reasons are exclusively queue-overflow and `RelayDown`; lossy drops
|
||
remain a property of edges.
|
||
|
||
### 5A.3 The forward algorithm
|
||
|
||
When the network receives `send(from, to, byte_len, sent_at_ns)` and
|
||
the configured route is relayed through `R`, the composition is:
|
||
|
||
1. Resolve the inbound edge `from → R`. Apply §5.4 for the inbound
|
||
leg. If that leg drops, the composed send drops with the same
|
||
reason; the relay's state is not consulted.
|
||
2. At time `arrival_at_R`, attempt to enqueue at the relay. If
|
||
`enqueued_bytes + byte_len > queue_depth_bytes`, the composed send
|
||
drops with reason `RelayQueueFull` and the relay emits a
|
||
`RelayDrop` record (§9.1).
|
||
3. Otherwise, compute the ingress serialization end:
|
||
`max(arrival_at_R, ingress_queue_tail_ns) +
|
||
(byte_len * 1_000_000_000) / ingress_capacity_bps`.
|
||
4. Compute the egress serialization start on the outbound link to
|
||
`to`: `max(ingress_end, egress_queue_tail_ns[to])`. If the relay
|
||
is `Booting`, add `cold_start_penalty_ns` and transition to
|
||
`Booted`. Compute `egress_end = egress_start + (byte_len *
|
||
1_000_000_000) / egress_capacity_bps_per_link`.
|
||
5. Resolve the outbound edge `R → to` per §5.4 with `sent_at_ns =
|
||
egress_end`. Its arrival is the composed send's arrival.
|
||
6. The relay emits a `RelayEnqueue` at `arrival_at_R` and a
|
||
`RelayDequeue` at `egress_end` (§9.1).
|
||
|
||
The composed send returns one `Arrive(at_ns)` or one `Drop(reason)`;
|
||
the relay's internal events are surfaced through the side channel
|
||
that §3.2 already names for cache state changes.
|
||
|
||
A relay's egress and ingress are independent: the ingress can be
|
||
serializing a new message while the egress is still draining an old
|
||
one. Head-of-line blocking arises only when two messages share an
|
||
egress link (or the single ingress).
|
||
|
||
### 5A.4 Determinism within the relay
|
||
|
||
The relay draws no randomness in the MVP. Iteration over per-egress-
|
||
link state inside a relay is by destination host ID in lexicographic
|
||
order, per §7.3.
|
||
|
||
### 5A.5 Behavioral tests
|
||
|
||
The relay's tests assert that it has the properties below; how each
|
||
is verified is the test author's call. (Tests live in
|
||
`tests/relay_invariants.rs`.)
|
||
|
||
- **Composition is transparent to hosts.** A send through a relayed
|
||
route returns one `SendOutcome` shaped identically to a direct
|
||
send's. The receiving host cannot distinguish a relayed delivery
|
||
from a direct one by the message it sees.
|
||
- **HOL is observable and bounded.** Two messages sharing an egress
|
||
arrive in send order, separated by at least the first message's
|
||
egress serialization time.
|
||
- **Ingress and egress are independent.** A message destined for peer
|
||
X does not delay a message destined for peer Y on the egress side.
|
||
- **Queue overflow is exact.** A send that would push
|
||
`enqueued_bytes` strictly above `queue_depth_bytes` at its
|
||
arrival-at-relay time drops with `RelayQueueFull` and emits a
|
||
`RelayDrop` record. A send that exactly fills the queue is
|
||
accepted.
|
||
- **Cold-start penalty is paid once per boot.** After a `RelayBoot`,
|
||
the first forwarded message includes the cold-start penalty; the
|
||
second does not.
|
||
- **Mutation invalidation is exact.** A `RelayKill` returns exactly
|
||
the deliveries in-flight through the relay at the mutation's
|
||
virtual time and no others. A `RelayCapacityChange` invalidates no
|
||
deliveries.
|
||
- **Ambiguous routes are rejected statically.** A scenario declaring
|
||
both a direct edge `A→B` and a relayed route `A→R→B`, or two
|
||
relayed routes for the same ordered host pair, is rejected by the
|
||
loader.
|
||
- **No multi-hop in MVP.** A scenario declaring a route that
|
||
traverses two relays is rejected by the loader.
|
||
- **Determinism.** Same topology, same seed, same query sequence ⇒
|
||
identical composed `SendOutcome` sequence and identical
|
||
`RelayEnqueue` / `RelayDequeue` / `RelayDrop` record streams.
|
||
|
||
---
|
||
|
||
## 6. Hosting an entity
|
||
|
||
### 6.1 The host trait
|
||
|
||
A host kind implements:
|
||
|
||
- `fn id(&self) -> HostId`.
|
||
- `fn kind_tag() -> &'static str`. Used for routing and bundle tagging.
|
||
- `fn tick(&mut self, now_ns: u64) -> Vec<Action>`.
|
||
- `fn recv(&mut self, message: HostMessage, now_ns: u64) -> Vec<Action>`.
|
||
- `fn snapshot(&self) -> SnapshotBytes`.
|
||
|
||
`HostMessage` is either a decoded inbound application message (the host
|
||
kind's own type, dispatched via the codec) or a `TimerFired(token)` or a
|
||
`SendFailed { to, reason }` envelope.
|
||
|
||
A host kind also exposes:
|
||
|
||
- `fn new_from_config(id: HostId, config: HostKindConfig, rng: HostRng) -> Self`.
|
||
- A codec (§3.3).
|
||
- A validation routine for its `HostKindConfig` (used by the scenario
|
||
loader; §8).
|
||
|
||
### 6.2 The SWIM host kind
|
||
|
||
The SWIM host wraps the production SWIM state machine without re-
|
||
implementing it. It:
|
||
|
||
- Constructs the production state machine with the scenario's per-peer
|
||
config.
|
||
- Installs the production diagnostics emitter against a shim that pushes
|
||
every emission into a per-tick / per-recv `Vec<Action>` as
|
||
`RecordEvent` actions.
|
||
- Installs the production tier-2 introspector against the host's
|
||
`snapshot()` method so the snapshot bytes are exactly what production
|
||
emits.
|
||
- Dispatches `recv` to the production handler matching the message kind
|
||
(ping, ack, ping-request, indirect-ack, join-request, join-response).
|
||
- Translates production state-machine output (outgoing messages, timer
|
||
requests) into `Send` and `ScheduleTimer` actions.
|
||
|
||
When the production state machine emits an output the host adapter does
|
||
not know how to route — a new message kind in a future SWIM version, for
|
||
example — the adapter panics. Silent fallback is exactly the class of bug
|
||
the simulator is meant to prevent.
|
||
|
||
The adapter does *not* substitute for any production logic. Its job is
|
||
purely translation between production data types and the host trait.
|
||
|
||
### 6.3 Adding a new host kind
|
||
|
||
Adding a new host kind is a strict superset operation:
|
||
|
||
1. Implement the host trait against the new algorithm.
|
||
2. Provide a codec.
|
||
3. Add a `kind` arm to the scenario loader's peer-declaration parser.
|
||
4. Register the kind with the engine's host-instantiation factory.
|
||
|
||
Existing host kinds continue to work without change. The network, the
|
||
bundle writer, the engine main loop, and the determinism contract are
|
||
host-kind-agnostic.
|
||
|
||
### 6.4 Behavioral tests
|
||
|
||
Host-kind tests come in two layers: kind-agnostic properties every
|
||
registered kind must satisfy, and per-kind properties specific to the
|
||
algorithm a kind hosts. The list below is what the tests must assert;
|
||
how is the test author's call.
|
||
|
||
**Trait conformance (every kind).** The kind exposes the §6.1 surface
|
||
with the §6.1 signatures. `kind_tag()` is a non-empty string unique
|
||
among registered kinds.
|
||
|
||
**Codec is invertible (every kind).** Encoding then decoding a message
|
||
is the identity on the kind's message type.
|
||
|
||
**Host determinism (every kind).** Same `HostKindConfig`, same RNG
|
||
seed, same `tick`/`recv` sequence ⇒ identical action sequence.
|
||
|
||
**SWIM emits no novel kinds.** Every event a SWIM host emits is of a
|
||
kind production's diagnostics also emits. The simulator invents no SWIM
|
||
event kind for itself.
|
||
|
||
**SWIM codec parity with production.** The SWIM host's encoding of an
|
||
outgoing message is byte-identical to the production transport's
|
||
encoding of the same message. Drift breaks the build.
|
||
|
||
**SWIM snapshot parity with production.** A SWIM host's `snapshot()`
|
||
conforms to the production tier-2 SWIM-state schema.
|
||
|
||
**SWIM unknown-output is loud.** A production state-machine output the
|
||
SWIM adapter does not route aborts the run with a structured error.
|
||
Silent fallback is a test failure.
|
||
|
||
---
|
||
|
||
## 6A. The pipeline-stage host kind
|
||
|
||
The stage host wraps the production stage supervisor — the worker
|
||
lifecycle implemented in `examples/pipeline-parallel-inference` and
|
||
running today as `pp-gpu-node`. The wrap is structurally analogous
|
||
to §6.2's SWIM host.
|
||
|
||
The MVP stage host does not engage in inter-stage application traffic
|
||
(activation forwarding, KV-cache updates). The failures Layer C
|
||
exhibits are lifecycle failures, not application-protocol failures;
|
||
inter-stage traffic is a strict superset and is out of scope for this
|
||
extension.
|
||
|
||
### 6A.2 Lifecycle
|
||
|
||
The stage host moves through four states, each transition emitting
|
||
exactly one diagnostic record:
|
||
|
||
- `Cold` — initial state on `new_from_config`. No actions until the
|
||
first `tick`.
|
||
- `Registering` — entered on the first `tick`. The host emits a
|
||
`register_name` event (§9.1) with the host's declared name and
|
||
address, then transitions to `Running`.
|
||
- `Running` — steady state. The host emits no further state-
|
||
transition records on its own.
|
||
- `Halted` — entered on receipt of a `WorkerExit { reason }` envelope
|
||
or after a `PeerKill` mutation. The host emits a `worker_exited`
|
||
event (§9.1) with the reason, then returns `Halt`.
|
||
|
||
Transitions are linear: `Cold → Registering → Running → Halted`.
|
||
There is no resurrection. A `PeerResurrect` mutation against a stage
|
||
host produces a fresh instance from the scenario's peer declaration
|
||
per §5.5; the resurrected instance starts in `Cold`.
|
||
|
||
### 6A.3 Internal-cause exit
|
||
|
||
A `WorkerExit { reason }` envelope arrives via the same recv path as
|
||
`TimerFired` and `SendFailed`. The stage host's `recv` for that
|
||
envelope returns exactly three actions, in order:
|
||
|
||
1. `RecordEvent` carrying a `worker_exited` event whose payload
|
||
names the reason verbatim.
|
||
2. `RecordEvent` carrying a `stage_lifecycle` event from the current
|
||
state to `Halted`.
|
||
3. `Halt`.
|
||
|
||
The order is normative: the event must reach the bundle writer
|
||
before the engine acts on the halt.
|
||
|
||
A `WorkerExit` mutation is dispatched **synchronously** — the
|
||
engine calls the target host's `recv` and processes the returned
|
||
actions inside the same `dispatch_mutation` call that records the
|
||
mutation, before the main loop pops the next event. This is the
|
||
only way to guarantee §6A.6's boundary case: a `WorkerExit` whose
|
||
`at_ns` equals the scenario's `duration_ns` must still produce a
|
||
`worker_exited` record. Routing the envelope through the queue
|
||
(an enqueued `LocalRecv`) would race with the construction-time
|
||
`Terminate` at the same virtual time and could silently lose the
|
||
event — the exact failure mode the boundary clause forbids.
|
||
|
||
The SWIM host kind has no `WorkerExit` semantics. A `WorkerExit`
|
||
mutation targeting a SWIM-kind peer aborts the run with a structured
|
||
error (`EngineAbort::WorkerExitOnWrongKind`). Silent acceptance is
|
||
the failure mode this simulator exists to prevent.
|
||
|
||
### 6A.4 Diagnostic surface
|
||
|
||
The stage host emits three event kinds in addition to anything the
|
||
production stage supervisor already emits:
|
||
|
||
- `register_name { name, address, peer_node_id }` — emitted exactly
|
||
once per stage instance, at the transition `Registering → Running`.
|
||
- `worker_exited { reason, status_code, signal }` — emitted exactly
|
||
once per stage instance, immediately before `Halt`.
|
||
- `stage_lifecycle { from, to }` — emitted on every state transition
|
||
the §6A.2 lifecycle declares.
|
||
|
||
Each kind's payload schema is named here for the production schema to
|
||
follow. The deployment report's items C.1 and C.3 name
|
||
`register_name` and `worker_exited` respectively; this spec fixes
|
||
their shapes so the sim and production cannot drift.
|
||
|
||
The stage host's `snapshot()` returns a JSON object with the fields:
|
||
|
||
- `state` — one of the four §6A.2 lifecycle states.
|
||
- `name_registry` — a map of `name → address` for every name this
|
||
host has registered. (The deployment report's item C.2 names the
|
||
absence of this field as a debugging gap; this spec requires it.)
|
||
- `last_exit_reason` — present only when `state == Halted`.
|
||
|
||
### 6A.5 Codec
|
||
|
||
The stage host kind's codec exists for parity with §3.3. In the MVP
|
||
its message-type is empty: the stage host produces no `Send` actions
|
||
during its lifecycle. When inter-stage traffic enters scope in a
|
||
later revision, the codec gains the production wire format.
|
||
|
||
### 6A.6 Behavioral tests
|
||
|
||
The stage host's tests assert that it has the properties below.
|
||
(Tests live in `tests/stage_host_invariants.rs`.)
|
||
|
||
- **Trait conformance.** `kind_tag()` returns `"stage"`, distinct
|
||
from every other registered kind's.
|
||
- **Lifecycle linearity.** Every stage instance traverses the §6A.2
|
||
states in declared order, never revisits a state, and never skips
|
||
one. Each transition emits exactly one `stage_lifecycle` event.
|
||
- **`register_name` exactly once.** Across the lifetime of one
|
||
instance, exactly one `register_name` event is emitted, at the
|
||
`Registering → Running` transition.
|
||
- **`worker_exited` exactly once.** Across the lifetime of one
|
||
instance, exactly one `worker_exited` event is emitted,
|
||
immediately before the `Halt` action that ends the instance. The
|
||
event's `reason` field is byte-equal to the mutation's `reason`.
|
||
- **Event-before-halt is observable.** A scenario whose
|
||
`duration_ns` is the same nanosecond as a `WorkerExit` mutation's
|
||
`at_ns` produces a bundle containing the `worker_exited` event.
|
||
- **`WorkerExit` against SWIM aborts.** A `WorkerExit` mutation
|
||
targeting a SWIM-kind peer aborts the run with a structured error
|
||
(`EngineAbort::WorkerExitOnWrongKind`) naming the host's kind and
|
||
the mutation's index.
|
||
- **Snapshot contains the registry.** A stage host's `snapshot()`
|
||
includes a `name_registry` field; every `register_name` event the
|
||
host has emitted appears in the map at every subsequent snapshot.
|
||
- **Determinism.** Same `HostKindConfig`, same RNG seed, same
|
||
envelope sequence ⇒ identical action sequence and identical
|
||
emitted event stream.
|
||
|
||
---
|
||
|
||
## 7. Determinism
|
||
|
||
This section is normative. A violation is a ship-blocker.
|
||
|
||
### 7.1 Forbidden inputs
|
||
|
||
No part of the simulator reads any of: host wall clock, host monotonic
|
||
clock, host process or thread ID, host hostname, environment variables
|
||
outside a documented sim-internal prefix, `/dev/urandom` or any host RNG
|
||
source, network interface state, filesystem state outside the bundle
|
||
output path.
|
||
|
||
The repo's existing lint scanner catches the static cases. Drift is
|
||
caught by the cross-architecture parity test (§7.6).
|
||
|
||
### 7.2 The randomness tree
|
||
|
||
All randomness derives from one root random stream seeded by the
|
||
scenario's `seed` field. Substreams are derived by hashing a fixed,
|
||
documented tuple with a constant-key siphash:
|
||
|
||
- `("link", from_id, to_id)` — per-edge substream used by the network.
|
||
- `("host", host_id, label)` — per-host substream used for tick offsets
|
||
and any RNG the host kind needs.
|
||
- `("mutation", index)` — per-mutation substream if a mutation needs
|
||
randomness (none currently do).
|
||
|
||
The hash function and the key are fixed in code. Substreams are stable
|
||
across runs and across host architectures.
|
||
|
||
Substream layout matters: editing one link's policy must not perturb the
|
||
draws on any other link, or every test edit becomes a new random universe
|
||
and bisection is impossible.
|
||
|
||
### 7.3 No hash-randomized iteration
|
||
|
||
Wherever any component iterates a collection, the order is determined by
|
||
the natural key order (host IDs sort lexicographically; pairs sort
|
||
lexicographically on the pair) or by an insertion-order-preserving
|
||
structure. Point lookups into hash maps remain allowed; iteration is the
|
||
divergence source.
|
||
|
||
### 7.4 No floating point in decisions
|
||
|
||
Latencies are integer nanoseconds. Probabilities are integer parts-per-
|
||
million of a fixed denominator. Jitter samples come from a precomputed
|
||
integer lookup table approximating a standard-normal distribution; the
|
||
table is checked in. Bandwidth math uses integer arithmetic with explicit
|
||
scaling; the precise formula is in §5.4.
|
||
|
||
Floating point is permitted in post-hoc calibration tools that read a
|
||
bundle. It is forbidden in the engine, the network, the host adapter, the
|
||
bundle writer, and the assertion evaluator's verdict computation.
|
||
|
||
### 7.5 No work outside the scheduler
|
||
|
||
No background thread, no async runtime, no timer that fires without the
|
||
engine's knowledge. Every effect is the consequence of a popped event.
|
||
|
||
### 7.6 The cross-architecture parity test
|
||
|
||
The test suite contains a reference scenario whose bundle output has a
|
||
known checksum, checked in. The test runs the simulator on the reference
|
||
scenario on every supported architecture and compares the checksum to the
|
||
stored value. A mismatch is either a deliberate spec amendment (with
|
||
justification) or a bug.
|
||
|
||
---
|
||
|
||
## 8. The scenario format
|
||
|
||
A scenario is a single TOML file. The file is the simulator's only input
|
||
and is byte-for-byte sufficient to reproduce any run. The format is TOML
|
||
because the repo already uses it; the choice is not load-bearing.
|
||
|
||
### 8.1 Schema
|
||
|
||
Top-level fields:
|
||
|
||
- `name: String`.
|
||
- `seed: u64`.
|
||
- `duration_ns: u64`.
|
||
- `early_terminate_on_all_assertions_resolved: bool` (default `false`).
|
||
|
||
A `[default_tick]` table:
|
||
|
||
- `period_ns: u64`.
|
||
|
||
A `[default_link]` table containing every field §5.2 names; per-edge
|
||
overrides under `[[links]]` may override any subset.
|
||
|
||
A `[[peers]]` array, each entry:
|
||
|
||
- `id: String`.
|
||
- `kind: String` — selects the host kind. The MVP recognises `"swim"`
|
||
and `"stage"`; the relay extension's `stage` kind_config is
|
||
`{ name: String, address: String }` (RELAY/STAGE §6A).
|
||
- `kind_config: { ... }` — host-kind-specific opaque table.
|
||
- `initial_state: String` — host-kind-specific. For `stage`, the only
|
||
legal value is `"cold"` (mirrors §6A.2's `Cold` initial state).
|
||
- `tick_period_ns_override: u64` (optional).
|
||
|
||
A `[[relays]]` array (relay extension), each entry:
|
||
|
||
- `id: String` — unique across the union of peer ids and relay ids.
|
||
- `ingress_capacity_bps: u64` — combined ingress bytes/sec.
|
||
- `egress_capacity_bps_per_link: u64` — per-outbound-link bytes/sec.
|
||
- `queue_depth_bytes: u64` — shared egress buffer bound.
|
||
- `cold_start_penalty_ns: u64` (default `0`).
|
||
|
||
A `[[links]]` array, each entry:
|
||
|
||
- `from: String`.
|
||
- `to: String`.
|
||
- `via: String` (optional) — the ID of a relay through which this
|
||
edge is routed. When `via` is set, the edge's `from` and `to` must
|
||
both be host IDs; the loader synthesizes the `from → via` and
|
||
`via → to` inbound and outbound legs and registers a relayed
|
||
`HostRoute` for the host pair. Multiple `via` shorthands through
|
||
the same relay are allowed iff their leg policies agree.
|
||
- Any subset of the §5.2 fields (overrides on top of `[default_link]`).
|
||
|
||
A `[[mutations]]` array, each entry:
|
||
|
||
- `at_ns: u64`.
|
||
- `kind: String` — one of the §5.5 variants.
|
||
- Variant-specific parameters.
|
||
|
||
A `[[snapshots]]` array, each entry:
|
||
|
||
- `at_ns: u64`.
|
||
|
||
A `[[assertions]]` array, each entry:
|
||
|
||
- `kind: String` — one of the §10.1 variants.
|
||
- Variant-specific parameters.
|
||
|
||
A `[base]` table with a single optional `extends: String` pointing to
|
||
another scenario file; the merge is deep, with child entries overriding
|
||
parent at the leaf.
|
||
|
||
### 8.2 Validation
|
||
|
||
The loader rejects:
|
||
|
||
- Duplicate peer IDs.
|
||
- Link, mutation, snapshot, or assertion references to undeclared peers.
|
||
- `duration_ns < max(mutation.at_ns)` or similar for snapshots /
|
||
assertions.
|
||
- Any host-kind config that fails the host kind's own validation routine.
|
||
For the SWIM kind, this includes `probe_interval < suspicion_timeout`.
|
||
- A `default_link` field that is non-integer, negative, or in a unit other
|
||
than the §5.2 names (e.g., `latency_ms` is rejected; only `latency_ns`).
|
||
|
||
Relay extension rules (added by RELAY/STAGE):
|
||
|
||
- Duplicate IDs across the union of `[[peers]]` and `[[relays]]`.
|
||
- A `via` reference to a peer ID rather than a relay ID.
|
||
- A peer pair declared with both a direct edge and a relayed route
|
||
(the loader-level ambiguity check).
|
||
- A `[[links]]` entry whose `from` or `to` is a relay and that also
|
||
carries a `via` field.
|
||
- A multi-hop relayed route (any direct relay→relay edge, which would
|
||
make a chained route possible).
|
||
- A `worker_exit` mutation whose target peer is not stage-kind.
|
||
- A `relay_capacity_change` mutation with no fields set (the mutation
|
||
must change at least one of the three policy fields).
|
||
- A `stage` peer whose `kind_config.name` is missing or whose
|
||
`kind_config.address` is missing or empty (delegated to
|
||
`StageHostKindValidator`).
|
||
|
||
Validation failures produce a structured error with the file path, the
|
||
offending field, and a one-line explanation.
|
||
|
||
### 8.3 Library structure
|
||
|
||
Scenarios live under a `scenarios/` directory next to the simulator, in
|
||
four subdirectories:
|
||
|
||
- `smoke/` — happy-path scenarios. Three-node mesh no impairments;
|
||
eight-node ring; chain; star. Each asserts continuous Alive.
|
||
- `reproduction/` — known live failures. Each is expected to fail until
|
||
its cause is fixed.
|
||
- `topology/` — partition-and-heal, rolling restart, peer churn. No
|
||
specific bug targeted.
|
||
- `calibration/` — paired with a captured production bundle. §11.
|
||
|
||
Every scenario carries a top-of-file prose comment naming what it
|
||
reproduces, the expected verdict (pass-now / fail-until-fix /
|
||
sensitivity-study), and any base scenario it extends.
|
||
|
||
### 8.4 Behavioral tests
|
||
|
||
The loader's contract is the schema and validation rules of §8. Its
|
||
tests assert that it has the properties below.
|
||
|
||
**Examples are well-formed.** Every shipped example scenario parses
|
||
and satisfies every rule in §8.2.
|
||
|
||
**Parse is invertible.** Parsing, re-emitting to TOML, and re-parsing
|
||
is the identity on scenario values.
|
||
|
||
**Validation is complete.** Every rule §8.2 names is enforced. A
|
||
scenario violating any rule is rejected; a scenario violating none is
|
||
accepted.
|
||
|
||
**Errors are structured.** A rejection names the file, the offending
|
||
field, and the violated rule in one line each. Generic errors are a
|
||
test failure.
|
||
|
||
**Host-kind validation is delegated.** A host-kind-config error
|
||
surfaces the kind's own rule, not a loader-generic one.
|
||
|
||
**Merge is leaves-override, lists-append.** Extending a base scenario
|
||
replaces leaf values and appends list entries, with no other effect.
|
||
|
||
**Loading is pure.** Loading the same file twice produces equal values
|
||
and performs no filesystem writes.
|
||
|
||
---
|
||
|
||
## 9. The bundle
|
||
|
||
### 9.1 Layout
|
||
|
||
A bundle is a directory laid out as follows:
|
||
|
||
```
|
||
<bundle_root>/
|
||
manifest.json
|
||
scenario.toml # echo of the scenario that produced the run
|
||
events.ndjson # newline-delimited JSON event stream
|
||
snapshots/
|
||
<host_id>/
|
||
<snapshot_seq>.json
|
||
verdicts.json # produced by the assertion evaluator (§10)
|
||
```
|
||
|
||
### 9.2 The event stream
|
||
|
||
Each line is one JSON object with the envelope:
|
||
|
||
```
|
||
{
|
||
"virtual_time_ns": <u64>,
|
||
"host_id": <string or null>, # null for engine-synth events not bound to a host
|
||
"kind_tag": <string>, # "swim", "engine", etc.
|
||
"event": <event-payload>
|
||
}
|
||
```
|
||
|
||
The event payload schema is exactly the production diagnostics schema for
|
||
that event kind. The simulator must not invent new event kinds; an event
|
||
the simulator emits is one production also emits.
|
||
|
||
Event kinds the MVP emits:
|
||
|
||
- SWIM state transitions, message-send and receive accounting, probe
|
||
lifecycle (sent / received / timed out), self-incarnation bumps.
|
||
- Engine-synthesized cache state changes, dial start and outcome, drop on
|
||
send, drop on delivery, send-failure errors.
|
||
- Mutation records.
|
||
- Stage host (RELAY/STAGE §6A.4): `register_name`, `worker_exited`,
|
||
`stage_lifecycle`. Each carries `kind_tag = "stage"` and the
|
||
emitting host's id.
|
||
- Relay subsystem (RELAY/STAGE §5A.3): `RelayEnqueue`, `RelayDequeue`,
|
||
`RelayDrop`. Each carries `kind_tag = "relay"` and `host_id = null`
|
||
(the relay is not a host).
|
||
|
||
The stage host's three kinds are introduced *ahead* of production on
|
||
the basis of the deployment report's items C.1–C.3. The production
|
||
stage supervisor must adopt the same schemas as part of landing this
|
||
spec; otherwise the contract test against the production diagnostics
|
||
emitter fails and the sim cannot reproduce Layer C without divergence.
|
||
|
||
### 9.3 Snapshots
|
||
|
||
A snapshot is exactly the production tier-2 SWIM-state JSON for that host
|
||
at that virtual time. The schema is unchanged from production.
|
||
|
||
### 9.4 The manifest
|
||
|
||
`manifest.json` records:
|
||
|
||
- Simulator version (commit hash).
|
||
- Path and SHA-256 of the scenario file.
|
||
- Seed.
|
||
- Duration in ns.
|
||
- Host architecture the run executed on.
|
||
- SHA-256 of `events.ndjson`.
|
||
- SHA-256 of each snapshot file, keyed by relative path.
|
||
|
||
Wall-time fields are derived from the virtual clock; the manifest declares
|
||
the unit so post-processors do not confuse virtual time with real time.
|
||
|
||
### 9.5 Renderability
|
||
|
||
The bundle is renderable by the same post-processor production uses. If
|
||
the renderer requires inputs the simulator does not have (collector-side
|
||
receive timestamps, for instance), the simulator substitutes the virtual-
|
||
clock equivalent and records the substitution in the manifest.
|
||
|
||
### 9.6 Behavioral tests
|
||
|
||
The writer's contract is the layout and schema of §9. Its tests assert
|
||
that it has the properties below.
|
||
|
||
**Layout.** Every produced bundle has the §9.1 entries. (`verdicts.json`
|
||
is the assertion evaluator's responsibility; §10.5.)
|
||
|
||
**Envelope conformance.** Every line of `events.ndjson` is valid JSON
|
||
conforming to the §9.2 envelope shape.
|
||
|
||
**Hash integrity.** Every manifest-recorded hash equals the actual hash
|
||
of the file it names.
|
||
|
||
**Snapshot organization.** Each `SnapshotRecord` corresponds to exactly
|
||
one file at `snapshots/<host_id>/<snapshot_seq>.json`. Sequence numbers
|
||
are monotonically increasing per host from zero.
|
||
|
||
**Ordering is deterministic and documented.** Identical record streams
|
||
produce byte-identical bundles. The ordering rule is named in §9 and the
|
||
writer obeys it.
|
||
|
||
**Arrival-order independence.** Records may arrive in any order; the
|
||
produced bundle depends only on the multiset of records and the
|
||
documented ordering rule, not on arrival order.
|
||
|
||
**Idempotency.** Writing the same record stream to a fresh output path
|
||
twice produces byte-identical bundles.
|
||
|
||
---
|
||
|
||
## 10. Assertions
|
||
|
||
### 10.1 The assertion catalog
|
||
|
||
Each assertion kind has a name and a parameter shape. The MVP catalog:
|
||
|
||
- `all_alive_at { at_ns, peers }` — at the given virtual time, every named
|
||
peer's view of every other named peer is Alive.
|
||
- `all_alive_throughout { window_start_ns, window_end_ns, peers }` — the
|
||
above, continuously, across a window.
|
||
- `convergence_after { after_ns, within_ns, peers }` — after the named
|
||
time, the cluster reaches a consistent membership view within the
|
||
bounded duration.
|
||
- `no_flap_while_probes_ok { peer, window_start_ns, window_end_ns }` — no
|
||
peer transitions Suspect → Alive → Suspect within a window in which the
|
||
peer's bidirectional probes are succeeding.
|
||
- `no_dead_when_probes_ok { peer, window_start_ns, window_end_ns }` — the
|
||
peer is never marked Dead in any other peer's view while bidirectional
|
||
probes are succeeding.
|
||
- `self_incarnation_bounded { peer, max_value }` — the peer's self-
|
||
incarnation counter never exceeds the bound.
|
||
- `message_size_bounded { kind, max_bytes }` — no sent message of the
|
||
named kind exceeds the byte threshold.
|
||
- `dead_peer_resurrects_within { peer, after_ns, within_ns }` — after the
|
||
peer becomes reachable again, the cluster marks it Alive within a
|
||
duration.
|
||
- `event_count { kind, max }` — bounds the absolute count of an event
|
||
kind across the run.
|
||
- `event_rate { kind, window_ns, max_per_window }` — bounds the rate of
|
||
an event kind.
|
||
- `relay_queue_depth_bounded { relay, max_bytes, window_start_ns?,
|
||
window_end_ns? }` (RELAY/STAGE) — across the window (defaults to
|
||
the whole run), the named relay's `enqueued_bytes` never exceeds
|
||
`max_bytes`. Fails on the first `RelayEnqueue` that pushes the
|
||
total above the bound.
|
||
- `worker_alive_throughout { peer, window_start_ns, window_end_ns }`
|
||
(RELAY/STAGE) — the named peer's lifecycle state remains `Running`
|
||
throughout the window. Fails on any `stage_lifecycle` event into
|
||
`Halted` whose time falls in the window. The window is inclusive
|
||
on both ends: a halt at `window_end_ns` (or at `duration_ns` when
|
||
the window spans the whole run) fails the assertion, because the
|
||
§6A.3 synchronous dispatch rule guarantees the
|
||
`stage_lifecycle → Halted` event appears in the bundle even at
|
||
the boundary.
|
||
- `name_resolves_within { name, observers, within_ns, from_ns }`
|
||
(RELAY/STAGE) — starting at `from_ns`, every observer in
|
||
`observers` produces a snapshot whose `name_registry` contains
|
||
`name` within `within_ns`. `Inconclusive` if no observer produces
|
||
a snapshot in the window.
|
||
|
||
Adding a kind is a deliberate amendment to this section.
|
||
|
||
### 10.2 The evaluator interface
|
||
|
||
The evaluator reads the bundle's `events.ndjson` and `snapshots/` and
|
||
evaluates each assertion. Per assertion it emits a verdict:
|
||
|
||
```
|
||
{
|
||
"name": "<assertion-name>",
|
||
"kind": "<assertion-kind>",
|
||
"parameters": { ... },
|
||
"outcome": "Pass" | "Fail" | "Inconclusive",
|
||
"evidence": [
|
||
{ "virtual_time_ns": <u64>, "event_or_snapshot_ref": "<path>" }
|
||
]
|
||
}
|
||
```
|
||
|
||
`Inconclusive` is reserved for assertions whose preconditions did not
|
||
fire during the run (e.g., a peer the assertion names never became
|
||
reachable).
|
||
|
||
`verdicts.json` is an array of verdict objects, one per declared
|
||
assertion, in the order the scenario declared them.
|
||
|
||
### 10.3 Library properties
|
||
|
||
A library property is a parameterized assertion kind evaluated across a
|
||
generated distribution of scenarios. The MVP ships one such property: the
|
||
gossip-flap detector. The property generates scenarios from a declared
|
||
space (peer count, latency range, jitter range, loss range, duration) and
|
||
applies `no_flap_while_probes_ok` to each.
|
||
|
||
The framework records the random seed that produced any failing scenario
|
||
so the failure is reproducible. The current SWIM source must fail this
|
||
property. The proposed fix must pass it.
|
||
|
||
Library properties are otherwise identical to per-scenario assertions in
|
||
output shape; their verdicts go into the property runner's own output,
|
||
not into a single bundle's `verdicts.json`.
|
||
|
||
### 10.4 Streaming evaluation (for early termination)
|
||
|
||
For early termination (§4.8) the evaluator exposes a streaming side: as
|
||
events are emitted, the evaluator may resolve assertions whose verdicts
|
||
are determinable from the prefix. The engine polls this side after each
|
||
event dispatch. The streaming side is an optimization; the post-run side
|
||
remains the authoritative source for `verdicts.json`.
|
||
|
||
### 10.5 Behavioral tests
|
||
|
||
The evaluator's contract is the assertion catalog of §10.1, the verdict
|
||
shape of §10.2, and the streaming side of §10.4. Its tests assert that
|
||
it has the properties below.
|
||
|
||
**Per-kind soundness.** For every kind in §10.1: `Pass` is returned
|
||
exactly when the kind's stated condition holds over the bundle; `Fail`
|
||
exactly when the condition is violated; `Inconclusive` exactly when the
|
||
preconditions did not fire.
|
||
|
||
**Verdict shape.** Every verdict conforms to §10.2. `Fail` verdicts
|
||
carry evidence referencing the event or snapshot responsible.
|
||
|
||
**Verdict order is scenario-declared.** `verdicts.json` lists verdicts
|
||
in the order the scenario declared the corresponding assertions.
|
||
|
||
**Streaming agrees with post-run.** On any bundle, the streaming side
|
||
either does not resolve or resolves to the same verdict the post-run
|
||
side will return. The two are never inconsistent.
|
||
|
||
**Streaming resolves as early as possible.** When a verdict is
|
||
determinable from a prefix, the streaming side resolves no later than
|
||
the end of that prefix.
|
||
|
||
**Property failures replay exactly.** A library-property failure
|
||
recorded with seed S, replayed with seed S, produces the identical
|
||
scenario and the identical `Fail` verdict.
|
||
|
||
---
|
||
|
||
## 11. Calibration
|
||
|
||
Calibration measures the simulator's fidelity against captured production
|
||
bundles.
|
||
|
||
### 11.1 The corpus
|
||
|
||
The MVP corpus is the three N3 vast.ai bundles described in the
|
||
deployment report. Each pairs with a scenario in `scenarios/calibration/`
|
||
that approximates the conditions under which the bundle was produced:
|
||
|
||
- `vastai-N3-1` ↔ `n3_canary_relay_real_worker.toml`. Canary relay,
|
||
real worker. The relay's policy is set to the canary's measured
|
||
shape (low egress per link, large queue, multi-hundred-millisecond
|
||
cold start). Stage hosts carry the worker-exit timing observed in
|
||
the bundle.
|
||
- `vastai-N3-2` ↔ `n3_own_relay_real_worker.toml`. Same topology
|
||
with the relay's policy widened to the own-relay's measured shape.
|
||
Stage hosts carry the same worker-exit timing.
|
||
- `vastai-N3-stub` ↔ `n3_own_relay_stub.toml`. Same topology,
|
||
widened relay, stage hosts with no `WorkerExit` mutations (full-
|
||
run survival). Reproduces the gossip-flap pathology without stage
|
||
death.
|
||
|
||
Each scenario carries a top-of-file prose comment naming the bundle
|
||
it pairs with, the placeholder tolerances that the first calibration
|
||
pass will replace with measured numbers, and any base scenario it
|
||
extends.
|
||
|
||
### 11.2 The procedure
|
||
|
||
For each pair, the calibration tool:
|
||
|
||
1. Runs the simulator with the declared scenario.
|
||
2. Loads the corresponding production bundle.
|
||
3. Computes the comparison metrics (§11.3) on both bundles.
|
||
4. Reports per-metric pass/fail against per-metric tolerances.
|
||
|
||
### 11.3 The metrics
|
||
|
||
- Per-peer event-timespan summaries (p50, p90, p99) per event kind.
|
||
- State-transition reason distribution across all peers.
|
||
- Message-size distribution per message kind.
|
||
- Message-count per kind, per peer pair.
|
||
- Self-incarnation trajectory per peer.
|
||
- Connection-cache hit count.
|
||
- Dial-started count.
|
||
- Per-peer fraction of run time in the Alive state.
|
||
|
||
Relay extension metrics (RELAY/STAGE §10.2):
|
||
|
||
- Relay queue-depth distribution over time (p50, p90, p99 of
|
||
`enqueued_bytes`).
|
||
- Per-link HOL delay decomposition: each delivery's delay attributed
|
||
to base latency, edge-bandwidth serialization, jitter, cold-dial
|
||
penalty, relay ingress, relay egress, and active mutations.
|
||
- Probe-success-vs-transition ratio: fraction of `Suspect`
|
||
transitions in which the observer's and the peer's bidirectional
|
||
probes were succeeding at the transition time.
|
||
- Worker-exit reason distribution per peer.
|
||
- Name-resolution latency per registered name (time from
|
||
`register_name` to first observer snapshot containing the name).
|
||
|
||
Tolerances ship as placeholders informed by intuition; the first
|
||
calibration pass against the N3 corpus sets the real numbers.
|
||
|
||
### 11.4 CI integration
|
||
|
||
Calibration runs on every change that touches the simulator or the SWIM
|
||
state machine. A regression — a previously-in-tolerance metric goes out
|
||
of tolerance — blocks merge. A widening of a tolerance is a separate,
|
||
justified commit.
|
||
|
||
---
|
||
|
||
## 12. Implementation phasing
|
||
|
||
Each phase ships independently. At every phase boundary the partial
|
||
simulator does something useful and is testable.
|
||
|
||
| Phase | Adds | Verifiable outcome |
|
||
|-------|------|-----|
|
||
| 1 | Scenario loader | Every shipped scenario parses; malformed input is rejected with a structured error. |
|
||
| 2 | Network (no engine, no host) | A microbenchmark queries `send` at scenario-scale rates and produces deterministic outputs. |
|
||
| 3 | Engine skeleton + a trivial echo host kind | The cross-architecture parity test passes on a reference scenario. |
|
||
| 4 | SWIM host kind + codec contract test | The smoke scenario runs to completion and asserts continuous Alive. |
|
||
| 5 | Assertion evaluator | The gossip-flap reproduction scenario fails on current SWIM, passes after the fix. |
|
||
| 6 | Bundle writer | A sim bundle renders through the production post-processor. |
|
||
| 7 | Calibration tool | At least one N3 pair passes calibration. |
|
||
| R1 | Relay vertex (§5A) without mutations | A direct send through a relay arrives later than the same send over a direct edge of equal policy by the relay's ingress + egress serialization time. |
|
||
| R2 | Relay mutations (§5.5 additions) | A `RelayKill` followed by a `RelayBoot` produces a bundle in which the in-flight messages at kill-time appear in the invalidated-deliveries list and no others. |
|
||
| R3 | Stage host kind (§6A) and `WorkerExit` mutation | A scenario with one stage host and one `WorkerExit` mutation produces a bundle containing exactly one `register_name`, one `worker_exited`, and three `stage_lifecycle` records, in §6A.2 order. |
|
||
| R4 | Assertion catalog additions (§10.1 R-tail) | The N3-1 calibration scenario fails `relay_queue_depth_bounded` against the canary-relay policy and passes against the own-relay policy. |
|
||
| R5 | Scenario loader additions (§8) and bundle additions (§9) | Every shipped calibration scenario parses; bundles render through the production post-processor with the new event kinds passed through unchanged. |
|
||
| R6 | Calibration tool extensions (§11) | At least one of the three N3 pairs passes calibration on every declared metric. |
|
||
|
||
MVP exit is the end of phase R6. Subsequent phases — proptest catalog
|
||
expansion, scenario library growth — are post-MVP.
|
||
|
||
Phases 1, 2, 6 are independently buildable by separate agents from this
|
||
spec alone; phases 3 onwards require the prior phase as input. The
|
||
relay phases R1–R3 are independently buildable from the §5A / §6A
|
||
sections alone; R4–R6 require their predecessors.
|
||
|
||
---
|
||
|
||
## 13. Open questions
|
||
|
||
These are deliberately unanswered; they are expected to resolve during
|
||
phases 1–3.
|
||
|
||
- SWIM's reactive probe mode introduces an internal safety-sweep timer.
|
||
Whether the engine needs a separate event kind for the safety sweep, or
|
||
whether driving the host on its tick interval is enough, depends on
|
||
details inside the production probe code that are easier to resolve
|
||
once the engine skeleton exists.
|
||
- One peer corresponds to one host ID in the MVP. Production permits a
|
||
single host to expose multiple endpoints. The MVP punts; if a
|
||
calibration scenario needs the multi-endpoint shape, the scenario grows
|
||
a per-peer endpoint list and the engine dispatches by endpoint.
|
||
- The starting tolerances in §11.3 are placeholders. The first
|
||
calibration run against the N3 corpus sets the real numbers; those
|
||
numbers replace the placeholders in a follow-up commit.
|
||
- The simulator's code lands under `crates/simulation/` after the current
|
||
cleanup of that crate. If the cleanup renames or relocates the
|
||
simulator, the phase plan in §12 needs a one-pass path refresh;
|
||
nothing else in this spec depends on the path.
|
||
|
||
---
|
||
|
||
## 14. Spec change protocol
|
||
|
||
Changes that relax a contract in §3 / §4 / §5 / §6 / §7 — a widened
|
||
tolerance, a removed assertion, a relaxed determinism rule — are
|
||
behaviour-changing and require a deliberate commit whose subject names
|
||
the relaxation and whose body justifies it in prose. A tightening change
|
||
— a new assertion kind, a narrower tolerance, a more restrictive
|
||
determinism rule — can land in any commit. Adding new scenarios, new
|
||
properties, or new phases does not require special treatment.
|
||
|
||
The lint scanner, the determinism digest, the cross-architecture parity
|
||
test, the encoding-symmetry contract test, and the calibration tolerances
|
||
are the load-bearing artefacts that enforce this spec. If any of them is
|
||
short-circuited — disabled in CI, allow-listed at the call site, silenced
|
||
with an exemption — the spec is being worked around, and the workaround
|
||
must surface in code review.
|
||
|
||
---
|
||
|
||
## 15. References
|
||
|
||
- `examples/pipeline-parallel-inference/N3_DEPLOYMENT_REPORT.md` —
|
||
source of truth for the live failures the simulator must reproduce.
|
||
Items A, B2, C.1, C.2, and C.3 of that report map to §5A,
|
||
§11.3 (metrics), §6A.4 + §9.1 (`register_name`), §6A.4 (snapshot
|
||
registry), and §6A.4 (`worker_exited`) respectively.
|
||
- `crates/simulation/NORTH_STAR.md` — the long-term simulator vision.
|
||
This MVP is a strict subset and does not retract any of its claims.
|
||
- `crates/simulation/BLOCKED.md` — the staged plan this MVP supersedes
|
||
for the immediate iteration. The deeper goals there (real quinn / iroh
|
||
on a sim facade, detector as a peer, full distribution-crate facade
|
||
migration) remain on the roadmap, just not gating SWIM-tuning.
|
||
- `crates/distribution/src/swim/` — the production SWIM state machine
|
||
the SWIM host kind wraps.
|
||
- `crates/distribution/src/diagnostics/` — the schema the bundle must
|
||
match and the renderer it must render through.
|
||
- `crates/distribution/src/bin/swactor-iroh-relay.rs` — the production
|
||
relay binary the §5A relay vertex models. The §9.2 parity contract
|
||
names this binary's emissions as the production side once it gains
|
||
matching schemas.
|
||
- `examples/pipeline-parallel-inference/src/bin/pp_gpu_node.rs` —
|
||
the production stage supervisor the §6A host kind wraps.
|