swactor/examples/pipeline-parallel-inference/SIM_ARCHITECTURE.md

Simulator architecture for SWIM (and beyond)

Goal: a SWIM/cluster simulator that catches bugs and is the substrate for behavioral optimization. Companion to N3_DEPLOYMENT_REPORT.md, which laid out the bugs the sim is meant to catch.

What we already have

crates/simulation exists and is more capable than I initially assumed. Strategic decisions below should be read against this baseline, not from zero.

• Depth: runs real DistributedNode (SWIM core + actor runtime + name registry) under feature = "distribution". Already option (b) from D1 below.
• Topology models: two, neither arbitrary.
  • Topology (closed enum: Ring / Star / FullMesh / Chain / Partitioned) — used by the gossip sim.
  • NetworkTopology (per-node NodeLocation of Public / Nat{group} / Firewalled, plus a list of relay-node indices) — used by the distribution sim. Richer, still not arbitrary per-link adjacency.
• Faults: scheduled Partition / Heal, global drop rate, per-link drop rate (uni- or bidirectional), relay penalty.
• Determinism: counter-based PRNG (LCG seeded constant). No Instant::now() in the sim path; rounds advance via an integer tick.
• Replay: SimulationTrace<EventKind, Snapshot> is the on-disk format, and the dashboard feature serves an HTML/JS replay UI. This is replay of sim-generated traces — not replay of a live diag bundle into the sim.
• Tests: ~9 integration tests covering cluster scenarios, deploy scenarios, lifecycle, registry, gossip convergence, properties, and adversarial topologies.

What it can't do today, mapped to bugs in N3_DEPLOYMENT_REPORT.md:

• Cannot reproduce the canary-buffering Layer A symptom — no bandwidth or queueing model, only per-message drops.
• Cannot reproduce the SWIM gossip flap (Layer B) at production timing — no latency distribution, no jitter, no per-link RTT. Drops alone don't reach the flap regime.
• Cannot consume a real diag bundle as input — no replay-from-production.

The key insight

The diagnostic bundle format is the right interface for this. We already have:

• DiagEvent — structured records for SwimTransition, MessageSent/Received, DialStarted/Outcome, ConnectionCacheHit/Miss, IrohConnTypeChanged, Probe*, etc.
• Per-node snapshots that capture full SWIM state (peers, incarnations, recent_messages, metadata version).
• The post-processor that turns a bundle into summary.md, reachability.tsv, per-direction timelines.

If the sim emits the same bundle format, then every tool we already wrote works against sim runs. More importantly: sim runs and live runs become visually comparable — open both summary.md files side-by-side and you can ask "does the sim's behavior match production?" — which is the operational test for whether the sim is realistic.

The bundle format is our lingua franca: sim and live both produce it, analysis tools consume it, and "fidelity" has a concrete definition.

Today the sim emits its own SimulationTrace JSON, not a diag bundle. That's a real gap if we want this insight to pay off.

Key design decisions, with tradeoffs

D1 — How deep does the sim go?

Decided: arbitrary connection topology that swactor might ever support.

This is broader than the depth question I originally framed. It means the topology model has to express any graph swactor can run on — not just the closed shapes in Topology, and not just NAT-group adjacency in NetworkTopology. The unit of expression should be a per-link descriptor (possibly directional, possibly with relay path) over the full N×N edge set, parameterizable per scenario.

What "supports" means:

• Public ↔ public, NAT ↔ NAT, NAT ↔ public via relay.
• Asymmetric reachability (A→B works, B→A doesn't).
• Per-link RTT, bandwidth, loss, jitter, and whether the path is relayed.
• A node can be reachable to some peers and not others — mirrors the per-link IrohConnTypeChanged reality.

The existing NetworkTopology is a starting point but folds reachability into NAT-group equality. Generalizing to per-edge descriptors is the right direction.

Depth in the protocol-stack sense (SWIM only vs. SWIM + actors + registry) is already settled: the sim runs the real DistributedNode, which includes all three.

D2 — Bit-exact determinism, or "mostly deterministic"?

Open.

• Bit-exact: same seed → byte-identical bundle. Requires banning sources of nondeterminism — HashMap iteration order (use BTreeMap), unspecified f64 operations, anything that depends on OS scheduling. Enables delta-debugging (binary search a seed range to find minimal failing input).
• Mostly deterministic: same seed → same outcome (convergence time, final state) but messages may interleave slightly differently. Cheaper but limits some advanced uses.

Current state: counter-based PRNG, no clock, integer-tick rounds. That's already most of the way to bit-exact for the gossip and distribution sims — but if we add wall-time clocks, latency distributions, and threaded delivery, that property is easy to lose.

My lean: bit-exact, because we're so close already. Cheap to preserve, expensive to claw back later.

D3 — Replay vs. synthesis

Decided: replay is required. Highest-fidelity sim possible; if replay runs are too expensive to run continuously, run them only when needed, but they must exist.

Two replay modes to keep distinct:

1. Trace replay (already shipped via dashboard): re-render a previously recorded SimulationTrace. Cheap. Useful for debugging sim runs.
2. Bundle replay (not yet built): consume a real production diag bundle and drive the sim's transport queue from its MessageSent/Received events, with the sim's protocol stack reacting. This is the one that reproduces a production failure deterministically.

The doc's earlier framing called replay a "nice-to-have." It's not — this is the decision.

D4 — Network model: how realistic?

Decided: high fidelity.

Existing model: partitions, drop rate (global + per-link), NAT/firewall + relay topology, relay penalty.

What's missing for high fidelity, in roughly the order each is required to reproduce a real bug we've seen:

• Latency distribution per link (e.g. Normal(mean, sd) or empirical CDF). SWIM probe timing is everything; without RTT variance, flap regimes don't reproduce.
• Bandwidth + queue depth per link. The 187s canary-buffering symptom was a queueing phenomenon, not a drop phenomenon — minimal-fidelity models silently skip past it.
• Reorder + jitter. Big SWIM ack bundles head-of-line block; reorder matters.
• Per-node CPU saturation. Plausibly relevant if name-registry GC or diag flushing competes with SWIM ticks. Lower priority than the link-level knobs.

D6 — Where does the sim live?

Decided: extend crates/simulation. That's where it already lives. The earlier "new crate" lean was wrong; the work is to evolve the existing crate, not to create a parallel one.

Concrete evolution path (for orientation, not commitment):

• Generalize topology to per-edge descriptors (D1).
• Add latency / bandwidth / reorder / jitter to NetworkState (D4).
• Add bundle-replay input pathway (D3).
• Add bundle-output sink so sim runs go through swactor-diag-postproc (the key insight above).
• Audit determinism once any of the above land (D2).

Open questions

• D2 (determinism level): bit-exact, or accept slight nondeterminism once latency/threads enter the picture? My lean: bit-exact, because we're close already.
• Anything else we want the sim to optimize for that hasn't surfaced in D1–D4?