swactor/docs/development_history/distribution/SIMULATION_TESTING.md

Simulation Testing — Development History

Covers the addition of network fault injection to the simulation harness and 15 new cluster scenario tests, informed by research into production distributed systems testing practices.

4 files changed · ~950 insertions

Branch: distribution-realization

---

Table of Contents

1. Overview & Motivation
2. What Was Built
3. Research Phase
4. Network Fault Injection
5. Cluster Scenario Tests
6. Key Findings
7. Design Decisions
8. Known Gaps & Future Work

---

1. Overview & Motivation

The simulation crate (crates/simulation/) had 6 distribution tests covering happy-path scenarios: cluster convergence, node death detection, node rejoin, and actor resolution. All tests assumed a perfect network — 100% delivery, zero latency variation, no partitions.

Real networks drop packets, partition nodes, and deliver messages out of order. The SWIM protocol's correctness under these conditions was untested. This work adds network fault simulation and exercises the protocol under adversarial conditions drawn from established testing methodologies.

---

2. What Was Built

Component	Location	Description
Network fault model	crates/simulation/src/distribution/sim.rs	Partition, heal, and message drop simulation
15 cluster scenario tests	crates/simulation/tests/cluster_scenarios.rs	Behavioral tests for failure modes
Research notes	CLAUDE/notes/research_simulation_testing.md	Survey of 7 codebases/frameworks

All 15 new tests run in ~1.4s total (well under the 2-minute cap). The original 6 distribution_sim tests are unaffected.

---

3. Research Phase

Seven codebases and frameworks were studied for their simulation testing methodology:

Source	Key Takeaway
FoundationDB	Deterministic simulation: single-threaded, seeded PRNG, simulated time. BUGGIFY injects faults inside production code at ~25% activation × 25% firing probability.
Hashicorp memberlist	~80 test functions. Lifeguard extensions: suspicion timer with log(k+1) decay, health-aware probe timeouts, dogpile confirmation.
Antithesis	Categorized fault injection: network, process, disk, timing. Emphasis on property-based invariant checking.
TigerBeetle	VOPR simulation + Vortex TCP proxy. Runs millions of seeds nightly.
Turmoil (tokio-rs)	Rust DST: sim.partition(a,b), sim.hold(a,b), sim.repair(a,b). Seeded RNG, simulated time.
MadSim	Rust DST used by RisingWave. FIRO scheduling, libc interception for true determinism.
Jepsen	Standard nemesis catalog: partition, kill, pause, clock skew, membership change.

Full notes: CLAUDE/notes/research_simulation_testing.md

---

4. Network Fault Injection

Three new types model network conditions:

pub struct Partition {
    pub side_a: Vec<usize>,   // node indices
    pub side_b: Vec<usize>,
    pub asymmetric: bool,     // if true, only side_a→side_b is blocked
}

pub enum NetworkFault {
    Partition { round: usize, partition: Partition },
    Heal { round: usize },
    SetDropRate { round: usize, rate: f64 },
}

NetworkState tracks blocked pairs (as a HashSet<(usize, usize)>) and applies probabilistic message dropping via a deterministic LCG PRNG (seed 0x853c49e6748fea9b). The should_deliver(from, to) method checks both partition membership and drop rate before allowing message delivery.

Faults are applied per-round in run_simulation before the tick/deliver cycle. Initial join and settle phases always use a clean NetworkState (no faults during cluster formation).

Backward Compatibility

DistributionSimConfig gained a network_faults: Vec<NetworkFault> field defaulting to an empty vec. Existing tests that don't set this field see no behavior change — the renamed deliver_actions_tagged_with_net function with a clean NetworkState is functionally identical to the original deliver_actions_tagged.

---

5. Cluster Scenario Tests

15 tests organized by failure category:

Partitions

Test	Scenario	Assertion
symmetric_partition_splits_membership_views	6 nodes split {0,1,2} vs {3,4,5}	Each side forms sub-cluster; dead-declared nodes not auto-rediscovered
asymmetric_partition_causes_one_sided_suspicion	5 nodes, one-way block	Recovery after heal
partition_plus_kill_in_minority_side	6 nodes, partition + kill in minority	Compound failure handled
sequential_partitions_fragment_cluster	Sequential partition events	Creates sub-clusters
actor_resolution_degrades_during_partition	Actors registered pre-partition	Cached resolutions survive partition

Message Loss

Test	Scenario	Assertion
cluster_converges_under_10_percent_message_loss	10% drop rate	Some membership maintained
heavy_message_loss_causes_membership_instability	30% drop rate	Degrades but doesn't crash
cluster_survives_brief_message_loss	15% loss for 15 rounds then heals	≥2 well-connected survivors

Node Failures

Test	Scenario	Assertion
cluster_survives_seed_node_death	Kill node 0 (seed)	4 survivors maintain ≥60% accuracy
simultaneous_two_node_failure_detected	Kill 2 of 7 at once	Both deaths detected
cascading_failures_leave_quorum_intact	Kill 3 of 7 sequentially	Survivors maintain membership
graceful_leave_detected_faster_than_crash	Crash detection timing	Bounded detection rounds

Scale & Churn

Test	Scenario	Assertion
cluster_of_fifty_converges	50-node cluster	≥90% accuracy
rapid_churn_maintains_partial_membership	8 nodes, 4 kill/revive cycles	Partial membership maintained
membership_changes_disseminate_to_all_nodes	10-node cluster, verify propagation	All survivors detect death

---

6. Key Findings

1. SWIM does not auto-rediscover dead-declared nodes. Once the suspicion timeout expires and a node is declared dead, it is permanently removed. Re-joining requires the join protocol. This is correct SWIM behavior, not a bug — but tests must account for it.

2. Message loss is highly destabilizing for SWIM because it affects both the direct probe AND indirect probes in the same cycle. Default config (suspicion_timeout=5, indirect_probes=1) cannot tolerate even 15% loss. Tuned config (suspicion_timeout=15–20, indirect_probes=2, probe_timeout=5) tolerates ~10%.

3. The LCG PRNG for message dropping needs a non-zero seed to avoid correlated early values (seed 0 always produces 0.0 as first output, causing deterministic first-message drop).

4. 50-node clusters converge quickly with the simulation's topology-aware join strategy, achieving ≥90% accuracy.

---

7. Design Decisions

Decision	Rationale
LCG instead of rand crate	Keeps simulation deterministic without adding dependencies; 64-bit LCG with Knuth constants is sufficient for drop-rate testing
Blocked pairs in HashSet	O(1) lookup per message; partition model maps directly to real network behavior
Clean NetworkState for join/settle	Faults during initial cluster formation would conflate test setup with test assertions
Loose accuracy thresholds for loss tests	SWIM's sensitivity to message loss means tight thresholds create flaky tests; the behavioral property being tested is "degrades gracefully" not "maintains perfect accuracy"
Tests verify SWIM's actual semantics	Rather than expecting auto-recovery after partition heal (which SWIM doesn't support), tests verify the sub-cluster formation that actually occurs

---

8. Known Gaps & Future Work

Gap	Priority	Notes
Property-based invariant checking	High	Formal completeness/accuracy as automated checks
Message reordering	Medium	Out-of-order delivery in network model
Kademlia-specific scenarios	Medium	Routing table convergence under churn, directory repair
Suspicion refutation tests	Medium	Incarnation bump prevents false death
Graceful leave protocol	Medium	Wire node.leave() into simulation
BUGGIFY-style injection	Low	Probabilistic faults at protocol decision points
Re-join after partition heal	Low	Auto-rediscovery mechanism (not standard SWIM)