# Progress

## Session 1 — Simulation Test Breadth (2026-02-12)

### Completed
1. **Research phase**: Studied Hashicorp memberlist, FoundationDB DST, Antithesis, TigerBeetle VOPR, Turmoil/MadSim, Jepsen nemeses
   - Notes in `CLAUDE/notes/research_simulation_testing.md`
2. **Enhanced simulation harness** (`crates/simulation/src/distribution/sim.rs`):
   - Added `NetworkFault` enum: `Partition`, `Heal`, `SetDropRate`
   - Added `Partition` struct with `side_a`, `side_b`, `asymmetric` fields
   - Added `NetworkState` with blocked-pair tracking and LCG-based message dropping
   - Modified `deliver_actions_tagged` → `deliver_actions_tagged_with_net` (respects network faults)
   - Existing 6 distribution tests unaffected (backward compatible)
3. **15 new cluster scenario tests** (`crates/simulation/tests/cluster_scenarios.rs`):
   - Symmetric partition (split-brain, each side forms sub-cluster)
   - Asymmetric partition (one-way communication)
   - 10% message loss (converges with tuned timeouts)
   - 30% message loss (degrades but doesn't crash)
   - Seed node death (cluster survives without seed)
   - Simultaneous 2-node failure
   - Cascading sequential failure (3 nodes killed over time)
   - Large cluster (50 nodes)
   - Rapid churn (kill/revive cycles)
   - Crash detection speed (bounded detection time)
   - Partition + kill in minority side
   - Actor resolution during partition
   - Dissemination completeness (10-node cluster, all detect death)
   - Sequential partitions (fragment cluster)
   - Brief message loss recovery

### Key Findings
- **SWIM does not auto-rediscover dead-declared nodes** after partition heals. Once the suspicion timeout expires and a node is declared dead, it's permanently removed. Re-discovery requires the join protocol.
- **Message loss is highly destabilizing** for SWIM because it affects both the direct probe AND the indirect probe simultaneously. Even 15% loss with default config can cause false deaths.
- **Tuning suspicion_timeout and indirect_probes** is critical for lossy networks. Higher values tolerate more loss but increase detection latency.
- **The LCG PRNG for message dropping needs a non-zero seed** to avoid correlated early values.

### Next Steps
1. **Depth: Property-based invariant checking** — Add formal SWIM invariants (completeness, accuracy) as automated property checks
2. **Message reordering** — Add out-of-order delivery to the network model
3. **Kademlia-specific scenarios** — Test routing table convergence under churn, directory repair after death
4. **Suspicion refutation tests** — Verify incarnation bump prevents false death declarations
5. **Graceful leave protocol** — Wire `node.leave()` into the simulation (currently only crash-stop)
6. **BUGGIFY-style injection** — Add probabilistic fault injection at protocol decision points
7. **Study more codebases** — tikv/raft-rs test harness, al8n/memberlist (Rust port)

### Open Questions
- Should we add a re-join mechanism that fires automatically when a partition heals? (FoundationDB does this; standard SWIM doesn't)
- Are the 3 pre-existing gossip MT test failures worth investigating? (convergence_curve_is_monotonic_mt, all_nodes_receive_all_keys_in_ring_1000_mt, partition_heals_and_converges_mt)
- How to model clock skew in a tick-based simulation?

### Blockers
- None currently