8d8e33c0e2
Add network partition, asymmetric partition, and message loss simulation to the distribution test harness. 15 new behavioral tests covering split-brain, cascading failure, seed node death, rapid churn, 50-node clusters, and actor resolution under partition. Research notes from studying FoundationDB DST, Hashicorp memberlist, Antithesis, TigerBeetle VOPR, Turmoil/MadSim, and Jepsen nemeses. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
52 lines
3.3 KiB
Markdown
52 lines
3.3 KiB
Markdown
# Progress
|
|
|
|
## Session 1 — Simulation Test Breadth (2026-02-12)
|
|
|
|
### Completed
|
|
1. **Research phase**: Studied Hashicorp memberlist, FoundationDB DST, Antithesis, TigerBeetle VOPR, Turmoil/MadSim, Jepsen nemeses
|
|
- Notes in `CLAUDE/notes/research_simulation_testing.md`
|
|
2. **Enhanced simulation harness** (`crates/simulation/src/distribution/sim.rs`):
|
|
- Added `NetworkFault` enum: `Partition`, `Heal`, `SetDropRate`
|
|
- Added `Partition` struct with `side_a`, `side_b`, `asymmetric` fields
|
|
- Added `NetworkState` with blocked-pair tracking and LCG-based message dropping
|
|
- Modified `deliver_actions_tagged` → `deliver_actions_tagged_with_net` (respects network faults)
|
|
- Existing 6 distribution tests unaffected (backward compatible)
|
|
3. **15 new cluster scenario tests** (`crates/simulation/tests/cluster_scenarios.rs`):
|
|
- Symmetric partition (split-brain, each side forms sub-cluster)
|
|
- Asymmetric partition (one-way communication)
|
|
- 10% message loss (converges with tuned timeouts)
|
|
- 30% message loss (degrades but doesn't crash)
|
|
- Seed node death (cluster survives without seed)
|
|
- Simultaneous 2-node failure
|
|
- Cascading sequential failure (3 nodes killed over time)
|
|
- Large cluster (50 nodes)
|
|
- Rapid churn (kill/revive cycles)
|
|
- Crash detection speed (bounded detection time)
|
|
- Partition + kill in minority side
|
|
- Actor resolution during partition
|
|
- Dissemination completeness (10-node cluster, all detect death)
|
|
- Sequential partitions (fragment cluster)
|
|
- Brief message loss recovery
|
|
|
|
### Key Findings
|
|
- **SWIM does not auto-rediscover dead-declared nodes** after partition heals. Once the suspicion timeout expires and a node is declared dead, it's permanently removed. Re-discovery requires the join protocol.
|
|
- **Message loss is highly destabilizing** for SWIM because it affects both the direct probe AND the indirect probe simultaneously. Even 15% loss with default config can cause false deaths.
|
|
- **Tuning suspicion_timeout and indirect_probes** is critical for lossy networks. Higher values tolerate more loss but increase detection latency.
|
|
- **The LCG PRNG for message dropping needs a non-zero seed** to avoid correlated early values.
|
|
|
|
### Next Steps
|
|
1. **Depth: Property-based invariant checking** — Add formal SWIM invariants (completeness, accuracy) as automated property checks
|
|
2. **Message reordering** — Add out-of-order delivery to the network model
|
|
3. **Kademlia-specific scenarios** — Test routing table convergence under churn, directory repair after death
|
|
4. **Suspicion refutation tests** — Verify incarnation bump prevents false death declarations
|
|
5. **Graceful leave protocol** — Wire `node.leave()` into the simulation (currently only crash-stop)
|
|
6. **BUGGIFY-style injection** — Add probabilistic fault injection at protocol decision points
|
|
7. **Study more codebases** — tikv/raft-rs test harness, al8n/memberlist (Rust port)
|
|
|
|
### Open Questions
|
|
- Should we add a re-join mechanism that fires automatically when a partition heals? (FoundationDB does this; standard SWIM doesn't)
|
|
- Are the 3 pre-existing gossip MT test failures worth investigating? (convergence_curve_is_monotonic_mt, all_nodes_receive_all_keys_in_ring_1000_mt, partition_heals_and_converges_mt)
|
|
- How to model clock skew in a tick-based simulation?
|
|
|
|
### Blockers
|
|
- None currently
|