8d8e33c0e2
Add network partition, asymmetric partition, and message loss simulation to the distribution test harness. 15 new behavioral tests covering split-brain, cascading failure, seed node death, rapid churn, 50-node clusters, and actor resolution under partition. Research notes from studying FoundationDB DST, Hashicorp memberlist, Antithesis, TigerBeetle VOPR, Turmoil/MadSim, and Jepsen nemeses. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
3.2 KiB
3.2 KiB
Simulation Testing Research
Sources Studied
- Hashicorp memberlist (Go SWIM) — test methodology, Lifeguard extensions
- FoundationDB — deterministic simulation, BUGGIFY fault injection
- Antithesis — fault injection categories
- TigerBeetle — VOPR simulation, Vortex TCP proxy testing
- Turmoil / MadSim — Rust DST frameworks
- Jepsen — standard nemeses for distributed systems
- Academic: SWIM paper, gossip protocol convergence properties
Key Concepts
FoundationDB DST Pattern
- Single-threaded, seeded PRNG, simulated time (discrete-event)
- Same binary for simulation and production (interface abstraction)
- BUGGIFY: two-phase internal fault injection (25% activation, 25% firing)
- 5 patterns: minimal work, error forcing, concurrency delays, knob randomization, damage control
- Test oracle: reference impl comparison, operation replay, invariant workloads
Hashicorp Memberlist Test Coverage
- Probe cycle: direct ping → indirect ping (PingReq) → TCP fallback → suspect
- Lifeguard: Suspicion timer with log(k+1) decay, health-aware probe timeouts, Dogpile confirmation
- State machine: Alive → Suspect → Dead with incarnation-based conflict resolution
- Tests: ~80 test functions covering join/leave, probe, state transitions, encryption, labels, metadata, PushPull sync
- Key missing from swactor: awareness/health scoring, nack-based probing, PushPull full state sync
Standard Failure Modes (from Jepsen/Antithesis/TigerBeetle)
- Network partition (symmetric)
- Asymmetric partition (A→B works, B→A drops)
- Message loss (random % drop)
- Message delay/reorder
- Process crash + restart
- Slow/degraded node (CPU starvation)
- Cascading failure (sequential kills)
- Split-brain (minority vs majority partition)
- Clock skew (not applicable to our tick-based sim)
Invariants to Check (SWIM+Kademlia)
- Completeness: Every failed node eventually detected by all survivors
- Accuracy: No healthy node permanently marked dead
- Convergence: Membership views agree within O(log N) rounds
- Dissemination: Membership updates reach all nodes
- Routing table consistency: k-buckets maintain closest-node invariant
- Directory repair: Dead node's entries re-replicated to surviving nodes
- Cache coherence: Dead node's cached locations invalidated
Gaps in Current Test Suite
| Gap | Priority | Notes |
|---|---|---|
| Network partition / split-brain | High | No partition testing exists |
| Message loss (% drop) | High | Sim delivers 100% reliably |
| Asymmetric partition | Medium | One-way failures |
| Seed node failure | High | Current tests only kill non-seed |
| Simultaneous multi-node failure | Medium | Only single kills tested |
| Cascading sequential failure | Medium | Real-world pattern |
| Large cluster (50+) | Medium | Only 5 and 20 tested |
| Rapid churn (join+leave+kill) | High | Realistic workload |
| Graceful leave protocol | Medium | leave() untested in sim |
| Dissemination completeness | High | Not directly verified |
| Suspicion refutation | Medium | Incarnation bump logic |
| Directory repair after death | Medium | repair_queue untested |
| Cache invalidation correctness | Low | Simple but important |