64 lines
3.2 KiB
Markdown
64 lines
3.2 KiB
Markdown
|
# Simulation Testing Research
|
||
|
|
|
||
|
|
## Sources Studied
|
||
|
|
- Hashicorp memberlist (Go SWIM) — test methodology, Lifeguard extensions
|
||
|
|
- FoundationDB — deterministic simulation, BUGGIFY fault injection
|
||
|
|
- Antithesis — fault injection categories
|
||
|
|
- TigerBeetle — VOPR simulation, Vortex TCP proxy testing
|
||
|
|
- Turmoil / MadSim — Rust DST frameworks
|
||
|
|
- Jepsen — standard nemeses for distributed systems
|
||
|
|
- Academic: SWIM paper, gossip protocol convergence properties
|
||
|
|
|
||
|
|
## Key Concepts
|
||
|
|
|
||
|
|
### FoundationDB DST Pattern
|
||
|
|
- Single-threaded, seeded PRNG, simulated time (discrete-event)
|
||
|
|
- Same binary for simulation and production (interface abstraction)
|
||
|
|
- BUGGIFY: two-phase internal fault injection (25% activation, 25% firing)
|
||
|
|
- 5 patterns: minimal work, error forcing, concurrency delays, knob randomization, damage control
|
||
|
|
- Test oracle: reference impl comparison, operation replay, invariant workloads
|
||
|
|
|
||
|
|
### Hashicorp Memberlist Test Coverage
|
||
|
|
- **Probe cycle**: direct ping → indirect ping (PingReq) → TCP fallback → suspect
|
||
|
|
- **Lifeguard**: Suspicion timer with log(k+1) decay, health-aware probe timeouts, Dogpile confirmation
|
||
|
|
- **State machine**: Alive → Suspect → Dead with incarnation-based conflict resolution
|
||
|
|
- **Tests**: ~80 test functions covering join/leave, probe, state transitions, encryption, labels, metadata, PushPull sync
|
||
|
|
- **Key missing from swactor**: awareness/health scoring, nack-based probing, PushPull full state sync
|
||
|
|
|
||
|
|
### Standard Failure Modes (from Jepsen/Antithesis/TigerBeetle)
|
||
|
|
1. Network partition (symmetric)
|
||
|
|
2. Asymmetric partition (A→B works, B→A drops)
|
||
|
|
3. Message loss (random % drop)
|
||
|
|
4. Message delay/reorder
|
||
|
|
5. Process crash + restart
|
||
|
|
6. Slow/degraded node (CPU starvation)
|
||
|
|
7. Cascading failure (sequential kills)
|
||
|
|
8. Split-brain (minority vs majority partition)
|
||
|
|
9. Clock skew (not applicable to our tick-based sim)
|
||
|
|
|
||
|
|
### Invariants to Check (SWIM+Kademlia)
|
||
|
|
- **Completeness**: Every failed node eventually detected by all survivors
|
||
|
|
- **Accuracy**: No healthy node permanently marked dead
|
||
|
|
- **Convergence**: Membership views agree within O(log N) rounds
|
||
|
|
- **Dissemination**: Membership updates reach all nodes
|
||
|
|
- **Routing table consistency**: k-buckets maintain closest-node invariant
|
||
|
|
- **Directory repair**: Dead node's entries re-replicated to surviving nodes
|
||
|
|
- **Cache coherence**: Dead node's cached locations invalidated
|
||
|
|
|
||
|
|
## Gaps in Current Test Suite
|
||
|
|
| Gap | Priority | Notes |
|
||
|
|
|-----|----------|-------|
|
||
|
|
| Network partition / split-brain | High | No partition testing exists |
|
||
|
|
| Message loss (% drop) | High | Sim delivers 100% reliably |
|
||
|
|
| Asymmetric partition | Medium | One-way failures |
|
||
|
|
| Seed node failure | High | Current tests only kill non-seed |
|
||
|
|
| Simultaneous multi-node failure | Medium | Only single kills tested |
|
||
|
|
| Cascading sequential failure | Medium | Real-world pattern |
|
||
|
|
| Large cluster (50+) | Medium | Only 5 and 20 tested |
|
||
|
|
| Rapid churn (join+leave+kill) | High | Realistic workload |
|
||
|
|
| Graceful leave protocol | Medium | leave() untested in sim |
|
||
|
|
| Dissemination completeness | High | Not directly verified |
|
||
|
|
| Suspicion refutation | Medium | Incarnation bump logic |
|
||
|
|
| Directory repair after death | Medium | repair_queue untested |
|
||
|
|
| Cache invalidation correctness | Low | Simple but important |
|