swactor/CLAUDE/notes/research_synthesis.md

# Research Synthesis: Competitor Analysis

## Frameworks Studied
1. **Ractor** (Rust) — async task-per-actor on tokio
2. **Tokio** (Rust) — work-stealing async runtime
3. **Erlang/OTP BEAM** — reduction-counted preemptive scheduler
4. **Linux CFS/EEVDF** — vruntime fairness, work stealing, adaptive ticks
5. **libuv/Node.js** — single-threaded event loop with phase-based execution

## Critical Finding: Swactor Fairness Bug

`tick_all` in `worker.rs` drains the ENTIRE mailbox for each actor before moving to the next:
```rust
while let Some(msg) = slot.mailbox.pop_front() {
    // processes ALL messages for actor A before moving to actor B
}
```

If actor A has 10,000 queued messages, all other actors on the same worker are completely starved
until A finishes. Every other runtime studied prevents this:
- **BEAM**: 4000 reductions per process, then preempt
- **Tokio**: 128-256 operation cooperative budget per task
- **libuv**: Round-robin across handlers; no single handler drains completely
- **Linux CFS**: vruntime-based fairness; time slices enforced

## Ranked Improvement Opportunities

### P0: Per-Actor Message Budget (Fairness)
- **Impact**: Prevents starvation; critical for production workloads
- **Effort**: Small — modify `tick_all` loop in `worker.rs`
- **Source**: BEAM reductions, tokio coop budget
- **Design**: Process up to N messages per actor per tick, configurable via RuntimeConfig

### P1: Improved Benchmarks
- **Impact**: Can't improve what you can't measure
- **Effort**: Medium — new benchmark scenarios
- **Source**: Ractor benchmarks, tokio benchmarks
- **New scenarios needed**:
  - Fairness: imbalanced load (1 hot actor + 99 cold actors)
  - Message size sensitivity (8B, 64B, 256B, 1KB)
  - Contention: many-to-one fanin
  - Latency percentiles (p50, p99, p999)
  - Cross-worker vs same-worker message delivery

### P2: Better Testing Coverage
- **Impact**: Catches regressions, validates fairness guarantees
- **Effort**: Medium
- **Source**: BEAM testing patterns, tokio Loom
- **New tests needed**:
  - Fairness: hot actor doesn't starve cold actors
  - Backpressure: tiny buffer under load
  - Concurrent spawn+send races
  - Multi-threaded delivery guarantees

### P3: Adaptive Backoff with Thread Parking
- **Impact**: Better latency under varying load; power savings
- **Effort**: Medium — modify `run` loop in `worker.rs`
- **Source**: Tokio parker, Linux NO_HZ
- **Design**: Replace spinning with condvar-based parking; use notification to wake

### P4: Work Stealing (Future)
- **Impact**: Dynamic load balancing
- **Effort**: Large — significant architectural change
- **Source**: Tokio steal-half, BEAM migration plans
- **Note**: Would require stealing actors between workers, which changes ownership

## Key Design Comparisons

| Dimension | Swactor | Ractor | Tokio | BEAM |
|-----------|---------|--------|-------|------|
| Scheduling | Sync tick | Async task-per-actor | Work-stealing | Reduction preemption |
| Fairness | None (drain all) | N/A (1 task = 1 actor) | Coop budget (128) | 4000 reductions |
| Backpressure | Bounded crossbeam ring | None (unbounded) | Bounded MPSC | Off-heap mailbox |
| Work stealing | None | Tokio handles it | Steal-half, N/2 searchers | Steal + migrate |
| Panic handling | catch_unwind + poison | AssertUnwindSafe + supervisor | N/A | Process isolation |
| Message passing | Box<dyn Any> downcast | Box<dyn Any> downcast | Typed channels | Term copying |

## Ractor Bug History Lessons
- Destructive `get_children()` — snapshot methods must not mutate state
- OutputPort silent drops — bounded channels need explicit backpressure, not silent overflow
- Remote actor latency regression — cross-runtime messaging needs careful ordering
- Memory bloat per actor — each channel/structure per actor adds up at scale

## Tokio Patterns to Adopt
1. LIFO slot for same-worker sends (cache locality)
2. Searcher count limiting (N/2 max) for cross-worker stealing
3. Steal-half strategy (amortize overhead)
4. Global queue interval checking (reduce contention)
5. Loom-style testing for lock-free code
6. Single allocation per actor context (hot/cold layout)