swactor/CLAUDE/notes/research_synthesis.md

# Research Synthesis: Competitor Analysis

## Frameworks Studied
1. **Ractor** (Rust) — async task-per-actor on tokio
2. **Tokio** (Rust) — work-stealing async runtime
3. **Erlang/OTP BEAM** — reduction-counted preemptive scheduler
4. **Linux CFS/EEVDF** — vruntime fairness, work stealing, adaptive ticks
5. **libuv/Node.js** — single-threaded event loop with phase-based execution

## Critical Finding: Swactor Fairness Bug

`tick_all` in `worker.rs` drains the ENTIRE mailbox for each actor before moving to the next:
```rust
while let Some(msg) = slot.mailbox.pop_front() {
    // processes ALL messages for actor A before moving to actor B
}
```

If actor A has 10,000 queued messages, all other actors on the same worker are completely starved
until A finishes. Every other runtime studied prevents this:
- **BEAM**: 4000 reductions per process, then preempt
- **Tokio**: 128-256 operation cooperative budget per task
- **libuv**: Round-robin across handlers; no single handler drains completely
- **Linux CFS**: vruntime-based fairness; time slices enforced

## Ranked Improvement Opportunities

### P0: Per-Actor Message Budget (Fairness)
- **Impact**: Prevents starvation; critical for production workloads
- **Effort**: Small — modify `tick_all` loop in `worker.rs`
- **Source**: BEAM reductions, tokio coop budget
- **Design**: Process up to N messages per actor per tick, configurable via RuntimeConfig

### P1: Improved Benchmarks
- **Impact**: Can't improve what you can't measure
- **Effort**: Medium — new benchmark scenarios
- **Source**: Ractor benchmarks, tokio benchmarks
- **New scenarios needed**:
  - Fairness: imbalanced load (1 hot actor + 99 cold actors)
  - Message size sensitivity (8B, 64B, 256B, 1KB)
  - Contention: many-to-one fanin
  - Latency percentiles (p50, p99, p999)
  - Cross-worker vs same-worker message delivery

### P2: Better Testing Coverage
- **Impact**: Catches regressions, validates fairness guarantees
- **Effort**: Medium
- **Source**: BEAM testing patterns, tokio Loom
- **New tests needed**:
  - Fairness: hot actor doesn't starve cold actors
  - Backpressure: tiny buffer under load
  - Concurrent spawn+send races
  - Multi-threaded delivery guarantees

### P3: Adaptive Backoff with Thread Parking
- **Impact**: Better latency under varying load; power savings
- **Effort**: Medium — modify `run` loop in `worker.rs`
- **Source**: Tokio parker, Linux NO_HZ
- **Design**: Replace spinning with condvar-based parking; use notification to wake

### P4: Work Stealing (Future)
- **Impact**: Dynamic load balancing
- **Effort**: Large — significant architectural change
- **Source**: Tokio steal-half, BEAM migration plans
- **Note**: Would require stealing actors between workers, which changes ownership

## Key Design Comparisons

| Dimension | Swactor | Ractor | Tokio | BEAM |
|-----------|---------|--------|-------|------|
| Scheduling | Sync tick | Async task-per-actor | Work-stealing | Reduction preemption |
| Fairness | None (drain all) | N/A (1 task = 1 actor) | Coop budget (128) | 4000 reductions |
| Backpressure | Bounded crossbeam ring | None (unbounded) | Bounded MPSC | Off-heap mailbox |
| Work stealing | None | Tokio handles it | Steal-half, N/2 searchers | Steal + migrate |
| Panic handling | catch_unwind + poison | AssertUnwindSafe + supervisor | N/A | Process isolation |
| Message passing | Box<dyn Any> downcast | Box<dyn Any> downcast | Typed channels | Term copying |

## Ractor Bug History Lessons
- Destructive `get_children()` — snapshot methods must not mutate state
- OutputPort silent drops — bounded channels need explicit backpressure, not silent overflow
- Remote actor latency regression — cross-runtime messaging needs careful ordering
- Memory bloat per actor — each channel/structure per actor adds up at scale

## Tokio Patterns to Adopt
1. LIFO slot for same-worker sends (cache locality)
2. Searcher count limiting (N/2 max) for cross-worker stealing
3. Steal-half strategy (amortize overhead)
4. Global queue interval checking (reduce contention)
5. Loom-style testing for lock-free code
6. Single allocation per actor context (hot/cold layout)

## Additional Frameworks Studied (Cycle 2)

### Kameo (v0.19)
- Fully async on tokio, one task per actor
- Dual mailbox: bounded (default 64) or unbounded tokio mpsc channels
- Proper backpressure via bounded mpsc sender blocking
- Typed signals (no Box<dyn Any>) — vtable dispatch, no downcast failures
- Erlang-style links for supervision (`on_link_died`)
- `on_panic` hook can restart actor (vs swactor's permanent poisoning)
- Bugs: deadlocks in link establishment, leaked ActorRef preventing stop

### Actix (v0.13)
- Context-as-Future model — each actor is a single pollable Future on an Arbiter
- **Custom Vyukov lock-free MPSC queue** (not tokio channels) — push is single atomic_swap
- Default mailbox capacity: 16 (tiny!)
- `do_send()` bypasses capacity for internal notifications
- Mailbox has 256-message assertion guard (similar to our budget approach!)
- vtable dispatch via `Box<dyn EnvelopeProxy<A>>` — no Any downcast
- SyncArbiter: crossbeam_channel thread pool for blocking actors
- WHY FAST: custom MPSC queue, no async overhead for message processing,
  same-thread actors avoid cross-thread coordination, SmallVec for futures

### Swactor Advantages (confirmed)
- Synchronous tick model: deterministic, no async overhead, simulation-friendly
- Hybrid channel: bounded ring + unbounded overflow = no message loss
- Per-actor message budget: validated by BEAM (4000 reds), tokio (128 ops), actix (256 assert)
- No tokio dependency: could run on bare metal
- Detailed per-phase timing stats (6-phase TickTiming)

### Swactor Weaknesses to Address
- Box<dyn Any> downcast can fail silently → type mismatch tracking needed (have it)
- No backpressure: senders never block → unbounded queue growth under sustained load
- Panicked actors permanently poisoned → no recovery path
- Spin/sleep backoff wastes CPU → condvar-based parking would be better
- No supervision trees