swactor/CLAUDE/notes/research_synthesis.md

Research Synthesis: Competitor Analysis

Frameworks Studied

1. Ractor (Rust) — async task-per-actor on tokio
2. Tokio (Rust) — work-stealing async runtime
3. Erlang/OTP BEAM — reduction-counted preemptive scheduler
4. Linux CFS/EEVDF — vruntime fairness, work stealing, adaptive ticks
5. libuv/Node.js — single-threaded event loop with phase-based execution

Critical Finding: Swactor Fairness Bug

tick_all in worker.rs drains the ENTIRE mailbox for each actor before moving to the next:

while let Some(msg) = slot.mailbox.pop_front() {
    // processes ALL messages for actor A before moving to actor B
}

If actor A has 10,000 queued messages, all other actors on the same worker are completely starved until A finishes. Every other runtime studied prevents this:

• BEAM: 4000 reductions per process, then preempt
• Tokio: 128-256 operation cooperative budget per task
• libuv: Round-robin across handlers; no single handler drains completely
• Linux CFS: vruntime-based fairness; time slices enforced

Ranked Improvement Opportunities

P0: Per-Actor Message Budget (Fairness)

• Impact: Prevents starvation; critical for production workloads
• Effort: Small — modify tick_all loop in worker.rs
• Source: BEAM reductions, tokio coop budget
• Design: Process up to N messages per actor per tick, configurable via RuntimeConfig

P1: Improved Benchmarks

• Impact: Can't improve what you can't measure
• Effort: Medium — new benchmark scenarios
• Source: Ractor benchmarks, tokio benchmarks
• New scenarios needed:
  • Fairness: imbalanced load (1 hot actor + 99 cold actors)
  • Message size sensitivity (8B, 64B, 256B, 1KB)
  • Contention: many-to-one fanin
  • Latency percentiles (p50, p99, p999)
  • Cross-worker vs same-worker message delivery

P2: Better Testing Coverage

• Impact: Catches regressions, validates fairness guarantees
• Effort: Medium
• Source: BEAM testing patterns, tokio Loom
• New tests needed:
  • Fairness: hot actor doesn't starve cold actors
  • Backpressure: tiny buffer under load
  • Concurrent spawn+send races
  • Multi-threaded delivery guarantees

P3: Adaptive Backoff with Thread Parking

• Impact: Better latency under varying load; power savings
• Effort: Medium — modify run loop in worker.rs
• Source: Tokio parker, Linux NO_HZ
• Design: Replace spinning with condvar-based parking; use notification to wake

P4: Work Stealing (Future)

• Impact: Dynamic load balancing
• Effort: Large — significant architectural change
• Source: Tokio steal-half, BEAM migration plans
• Note: Would require stealing actors between workers, which changes ownership

Key Design Comparisons

Dimension	Swactor	Ractor	Tokio	BEAM
Scheduling	Sync tick	Async task-per-actor	Work-stealing	Reduction preemption
Fairness	None (drain all)	N/A (1 task = 1 actor)	Coop budget (128)	4000 reductions
Backpressure	Bounded crossbeam ring	None (unbounded)	Bounded MPSC	Off-heap mailbox
Work stealing	None	Tokio handles it	Steal-half, N/2 searchers	Steal + migrate
Panic handling	catch_unwind + poison	AssertUnwindSafe + supervisor	N/A	Process isolation
Message passing	Box downcast	Box downcast	Typed channels	Term copying

Ractor Bug History Lessons

• Destructive get_children() — snapshot methods must not mutate state
• OutputPort silent drops — bounded channels need explicit backpressure, not silent overflow
• Remote actor latency regression — cross-runtime messaging needs careful ordering
• Memory bloat per actor — each channel/structure per actor adds up at scale

Tokio Patterns to Adopt

1. LIFO slot for same-worker sends (cache locality)
2. Searcher count limiting (N/2 max) for cross-worker stealing
3. Steal-half strategy (amortize overhead)
4. Global queue interval checking (reduce contention)
5. Loom-style testing for lock-free code
6. Single allocation per actor context (hot/cold layout)