swactor/CLAUDE/notes/research_synthesis.md

Research Synthesis: Competitor Analysis

Frameworks Studied

1. Ractor (Rust) — async task-per-actor on tokio
2. Tokio (Rust) — work-stealing async runtime
3. Erlang/OTP BEAM — reduction-counted preemptive scheduler
4. Linux CFS/EEVDF — vruntime fairness, work stealing, adaptive ticks
5. libuv/Node.js — single-threaded event loop with phase-based execution

Critical Finding: Swactor Fairness Bug

tick_all in worker.rs drains the ENTIRE mailbox for each actor before moving to the next:

while let Some(msg) = slot.mailbox.pop_front() {
    // processes ALL messages for actor A before moving to actor B
}

If actor A has 10,000 queued messages, all other actors on the same worker are completely starved until A finishes. Every other runtime studied prevents this:

• BEAM: 4000 reductions per process, then preempt
• Tokio: 128-256 operation cooperative budget per task
• libuv: Round-robin across handlers; no single handler drains completely
• Linux CFS: vruntime-based fairness; time slices enforced

Ranked Improvement Opportunities

P0: Per-Actor Message Budget (Fairness)

• Impact: Prevents starvation; critical for production workloads
• Effort: Small — modify tick_all loop in worker.rs
• Source: BEAM reductions, tokio coop budget
• Design: Process up to N messages per actor per tick, configurable via RuntimeConfig

P1: Improved Benchmarks

• Impact: Can't improve what you can't measure
• Effort: Medium — new benchmark scenarios
• Source: Ractor benchmarks, tokio benchmarks
• New scenarios needed:
  • Fairness: imbalanced load (1 hot actor + 99 cold actors)
  • Message size sensitivity (8B, 64B, 256B, 1KB)
  • Contention: many-to-one fanin
  • Latency percentiles (p50, p99, p999)
  • Cross-worker vs same-worker message delivery

P2: Better Testing Coverage

• Impact: Catches regressions, validates fairness guarantees
• Effort: Medium
• Source: BEAM testing patterns, tokio Loom
• New tests needed:
  • Fairness: hot actor doesn't starve cold actors
  • Backpressure: tiny buffer under load
  • Concurrent spawn+send races
  • Multi-threaded delivery guarantees

P3: Adaptive Backoff with Thread Parking

• Impact: Better latency under varying load; power savings
• Effort: Medium — modify run loop in worker.rs
• Source: Tokio parker, Linux NO_HZ
• Design: Replace spinning with condvar-based parking; use notification to wake

P4: Work Stealing (Future)

• Impact: Dynamic load balancing
• Effort: Large — significant architectural change
• Source: Tokio steal-half, BEAM migration plans
• Note: Would require stealing actors between workers, which changes ownership

Key Design Comparisons

Dimension	Swactor	Ractor	Tokio	BEAM
Scheduling	Sync tick	Async task-per-actor	Work-stealing	Reduction preemption
Fairness	None (drain all)	N/A (1 task = 1 actor)	Coop budget (128)	4000 reductions
Backpressure	Bounded crossbeam ring	None (unbounded)	Bounded MPSC	Off-heap mailbox
Work stealing	None	Tokio handles it	Steal-half, N/2 searchers	Steal + migrate
Panic handling	catch_unwind + poison	AssertUnwindSafe + supervisor	N/A	Process isolation
Message passing	Box downcast	Box downcast	Typed channels	Term copying

Ractor Bug History Lessons

• Destructive get_children() — snapshot methods must not mutate state
• OutputPort silent drops — bounded channels need explicit backpressure, not silent overflow
• Remote actor latency regression — cross-runtime messaging needs careful ordering
• Memory bloat per actor — each channel/structure per actor adds up at scale

Tokio Patterns to Adopt

1. LIFO slot for same-worker sends (cache locality)
2. Searcher count limiting (N/2 max) for cross-worker stealing
3. Steal-half strategy (amortize overhead)
4. Global queue interval checking (reduce contention)
5. Loom-style testing for lock-free code
6. Single allocation per actor context (hot/cold layout)

Additional Frameworks Studied (Cycle 2)

Kameo (v0.19)

• Fully async on tokio, one task per actor
• Dual mailbox: bounded (default 64) or unbounded tokio mpsc channels
• Proper backpressure via bounded mpsc sender blocking
• Typed signals (no Box) — vtable dispatch, no downcast failures
• Erlang-style links for supervision (on_link_died)
• on_panic hook can restart actor (vs swactor's permanent poisoning)
• Bugs: deadlocks in link establishment, leaked ActorRef preventing stop

Actix (v0.13)

• Context-as-Future model — each actor is a single pollable Future on an Arbiter
• Custom Vyukov lock-free MPSC queue (not tokio channels) — push is single atomic_swap
• Default mailbox capacity: 16 (tiny!)
• do_send() bypasses capacity for internal notifications
• Mailbox has 256-message assertion guard (similar to our budget approach!)
• vtable dispatch via Box<dyn EnvelopeProxy<A>> — no Any downcast
• SyncArbiter: crossbeam_channel thread pool for blocking actors
• WHY FAST: custom MPSC queue, no async overhead for message processing, same-thread actors avoid cross-thread coordination, SmallVec for futures

Swactor Advantages (confirmed)

• Synchronous tick model: deterministic, no async overhead, simulation-friendly
• Hybrid channel: bounded ring + unbounded overflow = no message loss
• Per-actor message budget: validated by BEAM (4000 reds), tokio (128 ops), actix (256 assert)
• No tokio dependency: could run on bare metal
• Detailed per-phase timing stats (6-phase TickTiming)

Swactor Weaknesses to Address

• Box downcast can fail silently → type mismatch tracking needed (have it)
• No backpressure: senders never block → unbounded queue growth under sustained load
• Panicked actors permanently poisoned → no recovery path
• Spin/sleep backoff wastes CPU → condvar-based parking would be better
• No supervision trees