swactor/CLAUDE/notes/progress.md

Progress Log

Current Stage: Phase 1 — Research + First Improvement Cycle

Status: Cycle 7 COMPLETE

Plan Overview

1. Phase 0: Codebase audit — understand current swactor architecture, existing tests, benchmarks ✅
2. Phase 1: Broad survey + interleaved improvements
3. Phase 2: Deeper improvements based on findings
4. Phase 3: Testing methodology improvements
5. Phase 4: Final evaluation & documentation

Completed This Session

Cycle 1: Fairness (Message Budget)

• Research: Studied ractor, tokio, Erlang/OTP BEAM, Linux CFS/EEVDF, libuv
• Finding: tick_all drained ENTIRE mailbox per actor per tick — critical fairness bug
  • BEAM uses 4000 reduction budget, tokio uses 128-op cooperative budget
  • Swactor had zero budget — one hot actor could starve all others on same worker
• Implementation: Added actor_message_budget to RuntimeConfig (default: 64)
  • Modified tick_all to break after budget messages per actor
  • budget=0 means unlimited (backward compatible)
• Tests: 3 new fairness tests (hot_actor_does_not_starve_cold_actor, unlimited_budget_drains_all, budget_messages_drain_across_multiple_ticks)
• Benchmarks: Added fairness benchmark group (cold_latency_under_pressure, throughput_by_budget)
• Fixes: Updated RuntimeConfig struct literals across crates (python, runtime-dashboard, mt_benchmarks)
• Result: 45 tests pass (42 original + 3 new), all workspace crates compile

Cycle 2: Stress Tests, Benchmarks, Research Expansion

• Research: Added Kameo and Actix analysis to synthesis
  • Actix uses custom Vyukov lock-free MPSC queue (why it's fastest)
  • Kameo has dual bounded/unbounded mailbox, default capacity 64
  • Both use vtable dispatch (not Box downcast)
  • Actix has 256-message assertion guard (validates our budget approach)
• Stress tests: 6 new tests
  • message_ordering_preserved_under_budget — FIFO order with budget=8
  • mt_stress_many_senders_one_receiver — 50 senders × 100 msgs, 4 threads
  • mt_stress_concurrent_spawn_and_send — 200 concurrent spawn+send, 4 threads
  • mt_chain_spawning_under_load — 50-level chain across 2 workers
  • mt_panic_isolation_under_load — 10 panicking + 10 healthy actors, 4 threads
  • sustained_throughput_does_not_drop_messages — 10 batches × 100 msgs
• Benchmarks: 2 new benchmark groups
  • msg_size: throughput and send_latency by message size (8B-4KB)
  • contention: fanin (1-100 senders to 1 sink), cross_worker (1-4 threads)
• Result: 51 tests pass (42 original + 3 fairness + 6 stress), all workspace compiles

Cycle 3: Thread Parking (Adaptive Backoff)

• Implementation: Replaced thread::sleep with thread::park_timeout in worker run loop
  • Workers register thread::current() via OnceLock<Thread> on startup
  • send_to and spawn call Thread::unpark() on target worker
  • Cross-worker sends from WorkerContext also unpark target
  • Zero new dependencies (uses std::sync::OnceLock + std::thread::park_timeout)
• Design source: Tokio's parker state machine, Linux NO_HZ adaptive ticks
• Benefits: Parked workers wake instantly when work arrives (vs waiting for sleep timer)
  • Reduces idle-to-active latency from up to 1ms to near-zero
  • No overhead on hot path — unpark() is no-op if thread isn't parked
• Tests: 1 new test (mt_parked_worker_wakes_on_send)
• Result: 52 tests pass (51 + 1 new), all workspace compiles

Cycle 4: Shutdown Fix + Bug-Inspired Tests

• Shutdown improvement: shutdown() now unparks all workers for immediate exit
  • Previously, parked workers wouldn't notice shutdown until park_timeout expired
• Bug-inspired tests (5 new, from competitor bug reports):
  • stats_snapshot_is_read_only — from ractor #310 (destructive get_children)
  • stats_under_load_do_not_interfere_with_processing — stats don't affect msg processing
  • shutdown_wakes_parked_workers_immediately — validates fast shutdown with parking
  • mt_send_after_run_delivers_to_running_actors — from kameo #185 (startup delivery)
  • budget_respected_even_with_self_sends — from actix #515 (mailbox bypass)
• Result: 57 tests pass, all workspace compiles

Cycle 5: Work Stealing Research + Load-Aware Placement

• Research: Deep analysis of work stealing in Tokio, Go, BEAM, ForkJoinPool
  • Tokio: fixed 256-slot ring, steal-half, LIFO slot (3-use starvation cap), N/2 searcher limit
  • Go: M:N scheduler, runnext + 256-slot local queue, steal-half, 4 tries with random permutation
  • BEAM: unique dual approach — reactive stealing + proactive migration via check_balance()
  • ForkJoinPool: owner LIFO / thief FIFO deque, even/odd queue indexing
• Feasibility analysis: Full actor migration IS mechanically possible (ActorSlot is Send), but:
  • Requires push-based donation (ActorPool not Sync → no pull stealing)
  • 1-tick message loss window during migration
  • Significant complexity for uncertain benefit
• Implementation: Load-aware placement replaces blind round-robin
  • Placement::next_worker() now reads per-worker stats (num_actors + mailbox_depth)
  • Scan starts from rotating position → round-robin when all stats equal (initial burst)
  • O(N) relaxed atomic loads per spawn, trivial for N≤8 workers
• Tests: 3 new tests
  • load_aware_placement_prefers_lighter_worker — imbalanced load biases toward lighter worker
  • load_aware_placement_single_worker_degrades_gracefully — single-thread works correctly
  • load_aware_placement_falls_back_to_round_robin_on_fresh_runtime — even distribution before ticks
• Benchmarks: 1 new group — placement/spawn_under_load (2t, 4t)
• Result: 60 tests pass, all workspace compiles

Cycle 7: Actor Recovery (Factory Restart)

• Research: Deep analysis of supervision/recovery across Erlang (supervision trees, restart intensity), Akka (Resume/Restart/Stop/Escalate), Kameo (on_panic hook), Actix (Supervised trait), Ractor (SupervisionEvent)
  • Erlang: fresh process via factory (MFA tuple), mailbox lost, PID changes
  • Akka: replace internals but keep ActorRef stable, mailbox preserved (docs say this is usually wrong)
  • Kameo: on_panic(&mut self) — risky with corrupt state after panic
  • Decision: factory-based restart (Erlang-style), safest approach
• Implementation: spawn_restartable(actor, factory, max_restarts) on Runtime and Ctx
  • Actor<A> expanded from tuple struct to named fields: inner, restart_factory, max_restarts, restart_count
  • AnyActor::try_restart(&self) trait method (default None, backward compatible)
  • Factory stored as Arc<dyn Fn() -> A + Send + Sync> — cloned into fresh Actor on restart
  • tick_all panic handler: try_restart before poisoning, clear mailbox, fresh state
  • restarts counter added to WorkerStats and WorkerInfo
• Safety: Factory fields are "cold" (never touched by handle_any), safe to read after catch_unwind
• Tests: 4 new tests
  • restartable_actor_recovers_after_panic — basic restart works
  • restartable_actor_resets_state_on_restart — fresh state post-restart
  • restartable_actor_respects_max_restarts — 2 restarts then permanent poison
  • non_restartable_actor_still_poisons_on_panic — backward compatibility
• Result: 68 tests pass, all workspace compiles

Cycle 6: Mailbox Backpressure

• Research: Compared backpressure across Erlang (unbounded, pobox), Actix (cap 16, do_send bypass), Kameo (cap 64, bounded), Tokio mpsc (bounded, permit pattern), Go channels (blocking)
  • Consensus: bounded by default, configurable overflow policy
• Implementation: Per-actor bounded mailboxes with configurable overflow
  • Added MailboxOverflow enum: DropNewest (discard incoming) and DropOldest (evict oldest)
  • Added default_mailbox_capacity and mailbox_overflow to RuntimeConfig
  • Default: capacity=0 (unbounded) — 100% backward compatible
  • ActorSlot stores per-actor capacity and policy (from runtime defaults)
  • deliver() enforces bounds; dropped messages tracked via drops_this_tick counter
  • messages_dropped: AtomicU64 added to WorkerStats and WorkerInfo
• Tests: 4 new tests
  • bounded_mailbox_drop_newest_caps_at_capacity — 50 msgs, cap 10 → only 10 delivered
  • bounded_mailbox_drop_oldest_keeps_newest — 10 msgs, cap 5 → newest 5 kept
  • unbounded_mailbox_delivers_all_messages — backward compatibility check
  • bounded_mailbox_refills_after_processing — cap 5, process, refill works
• Result: 64 tests pass, all workspace compiles

Research Notes

• Full analysis in CLAUDE/notes/research_synthesis.md
• Baseline benchmarks in CLAUDE/notes/baseline_benchmarks.md
• Constraints in CLAUDE/notes/constraints.md

Next Steps

• Cycle 2: Stress testing + property-based tests ✅
• Cycle 3: Adaptive backoff with thread parking ✅
• Cycle 4: Enhanced benchmarks + bug-inspired tests ✅
• Cycle 5: Work stealing research + load-aware placement ✅
• Cycle 6: Mailbox backpressure ✅
• Cycle 7: Actor recovery (factory restart) ✅
• Cycle 8: Next improvement
  • Candidates: arena-allocated ActorPool, per-actor mailbox config, tracing integration
  • LIFO slot rejected (0-5% benefit for typical workloads, not worth complexity)

Open Questions

• Should budget be configurable per-actor (not just per-runtime)?
• Is 64 the right default budget? Benchmarks show budget=32 slightly faster for throughput
• Thread parking: notification mechanism RESOLVED: OnceLock + unpark()
• Should load-aware placement weight mailbox depth more than actor count?
• LIFO slot for same-worker sends: worth the complexity?

Blockers

• (none)