# Progress Log ## Current Stage: Phase 1 — Research + First Improvement Cycle ### Status: Cycle 10 COMPLETE ## Plan Overview 1. **Phase 0**: Codebase audit — understand current swactor architecture, existing tests, benchmarks ✅ 2. **Phase 1**: Broad survey + interleaved improvements 3. **Phase 2**: Deeper improvements based on findings 4. **Phase 3**: Testing methodology improvements 5. **Phase 4**: Final evaluation & documentation ## Completed This Session ### Cycle 1: Fairness (Message Budget) - **Research**: Studied ractor, tokio, Erlang/OTP BEAM, Linux CFS/EEVDF, libuv - **Finding**: `tick_all` drained ENTIRE mailbox per actor per tick — critical fairness bug - BEAM uses 4000 reduction budget, tokio uses 128-op cooperative budget - Swactor had zero budget — one hot actor could starve all others on same worker - **Implementation**: Added `actor_message_budget` to `RuntimeConfig` (default: 64) - Modified `tick_all` to break after `budget` messages per actor - `budget=0` means unlimited (backward compatible) - **Tests**: 3 new fairness tests (hot_actor_does_not_starve_cold_actor, unlimited_budget_drains_all, budget_messages_drain_across_multiple_ticks) - **Benchmarks**: Added fairness benchmark group (cold_latency_under_pressure, throughput_by_budget) - **Fixes**: Updated RuntimeConfig struct literals across crates (python, runtime-dashboard, mt_benchmarks) - **Result**: 45 tests pass (42 original + 3 new), all workspace crates compile ### Cycle 2: Stress Tests, Benchmarks, Research Expansion - **Research**: Added Kameo and Actix analysis to synthesis - Actix uses custom Vyukov lock-free MPSC queue (why it's fastest) - Kameo has dual bounded/unbounded mailbox, default capacity 64 - Both use vtable dispatch (not Box downcast) - Actix has 256-message assertion guard (validates our budget approach) - **Stress tests**: 6 new tests - `message_ordering_preserved_under_budget` — FIFO order with budget=8 - `mt_stress_many_senders_one_receiver` — 50 senders × 100 msgs, 4 threads - `mt_stress_concurrent_spawn_and_send` — 200 concurrent spawn+send, 4 threads - `mt_chain_spawning_under_load` — 50-level chain across 2 workers - `mt_panic_isolation_under_load` — 10 panicking + 10 healthy actors, 4 threads - `sustained_throughput_does_not_drop_messages` — 10 batches × 100 msgs - **Benchmarks**: 2 new benchmark groups - `msg_size`: throughput and send_latency by message size (8B-4KB) - `contention`: fanin (1-100 senders to 1 sink), cross_worker (1-4 threads) - **Result**: 51 tests pass (42 original + 3 fairness + 6 stress), all workspace compiles ### Cycle 3: Thread Parking (Adaptive Backoff) - **Implementation**: Replaced `thread::sleep` with `thread::park_timeout` in worker run loop - Workers register `thread::current()` via `OnceLock` on startup - `send_to` and `spawn` call `Thread::unpark()` on target worker - Cross-worker sends from `WorkerContext` also unpark target - Zero new dependencies (uses `std::sync::OnceLock` + `std::thread::park_timeout`) - **Design source**: Tokio's parker state machine, Linux NO_HZ adaptive ticks - **Benefits**: Parked workers wake instantly when work arrives (vs waiting for sleep timer) - Reduces idle-to-active latency from up to 1ms to near-zero - No overhead on hot path — `unpark()` is no-op if thread isn't parked - **Tests**: 1 new test (`mt_parked_worker_wakes_on_send`) - **Result**: 52 tests pass (51 + 1 new), all workspace compiles ### Cycle 4: Shutdown Fix + Bug-Inspired Tests - **Shutdown improvement**: `shutdown()` now unparks all workers for immediate exit - Previously, parked workers wouldn't notice shutdown until park_timeout expired - **Bug-inspired tests** (5 new, from competitor bug reports): - `stats_snapshot_is_read_only` — from ractor #310 (destructive get_children) - `stats_under_load_do_not_interfere_with_processing` — stats don't affect msg processing - `shutdown_wakes_parked_workers_immediately` — validates fast shutdown with parking - `mt_send_after_run_delivers_to_running_actors` — from kameo #185 (startup delivery) - `budget_respected_even_with_self_sends` — from actix #515 (mailbox bypass) - **Result**: 57 tests pass, all workspace compiles ### Cycle 5: Work Stealing Research + Load-Aware Placement - **Research**: Deep analysis of work stealing in Tokio, Go, BEAM, ForkJoinPool - Tokio: fixed 256-slot ring, steal-half, LIFO slot (3-use starvation cap), N/2 searcher limit - Go: M:N scheduler, runnext + 256-slot local queue, steal-half, 4 tries with random permutation - BEAM: unique dual approach — reactive stealing + proactive migration via check_balance() - ForkJoinPool: owner LIFO / thief FIFO deque, even/odd queue indexing - **Feasibility analysis**: Full actor migration IS mechanically possible (ActorSlot is Send), but: - Requires push-based donation (ActorPool not Sync → no pull stealing) - 1-tick message loss window during migration - Significant complexity for uncertain benefit - **Implementation**: Load-aware placement replaces blind round-robin - `Placement::next_worker()` now reads per-worker stats (num_actors + mailbox_depth) - Scan starts from rotating position → round-robin when all stats equal (initial burst) - O(N) relaxed atomic loads per spawn, trivial for N≤8 workers - **Tests**: 3 new tests - `load_aware_placement_prefers_lighter_worker` — imbalanced load biases toward lighter worker - `load_aware_placement_single_worker_degrades_gracefully` — single-thread works correctly - `load_aware_placement_falls_back_to_round_robin_on_fresh_runtime` — even distribution before ticks - **Benchmarks**: 1 new group — `placement/spawn_under_load` (2t, 4t) - **Result**: 60 tests pass, all workspace compiles ### Cycle 9: Lifecycle Hooks + Graceful Stop - **Research**: Cross-framework lifecycle analysis (Erlang init/terminate, Akka preStart/postStop, Actix started/stopping/stopped, Kameo on_start/on_stop/on_panic, Ractor pre_start/post_stop, Stakker state-based Prep/Ready/Zombie, CAF on_exit) - Also researched graceful stop across Erlang (gen_server:stop, exit, kill), Akka (stop, PoisonPill, Kill, gracefulStop), Actix (ctx.stop, Running::Stop), Kameo (stop_gracefully, kill), Go (context.Done) - Key finding: most frameworks have on_stop NOT called on panic (state may be corrupt) - Key finding: self-stop should be immediate (after current message), external stop is queued - **Implementation**: Lifecycle hooks + dual-mode graceful stop - `ActorInterface::on_start()` and `on_stop()` — default no-ops, backward compatible - `AnyActor::on_start()`/`on_stop()` forwarded from `Actor` impl - `ctx.stop_self()` — immediate stop after current message via `request_stop` buffer - `runtime.stop_actor(addr)` — external stop via StopSignal message (PoisonPill semantics) - `ActorSlot` gains `started: bool` and `stopping: bool` flags - `on_start` called in tick_all before first message; panic in on_start → immediate poison - `on_stop` called in cleanup_dead for stopping (not poisoned) actors, wrapped in catch_unwind - Restarted actors get `started=false` so on_start fires again on fresh instance - `stops: AtomicU64` added to WorkerStats and WorkerInfo - `ContextInner::request_stop()` method for same-worker immediate stop - Phase 7 cleanup_dead now handles both poisoned AND stopping actors, with on_stop context - **Tests**: 12 new tests - `on_start_called_before_first_message` — on_start fires on first tick, before messages - `on_start_called_per_actor` — 5 actors each get one on_start call - `on_start_panic_poisons_actor` — panic in on_start → poisoned, no messages processed - `actor_can_stop_self` — 5 msgs sent, stops after 3, only 3 processed, on_stop called - `runtime_can_stop_actor` — external stop via runtime, on_stop called, actor removed - `send_to_stopped_actor_returns_error` — stopped actor gone from address map - `stop_vs_panic_tracked_separately_in_stats` — stops and panics counted independently - `on_stop_can_send_messages` — farewell message sent during on_stop is delivered - `on_start_called_again_after_restart` — restartable actor gets on_start on fresh instance - `external_stop_is_queued_after_pending_messages` — PoisonPill semantics for external stop - `external_stop_before_new_messages_prevents_processing` — stop before send blocks msgs - `stop_nonexistent_actor_returns_error` — stop on bad address returns Err - **Result**: 82 tests pass, all workspace compiles ### Cycle 10: Actor Timers (Tick-Counting) - **Research**: Studied timer/scheduling patterns across Erlang (timer:send_after, erlang:start_timer), Akka (scheduleOnce, scheduler), Actix (ctx.run_later, ctx.run_interval), Kameo (tokio::time::sleep), Tokio (tokio::time), Go (time.After, time.NewTicker) - Also researched priority messages (REJECTED: lifecycle hooks cover 95% of use cases) - Also researched SmallBox optimization (DEFERRED: measure allocation cost first) - Key finding: per-worker tick-counting is ideal for swactor's synchronous model (deterministic) - **Implementation**: Per-worker `TimerWheel` with deterministic tick-based scheduling - `OnceTimer`: fire once at `fire_at` tick, consumed after firing - `IntervalTimer`: fire every `period` ticks, message cloned via `CloneMsg` trait - `CloneMsg` trait: type-erased clone for interval timer messages (blanket impl for `Message`) - `TimerRequest` enum: `Once { dest, msg, ticks }` | `Interval { dest, msg, period }` - `ctx.send_after_ticks(addr, msg, ticks)` — one-shot timer API - `ctx.send_interval_ticks(addr, msg, period)` — interval timer API - Phase 2.5 in tick_once: fire due timers, route through full delivery system (pool.deliver for local actors, transfer_txs for cross-worker, inbox_registry for inboxes) - Phase 5.5: drain timer requests from handler buffer into TimerWheel - GC: interval timers for removed actors cleaned up after cleanup_dead - `schedule_timer` on Runtime's ContextInner: no-op with warning (timers are per-worker only) - **Bug fixed**: `gc_dead_intervals` was over-aggressive — removed timers for ANY address not in the local pool, including inboxes and cross-worker actors. Fixed to only GC timers for addresses in the `dead` set from cleanup_dead. - **Tests**: 6 new tests - `one_shot_timer_fires_after_n_ticks` — timer with delay=3 fires on tick 4 - `handler_can_schedule_one_shot_timer` — timer scheduled from handler, fires correctly - `one_shot_timer_fires_only_once` — consumed after firing, doesn't repeat - `interval_timer_fires_repeatedly` — period=2, fires every 2 ticks (3 fires verified) - `interval_timer_cleaned_up_when_actor_dies` — GC removes orphaned timers - `timer_with_zero_delay_fires_next_tick` — delay=0 fires on next tick - **Result**: 88 tests pass, all workspace compiles, zero warnings ### Cycle 8: Dead Actor Cleanup (Memory Leak Fix) - **Research**: Audited remaining improvement gaps, found ActorPool and AddressMap both leak permanently poisoned actors. Known class of bug in Akka (#22990), CAF (#420). - **Implementation**: Automatic cleanup of poisoned actors after tick_all - `AddressMap::remove()` added to delivery.rs - `ActorPool::cleanup_dead()` collects and removes poisoned actors, returns their addresses - Phase 7 in tick_once: cleanup_dead → remove from address_map → update num_actors stat - Re-publish num_actors after cleanup so stats immediately reflect removal - **Behavior change**: Sends to poisoned actors now return Err (address not found) instead of silently discarding. This is better — callers learn the actor is gone. - **Tests**: 2 new tests + 2 existing tests updated - `dead_actor_cleaned_up_from_stats_and_address_map` — good actor persists, bad actor removed - `bulk_dead_actor_cleanup` — 20 panicked actors all cleaned up - Updated `send_to_poisoned_actor_is_a_silent_black_hole` — now asserts send returns Err - Updated `poisoned_actor_messages_not_counted_as_processed` — sends fail to cleaned-up actor - **Result**: 70 tests pass, all workspace compiles ### Cycle 7: Actor Recovery (Factory Restart) - **Research**: Deep analysis of supervision/recovery across Erlang (supervision trees, restart intensity), Akka (Resume/Restart/Stop/Escalate), Kameo (on_panic hook), Actix (Supervised trait), Ractor (SupervisionEvent) - Erlang: fresh process via factory (MFA tuple), mailbox lost, PID changes - Akka: replace internals but keep ActorRef stable, mailbox preserved (docs say this is usually wrong) - Kameo: on_panic(&mut self) — risky with corrupt state after panic - Decision: factory-based restart (Erlang-style), safest approach - **Implementation**: `spawn_restartable(actor, factory, max_restarts)` on Runtime and Ctx - `Actor` expanded from tuple struct to named fields: inner, restart_factory, max_restarts, restart_count - `AnyActor::try_restart(&self)` trait method (default None, backward compatible) - Factory stored as `Arc A + Send + Sync>` — cloned into fresh Actor on restart - `tick_all` panic handler: try_restart before poisoning, clear mailbox, fresh state - `restarts` counter added to `WorkerStats` and `WorkerInfo` - **Safety**: Factory fields are "cold" (never touched by handle_any), safe to read after catch_unwind - **Tests**: 4 new tests - `restartable_actor_recovers_after_panic` — basic restart works - `restartable_actor_resets_state_on_restart` — fresh state post-restart - `restartable_actor_respects_max_restarts` — 2 restarts then permanent poison - `non_restartable_actor_still_poisons_on_panic` — backward compatibility - **Result**: 68 tests pass, all workspace compiles ### Cycle 6: Mailbox Backpressure - **Research**: Compared backpressure across Erlang (unbounded, pobox), Actix (cap 16, do_send bypass), Kameo (cap 64, bounded), Tokio mpsc (bounded, permit pattern), Go channels (blocking) - Consensus: bounded by default, configurable overflow policy - **Implementation**: Per-actor bounded mailboxes with configurable overflow - Added `MailboxOverflow` enum: `DropNewest` (discard incoming) and `DropOldest` (evict oldest) - Added `default_mailbox_capacity` and `mailbox_overflow` to `RuntimeConfig` - Default: capacity=0 (unbounded) — 100% backward compatible - `ActorSlot` stores per-actor capacity and policy (from runtime defaults) - `deliver()` enforces bounds; dropped messages tracked via `drops_this_tick` counter - `messages_dropped: AtomicU64` added to `WorkerStats` and `WorkerInfo` - **Tests**: 4 new tests - `bounded_mailbox_drop_newest_caps_at_capacity` — 50 msgs, cap 10 → only 10 delivered - `bounded_mailbox_drop_oldest_keeps_newest` — 10 msgs, cap 5 → newest 5 kept - `unbounded_mailbox_delivers_all_messages` — backward compatibility check - `bounded_mailbox_refills_after_processing` — cap 5, process, refill works - **Result**: 64 tests pass, all workspace compiles ### Research Notes - Full analysis in `CLAUDE/notes/research_synthesis.md` - Baseline benchmarks in `CLAUDE/notes/baseline_benchmarks.md` - Constraints in `CLAUDE/notes/constraints.md` ## Next Steps - [x] **Cycle 2: Stress testing + property-based tests** ✅ - [x] **Cycle 3: Adaptive backoff with thread parking** ✅ - [x] **Cycle 4: Enhanced benchmarks + bug-inspired tests** ✅ - [x] **Cycle 5: Work stealing research + load-aware placement** ✅ - [x] **Cycle 6: Mailbox backpressure** ✅ - [x] **Cycle 7: Actor recovery (factory restart)** ✅ - [x] **Cycle 8: Dead actor cleanup** ✅ - [x] **Cycle 9: Lifecycle hooks + graceful stop** ✅ - [x] **Cycle 10: Actor timers (tick-counting)** ✅ - [ ] **Cycle 11: Next improvement** - Candidates: SmallBox optimization, property-based tests, named actors/registry, actor groups - Priority messages REJECTED (lifecycle hooks cover 95% of cases) - LIFO slot rejected (0-5% benefit for typical workloads, not worth complexity) ## Open Questions - Should budget be configurable per-actor (not just per-runtime)? - Is 64 the right default budget? Benchmarks show budget=32 slightly faster for throughput - ~~Thread parking: notification mechanism~~ RESOLVED: OnceLock + unpark() - Should load-aware placement weight mailbox depth more than actor count? - LIFO slot for same-worker sends: worth the complexity? ## Blockers - (none)