swactor/CLAUDE/notes/progress.md

# Progress Log

## Current Stage: Phase 1 — Research + First Improvement Cycle

### Status: Cycle 6 COMPLETE

## Plan Overview
1. **Phase 0**: Codebase audit — understand current swactor architecture, existing tests, benchmarks ✅
2. **Phase 1**: Broad survey + interleaved improvements
3. **Phase 2**: Deeper improvements based on findings
4. **Phase 3**: Testing methodology improvements
5. **Phase 4**: Final evaluation & documentation

## Completed This Session

### Cycle 1: Fairness (Message Budget)
- **Research**: Studied ractor, tokio, Erlang/OTP BEAM, Linux CFS/EEVDF, libuv
- **Finding**: `tick_all` drained ENTIRE mailbox per actor per tick — critical fairness bug
  - BEAM uses 4000 reduction budget, tokio uses 128-op cooperative budget
  - Swactor had zero budget — one hot actor could starve all others on same worker
- **Implementation**: Added `actor_message_budget` to `RuntimeConfig` (default: 64)
  - Modified `tick_all` to break after `budget` messages per actor
  - `budget=0` means unlimited (backward compatible)
- **Tests**: 3 new fairness tests (hot_actor_does_not_starve_cold_actor, unlimited_budget_drains_all, budget_messages_drain_across_multiple_ticks)
- **Benchmarks**: Added fairness benchmark group (cold_latency_under_pressure, throughput_by_budget)
- **Fixes**: Updated RuntimeConfig struct literals across crates (python, runtime-dashboard, mt_benchmarks)
- **Result**: 45 tests pass (42 original + 3 new), all workspace crates compile

### Cycle 2: Stress Tests, Benchmarks, Research Expansion
- **Research**: Added Kameo and Actix analysis to synthesis
  - Actix uses custom Vyukov lock-free MPSC queue (why it's fastest)
  - Kameo has dual bounded/unbounded mailbox, default capacity 64
  - Both use vtable dispatch (not Box<dyn Any> downcast)
  - Actix has 256-message assertion guard (validates our budget approach)
- **Stress tests**: 6 new tests
  - `message_ordering_preserved_under_budget` — FIFO order with budget=8
  - `mt_stress_many_senders_one_receiver` — 50 senders × 100 msgs, 4 threads
  - `mt_stress_concurrent_spawn_and_send` — 200 concurrent spawn+send, 4 threads
  - `mt_chain_spawning_under_load` — 50-level chain across 2 workers
  - `mt_panic_isolation_under_load` — 10 panicking + 10 healthy actors, 4 threads
  - `sustained_throughput_does_not_drop_messages` — 10 batches × 100 msgs
- **Benchmarks**: 2 new benchmark groups
  - `msg_size`: throughput and send_latency by message size (8B-4KB)
  - `contention`: fanin (1-100 senders to 1 sink), cross_worker (1-4 threads)
- **Result**: 51 tests pass (42 original + 3 fairness + 6 stress), all workspace compiles

### Cycle 3: Thread Parking (Adaptive Backoff)
- **Implementation**: Replaced `thread::sleep` with `thread::park_timeout` in worker run loop
  - Workers register `thread::current()` via `OnceLock<Thread>` on startup
  - `send_to` and `spawn` call `Thread::unpark()` on target worker
  - Cross-worker sends from `WorkerContext` also unpark target
  - Zero new dependencies (uses `std::sync::OnceLock` + `std::thread::park_timeout`)
- **Design source**: Tokio's parker state machine, Linux NO_HZ adaptive ticks
- **Benefits**: Parked workers wake instantly when work arrives (vs waiting for sleep timer)
  - Reduces idle-to-active latency from up to 1ms to near-zero
  - No overhead on hot path — `unpark()` is no-op if thread isn't parked
- **Tests**: 1 new test (`mt_parked_worker_wakes_on_send`)
- **Result**: 52 tests pass (51 + 1 new), all workspace compiles

### Cycle 4: Shutdown Fix + Bug-Inspired Tests
- **Shutdown improvement**: `shutdown()` now unparks all workers for immediate exit
  - Previously, parked workers wouldn't notice shutdown until park_timeout expired
- **Bug-inspired tests** (5 new, from competitor bug reports):
  - `stats_snapshot_is_read_only` — from ractor #310 (destructive get_children)
  - `stats_under_load_do_not_interfere_with_processing` — stats don't affect msg processing
  - `shutdown_wakes_parked_workers_immediately` — validates fast shutdown with parking
  - `mt_send_after_run_delivers_to_running_actors` — from kameo #185 (startup delivery)
  - `budget_respected_even_with_self_sends` — from actix #515 (mailbox bypass)
- **Result**: 57 tests pass, all workspace compiles

### Cycle 5: Work Stealing Research + Load-Aware Placement
- **Research**: Deep analysis of work stealing in Tokio, Go, BEAM, ForkJoinPool
  - Tokio: fixed 256-slot ring, steal-half, LIFO slot (3-use starvation cap), N/2 searcher limit
  - Go: M:N scheduler, runnext + 256-slot local queue, steal-half, 4 tries with random permutation
  - BEAM: unique dual approach — reactive stealing + proactive migration via check_balance()
  - ForkJoinPool: owner LIFO / thief FIFO deque, even/odd queue indexing
- **Feasibility analysis**: Full actor migration IS mechanically possible (ActorSlot is Send), but:
  - Requires push-based donation (ActorPool not Sync → no pull stealing)
  - 1-tick message loss window during migration
  - Significant complexity for uncertain benefit
- **Implementation**: Load-aware placement replaces blind round-robin
  - `Placement::next_worker()` now reads per-worker stats (num_actors + mailbox_depth)
  - Scan starts from rotating position → round-robin when all stats equal (initial burst)
  - O(N) relaxed atomic loads per spawn, trivial for N≤8 workers
- **Tests**: 3 new tests
  - `load_aware_placement_prefers_lighter_worker` — imbalanced load biases toward lighter worker
  - `load_aware_placement_single_worker_degrades_gracefully` — single-thread works correctly
  - `load_aware_placement_falls_back_to_round_robin_on_fresh_runtime` — even distribution before ticks
- **Benchmarks**: 1 new group — `placement/spawn_under_load` (2t, 4t)
- **Result**: 60 tests pass, all workspace compiles

### Cycle 6: Mailbox Backpressure
- **Research**: Compared backpressure across Erlang (unbounded, pobox), Actix (cap 16, do_send bypass),
  Kameo (cap 64, bounded), Tokio mpsc (bounded, permit pattern), Go channels (blocking)
  - Consensus: bounded by default, configurable overflow policy
- **Implementation**: Per-actor bounded mailboxes with configurable overflow
  - Added `MailboxOverflow` enum: `DropNewest` (discard incoming) and `DropOldest` (evict oldest)
  - Added `default_mailbox_capacity` and `mailbox_overflow` to `RuntimeConfig`
  - Default: capacity=0 (unbounded) — 100% backward compatible
  - `ActorSlot` stores per-actor capacity and policy (from runtime defaults)
  - `deliver()` enforces bounds; dropped messages tracked via `drops_this_tick` counter
  - `messages_dropped: AtomicU64` added to `WorkerStats` and `WorkerInfo`
- **Tests**: 4 new tests
  - `bounded_mailbox_drop_newest_caps_at_capacity` — 50 msgs, cap 10 → only 10 delivered
  - `bounded_mailbox_drop_oldest_keeps_newest` — 10 msgs, cap 5 → newest 5 kept
  - `unbounded_mailbox_delivers_all_messages` — backward compatibility check
  - `bounded_mailbox_refills_after_processing` — cap 5, process, refill works
- **Result**: 64 tests pass, all workspace compiles

### Research Notes
- Full analysis in `CLAUDE/notes/research_synthesis.md`
- Baseline benchmarks in `CLAUDE/notes/baseline_benchmarks.md`
- Constraints in `CLAUDE/notes/constraints.md`

## Next Steps
- [x] **Cycle 2: Stress testing + property-based tests** ✅
- [x] **Cycle 3: Adaptive backoff with thread parking** ✅
- [x] **Cycle 4: Enhanced benchmarks + bug-inspired tests** ✅
- [x] **Cycle 5: Work stealing research + load-aware placement** ✅
- [x] **Cycle 6: Mailbox backpressure** ✅
- [ ] **Cycle 7: Next improvement**
  - Candidates: actor recovery (factory restart), LIFO slot, VecDeque→ring buffer optimization
  - Pick based on highest impact-to-effort ratio

## Open Questions
- Should budget be configurable per-actor (not just per-runtime)?
- Is 64 the right default budget? Benchmarks show budget=32 slightly faster for throughput
- ~~Thread parking: notification mechanism~~ RESOLVED: OnceLock<Thread> + unpark()
- Should load-aware placement weight mailbox depth more than actor count?
- LIFO slot for same-worker sends: worth the complexity?

## Blockers
- (none)