docs: add development history pages for all 19 cfuzz improvement cycles
One page per cycle covering motivation, competitor analysis, implementation, design decisions, tests added, and results. Plus an overview/index page. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
c688f0a60e
commit
9ad90aed12
20 changed files with 1428 additions and 0 deletions
95
docs/development_history/CFUZZ_OVERVIEW.md
Normal file
95
docs/development_history/CFUZZ_OVERVIEW.md
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
# cfuzz Branch — Development History Overview
|
||||
|
||||
> 19 improvement cycles on the `cfuzz` branch.
|
||||
> Research-driven methodology: study competitors → identify gap → implement → test → benchmark.
|
||||
> Grew test suite from 42 → 148 passing tests.
|
||||
|
||||
---
|
||||
|
||||
## Methodology
|
||||
|
||||
Each cycle followed a consistent pattern:
|
||||
|
||||
1. **Research** — Study how competitors (Erlang/OTP, Tokio, Akka, Ractor, Actix, Kameo) handle the problem
|
||||
2. **Identify gap** — Find a specific deficiency in swactor
|
||||
3. **Implement** — Fix the gap with minimal, targeted changes
|
||||
4. **Test** — Write behavioral tests (Given/When/Then) from the consumer's perspective
|
||||
5. **Benchmark** — Measure impact where applicable
|
||||
|
||||
### Constraints
|
||||
|
||||
- `src/` structure is frozen — no new files or modules, only modify existing files in-place
|
||||
- No new dependencies on the root crate
|
||||
- Behavioral tests only — no white-box/structural tests
|
||||
- All `cargo test` must pass before each commit
|
||||
- Never delete tests for active code
|
||||
|
||||
---
|
||||
|
||||
## Baseline Benchmarks (Pre-Improvement)
|
||||
|
||||
| Benchmark | Time | Throughput |
|
||||
|-----------|------|-----------|
|
||||
| spawn | 1.28 µs | — |
|
||||
| message_roundtrip | 2.24 µs | — |
|
||||
| send_fire_and_forget | 1.50 µs | — |
|
||||
| single_actor/1000 | 57.5 µs | 17.4 Melem/s |
|
||||
| multi_actor/100x100 | 610.6 µs | 16.4 Melem/s |
|
||||
| ring/100 | 99.9 µs | 1.01 Melem/s |
|
||||
|
||||
---
|
||||
|
||||
## Cycle Summary
|
||||
|
||||
| Cycle | Commit | Topic | Tests Added | Cumulative Tests |
|
||||
|-------|--------|-------|-------------|-----------------|
|
||||
| 1 | `ef87f7e` | [Fairness (message budget)](CYCLE_01_FAIRNESS.md) | 3 | 45 |
|
||||
| 2 | `10cb078` | [Stress tests + benchmarks](CYCLE_02_STRESS_TESTS.md) | 6 | 51 |
|
||||
| 3 | `acacc1b` | [Thread parking](CYCLE_03_THREAD_PARKING.md) | 1 | 52 |
|
||||
| 4 | `cf61619` | [Shutdown fix + bug-inspired tests](CYCLE_04_SHUTDOWN_FIX.md) | 5 | 57 |
|
||||
| 5 | `7d00e65` | [Load-aware placement](CYCLE_05_LOAD_AWARE_PLACEMENT.md) | 3 | 60 |
|
||||
| 6 | `265992c` | [Mailbox backpressure](CYCLE_06_BACKPRESSURE.md) | 4 | 64 |
|
||||
| 7 | `1779ad6` | [Actor recovery](CYCLE_07_ACTOR_RECOVERY.md) | 4 | 68 |
|
||||
| 8 | `0213938` | [Dead actor cleanup](CYCLE_08_DEAD_ACTOR_CLEANUP.md) | 2 (+2 updated) | 70 |
|
||||
| 9 | `e28aca0` | [Lifecycle hooks + graceful stop](CYCLE_09_LIFECYCLE_HOOKS.md) | 12 | 82 |
|
||||
| 10 | `d58a999` | [Actor timers](CYCLE_10_TIMERS.md) | 6 | 88 |
|
||||
| 11 | `9b1518b` | [Property-based testing](CYCLE_11_PROPERTY_TESTING.md) | 7 | 95 |
|
||||
| 12 | `66a8523` | [Named actor registry](CYCLE_12_NAMED_REGISTRY.md) | 11 | 106 |
|
||||
| 13 | `8782638` | [Actor monitoring](CYCLE_13_MONITORING.md) | 7 | 113 |
|
||||
| 14 | `4d18874` | [Actor groups](CYCLE_14_GROUPS.md) | 9 | 122 |
|
||||
| 15 | `902471b` | [Ask pattern](CYCLE_15_ASK_PATTERN.md) | 5 | 127 |
|
||||
| 16 | `0ef6df9` | [Registry benchmarks](CYCLE_16_REGISTRY_BENCHMARKS.md) | 0 | 127 |
|
||||
| 17 | `a70bd86` | [Supervision trees](CYCLE_17_SUPERVISION.md) | 10 | 138 |
|
||||
| 18 | `771c38c` | [OneForAll + RestForOne](CYCLE_18_SUPERVISOR_STRATEGIES.md) | 3 | 141 |
|
||||
| 19 | `c688f0a` | [Router](CYCLE_19_ROUTER.md) | 7 | 148 |
|
||||
|
||||
---
|
||||
|
||||
## Thematic Groupings
|
||||
|
||||
### Scheduling & Performance (Cycles 1–5)
|
||||
Foundation work: fairness guarantees, stress testing, thread parking, shutdown reliability, and load-aware actor placement. Research thread: BEAM reductions → tokio coop budget → Kameo/Actix mailboxes → tokio parker → work stealing survey.
|
||||
|
||||
### Resilience & Lifecycle (Cycles 6–10)
|
||||
Production hardening: backpressure, crash recovery, memory leak fix, lifecycle hooks, and deterministic timers. Narrative arc: from "actors crash permanently" to "actors have a fully managed lifecycle."
|
||||
|
||||
### Testing & Service Discovery (Cycles 11–16)
|
||||
Property-based testing for invariant verification, plus four registry features (names, monitoring, groups, ask pattern) and benchmarks to validate them. Research shifted from scheduling to service discovery patterns.
|
||||
|
||||
### Supervision (Cycles 17–19)
|
||||
Capstone features built on everything preceding: supervision trees with configurable restart strategies, and routers for actor pool management. Directly modeled on Erlang/OTP supervision trees.
|
||||
|
||||
---
|
||||
|
||||
## Frameworks Studied
|
||||
|
||||
| Framework | Language | Key Lessons |
|
||||
|-----------|----------|-------------|
|
||||
| Erlang/OTP BEAM | Erlang | 4000-reduction budget, supervision trees, pg groups, gen_server:call |
|
||||
| Tokio | Rust | 128-op coop budget, work-stealing, parker state machine |
|
||||
| Akka | Scala/Java | SupervisorStrategy, Router actors, PoisonPill |
|
||||
| Ractor | Rust | String-based registry, SupervisionEvent, bug history |
|
||||
| Actix | Rust | Vyukov MPSC queue, 256-message guard, ctx.stop() |
|
||||
| Kameo | Rust | Dual mailbox (bounded/unbounded), on_panic hook, ActorPool |
|
||||
| Linux CFS/EEVDF | C | vruntime fairness, NO_HZ adaptive ticks |
|
||||
| libuv/Node.js | C | Phase-based event loop, round-robin handlers |
|
||||
61
docs/development_history/CYCLE_01_FAIRNESS.md
Normal file
61
docs/development_history/CYCLE_01_FAIRNESS.md
Normal file
|
|
@ -0,0 +1,61 @@
|
|||
# Cycle 1: Per-Actor Message Budget for Tick Fairness — Development History
|
||||
|
||||
> Commit: `ef87f7e` · 8 files · Priority: P0 (critical bug fix)
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
The `tick_all` function in `worker.rs` drained the **entire mailbox** for each actor before moving to the next:
|
||||
|
||||
```rust
|
||||
while let Some(msg) = slot.mailbox.pop_front() {
|
||||
// processes ALL messages for actor A before moving to actor B
|
||||
}
|
||||
```
|
||||
|
||||
If actor A had 10,000 queued messages, all other actors on the same worker were completely starved until A finished. This is a critical fairness bug — every other runtime studied prevents this.
|
||||
|
||||
## Competitor Analysis
|
||||
|
||||
| Runtime | Fairness Mechanism | Budget |
|
||||
|---------|-------------------|--------|
|
||||
| Erlang/OTP BEAM | Reduction counting, preemptive | 4,000 reductions |
|
||||
| Tokio | Cooperative budgeting | 128–256 operations |
|
||||
| libuv/Node.js | Round-robin across handlers | No single handler drains completely |
|
||||
| Linux CFS | vruntime-based fairness | Time slices enforced |
|
||||
| Ractor | N/A (1 task = 1 actor via tokio) | Inherited from tokio |
|
||||
| **Swactor (before)** | **None** | **Unlimited drain** |
|
||||
|
||||
The BEAM's reduction budget (4,000 per process before preemption) is the gold standard for actor fairness. Tokio's cooperative budget (128 ops) serves a similar purpose for async tasks. Actix has a 256-message assertion guard that validates the approach.
|
||||
|
||||
## Implementation
|
||||
|
||||
- Added `actor_message_budget: usize` to `RuntimeConfig` (default: 64)
|
||||
- Modified `tick_all` in `worker.rs` to break after `budget` messages per actor per tick
|
||||
- `budget=0` means unlimited (100% backward compatible)
|
||||
- Updated `RuntimeConfig` struct literals across all crates (python, runtime-dashboard, mt_benchmarks)
|
||||
|
||||
**Key files modified:** `src/worker.rs`, `src/config.rs`, `benches/runtime_benchmarks.rs`, `tests/runtime_api.rs`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **Budget of 64 chosen** as default — between BEAM's 4,000 (too generous for swactor's coarser granularity) and tokio's 128 (per-op vs per-message). Benchmarks showed budget=32 was slightly faster for throughput, but 64 provides more fairness headroom.
|
||||
- **Per-runtime, not per-actor** — simpler configuration, matching the BEAM model where the reduction budget is global. Per-actor budgets could be added later as an extension.
|
||||
- **budget=0 means unlimited** — backward compatibility for users who want the old behavior.
|
||||
|
||||
## Tests Added
|
||||
|
||||
3 new behavioral tests (42 → 45 total):
|
||||
|
||||
- `hot_actor_does_not_starve_cold_actor` — hot actor with many messages doesn't prevent cold actor from processing
|
||||
- `unlimited_budget_drains_all` — budget=0 preserves old behavior
|
||||
- `budget_messages_drain_across_multiple_ticks` — excess messages carry over to next tick
|
||||
|
||||
**Benchmarks added:** `fairness/cold_latency_under_pressure`, `fairness/throughput_by_budget`
|
||||
|
||||
## Result
|
||||
|
||||
- 45 tests pass (42 original + 3 new)
|
||||
- All workspace crates compile
|
||||
- Baseline benchmarks established for future comparison
|
||||
74
docs/development_history/CYCLE_02_STRESS_TESTS.md
Normal file
74
docs/development_history/CYCLE_02_STRESS_TESTS.md
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
# Cycle 2: Stress Tests, Expanded Benchmarks, and Research Extension — Development History
|
||||
|
||||
> Commit: `10cb078` · 4 files · 517 insertions
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
After fixing the fairness bug in Cycle 1, the runtime needed stress testing under adversarial conditions to find edge cases. Additionally, the competitor survey was extended to cover Kameo and Actix — two frameworks with distinct approaches to mailbox management and message dispatch.
|
||||
|
||||
## Competitor Analysis
|
||||
|
||||
### Kameo (v0.19)
|
||||
- Fully async on tokio, one task per actor
|
||||
- Dual mailbox: bounded (default 64) or unbounded tokio mpsc channels
|
||||
- Typed signals via vtable dispatch (no `Box<dyn Any>` downcast)
|
||||
- Erlang-style links for supervision (`on_link_died`)
|
||||
- `on_panic` hook can restart actor (vs swactor's then-permanent poisoning)
|
||||
- Known bugs: deadlocks in link establishment, leaked ActorRef preventing stop
|
||||
|
||||
### Actix (v0.13)
|
||||
- Context-as-Future model — each actor is a single pollable Future on an Arbiter
|
||||
- **Custom Vyukov lock-free MPSC queue** (not tokio channels) — push is single atomic_swap
|
||||
- Default mailbox capacity: 16 (tiny)
|
||||
- `do_send()` bypasses capacity for internal notifications
|
||||
- **256-message assertion guard** — validates swactor's budget approach
|
||||
- vtable dispatch via `Box<dyn EnvelopeProxy<A>>` — no Any downcast
|
||||
- WHY FAST: custom MPSC queue, no async overhead, same-thread actors avoid cross-thread coordination
|
||||
|
||||
### Key Insight
|
||||
Both frameworks use vtable dispatch instead of `Box<dyn Any>` downcast. Actix's 256-message assertion guard independently validates the per-actor budget concept from Cycle 1.
|
||||
|
||||
## Implementation
|
||||
|
||||
### Stress Tests (6 new)
|
||||
- `message_ordering_preserved_under_budget` — FIFO order with budget=8
|
||||
- `mt_stress_many_senders_one_receiver` — 50 senders × 100 msgs on 4 threads
|
||||
- `mt_stress_concurrent_spawn_and_send` — 200 concurrent spawn+send on 4 threads
|
||||
- `mt_chain_spawning_under_load` — 50-level chain across 2 workers
|
||||
- `mt_panic_isolation_under_load` — 10 panicking + 10 healthy actors on 4 threads
|
||||
- `sustained_throughput_does_not_drop_messages` — 10 batches × 100 msgs
|
||||
|
||||
### Benchmarks (2 new groups)
|
||||
- `msg_size` group: throughput and send_latency by message size (8B, 64B, 256B, 1KB, 4KB)
|
||||
- `contention` group: fanin (1–100 senders to 1 sink), cross_worker (1–4 threads)
|
||||
|
||||
**Key files modified:** `tests/runtime_api.rs`, `benches/runtime_benchmarks.rs`, `CLAUDE/notes/research_synthesis.md`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **Multi-threaded stress tests** included because single-threaded testing can't catch cross-worker races
|
||||
- **Panic isolation test** inspired by Actix's SyncArbiter model — ensures one panicking actor doesn't take down healthy actors on other workers
|
||||
- **Message ordering test** validates that the budget mechanism (Cycle 1) doesn't break FIFO guarantees
|
||||
- **Chain spawning** tests the spawn+send-in-same-handler pattern across worker boundaries
|
||||
|
||||
## Tests Added
|
||||
|
||||
6 new stress tests (45 → 51 total):
|
||||
|
||||
| Test | Pattern | Purpose |
|
||||
|------|---------|---------|
|
||||
| `message_ordering_preserved_under_budget` | FIFO verification | Budget doesn't break ordering |
|
||||
| `mt_stress_many_senders_one_receiver` | Fan-in | 50:1 contention on 4 threads |
|
||||
| `mt_stress_concurrent_spawn_and_send` | Concurrent spawn | Race condition hunting |
|
||||
| `mt_chain_spawning_under_load` | Cascading spawn | Cross-worker chain delivery |
|
||||
| `mt_panic_isolation_under_load` | Fault isolation | Panics don't spread |
|
||||
| `sustained_throughput_does_not_drop_messages` | Sustained load | No message loss over time |
|
||||
|
||||
## Result
|
||||
|
||||
- 51 tests pass (42 original + 3 fairness + 6 stress)
|
||||
- All workspace crates compile
|
||||
- No bugs found — the runtime handles adversarial conditions correctly
|
||||
- Benchmark data provides baselines for message size sensitivity and contention scaling
|
||||
51
docs/development_history/CYCLE_03_THREAD_PARKING.md
Normal file
51
docs/development_history/CYCLE_03_THREAD_PARKING.md
Normal file
|
|
@ -0,0 +1,51 @@
|
|||
# Cycle 3: Thread Parking for Instant Worker Wakeup — Development History
|
||||
|
||||
> Commit: `acacc1b` · 4 files · 59 insertions, 10 deletions
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
Before this change, idle workers used `thread::sleep` with a fixed timeout to wait for new work. This meant an idle worker wouldn't notice new messages until its sleep timer expired — up to 1ms of unnecessary latency on the idle-to-active transition. Under bursty workloads, this sleep-based backoff wastes both time and power.
|
||||
|
||||
## Competitor Analysis
|
||||
|
||||
| Runtime | Idle Strategy | Wakeup Mechanism |
|
||||
|---------|--------------|-----------------|
|
||||
| Tokio | Parker state machine (notified/sleeping/empty) | `unpark()` via atomic CAS |
|
||||
| Linux | NO_HZ adaptive ticks (stop tick when idle) | Interrupt on new work |
|
||||
| Go | `notewakeup` / futex | OS-level wake |
|
||||
| BEAM | Scheduler sleep + signal | Thread signal |
|
||||
| **Swactor (before)** | **`thread::sleep(1ms)`** | **Timer expiry only** |
|
||||
|
||||
Tokio's parker uses a 3-state machine (notified → sleeping → empty) with atomic transitions. The key insight: `unpark()` is a **no-op** if the thread isn't parked, so callers pay zero cost on the hot path.
|
||||
|
||||
## Implementation
|
||||
|
||||
- Replaced `thread::sleep` with `thread::park_timeout` in worker run loop
|
||||
- Workers register `thread::current()` via `OnceLock<Thread>` on startup
|
||||
- `send_to` and `spawn` call `Thread::unpark()` on target worker after enqueuing work
|
||||
- Cross-worker sends from `WorkerContext` also unpark the target
|
||||
- Zero new dependencies — uses only `std::sync::OnceLock` + `std::thread::park_timeout`
|
||||
|
||||
**Key files modified:** `src/worker.rs`, `src/runtime.rs`, `src/delivery.rs`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **`OnceLock<Thread>` for thread handle storage** — set-once semantics match the worker lifecycle (one thread per worker, never changes). Simpler than `Mutex<Option<Thread>>`.
|
||||
- **`park_timeout` instead of `park`** — timeout ensures workers periodically wake even without explicit unpark, preventing permanent sleep if an unpark is missed.
|
||||
- **Unpark on `send_to` and `spawn`** — these are the two operations that create work for a worker. The cost is a single atomic store (no-op if thread is already running).
|
||||
- **No condvar** — `thread::park/unpark` is simpler and avoids the spurious wakeup complexity of condition variables. Tokio's parker validates this approach.
|
||||
|
||||
## Tests Added
|
||||
|
||||
1 new test (51 → 52 total):
|
||||
|
||||
- `mt_parked_worker_wakes_on_send` — verifies that a parked worker processes a message immediately after send (not after timeout)
|
||||
|
||||
## Result
|
||||
|
||||
- 52 tests pass
|
||||
- All workspace crates compile
|
||||
- Idle-to-active latency reduced from up to 1ms to near-zero
|
||||
- No overhead on hot path — `unpark()` is a no-op when thread isn't parked
|
||||
54
docs/development_history/CYCLE_04_SHUTDOWN_FIX.md
Normal file
54
docs/development_history/CYCLE_04_SHUTDOWN_FIX.md
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
# Cycle 4: Shutdown Fix + Bug-Inspired Tests — Development History
|
||||
|
||||
> Commit: `cf61619` · 3 files · 161 insertions, 12 deletions
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
Cycle 3 introduced thread parking, but created a new problem: `shutdown()` didn't unpark workers. Parked workers wouldn't notice the shutdown signal until their `park_timeout` expired, causing delayed shutdown. Additionally, studying bug reports from competitor projects (Ractor, Kameo, Actix) revealed specific failure modes worth testing in swactor.
|
||||
|
||||
## Competitor Bug Analysis
|
||||
|
||||
The 5 new tests were directly inspired by real bug reports from other actor frameworks:
|
||||
|
||||
| Test | Inspired By | Bug |
|
||||
|------|-------------|-----|
|
||||
| `stats_snapshot_is_read_only` | Ractor #310 | `get_children()` was destructive — moved children out of supervisor |
|
||||
| `stats_under_load_do_not_interfere_with_processing` | General | Stats collection shouldn't slow down message processing |
|
||||
| `shutdown_wakes_parked_workers_immediately` | Cycle 3 regression | Parked workers must notice shutdown promptly |
|
||||
| `mt_send_after_run_delivers_to_running_actors` | Kameo #185 | Messages sent after `run()` weren't delivered during startup race |
|
||||
| `budget_respected_even_with_self_sends` | Actix #515 | Self-sends bypassed mailbox capacity, defeating backpressure |
|
||||
|
||||
## Implementation
|
||||
|
||||
### Shutdown Fix
|
||||
- `shutdown()` now iterates all workers and calls `unpark()` on each thread handle
|
||||
- Parked workers wake immediately and check the shutdown flag
|
||||
- Workers that aren't parked are unaffected (unpark is a no-op)
|
||||
|
||||
### Bug-Inspired Tests
|
||||
Each test encodes a real bug class discovered in competitor frameworks, ensuring swactor doesn't have the same vulnerability.
|
||||
|
||||
**Key files modified:** `src/runtime.rs`, `tests/runtime_api.rs`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **Unpark-all on shutdown** rather than a dedicated shutdown condvar — simpler, reuses existing parking infrastructure from Cycle 3
|
||||
- **Bug-inspired testing methodology** — studying competitor bug trackers yields high-value test cases that target real failure modes, not theoretical ones
|
||||
|
||||
## Tests Added
|
||||
|
||||
5 new tests (52 → 57 total):
|
||||
|
||||
- `stats_snapshot_is_read_only` — reading stats doesn't mutate runtime state (from Ractor #310)
|
||||
- `stats_under_load_do_not_interfere_with_processing` — stats don't affect message processing throughput
|
||||
- `shutdown_wakes_parked_workers_immediately` — validates fast shutdown with thread parking
|
||||
- `mt_send_after_run_delivers_to_running_actors` — messages sent after run() are delivered (from Kameo #185)
|
||||
- `budget_respected_even_with_self_sends` — self-sends don't bypass budget (from Actix #515)
|
||||
|
||||
## Result
|
||||
|
||||
- 57 tests pass
|
||||
- All workspace crates compile
|
||||
- Shutdown latency with parked workers reduced from up to 1ms to near-zero
|
||||
66
docs/development_history/CYCLE_05_LOAD_AWARE_PLACEMENT.md
Normal file
66
docs/development_history/CYCLE_05_LOAD_AWARE_PLACEMENT.md
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
# Cycle 5: Load-Aware Actor Placement + Work Stealing Research — Development History
|
||||
|
||||
> Commit: `7d00e65` · 6 files · 184 insertions, 13 deletions
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
With fairness (Cycle 1), thread parking (Cycle 3), and shutdown (Cycle 4) resolved, the next bottleneck was actor placement. Swactor used blind round-robin to assign actors to workers — ignoring current load. If actors have unequal workloads, round-robin produces persistent imbalance. This cycle also included deep research into work stealing to decide whether full actor migration was worthwhile.
|
||||
|
||||
## Competitor Analysis: Work Stealing Deep Dive
|
||||
|
||||
| Aspect | Tokio | Go | BEAM | ForkJoinPool |
|
||||
|--------|-------|-----|------|-------------|
|
||||
| Queue | Fixed 256-slot ring | 256-slot ring + runnext | Per-priority linked | Growable array deque |
|
||||
| Steal granularity | Half victim's queue | Half victim's runq | Individual processes | One task at a time |
|
||||
| LIFO fast-path | Dedicated slot (3-use cap) | runnext (stealable 4th try) | None | Owner pops from top |
|
||||
| Global queue | Mutex intrusive list | Checked 1/61 ticks | Per-priority migration | Even-indexed queues |
|
||||
| Searcher limit | N/2 workers | GOMAXPROCS/2 | N/A (proactive migration) | Idle stack in ctl |
|
||||
| Balance strategy | Reactive steal | Reactive steal | **Proactive migration** + reactive | Reactive scan |
|
||||
|
||||
### Key Patterns Discovered
|
||||
|
||||
1. **LIFO slot** — every runtime has one; improves cache locality by running the recipient immediately after the sender. Tokio caps at 3 consecutive uses to prevent starvation.
|
||||
2. **Steal-half** — Tokio and Go both steal half the victim's queue, amortizing cross-thread coordination overhead.
|
||||
3. **N/2 searcher limit** — both Tokio and Go cap concurrent searchers to prevent thundering herd (O(N²) cache-line bouncing).
|
||||
4. **BEAM's migration** — unique dual approach: reactive stealing when idle + proactive migration via periodic `check_balance()`.
|
||||
|
||||
### Feasibility for Swactor
|
||||
|
||||
- **Full actor migration**: Mechanically possible (ActorSlot is `Send`), but has a 1-tick message loss window during migration and requires push-based donation (`ActorPool` is not `Sync` → no pull stealing)
|
||||
- **Message stealing without actors**: Impossible — the actor IS the state; messages without the actor are meaningless
|
||||
- **Decision: Load-aware placement over work stealing** — zero correctness risk, handles the primary imbalance source (uneven spawn distribution), full work stealing deferred
|
||||
|
||||
## Implementation
|
||||
|
||||
- `Placement::next_worker()` now reads per-worker stats (`num_actors` + `mailbox_depth`)
|
||||
- Selects the worker with lowest combined load
|
||||
- Scan starts from a rotating position → round-robin fallback when all stats are equal (initial burst, before first tick publishes stats)
|
||||
- O(N) relaxed atomic loads per spawn — trivial for N ≤ 8 workers
|
||||
|
||||
**Key files modified:** `src/delivery.rs`, `tests/runtime_api.rs`, `benches/runtime_benchmarks.rs`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **Load-aware placement instead of work stealing** — zero message loss risk, no ordering changes, trivial implementation cost. Handles the #1 source of imbalance: uneven spawn distribution.
|
||||
- **Combined metric (actors + depth)** — neither actor count alone nor mailbox depth alone captures load accurately. Combined metric approximates total pending work per worker.
|
||||
- **Relaxed atomics for stat reads** — stats are advisory (best-effort), so relaxed ordering is sufficient. No need for acquire/release which would add synchronization cost.
|
||||
- **Round-robin fallback** — before the first tick, all workers report zero stats. Falling back to round-robin ensures even initial distribution rather than always picking worker 0.
|
||||
- **Full work stealing deferred** — would require migration channels, address map coordination, forwarding tombstones, and a message loss window. Benefit uncertain for N ≤ 8 workers.
|
||||
|
||||
## Tests Added
|
||||
|
||||
3 new tests (57 → 60 total):
|
||||
|
||||
- `load_aware_placement_prefers_lighter_worker` — imbalanced load biases spawn toward the lighter worker
|
||||
- `load_aware_placement_single_worker_degrades_gracefully` — single-thread mode works correctly
|
||||
- `load_aware_placement_falls_back_to_round_robin_on_fresh_runtime` — even distribution before ticks produce stats
|
||||
|
||||
**Benchmark added:** `placement/spawn_under_load` (2-thread and 4-thread variants)
|
||||
|
||||
## Result
|
||||
|
||||
- 60 tests pass
|
||||
- All workspace crates compile
|
||||
- Comprehensive work-stealing research documented for future reference
|
||||
56
docs/development_history/CYCLE_06_BACKPRESSURE.md
Normal file
56
docs/development_history/CYCLE_06_BACKPRESSURE.md
Normal file
|
|
@ -0,0 +1,56 @@
|
|||
# Cycle 6: Per-Actor Mailbox Backpressure — Development History
|
||||
|
||||
> Commit: `265992c` · 6 files · 163 insertions, 8 deletions
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
Before this change, swactor mailboxes were unbounded — a fast producer could flood a slow consumer's mailbox without limit, eventually exhausting memory. Every production actor framework provides some form of backpressure. This was identified as a key weakness in the competitor analysis.
|
||||
|
||||
## Competitor Analysis
|
||||
|
||||
| Framework | Default Capacity | Overflow Policy | Backpressure Model |
|
||||
|-----------|-----------------|----------------|-------------------|
|
||||
| Erlang/OTP | Unbounded | N/A (pobox for opt-in bounding) | Process isolation limits blast radius |
|
||||
| Actix | 16 | `do_send()` bypasses for internal msgs | Tiny default, force callers to handle |
|
||||
| Kameo | 64 | Bounded tokio mpsc (sender blocks) | Blocking backpressure |
|
||||
| Tokio mpsc | User-specified | Bounded (sender blocks or permit pattern) | Blocking or try_send |
|
||||
| Go channels | User-specified | Blocking send / non-blocking select | Blocking backpressure |
|
||||
| **Swactor (before)** | **Unbounded** | **None** | **None** |
|
||||
|
||||
Key observation: Actix's default capacity of 16 is aggressive — it forces callers to think about message flow. Kameo's 64 matches swactor's message budget. The consensus across frameworks: bounded by default, with configurable overflow policy.
|
||||
|
||||
## Implementation
|
||||
|
||||
- Added `MailboxOverflow` enum: `DropNewest` (discard incoming when full) and `DropOldest` (evict oldest to make room)
|
||||
- Added `default_mailbox_capacity` and `mailbox_overflow` to `RuntimeConfig`
|
||||
- Default: `capacity=0` (unbounded) — 100% backward compatible
|
||||
- `ActorSlot` stores per-actor capacity and policy (initialized from runtime defaults at spawn time)
|
||||
- `deliver()` in worker enforces bounds; dropped messages tracked via `drops_this_tick` counter
|
||||
- `messages_dropped: AtomicU64` added to `WorkerStats` and `WorkerInfo`
|
||||
|
||||
**Key files modified:** `src/config.rs`, `src/worker.rs`, `src/runtime.rs`, `src/stats.rs`, `tests/runtime_api.rs`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **DropNewest vs DropOldest (not blocking)** — swactor's synchronous tick model can't block the sender (it would deadlock the entire worker). Drop policies are the only viable option for a sync runtime.
|
||||
- **Default unbounded** — backward compatibility. Users opt into backpressure by setting capacity > 0.
|
||||
- **Per-runtime defaults, not per-actor** — simpler configuration. Per-actor overrides could be added later via a builder pattern on spawn.
|
||||
- **Drop counting** — critical for observability. Without it, users can't tell if their system is losing messages.
|
||||
- **No DropRandom** — the two policies cover the common cases. DropNewest protects against producer floods (newest messages are redundant). DropOldest keeps the freshest state (useful for sensor/status actors).
|
||||
|
||||
## Tests Added
|
||||
|
||||
4 new tests (60 → 64 total):
|
||||
|
||||
- `bounded_mailbox_drop_newest_caps_at_capacity` — 50 msgs sent, capacity 10 → only 10 delivered (oldest 10)
|
||||
- `bounded_mailbox_drop_oldest_keeps_newest` — 10 msgs sent, capacity 5 → newest 5 kept
|
||||
- `unbounded_mailbox_delivers_all_messages` — backward compatibility: capacity=0 delivers everything
|
||||
- `bounded_mailbox_refills_after_processing` — capacity 5, process batch, refill works correctly
|
||||
|
||||
## Result
|
||||
|
||||
- 64 tests pass
|
||||
- All workspace crates compile
|
||||
- Swactor weakness "no backpressure" resolved
|
||||
59
docs/development_history/CYCLE_07_ACTOR_RECOVERY.md
Normal file
59
docs/development_history/CYCLE_07_ACTOR_RECOVERY.md
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
# Cycle 7: Actor Recovery via Factory-Based Restart — Development History
|
||||
|
||||
> Commit: `1779ad6` · 6 files · 167 insertions, 10 deletions
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
Before this change, a panicking actor was permanently poisoned — it could never process messages again. Its address remained in the address map but silently discarded all messages. In production, this means a single panic permanently degrades the system. Every mature actor framework provides some form of crash recovery.
|
||||
|
||||
## Competitor Analysis
|
||||
|
||||
| Framework | Recovery Model | State After Restart | Mailbox After Restart |
|
||||
|-----------|---------------|--------------------|-----------------------|
|
||||
| Erlang/OTP | Factory (MFA tuple), fresh process | Fresh (new init/1) | Lost (new PID) |
|
||||
| Akka | Replace internals, keep ActorRef | Fresh (preRestart hook) | Preserved (docs say "usually wrong") |
|
||||
| Kameo | `on_panic(&mut self)` hook | Potentially corrupt | Preserved |
|
||||
| Actix | `Supervised` trait, re-create context | Fresh | Lost |
|
||||
| Ractor | `SupervisionEvent` callback | Up to supervisor | Up to supervisor |
|
||||
| **Swactor (before)** | **None — permanent poison** | **N/A** | **Silently discarded** |
|
||||
|
||||
### Key Insight
|
||||
Akka's approach of preserving state by replacing internals is documented as "usually wrong" — the state that caused the panic is likely corrupt. Kameo's `on_panic(&mut self)` is risky for the same reason. Erlang's factory-based restart (fresh process from MFA tuple) is the safest approach: guaranteed clean state.
|
||||
|
||||
## Implementation
|
||||
|
||||
- `Actor<A>` expanded from tuple struct to named fields: `inner`, `restart_factory`, `max_restarts`, `restart_count`
|
||||
- `AnyActor::try_restart(&self) -> Option<Box<dyn AnyActor>>` trait method (default `None`, backward compatible)
|
||||
- Factory stored as `Arc<dyn Fn() -> A + Send + Sync>` — called to produce fresh actor instance on restart
|
||||
- `spawn_restartable(actor, factory, max_restarts)` added to both `Runtime` and `Ctx`
|
||||
- `tick_all` panic handler: `try_restart()` before poisoning; on success, replace actor, clear mailbox, reset state
|
||||
- `restarts` counter added to `WorkerStats` and `WorkerInfo`
|
||||
|
||||
**Key files modified:** `src/actor.rs`, `src/runtime.rs`, `src/worker.rs`, `src/stats.rs`, `tests/runtime_api.rs`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **Factory-based restart (Erlang model)** — safest approach, guaranteed clean state. Factory closure is `Arc<dyn Fn() -> A>`, cloned into fresh `Actor<A>` on each restart.
|
||||
- **max_restarts limit** — prevents infinite restart loops. When exceeded, actor is permanently poisoned. Mirrors Erlang's restart intensity.
|
||||
- **Mailbox cleared on restart** — messages that triggered the panic are discarded. Fresh actor starts with empty mailbox. (Erlang does this too — new PID means new mailbox.)
|
||||
- **Same address preserved** — unlike Erlang (new PID), the restarted actor keeps its `ActorAddress`. This is simpler for callers and matches Akka's model.
|
||||
- **Factory fields are "cold"** — `restart_factory` and `max_restarts` are never touched by `handle_any` (the hot path). After `catch_unwind`, these fields are guaranteed safe to read.
|
||||
- **Non-restartable actors unchanged** — `try_restart()` returns `None` by default, preserving the existing poison-on-panic behavior.
|
||||
|
||||
## Tests Added
|
||||
|
||||
4 new tests (64 → 68 total):
|
||||
|
||||
- `restartable_actor_recovers_after_panic` — basic restart works: panic, recover, process new messages
|
||||
- `restartable_actor_resets_state_on_restart` — fresh state confirmed post-restart (counter resets to zero)
|
||||
- `restartable_actor_respects_max_restarts` — 2 restarts allowed, 3rd panic → permanent poison
|
||||
- `non_restartable_actor_still_poisons_on_panic` — backward compatibility: default actors still poison
|
||||
|
||||
## Result
|
||||
|
||||
- 68 tests pass
|
||||
- All workspace crates compile
|
||||
- Swactor weakness "panicked actors permanently poisoned" resolved
|
||||
- Foundation laid for supervision trees (Cycle 17)
|
||||
54
docs/development_history/CYCLE_08_DEAD_ACTOR_CLEANUP.md
Normal file
54
docs/development_history/CYCLE_08_DEAD_ACTOR_CLEANUP.md
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
# Cycle 8: Dead Actor Cleanup (Memory Leak Fix) — Development History
|
||||
|
||||
> Commit: `0213938` · 4 files · 120 insertions, 14 deletions
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
After Cycles 7 (recovery) and the pre-existing poison-on-panic behavior, dead actors accumulated in both `ActorPool` and `AddressMap` forever. Their slots were never reclaimed, their addresses remained registered, and the system gradually leaked memory. This is a known bug class in actor frameworks.
|
||||
|
||||
## Competitor Analysis
|
||||
|
||||
| Framework | Dead Actor Handling | Known Bugs |
|
||||
|-----------|-------------------|------------|
|
||||
| Akka | Automatic cleanup via DeathWatch | #22990 — ActorRef leak in certain paths |
|
||||
| CAF | Manual cleanup expected | #420 — actor leak in specific failure modes |
|
||||
| Erlang/OTP | Automatic — process exits free all resources | N/A (VM handles cleanup) |
|
||||
| Ractor | Supervisor-driven cleanup | Memory bloat per actor at scale |
|
||||
| **Swactor (before)** | **None — permanent leak** | **Both ActorPool and AddressMap leak** |
|
||||
|
||||
## Implementation
|
||||
|
||||
- Added `AddressMap::remove(addr)` to `delivery.rs` — O(1) removal from address map
|
||||
- Added `ActorPool::cleanup_dead()` to `worker.rs` — collects and removes poisoned actors, returns their addresses
|
||||
- Added Phase 7 to `tick_once`: `cleanup_dead` → remove from address_map → re-publish `num_actors` stat
|
||||
- Stats immediately reflect removal (no stale counts)
|
||||
|
||||
**Key files modified:** `src/delivery.rs`, `src/worker.rs`, `tests/runtime_api.rs`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **Automatic cleanup in tick_once** — no manual API needed. Dead actors are cleaned up every tick, preventing accumulation.
|
||||
- **Phase 7 (after all message processing)** — cleanup happens after `tick_all` and `pending_local`, so any final messages to dead actors correctly fail. No risk of cleaning up an actor that's about to receive a message.
|
||||
- **Re-publish `num_actors` after cleanup** — ensures stats are immediately consistent. Without this, stats would show stale actor counts until the next tick.
|
||||
|
||||
### Behavior Change
|
||||
- **Before**: Sending to a poisoned actor silently discarded the message (address still in map, delivery succeeded, but processing was skipped)
|
||||
- **After**: Sending to a cleaned-up actor returns `Err` (address removed from map, send fails)
|
||||
- This is **better** — callers learn the actor is gone instead of silently losing messages.
|
||||
|
||||
## Tests Added
|
||||
|
||||
2 new tests + 2 existing tests updated (68 → 70 total):
|
||||
|
||||
- `dead_actor_cleaned_up_from_stats_and_address_map` — good actor persists, bad actor's address is removed
|
||||
- `bulk_dead_actor_cleanup` — 20 panicked actors all cleaned up in one tick
|
||||
- Updated `send_to_poisoned_actor_is_a_silent_black_hole` → now asserts send returns `Err` (behavior change)
|
||||
- Updated `poisoned_actor_messages_not_counted_as_processed` → sends fail to cleaned-up actor
|
||||
|
||||
## Result
|
||||
|
||||
- 70 tests pass
|
||||
- All workspace crates compile
|
||||
- Memory leak closed: dead actors no longer accumulate in ActorPool or AddressMap
|
||||
79
docs/development_history/CYCLE_09_LIFECYCLE_HOOKS.md
Normal file
79
docs/development_history/CYCLE_09_LIFECYCLE_HOOKS.md
Normal file
|
|
@ -0,0 +1,79 @@
|
|||
# Cycle 9: Lifecycle Hooks and Graceful Actor Stop — Development History
|
||||
|
||||
> Commit: `e28aca0` · 8 files · 427 insertions, 27 deletions
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
Before this change, actors had no initialization or teardown callbacks and no way to stop gracefully. An actor started processing messages immediately (no setup phase) and could only die by panicking. Every mature actor framework provides lifecycle hooks for resource management and graceful shutdown.
|
||||
|
||||
## Competitor Analysis
|
||||
|
||||
| Framework | on_start | on_stop | on_panic | Self-stop | External stop |
|
||||
|-----------|----------|---------|----------|-----------|---------------|
|
||||
| Erlang | `init/1` | `terminate/2` (NOT on crash) | N/A | `{stop,Reason,State}` | `gen_server:stop` |
|
||||
| Akka | `preStart` | `postStop` (always) | `preRestart` | `context.stop(self)` | PoisonPill / stop |
|
||||
| Actix | `started` | `stopped` | N/A | `ctx.stop()` | `addr.do_send(Stop)` |
|
||||
| Kameo | `on_start` | `on_stop` | `on_panic` | `Context::stop()` | `stop_gracefully/kill` |
|
||||
| Ractor | `pre_start` | `post_stop` (NOT on kill/panic) | N/A | `stop()` | `Signal::Kill` |
|
||||
| **Swactor** | **`on_start`** | **`on_stop` (NOT on panic)** | N/A | **`ctx.stop_self()`** | **`runtime.stop_actor()`** |
|
||||
|
||||
### Key Findings
|
||||
- Most frameworks do NOT call `on_stop` on panic — state may be corrupt, running teardown on corrupt state is unsafe. Erlang and Ractor agree. Akka is the outlier (always calls `postStop`).
|
||||
- Self-stop should be immediate (after current message). External stop should be queued (PoisonPill semantics — process pending messages first).
|
||||
- Restarted actors should get `on_start` called again on the fresh instance.
|
||||
|
||||
## Implementation
|
||||
|
||||
### Lifecycle Hooks
|
||||
- `ActorInterface::on_start(&mut self, ctx: &Ctx)` — default no-op, called on first tick before any messages
|
||||
- `ActorInterface::on_stop(&mut self, ctx: &Ctx)` — default no-op, called during cleanup for gracefully-stopped actors
|
||||
- `AnyActor::on_start()`/`on_stop()` — forwarded from `Actor<A>` implementation
|
||||
- `ActorSlot` gains `started: bool` flag — tracks whether `on_start` has been called
|
||||
- `on_start` called in `tick_all` before first message; panic in `on_start` → immediate poison
|
||||
- `on_stop` called in `cleanup_dead` for stopping (not poisoned) actors, wrapped in `catch_unwind`
|
||||
- Restarted actors get `started=false` so `on_start` fires again on fresh instance
|
||||
|
||||
### Graceful Stop (Dual Mode)
|
||||
- `ctx.stop_self()` — **immediate** stop after current message via `request_stop` buffer
|
||||
- `runtime.stop_actor(addr)` — **external** stop via `StopSignal` message (PoisonPill semantics: queued after existing messages)
|
||||
- `ActorSlot` gains `stopping: bool` flag
|
||||
- Phase 7 `cleanup_dead` now handles both poisoned AND stopping actors
|
||||
|
||||
### Stats
|
||||
- `stops: AtomicU64` added to `WorkerStats` and `WorkerInfo` — tracks graceful stops separately from panics
|
||||
|
||||
**Key files modified:** `src/actor.rs`, `src/worker.rs`, `src/runtime.rs`, `src/delivery.rs`, `src/stats.rs`, `tests/runtime_api.rs`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **`on_stop` NOT called on panic** — matches Erlang and Ractor. Corrupt state after panic makes teardown unsafe. If you need cleanup, use `spawn_restartable` (Cycle 7) to get a fresh instance.
|
||||
- **Dual stop modes** — `ctx.stop_self()` is immediate (actor decides "I'm done after this message"). `runtime.stop_actor()` is queued (external signal processed after pending messages). This matches Erlang's `{stop, Reason, State}` vs `gen_server:stop`.
|
||||
- **StopSignal as a message** — external stop uses the same delivery pipeline as regular messages. No special-case routing needed. The PoisonPill pattern (Akka) is well-proven.
|
||||
- **`on_start` panic → immediate poison** — initialization failure is fatal. No restart attempted because the factory might produce the same broken actor. Matches Erlang's `{stop, Reason}` from `init/1`.
|
||||
- **Default no-ops** — both hooks are optional. Existing actors don't need to change. 100% backward compatible.
|
||||
|
||||
## Tests Added
|
||||
|
||||
12 new tests (70 → 82 total):
|
||||
|
||||
- `on_start_called_before_first_message` — on_start fires on first tick, before messages
|
||||
- `on_start_called_per_actor` — 5 actors each get exactly one on_start call
|
||||
- `on_start_panic_poisons_actor` — panic in on_start → poisoned, no messages processed
|
||||
- `actor_can_stop_self` — 5 msgs sent, stops after 3, only 3 processed, on_stop called
|
||||
- `runtime_can_stop_actor` — external stop via runtime, on_stop called, actor removed
|
||||
- `send_to_stopped_actor_returns_error` — stopped actor gone from address map
|
||||
- `stop_vs_panic_tracked_separately_in_stats` — stops and panics counted independently
|
||||
- `on_stop_can_send_messages` — farewell message sent during on_stop is delivered
|
||||
- `on_start_called_again_after_restart` — restartable actor gets on_start on fresh instance
|
||||
- `external_stop_is_queued_after_pending_messages` — PoisonPill semantics verified
|
||||
- `external_stop_before_new_messages_prevents_processing` — stop before send blocks new msgs
|
||||
- `stop_nonexistent_actor_returns_error` — stop on bad address returns Err
|
||||
|
||||
## Result
|
||||
|
||||
- 82 tests pass
|
||||
- All workspace crates compile
|
||||
- Swactor weaknesses "no lifecycle hooks" and "no graceful stop" both resolved
|
||||
- Foundation for supervision (Cycle 17) — `on_stop` enables resource cleanup, `stop_actor` enables supervisor-controlled shutdown
|
||||
76
docs/development_history/CYCLE_10_TIMERS.md
Normal file
76
docs/development_history/CYCLE_10_TIMERS.md
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
# Cycle 10: Per-Worker Tick-Counting Timers — Development History
|
||||
|
||||
> Commit: `d58a999` · 5 files · 247 insertions, 5 deletions
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
Actors often need to schedule delayed or periodic work (timeouts, heartbeats, polling intervals). Before this change, swactor had no timer mechanism — actors had to manually count ticks or rely on external scheduling. The synchronous tick model makes wall-clock timers inappropriate, but tick-counting timers are a natural fit and provide deterministic behavior.
|
||||
|
||||
## Competitor Analysis
|
||||
|
||||
| Framework | Timer Model | Deterministic? |
|
||||
|-----------|------------|---------------|
|
||||
| Erlang | `timer:send_after`, `erlang:start_timer` (wall-clock ms) | No |
|
||||
| Akka | `scheduleOnce`, `scheduler` (wall-clock duration) | No |
|
||||
| Actix | `ctx.run_later`, `ctx.run_interval` (wall-clock) | No |
|
||||
| Kameo | `tokio::time::sleep` (wall-clock) | No |
|
||||
| Tokio | `tokio::time` (wall-clock, pausable for testing) | With `time::pause()` |
|
||||
| Go | `time.After`, `time.NewTicker` (wall-clock) | No |
|
||||
| **Swactor** | **Tick-counting** | **Yes — fully deterministic** |
|
||||
|
||||
### Key Insight
|
||||
Swactor's synchronous tick model makes tick-counting timers uniquely valuable: a timer scheduled for "5 ticks from now" fires at exactly tick N+5, regardless of wall-clock speed. This makes timer behavior reproducible in tests and simulations — something no other framework provides natively.
|
||||
|
||||
Also researched but **rejected**: priority messages (lifecycle hooks from Cycle 9 cover 95% of use cases) and SmallBox optimization (deferred: measure allocation cost first before adding unsafe code).
|
||||
|
||||
## Implementation
|
||||
|
||||
### Timer Types
|
||||
- `OnceTimer` — fire once at `fire_at` tick, consumed after firing
|
||||
- `IntervalTimer` — fire every `period` ticks, message cloned via `CloneMsg` trait
|
||||
|
||||
### Timer Infrastructure
|
||||
- `CloneMsg` trait — type-erased clone for interval timer messages (blanket impl for `Message + Clone`)
|
||||
- `TimerRequest` enum: `Once { dest, msg, ticks }` | `Interval { dest, msg, period }`
|
||||
- Per-worker `TimerWheel` — stores pending timers, checked each tick
|
||||
|
||||
### Integration into tick_once
|
||||
- **Phase 2.5**: Fire due timers, route through full delivery system (pool.deliver for local actors, transfer_txs for cross-worker, inbox_registry for inboxes)
|
||||
- **Phase 5.5**: Drain timer requests from handler buffer into TimerWheel
|
||||
- **After cleanup_dead**: GC interval timers for dead actors
|
||||
|
||||
### API
|
||||
- `ctx.send_after_ticks(addr, msg, ticks)` — one-shot timer
|
||||
- `ctx.send_interval_ticks(addr, msg, period)` — interval timer
|
||||
- `Runtime::schedule_timer()` — no-op with warning (timers are per-worker only, must be scheduled from within a handler)
|
||||
|
||||
**Key files modified:** `src/actor.rs`, `src/worker.rs`, `src/runtime.rs`, `src/delivery.rs`, `tests/runtime_api.rs`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **Tick-counting, not wall-clock** — deterministic behavior is a core swactor advantage. Wall-clock timers would break test reproducibility and simulation fidelity.
|
||||
- **Per-worker timer wheel** — timers are local to the worker that owns the actor. No cross-worker synchronization needed. Timer routing uses the same delivery system as regular messages.
|
||||
- **CloneMsg trait** — interval timers need to clone the message for each firing. A blanket impl covers all `Message + Clone` types, so users don't need to implement anything extra.
|
||||
- **Timer GC for dead actors** — interval timers must be cleaned up when their target actor dies, otherwise they fire forever into the void.
|
||||
|
||||
### Bug Fixed
|
||||
`gc_dead_intervals` was initially over-aggressive — it removed timers for ANY address not in the local pool, including inboxes and cross-worker actors. Fixed to only GC timers for addresses in the `dead` set from `cleanup_dead`.
|
||||
|
||||
## Tests Added
|
||||
|
||||
6 new tests (82 → 88 total):
|
||||
|
||||
- `one_shot_timer_fires_after_n_ticks` — timer with delay=3 fires on tick 4
|
||||
- `handler_can_schedule_one_shot_timer` — timer scheduled from within a handler fires correctly
|
||||
- `one_shot_timer_fires_only_once` — consumed after firing, doesn't repeat
|
||||
- `interval_timer_fires_repeatedly` — period=2, fires every 2 ticks (3 firings verified)
|
||||
- `interval_timer_cleaned_up_when_actor_dies` — GC removes orphaned interval timers
|
||||
- `timer_with_zero_delay_fires_next_tick` — delay=0 fires on next tick (not same tick)
|
||||
|
||||
## Result
|
||||
|
||||
- 88 tests pass
|
||||
- All workspace crates compile
|
||||
- Bug found and fixed: over-aggressive timer GC for cross-worker addresses
|
||||
84
docs/development_history/CYCLE_11_PROPERTY_TESTING.md
Normal file
84
docs/development_history/CYCLE_11_PROPERTY_TESTING.md
Normal file
|
|
@ -0,0 +1,84 @@
|
|||
# Cycle 11: Property-Based Testing and Extended Fuzz Targets — Development History
|
||||
|
||||
> Commit: `9b1518b` · 5 files · 534 insertions, 3 deletions
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
After 10 cycles of behavioral tests, the test suite relied entirely on manually-written scenarios. Property-based testing can explore state spaces that humans wouldn't think to test, automatically finding minimal failing cases. With swactor's deterministic tick model, property-based testing is an especially good fit — no concurrency noise to mask bugs.
|
||||
|
||||
## Competitor Analysis
|
||||
|
||||
| Framework/Tool | Testing Approach | Fit for Swactor |
|
||||
|----------------|-----------------|-----------------|
|
||||
| Tokio + Loom | Model-checking for lock-free code | Poor fit — swactor isn't lock-free |
|
||||
| Erlang + PropEr/QuickCheck | Property-based with shrinking | Good model for swactor |
|
||||
| Shuttle | Concurrency permutation testing | Moderate — useful for MT tests |
|
||||
| proptest-state-machine | Stateful property testing for Rust | **Perfect fit** — deterministic ticks |
|
||||
| cargo-fuzz | Coverage-guided fuzzing | Already in use, extended here |
|
||||
|
||||
### Ranked Approaches
|
||||
1. **proptest-state-machine** — perfect fit for deterministic ticks, generates random operation sequences, automatic shrinking
|
||||
2. Extend cargo-fuzz with new action types
|
||||
3. Simple proptest (stateless properties)
|
||||
4. Shuttle (concurrency permutations)
|
||||
5. Loom (lock-free verification)
|
||||
|
||||
### Key Finding: Feature Gap Analysis
|
||||
While researching testing approaches, also surveyed remaining feature gaps: named actors/registry, actor monitoring/death watch, actor groups/pub-sub, and ask pattern. These became Cycles 12–15.
|
||||
|
||||
## Implementation
|
||||
|
||||
### Property-Based Tests (proptest)
|
||||
Added `proptest` and `proptest-state-machine` to dev-dependencies. New test file: `tests/proptest_runtime.rs` with 7 tests:
|
||||
|
||||
| Test | Property Verified |
|
||||
|------|-------------------|
|
||||
| `fifo_ordering_for_any_message_sequence` | FIFO preserved for 1–100 random messages |
|
||||
| `budget_limits_per_actor_processing` | Budget caps per-tick processing for 2–10 actors |
|
||||
| `one_shot_timer_fires_at_correct_tick` | Timer with delay 1–20 fires at exact right tick |
|
||||
| `interval_timer_fires_at_correct_period` | Period 1–10, verifies 3 consecutive firings |
|
||||
| `bounded_mailbox_never_exceeds_capacity` | Capacity 1–20, 1–200 messages, never exceeds |
|
||||
| `spawn_n_actors_all_tracked` | 1–50 actors, all unique, all in stats |
|
||||
| `swactor_state_machine` | Random Spawn/Send/Tick/Stop/CheckStats sequences |
|
||||
|
||||
### State Machine Test
|
||||
The `swactor_state_machine` test is the most sophisticated:
|
||||
- **Reference model**: `HashMap<id, alive>` tracking expected actor lifecycle
|
||||
- **Operations**: random Spawn, Send, Tick, Stop, CheckStats transitions (up to 40 per test, 128 cases)
|
||||
- **Invariants checked after every transition**: worker count, actor placement, mailbox safety
|
||||
- **Automatic shrinking**: finds minimal failing sequences when invariants break
|
||||
|
||||
### Extended Fuzz Targets
|
||||
Added 4 new `RawAction` variants to `fuzz/fuzz_targets/fuzz_runtime.rs`:
|
||||
- `StopActor` — graceful stop via `runtime.stop_actor`
|
||||
- `SpawnRestartable` — `spawn_restartable` with configurable `max_restarts`
|
||||
- `ScheduleTimer` — one-shot timer via TimerSchedulerActor
|
||||
- `ScheduleInterval` — interval timer via IntervalSchedulerActor
|
||||
|
||||
3 new actor types added to fuzz: `TimerSchedulerActor`, `IntervalSchedulerActor`, `RestartableEchoActor`
|
||||
|
||||
**Key files modified:** `Cargo.toml`, `tests/proptest_runtime.rs` (new), `fuzz/fuzz_targets/fuzz_runtime.rs`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **proptest-state-machine over Loom** — Loom is designed for lock-free concurrent data structures. Swactor's primary correctness properties are sequential (within a tick). The state machine approach tests the actor lifecycle model, which is where bugs are most likely.
|
||||
- **Reference model pattern** — the state machine test maintains a separate `HashMap` as the "expected" state and compares it against the runtime's actual state after each operation. This catches any divergence between the mental model and reality.
|
||||
- **Extending existing fuzz targets** — rather than creating new fuzz targets, extended the existing `fuzz_runtime.rs` with new action variants. This means the fuzzer explores interactions between the new features (timers, restart, stop) and existing operations (spawn, send, tick).
|
||||
|
||||
### Bug Found
|
||||
The state machine test immediately caught an invariant mismatch: `address_map` tracks spawned actors immediately (on spawn), but per-worker `num_actors` lags until the first tick (when the spawn is drained). Fixed the invariant to use `<=` check instead of exact equality.
|
||||
|
||||
## Tests Added
|
||||
|
||||
7 new property tests (88 → 95 total):
|
||||
|
||||
- 6 stateless property tests covering FIFO, budget, timers, mailbox bounds, and spawn tracking
|
||||
- 1 stateful state machine test covering random operation sequences
|
||||
|
||||
## Result
|
||||
|
||||
- 95 tests pass (88 behavioral + 7 proptest)
|
||||
- Fuzz targets compile with new action variants
|
||||
- Bug found: stats lag vs address_map on spawn (invariant relaxed)
|
||||
83
docs/development_history/CYCLE_12_NAMED_REGISTRY.md
Normal file
83
docs/development_history/CYCLE_12_NAMED_REGISTRY.md
Normal file
|
|
@ -0,0 +1,83 @@
|
|||
# Cycle 12: Named Actor Registry with Auto-Cleanup — Development History
|
||||
|
||||
> Commit: `66a8523` · 6 files · 267 insertions, 7 deletions
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
Actors in swactor were only addressable by opaque `ActorAddress` values returned from spawn. There was no way to look up an actor by name — callers needed to pass addresses around manually. Named registration is one of the most fundamental actor runtime features, enabling service discovery within a runtime.
|
||||
|
||||
## Competitor Analysis
|
||||
|
||||
| Framework | Key Type | Storage | Scope | Auto-Cleanup |
|
||||
|-----------|----------|---------|-------|-------------|
|
||||
| Erlang | Atom | ETS table | Per-node or global | Yes (on process exit) |
|
||||
| Actix | TypeId | SystemRegistry | Per-Arbiter | Yes (on actor stop) |
|
||||
| Bastion | Path | Hierarchy | Global | Yes (structural) |
|
||||
| Ractor | String | DashMap (global static) | Global | Yes (on actor death) |
|
||||
| xactor | TypeId | Singleton registry | Global | N/A (singletons) |
|
||||
| Akka | ServiceKey[T] | Receptionist | Cluster-wide | Yes (via DeathWatch) |
|
||||
| **Swactor** | **String** | **RwLock\<HashMap\>** | **Per-runtime** | **Yes (on death)** |
|
||||
|
||||
### Key Findings
|
||||
- **TypeId keys** (Actix, xactor) don't fit swactor's type-erased model — multiple actors of the same type can't share a TypeId key
|
||||
- **Global static** (Ractor) breaks multi-runtime scenarios (tests, embedding)
|
||||
- **Erlang's `register/whereis`** is the gold standard: atom keys, per-node scope, automatic cleanup on process exit
|
||||
|
||||
## Implementation
|
||||
|
||||
### NameRegistry
|
||||
- `NameRegistry` in `delivery.rs` with forward + reverse maps:
|
||||
- `names: RwLock<HashMap<String, ActorAddress>>` — name → address lookup
|
||||
- `addrs: RwLock<HashMap<ActorAddress, String>>` — address → name (for O(1) cleanup)
|
||||
- Added to `Runtime` as `Arc<NameRegistry>`, threaded through `TickContext`
|
||||
|
||||
### Runtime API
|
||||
- `spawn_named(name, actor)` — spawn and register atomically
|
||||
- `where_is(name)` — look up address by name
|
||||
- `unregister(name)` — manual unregistration (actor keeps running)
|
||||
- `registered_names()` — list all registered names
|
||||
|
||||
### Context API
|
||||
- `ctx.spawn_named(name, actor)` — register from within a handler
|
||||
- `ctx.where_is(name)` — look up from within a handler
|
||||
|
||||
### Auto-Cleanup
|
||||
- `cleanup_dead` phase calls `name_registry.unregister_by_addr()` for each dead actor
|
||||
- Name is freed immediately — can be reused for a replacement actor
|
||||
|
||||
### TOCTOU Prevention
|
||||
- Name reservation is immediate (before spawn queue push) — prevents race between checking name availability and registering it
|
||||
|
||||
**Key files modified:** `src/delivery.rs`, `src/runtime.rs`, `src/actor.rs`, `src/worker.rs`, `tests/runtime_api.rs`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **String keys** — most flexible. Atoms (Erlang) aren't idiomatic in Rust. TypeId (Actix) is too restrictive. Strings allow any naming convention.
|
||||
- **Per-runtime scope** — matches swactor's architecture (one runtime per application). Global registries (Ractor) cause problems in tests and embedded scenarios.
|
||||
- **RwLock\<HashMap\>** — matches the existing `AddressMap` and `InboxRegistry` pattern. RwLock allows concurrent reads (lookups) with exclusive writes (registration).
|
||||
- **Collision returns error** — `spawn_named` returns `Err` if the name is already taken. The original binding is preserved. This is explicit and predictable, matching Erlang's behavior.
|
||||
- **Reverse map for O(1) cleanup** — without the reverse map, cleanup would require scanning all entries. The reverse map adds memory proportional to registered actors but makes cleanup constant-time.
|
||||
- **Immediate reservation** — name is reserved before the spawn is queued, preventing TOCTOU races where two `spawn_named` calls for the same name could both succeed.
|
||||
|
||||
## Tests Added
|
||||
|
||||
11 new tests (95 → 106 total):
|
||||
|
||||
- `named_actor_lookup_returns_spawn_address` — spawn_named → where_is roundtrip
|
||||
- `named_actor_receives_messages_via_lookup` — send to looked-up address works
|
||||
- `duplicate_name_returns_error` — collision error, original binding preserved
|
||||
- `where_is_returns_none_for_unknown_name` — nonexistent name → None
|
||||
- `name_auto_unregistered_on_actor_death` — stop_actor → name freed
|
||||
- `name_can_be_reused_after_actor_death` — death → respawn with same name succeeds
|
||||
- `name_auto_unregistered_on_panic` — panic → name freed
|
||||
- `registered_names_lists_all` — all registered names returned
|
||||
- `manual_unregister_frees_name_but_actor_lives` — unregister doesn't kill the actor
|
||||
- `ctx_where_is_resolves_inside_handler` — where_is works from handler context
|
||||
- `ctx_spawn_named_registers_from_handler` — spawn_named works from handler context
|
||||
|
||||
## Result
|
||||
|
||||
- 106 tests pass (99 behavioral + 7 proptest)
|
||||
- All workspace crates compile, zero warnings
|
||||
74
docs/development_history/CYCLE_13_MONITORING.md
Normal file
74
docs/development_history/CYCLE_13_MONITORING.md
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
# Cycle 13: Actor Monitoring with Down Message Notifications — Development History
|
||||
|
||||
> Commit: `8782638` · 6 files · 268 insertions, 10 deletions
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
Actors had no way to know when other actors died. If actor A depended on actor B, and B panicked or was stopped, A would continue sending messages into the void with no notification. Monitoring (also called "death watch") is essential for building fault-tolerant systems — it's the foundation that supervision trees are built on.
|
||||
|
||||
## Competitor Analysis
|
||||
|
||||
| Framework | Mechanism | Direction | Notification |
|
||||
|-----------|-----------|-----------|-------------|
|
||||
| Erlang | `monitor/2` | Unidirectional | `DOWN` message |
|
||||
| Akka | `watch` | Unidirectional | `Terminated` message |
|
||||
| Ractor | `link` | Bidirectional | `SupervisionEvent` |
|
||||
| Actix | None built-in | N/A | N/A |
|
||||
| Kameo | `link` | Bidirectional | `on_link_died` callback |
|
||||
| **Swactor** | **`ctx.monitor()`** | **Unidirectional** | **`Down` message** |
|
||||
|
||||
### Key Findings
|
||||
- **Erlang's unidirectional monitor + message delivery** is the best fit for swactor — it reuses the existing type-erased message handler, requires zero trait changes, and is composable
|
||||
- **Callbacks** (Ractor/Kameo style) rejected — would require adding a new method to `AnyActor`/`ActorInterface` traits, forcing all actors to implement it
|
||||
- **Bidirectional links** deferred — can be layered on top of monitors later
|
||||
- **Stacking** (Erlang) — multiple monitors of the same target produce independent notifications
|
||||
|
||||
## Implementation
|
||||
|
||||
### Types (in `actor.rs`)
|
||||
- `MonitorRef(u64)` — unique token from `AtomicU64` counter, used for demonitor
|
||||
- `Down { addr: ActorAddress, reason: StopReason }` — delivered as normal mailbox message
|
||||
- `StopReason` enum: `Normal` (graceful stop) | `Panicked` (panic, not restartable)
|
||||
|
||||
### MonitorRegistry (in `delivery.rs`)
|
||||
- `watchers: RwLock<HashMap<ActorAddress, Vec<(MonitorRef, ActorAddress)>>>` — watched → list of (ref, watcher)
|
||||
- `refs: RwLock<HashMap<MonitorRef, ActorAddress>>` — ref → watched (for O(1) demonitor)
|
||||
|
||||
### API
|
||||
- `ctx.monitor(target) → MonitorRef` — subscribe to death notifications
|
||||
- `ctx.demonitor(mref)` — cancel a subscription
|
||||
|
||||
### Integration with cleanup_dead
|
||||
- `cleanup_dead` now returns `Vec<(ActorAddress, StopReason)>` instead of `Vec<ActorAddress>`
|
||||
- After cleanup: iterate dead actors, take monitors from registry, route `Down` through normal delivery (pool.deliver for same-worker, transfer_txs for cross-worker, inbox_registry for inboxes)
|
||||
- Dead watcher cleanup: `remove_watcher()` strips monitor subscriptions for dead watchers (prevents ghost subscriptions)
|
||||
|
||||
**Key files modified:** `src/actor.rs`, `src/delivery.rs`, `src/runtime.rs`, `src/worker.rs`, `tests/runtime_api.rs`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **Unidirectional monitors (Erlang model)** — simpler than bidirectional links, no cascading death. The watcher is notified but doesn't automatically die. This gives the watcher full control over how to react.
|
||||
- **Down as a regular message** — delivered through the same mailbox as other messages. Actors with `Incoming = Down` receive it via `handle()`. This reuses the entire existing delivery pipeline with zero special-case code.
|
||||
- **MonitorRef for demonitor** — each monitor subscription gets a unique ref. This supports stacking (multiple monitors of the same target) and precise cancellation.
|
||||
- **StopReason distinguishes Normal vs Panicked** — watchers can decide how to react based on whether the death was graceful or a crash. Matches Erlang's `DOWN` message which includes the exit reason.
|
||||
- **Dead watcher cleanup** — if the watcher dies before the watched actor, its monitor subscriptions are cleaned up. Without this, dead watchers would accumulate as ghost entries in the registry.
|
||||
|
||||
## Tests Added
|
||||
|
||||
7 new tests (106 → 113 total):
|
||||
|
||||
- `monitor_notifies_on_graceful_stop` — Down{reason: Normal} on graceful stop
|
||||
- `monitor_notifies_on_panic` — Down{reason: Panicked} on panic
|
||||
- `multiple_watchers_all_notified` — two watchers both receive Down
|
||||
- `demonitor_cancels_notification` — demonitor → no Down delivered
|
||||
- `dead_watcher_does_not_receive_down` — dead watcher's monitors cleaned up
|
||||
- `down_delivered_to_external_inbox` — Down forwarded through inbox
|
||||
- `stacked_monitors_produce_multiple_notifications` — two monitors on same target → two Downs
|
||||
|
||||
## Result
|
||||
|
||||
- 113 tests pass (106 behavioral + 7 proptest)
|
||||
- All workspace crates compile, zero warnings
|
||||
- Foundation for supervision trees (Cycle 17) — monitors provide the death detection mechanism
|
||||
83
docs/development_history/CYCLE_14_GROUPS.md
Normal file
83
docs/development_history/CYCLE_14_GROUPS.md
Normal file
|
|
@ -0,0 +1,83 @@
|
|||
# Cycle 14: Actor Groups with Pub-Sub Broadcast — Development History
|
||||
|
||||
> Commit: `4d18874` · 5 files · 307 insertions, 8 deletions
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
Named registry (Cycle 12) provides one-to-one name→actor mapping. Many patterns require one-to-many: broadcasting events to subscribers, load distribution across a pool, or topic-based message routing. Actor groups provide this — a named collection of actors that can receive messages as a group.
|
||||
|
||||
## Competitor Analysis
|
||||
|
||||
| Framework | Mechanism | Key Design | Auto-Cleanup |
|
||||
|-----------|-----------|------------|-------------|
|
||||
| Erlang `pg` | Scopes, join/leave/get_members | Flat groups, atom keys | Yes (on process exit) |
|
||||
| Akka | DistributedPubSub (mediator, topics) | Cluster-wide pub-sub | Yes (via DeathWatch) |
|
||||
| Ractor | `pg` module (join/leave/broadcast) | Erlang-style, global | Yes |
|
||||
| Bastion | Dispatcher | Hierarchy-based routing | Structural |
|
||||
| Redis pub/sub | Channels, patterns | External service | N/A |
|
||||
| **Swactor** | **GroupRegistry** | **Erlang pg-style, per-runtime** | **Yes (on death)** |
|
||||
|
||||
### Common Patterns Across Frameworks
|
||||
- Auto-cleanup on death (universal)
|
||||
- At-most-once delivery (no re-delivery guarantees)
|
||||
- String-based naming (flat, not hierarchical)
|
||||
- Lazy group creation/deletion (groups created on first join, deleted when empty)
|
||||
|
||||
## Implementation
|
||||
|
||||
### GroupRegistry (in `delivery.rs`)
|
||||
- Forward map: `groups: RwLock<HashMap<String, HashSet<ActorAddress>>>` — group → members
|
||||
- Reverse map: `memberships: RwLock<HashMap<ActorAddress, HashSet<String>>>` — actor → groups (for cleanup)
|
||||
- Groups auto-create on first join, auto-delete when empty
|
||||
|
||||
### Runtime API
|
||||
- `join_group(addr, name)` — add actor to group
|
||||
- `leave_group(addr, name)` — remove actor from group
|
||||
- `publish_to(group, msg)` — broadcast to all group members
|
||||
- `group_members(group)` — list members
|
||||
- `groups()` — list all groups
|
||||
|
||||
### Context API (from handler)
|
||||
- `ctx.join_group(name)` — join from inside handler
|
||||
- `ctx.leave_group(name)` — leave from inside handler
|
||||
- `ctx.publish(group, msg)` — broadcast from inside handler
|
||||
- `ctx.group_members(group)` — query from inside handler
|
||||
|
||||
### Message Delivery
|
||||
- `publish` clones message at the typed level (`Message: Clone`), sends to each member via normal routing
|
||||
- Uses the same delivery pipeline as regular messages (pool.deliver, transfer_txs, inbox_registry)
|
||||
|
||||
### Auto-Cleanup
|
||||
- `group_registry.cleanup(&addr)` called in `cleanup_dead` phase
|
||||
- Uses reverse map to find all groups the dead actor belonged to, removes from each
|
||||
|
||||
**Key files modified:** `src/delivery.rs`, `src/runtime.rs`, `src/actor.rs`, `src/worker.rs`, `tests/runtime_api.rs`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **Erlang `pg` model** — flat groups with string keys. Simpler than Akka's mediator/topic model, and sufficient for the common use cases (event broadcasting, worker pools).
|
||||
- **Clone-based broadcast** — message is cloned for each recipient. This is O(N) but straightforward and type-safe. Alternative (shared Arc) would complicate the message pipeline.
|
||||
- **Reverse map for cleanup** — without it, cleaning up a dead actor would require scanning all groups. O(1) per group membership vs O(groups) scan.
|
||||
- **Lazy lifecycle** — groups are created implicitly on first join and deleted when the last member leaves. No explicit create/delete API needed. Matches Erlang `pg`.
|
||||
- **publish requires `Message: Clone`** — enforced at the type level. If a message type isn't Clone, it can't be broadcast. This is a compile-time safety guarantee.
|
||||
|
||||
## Tests Added
|
||||
|
||||
9 new tests (113 → 122 total):
|
||||
|
||||
- `group_members_returns_joined_actors` — join + query returns members
|
||||
- `empty_group_returns_no_members` — nonexistent group → empty set
|
||||
- `publish_broadcasts_to_all_members` — 2 members, both receive the message
|
||||
- `leave_group_stops_receiving_publishes` — leave → excluded from future broadcasts
|
||||
- `dead_actor_auto_removed_from_group` — stop → removed from group
|
||||
- `actor_removed_from_all_groups_on_death` — multi-group membership cleanup
|
||||
- `empty_group_auto_deleted` — last member leaves → group removed from `groups()`
|
||||
- `ctx_join_group_from_handler` — join via on_start
|
||||
- `ctx_publish_broadcasts_from_handler` — publish via handler
|
||||
|
||||
## Result
|
||||
|
||||
- 122 tests pass (115 behavioral + 7 proptest)
|
||||
- All workspace crates compile, zero warnings
|
||||
67
docs/development_history/CYCLE_15_ASK_PATTERN.md
Normal file
67
docs/development_history/CYCLE_15_ASK_PATTERN.md
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
# Cycle 15: Ask Pattern for Typed Request-Response — Development History
|
||||
|
||||
> Commit: `902471b` · 3 files · 166 insertions, 1 deletion
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
Request-response is one of the most common actor communication patterns: "send a question, wait for the answer." Before this change, implementing request-response in swactor required manual inbox creation, message construction with a reply-to address, sending, ticking, and polling — a verbose 5-step process. Every mature actor framework provides a convenience wrapper for this pattern.
|
||||
|
||||
## Competitor Analysis
|
||||
|
||||
| Framework | Pattern | Mechanism | Synchronous? |
|
||||
|-----------|---------|-----------|-------------|
|
||||
| Erlang | `gen_server:call` | `From` + `gen_server:reply` | Blocks caller (with timeout) |
|
||||
| Akka | `ask` | Temporary actor + `Future` | Returns Future |
|
||||
| Ractor | `call` | `RpcReplyPort` (oneshot channel) | Returns JoinHandle |
|
||||
| Kameo | `ask` | Async + `Reply` trait | Returns Future |
|
||||
| xactor | `Handler::handle` | Return value auto-routed | Implicit |
|
||||
| **Swactor** | **`rt.ask()`** | **Inbox + closure** | **`recv_ticking` (tick-driven)** |
|
||||
|
||||
### Key Findings
|
||||
- Swactor's synchronous tick model requires explicit `reply_to` — there's no async runtime to suspend the caller
|
||||
- **Implicit auto-reply rejected** — would add magic to the message pipeline and complicate the actor interface
|
||||
- **Decision**: convenience wrapper over existing inbox pattern (not a new mechanism)
|
||||
|
||||
## Implementation
|
||||
|
||||
### Ask\<R\> Struct
|
||||
- Wraps an `Inbox<R>` with convenience methods
|
||||
- `try_recv()` — poll without ticking (works in both single and multi-threaded modes)
|
||||
- `recv_ticking(rt, max_ticks)` — tick the runtime until a response arrives or timeout (single-threaded only)
|
||||
- `reply_addr()` — access the inbox address for manual use
|
||||
|
||||
### Runtime::ask()
|
||||
- `rt.ask(addr, |reply_to| Msg { reply_to })` — one-line request-response
|
||||
- Creates inbox, builds message via closure (user provides the reply_to field), sends, returns `Ask<R>`
|
||||
- Purely sugar over the existing `new_inbox → send_to → tick → try_recv` pattern
|
||||
|
||||
### No Internal Changes
|
||||
- Zero changes to `ContextInner` or `ActorInterface`
|
||||
- No implicit auto-reply magic
|
||||
- Actors reply by explicitly sending to the `reply_to` address (same as before)
|
||||
|
||||
**Key files modified:** `src/runtime.rs`, `tests/runtime_api.rs`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **Closure-based message construction** — `rt.ask(addr, |reply_to| Msg { reply_to })` lets the user embed the reply address in any message shape. No trait requirements on the message type (beyond `Message`).
|
||||
- **`recv_ticking` for single-threaded** — in single-threaded mode, the runtime must be ticked for the target actor to process the request and reply. `recv_ticking` does this automatically. In multi-threaded mode, use `try_recv` with your own tick loop.
|
||||
- **No implicit reply** — frameworks like xactor auto-route the handler's return value as a reply. This is magical and doesn't fit swactor's explicit model. The ask pattern wraps existing mechanics without adding new ones.
|
||||
- **max_ticks timeout** — instead of wall-clock timeout, uses tick count for deterministic behavior (consistent with Cycle 10 timers).
|
||||
|
||||
## Tests Added
|
||||
|
||||
5 new tests (122 → 127 total):
|
||||
|
||||
- `ask_recv_ticking_returns_response` — basic PingPong ask roundtrip
|
||||
- `ask_multiple_times_tracks_state` — 3 sequential asks to CounterActor, state increments
|
||||
- `ask_timeout_when_no_response` — ask dead actor → timeout error
|
||||
- `ask_try_recv_returns_none_before_tick` — poll before tick → None, after tick → Some
|
||||
- `ask_reply_addr_is_accessible` — reply address is valid for manual use
|
||||
|
||||
## Result
|
||||
|
||||
- 127 tests pass (120 behavioral + 7 proptest)
|
||||
- All workspace crates compile, zero warnings
|
||||
59
docs/development_history/CYCLE_16_REGISTRY_BENCHMARKS.md
Normal file
59
docs/development_history/CYCLE_16_REGISTRY_BENCHMARKS.md
Normal file
|
|
@ -0,0 +1,59 @@
|
|||
# Cycle 16: Registry Benchmarks for Named Actors, Groups, Monitors, and Ask — Development History
|
||||
|
||||
> Commit: `0ef6df9` · 2 files · 130 insertions, 1 deletion
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
Cycles 12–15 added four new features (named registry, monitoring, groups, ask pattern) without performance measurement. Before building more features on top of these primitives, it was important to quantify their overhead and ensure they're efficient enough for production use.
|
||||
|
||||
## Benchmark Results
|
||||
|
||||
| Benchmark | Time | Analysis |
|
||||
|-----------|------|----------|
|
||||
| `named_spawn_lookup` | ~2.4 µs | vs bare spawn 1.9 µs → **+0.5 µs** overhead for name registration |
|
||||
| `where_is_100_names` | ~9.0 µs | Includes setup overhead; per-lookup cost is negligible |
|
||||
| `group_publish/10` | ~4.8 µs | O(N) message cloning |
|
||||
| `group_publish/50` | ~15.5 µs | Linear scaling confirmed |
|
||||
| `group_publish/100` | ~60 µs | Linear with O(N) clones |
|
||||
| `monitor_setup` | ~13.4 µs | monitor + stop + cleanup full cycle |
|
||||
| `ask_roundtrip` | ~4.5 µs | vs manual roundtrip 3.0 µs → **+1.5 µs** for inbox creation |
|
||||
|
||||
### Analysis
|
||||
|
||||
- **Named lookup**: +0.5 µs over bare spawn — the `RwLock<HashMap>` insert is fast. Acceptable for a feature used at spawn time, not on the hot path.
|
||||
- **Group publish**: scales linearly with group size, as expected for O(N) message cloning. No optimization needed — the bottleneck is inherent (must clone and deliver N messages).
|
||||
- **Monitor setup**: 13.4 µs covers the full lifecycle (monitor → stop → cleanup → Down delivery). The monitoring machinery adds minimal per-message overhead.
|
||||
- **Ask roundtrip**: +1.5 µs over manual inbox pattern (4.5 µs vs 3.0 µs). The overhead is inbox creation. Acceptable for a convenience pattern — users who need maximum throughput can use the manual pattern.
|
||||
|
||||
## Implementation
|
||||
|
||||
5 new criterion benchmark functions added to `benches/runtime_benchmarks.rs` in a `registry` group:
|
||||
|
||||
- `named_spawn_lookup` — spawn_named + where_is roundtrip
|
||||
- `where_is_100_names` — lookup in 100-name registry
|
||||
- `group_publish/{10,50,100}` — broadcast to N group members
|
||||
- `monitor_setup` — monitor + stop + Down delivery cycle
|
||||
- `ask_roundtrip` — ask + recv_ticking response
|
||||
|
||||
**Key files modified:** `benches/runtime_benchmarks.rs`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **Full-cycle benchmarks** — each benchmark measures the complete operation (not just the fast path). For example, `monitor_setup` includes stop and cleanup, not just the monitor call, because that's the real-world cost.
|
||||
- **Parameterized group publish** — three group sizes (10, 50, 100) to verify linear scaling and catch any unexpected superlinear behavior.
|
||||
- **No optimization undertaken** — all operations are efficient enough. The benchmark results serve as baselines for future changes.
|
||||
|
||||
## Tests Added
|
||||
|
||||
No new tests (benchmarks only). Test count remains at 127.
|
||||
|
||||
## Result
|
||||
|
||||
- All benchmarks run cleanly
|
||||
- 127 tests pass, zero warnings
|
||||
- All registry operations confirmed efficient for production use
|
||||
- Named lookup: <1 µs overhead over bare spawn
|
||||
- Ask: ~50% overhead over manual inbox pattern (acceptable for convenience)
|
||||
- Group publish: linear O(N) as expected
|
||||
89
docs/development_history/CYCLE_17_SUPERVISION.md
Normal file
89
docs/development_history/CYCLE_17_SUPERVISION.md
Normal file
|
|
@ -0,0 +1,89 @@
|
|||
# Cycle 17: Supervision Trees with handle_down and Supervisor Actor — Development History
|
||||
|
||||
> Commit: `a70bd86` · 4 files · 754 insertions, 7 deletions
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
With monitoring (Cycle 13), lifecycle hooks (Cycle 9), and factory-based restart (Cycle 7) in place, swactor had all the building blocks for supervision trees — the signature feature of Erlang/OTP. Supervision trees provide structured fault tolerance: a parent actor (supervisor) monitors children and restarts them according to configurable policies when they fail.
|
||||
|
||||
## Competitor Analysis
|
||||
|
||||
| Framework | Supervisor Model | Strategies | Child Spec | Meltdown Protection |
|
||||
|-----------|-----------------|------------|------------|---------------------|
|
||||
| Erlang/OTP | Built-in `supervisor` behaviour | one_for_one, one_for_all, rest_for_one, simple_one_for_one | `{Id, MFA, Restart, Shutdown, Type}` | Intensity/period limits |
|
||||
| Akka | SupervisorStrategy | Resume, Restart, Stop, Escalate + BackoffSupervisor | N/A (inline) | MaxNrOfRetries/withinTimeRange |
|
||||
| Ractor | `ractor-supervisor` crate | External crate, event-based | SupervisionEvent callback | N/A |
|
||||
| Bastion | Built-in hierarchy | Redundancy groups | Structural (parent-child) | N/A |
|
||||
| CAF | No built-in supervisor | Monitor-based (manual) | N/A | N/A |
|
||||
| **Swactor** | **User-space `Supervisor` actor** | **OneForOne** (Cycle 17), **OneForAll/RestForOne** (Cycle 18) | **`ChildSpec`** | **max_restarts budget** |
|
||||
|
||||
### Key Findings
|
||||
- Swactor has all the building blocks: monitor (Cycle 13), `spawn_restartable` (Cycle 7), lifecycle hooks (Cycle 9), `Down` messages (Cycle 13)
|
||||
- **Decision**: Supervisor as a user-space actor built on existing primitives (like Ractor's `ractor-supervisor` crate), not a special runtime construct
|
||||
- **`handle_down` callback** enables any actor to react to monitored deaths without requiring `Incoming = Down` — this is the key API gap that needed filling
|
||||
|
||||
## Implementation
|
||||
|
||||
### 1. `handle_down` Callback on ActorInterface
|
||||
|
||||
The core API addition enabling supervision:
|
||||
|
||||
- `fn handle_down(&mut self, ctx: &Ctx, down: Down)` — default no-op, called when a monitored actor dies and the actor's `Incoming` type is NOT `Down`
|
||||
- Implemented via second downcast attempt in `handle_any`: if the message is `Down` and the actor's `Incoming` type doesn't match, call `handle_down` instead of `handle`
|
||||
- Fully backward-compatible: actors with `Incoming = Down` still receive via `handle()` as before
|
||||
- This decouples supervision logic from the actor's primary message type
|
||||
|
||||
### 2. `ctx.stop_actor(addr)` — Stop Another Actor
|
||||
|
||||
- Sends graceful stop to another actor from handler context
|
||||
- Uses `StopSignal` through normal message routing (PoisonPill semantics)
|
||||
- Enables supervisor-controlled shutdown of children
|
||||
|
||||
### 3. `Supervisor` Actor
|
||||
|
||||
A user-space actor managing child actors:
|
||||
|
||||
- **`SupervisorStrategy::OneForOne`** — only the failed child is restarted (Cycle 17)
|
||||
- **`RestartPolicy`**: `Permanent` (always restart), `Transient` (restart only on panic, not normal stop), `Temporary` (never restart)
|
||||
- **`ChildSpec`** — `{ id: String, restart: RestartPolicy, factory: Fn(&Ctx) -> Result<ActorAddress> }`
|
||||
- Children spawned in `on_start`, monitored via `ctx.monitor()`
|
||||
- Death detected via `handle_down`, restart policy consulted, factory invoked for replacement
|
||||
- **Meltdown detection**: stops itself when `total_restarts > max_restarts`
|
||||
- **Cascading shutdown**: `on_stop` sends stop signals to all living children
|
||||
|
||||
### ActiveChild Struct
|
||||
- Tracks `addr: ActorAddress` and `monitor_ref: MonitorRef` per child
|
||||
- Reused by Router (Cycle 19)
|
||||
|
||||
**Key files modified:** `src/actor.rs`, `tests/runtime_api.rs`, `docs/runtime.md`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **User-space actor (not runtime primitive)** — the Supervisor is just an actor that uses existing APIs (monitor, spawn, stop). No special runtime support needed. This validates the composability of the monitoring and lifecycle systems.
|
||||
- **`handle_down` as opt-in callback** — adding `handle_down` to `ActorInterface` with a default no-op means existing actors don't need to change. Actors that want to react to deaths override it. The alternative (requiring `Incoming = Down`) would force actors to handle `Down` as their primary message type.
|
||||
- **Factory takes `&Ctx`** — the factory closure receives the context so it can use `ctx.spawn`, `ctx.monitor`, etc. during child creation. This enables the supervisor to monitor new children immediately.
|
||||
- **Meltdown protection** — if children keep crashing faster than they can be restarted, the supervisor stops itself rather than looping forever. Matches Erlang's intensity/period limits.
|
||||
- **Cascading shutdown** — when the supervisor stops, all living children receive stop signals. This prevents orphaned actors.
|
||||
|
||||
## Tests Added
|
||||
|
||||
10 new tests (127 → 138 total, counting 130 behavioral + 7 proptest + 1 doctest):
|
||||
|
||||
- `handle_down_receives_death_notification` — handle_down callback fires on monitored death
|
||||
- `handle_down_skipped_when_incoming_is_down` — backward compat: Incoming=Down uses handle()
|
||||
- `ctx_stop_actor_stops_target` — one actor stops another via ctx.stop_actor()
|
||||
- `supervisor_restarts_permanent_child_on_panic` — panic → restart (OneForOne + Permanent)
|
||||
- `supervisor_does_not_restart_transient_child_on_normal_stop` — Normal stop → no restart
|
||||
- `supervisor_restarts_transient_child_on_panic` — Panicked → restart (Transient)
|
||||
- `supervisor_never_restarts_temporary_child` — Temporary → never restart
|
||||
- `supervisor_meltdown_after_max_restarts` — exceeding max_restarts stops supervisor
|
||||
- `supervisor_one_for_one_only_restarts_failed_child` — multi-child, only crashed child restarted
|
||||
- `supervisor_on_stop_kills_children` — supervisor shutdown cascades to children
|
||||
|
||||
## Result
|
||||
|
||||
- 138 tests pass (130 behavioral + 7 proptest + 1 doctest)
|
||||
- Zero warnings, full workspace compiles
|
||||
- Supervisor validates the composability of Cycles 7 (recovery), 9 (lifecycle), and 13 (monitoring)
|
||||
86
docs/development_history/CYCLE_18_SUPERVISOR_STRATEGIES.md
Normal file
86
docs/development_history/CYCLE_18_SUPERVISOR_STRATEGIES.md
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
# Cycle 18: OneForAll and RestForOne Supervisor Strategies — Development History
|
||||
|
||||
> Commit: `771c38c` · 4 files · 277 insertions, 2 deletions
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
Cycle 17 introduced supervision with the `OneForOne` strategy (only the failed child is restarted). Erlang/OTP defines two additional coordinated restart strategies that handle interdependent children:
|
||||
|
||||
- **`one_for_all`** — when one child fails, ALL children are restarted (for tightly coupled children that share state assumptions)
|
||||
- **`rest_for_one`** — when one child fails, it and all children started AFTER it are restarted (for chains where later children depend on earlier ones)
|
||||
|
||||
These strategies require coordinated shutdown: the supervisor must stop living siblings, wait for all of them to die, then restart the affected set in the original spec order.
|
||||
|
||||
### Research Detour: SmallBox/InlineAny Optimization
|
||||
Before choosing this cycle's topic, investigated SmallBox optimization for message dispatch — a 44% queue throughput improvement was measured. However, it was deferred because:
|
||||
- Requires `unsafe` code in a core path
|
||||
- Would touch 32+ call sites across the codebase
|
||||
- Violates the "src/ structure frozen" constraint
|
||||
|
||||
Extended the Supervisor with coordinated strategies instead — higher value, zero risk.
|
||||
|
||||
## Competitor Analysis
|
||||
|
||||
| Framework | OneForAll | RestForOne | Coordinated Shutdown |
|
||||
|-----------|-----------|------------|---------------------|
|
||||
| Erlang/OTP | Yes | Yes | Built into supervisor behaviour |
|
||||
| Akka | No (different model: Resume/Restart/Stop/Escalate) | No | N/A |
|
||||
| Ractor | No | No | N/A |
|
||||
| Bastion | Implicit (redundancy groups) | No | Implicit |
|
||||
| **Swactor** | **Yes** | **Yes** | **Phase-based state machine** |
|
||||
|
||||
### Erlang's Coordinated Restart
|
||||
In Erlang, `one_for_all` and `rest_for_one` stop affected children in reverse start order, wait for all to terminate, then restart in start order. This guarantees initialization dependencies are respected.
|
||||
|
||||
## Implementation
|
||||
|
||||
### SupervisorPhase State Machine
|
||||
- `Normal` — steady state, processing handle_down events normally
|
||||
- `Stopping { awaiting: HashSet<ActorAddress>, restart_set: Vec<usize> }` — coordinated shutdown in progress
|
||||
|
||||
### SupervisorStrategy Extensions
|
||||
- `SupervisorStrategy::OneForAll` — all children restarted when one fails
|
||||
- `SupervisorStrategy::RestForOne` — failed child + all children after it (in spec order) restarted
|
||||
|
||||
### Coordinated Restart Flow
|
||||
1. Child dies → `handle_down` called
|
||||
2. Strategy determines affected indices (OneForAll: all, RestForOne: failed + later)
|
||||
3. `begin_coordinated_restart(ctx, indices)`:
|
||||
- Sends stop signals to living siblings in the restart set
|
||||
- Transitions to `Stopping` phase with `awaiting` set
|
||||
- Already-dead children handled: if all targets are already dead, skip to immediate restart
|
||||
4. Subsequent `handle_down` calls during `Stopping` phase:
|
||||
- Remove from `awaiting` set
|
||||
- When `awaiting` is empty → all stopped
|
||||
5. `finish_restart(ctx)`:
|
||||
- Restart all children in the restart set, in spec order
|
||||
- Transition back to `Normal` phase
|
||||
|
||||
### Refactoring
|
||||
- `check_intensity()` factored out of `handle_down` for restart budget checking — shared by all strategies
|
||||
|
||||
**Key files modified:** `src/actor.rs`, `tests/runtime_api.rs`, `docs/runtime.md`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **Phase-based state machine** — the `Stopping` phase cleanly separates "waiting for siblings to die" from "normal operation." This prevents races where a new death arrives while a coordinated restart is in progress.
|
||||
- **Stop signals (not kill)** — affected siblings are stopped gracefully (PoisonPill semantics), giving them a chance to run `on_stop` for cleanup. This matches Erlang's `terminate/2` being called during supervised shutdown.
|
||||
- **Restart in spec order** — children are restarted in the order they appear in the ChildSpec list, regardless of which child triggered the restart. This preserves initialization dependencies.
|
||||
- **Already-dead optimization** — if all children in the restart set are already dead (e.g., cascading failures), skip the `Stopping` phase entirely and restart immediately. Without this, the supervisor would wait forever for Down messages that already arrived.
|
||||
- **Meltdown protection shared** — the same `max_restarts` budget applies across all strategies. OneForAll restarts count as one restart event (not N), matching Erlang's behavior.
|
||||
|
||||
## Tests Added
|
||||
|
||||
3 new tests (138 → 141 total):
|
||||
|
||||
- `supervisor_one_for_all_restarts_all_on_single_failure` — one child panics, all 3 get new addresses
|
||||
- `supervisor_rest_for_one_restarts_rest_after_failed` — child_b panics, child_a unchanged, child_b + child_c restarted
|
||||
- `supervisor_one_for_all_waits_for_all_downs_before_restart` — verifies coordinated shutdown completes before restart begins
|
||||
|
||||
## Result
|
||||
|
||||
- 141 tests pass (133 behavioral + 7 proptest + 1 doctest)
|
||||
- Zero warnings, full workspace compiles
|
||||
- All three Erlang-standard supervision strategies now available: OneForOne, OneForAll, RestForOne
|
||||
78
docs/development_history/CYCLE_19_ROUTER.md
Normal file
78
docs/development_history/CYCLE_19_ROUTER.md
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
# Cycle 19: Router Actor for Pooled Message Distribution — Development History
|
||||
|
||||
> Commit: `c688f0a` · 4 files · 528 insertions, 3 deletions
|
||||
|
||||
---
|
||||
|
||||
## Motivation
|
||||
|
||||
Many workloads benefit from distributing messages across a pool of identical worker actors. Before this change, users had to manually manage actor pools: spawn N workers, track their addresses, implement distribution logic, and handle worker replacement on failure. A Router actor encapsulates this pattern — it receives messages and transparently forwards them to pool members using a configurable strategy.
|
||||
|
||||
## Competitor Analysis
|
||||
|
||||
| Framework | Pool/Router Model | Strategies | Auto-Replace |
|
||||
|-----------|------------------|------------|-------------|
|
||||
| Erlang | `poolboy` (checkout/checkin), `wpool` (transparent forwarding, 6 strategies + custom) | RoundRobin, Random, BestWorker, Hash, Available, custom | Manual |
|
||||
| Akka | Router actors (Pool vs Group), Resizer for dynamic sizing | RoundRobin, Random, SmallestMailbox, Balancing, Broadcast, ScatterGather, TailChopping, ConsistentHashing | Pool auto-creates, Group manual |
|
||||
| Actix | SyncArbiter (shared queue, implicit work-stealing) | N/A (shared queue) | N/A |
|
||||
| Kameo | ActorPool (least-connections, auto-replace dead workers) | Least-connections | Yes |
|
||||
| Ractor | No built-in router (process groups only) | N/A | N/A |
|
||||
| **Swactor** | **`Router<M>` actor** | **RoundRobin, Random, Broadcast** | **Yes (via monitor + handle_down)** |
|
||||
|
||||
### Key Findings
|
||||
- **Router-as-actor** with transparent forwarding (wpool/Akka style) is the best fit — the router looks like a regular actor to callers
|
||||
- **User-space actor** like Supervisor (Cycle 17), reusing monitor + handle_down for worker replacement
|
||||
- **SmallestMailbox deferred** — requires runtime stats access not available in user-space
|
||||
- **ConsistentHashing deferred** — requires a hash function parameter, can be added later as a builder method
|
||||
|
||||
## Implementation
|
||||
|
||||
### Router\<M\> Actor
|
||||
- Generic over `M: Message` — same `Incoming` type as workers, enabling transparent forwarding
|
||||
- Workers spawned in `on_start`, monitored via `ctx.monitor()`, auto-replaced via `handle_down`
|
||||
- Reuses `ActiveChild` struct from Supervisor (addr + monitor_ref)
|
||||
|
||||
### Routing Strategies
|
||||
- `RoutingStrategy::RoundRobin` — sequential circular distribution via counter
|
||||
- `RoutingStrategy::Random` — random worker selection via `get_random()` helper
|
||||
- `RoutingStrategy::Broadcast` — clone message to all live workers (`M: Clone` required)
|
||||
|
||||
### Fault Tolerance
|
||||
- Dead worker detected via `handle_down` → factory invoked → new worker spawned and monitored
|
||||
- **Meltdown protection**: `total_restarts > max_restarts` → `ctx.stop_self()`
|
||||
- **Cascading shutdown**: `on_stop` sends stop signals to all workers
|
||||
|
||||
### Configuration
|
||||
- `Router::new(pool_size, strategy, factory, max_restarts)` — all-in-one constructor
|
||||
- Factory: `Arc<dyn Fn(&Ctx) -> Result<ActorAddress, Error> + Send + Sync>`
|
||||
|
||||
**Key files modified:** `src/actor.rs`, `tests/runtime_api.rs`, `docs/runtime.md`
|
||||
|
||||
## Design Decisions
|
||||
|
||||
- **Router-as-actor (transparent forwarding)** — callers send messages to the router's address as if it were a regular actor. The router forwards to pool members. This is the cleanest API: no special send function, no pool handle, just an address.
|
||||
- **User-space actor (not runtime primitive)** — like Supervisor, Router is built entirely on existing APIs (spawn, monitor, handle_down, stop). This validates the actor system's composability.
|
||||
- **Generic over M** — `Router<M>` has `Incoming = M`, same as the workers. Messages are forwarded with zero transformation. Type safety is enforced at compile time.
|
||||
- **Broadcast requires Clone** — broadcasting clones the message for each worker. The Clone bound is only required when using the Broadcast strategy, enforced at the type level.
|
||||
- **SmallestMailbox deferred** — would require reading per-actor mailbox depth from runtime stats, which isn't available from within a handler. Could be added with a stats query API.
|
||||
- **ConsistentHashing deferred** — requires a hash function parameter (user must define which part of the message determines the routing key). Better to add as a builder method with a closure parameter.
|
||||
- **Reuses ActiveChild from Supervisor** — the pattern of "track address + monitor ref, replace on death" is identical. Code sharing confirms the design consistency between Supervisor and Router.
|
||||
|
||||
## Tests Added
|
||||
|
||||
7 new tests (141 → 148 total):
|
||||
|
||||
- `router_round_robin_distributes_across_workers` — 6 msgs to 3 workers, each gets 2
|
||||
- `router_broadcast_sends_to_all_workers` — 1 msg, all 3 workers receive
|
||||
- `router_random_delivers_to_some_worker` — 30 msgs across 3 workers, at least 2 workers used
|
||||
- `router_replaces_dead_worker` — panicked worker auto-replaced, pool size maintained
|
||||
- `router_meltdown_after_max_restarts` — 3 deaths with max_restarts=2 → router stops
|
||||
- `router_on_stop_kills_workers` — stopping router cascades to all workers
|
||||
- `router_broadcast_multiple_messages_all_received` — 5 msgs × 3 workers = 15 received
|
||||
|
||||
## Result
|
||||
|
||||
- 148 tests pass (140 behavioral + 7 proptest + 1 doctest)
|
||||
- Zero warnings, full workspace compiles
|
||||
- Router validates the composability of the entire cfuzz feature set: monitoring (Cycle 13), lifecycle hooks (Cycle 9), handle_down (Cycle 17), and the ActiveChild pattern (Cycle 17)
|
||||
- The cfuzz branch concludes with a comprehensive actor runtime featuring: fairness, backpressure, recovery, lifecycle management, timers, named registry, monitoring, groups, ask pattern, supervision trees, and routers
|
||||
Loading…
Reference in a new issue