swactor/docs/specs/drafts/MULTICORE_SPEC.md

# swactor process-local multicore runtime — specification

Id: 2
Last modified:
Last reviewed:

**Scope:** multicore (multi-worker) execution and message delivery within a single process (shared address space).

## 1. Scope

**In scope**

- How N workers run concurrently on N cores within one process.
- How a message is routed and delivered between actors on different workers (foreign-thread, shared memory).
- How a message is delivered between actors on the same worker.
- The seam between core and the hosting engine.

**Out of scope (deferred to separate specs)**

- Cross-isolation delivery (JS web workers) — no shared heap; requires serialization.
- Cross-process / WAN delivery — owned by the `transport` and `distribution` crates.
- Actor migration, work-stealing, and load balancing beyond spawn-time worker selection.
- A ready-queue / runnable-set optimization.

**Constraint.** Core (`src/`) stays free of any specific engine: no tokio dependency, no owned thread pool. Any engine that can host a blocking or async receiver can host a worker.

## 2. Model

- A **worker** owns a disjoint set of actors and processes them one at a time. It is the unit of parallelism: N workers on N cores run up to N actors concurrently.
- An actor is **pinned**: assigned to one worker at spawn, never moved. Worker selection at spawn is deliberately simple: an actor spawned from within a worker (`ctx.spawn`) pins to that same worker; an actor spawned from outside the runtime (via the runtime handle) is assigned round-robin across workers. A side effect is that a parent and the children it talks to stay co-located, so their traffic stays on the same-worker fast path.
- Multicore parallelizes *different actors*. One actor's work is never split across cores. This preserves the single-writer invariant: at most one message is handled per actor at any instant, across all workers.
- Workers are **autonomous and independent**: each runs its own loop. There is no global tick, no barrier, no per-step cross-worker synchronization.
- Workers are **reactive**: when idle they wait; when work arrives they run a pass.

## 3. Responsibilities (the core / engine seam)

- **Runtime** (core): owns the actor address space, the `address → worker` routing map, the per-worker inbox deposit handles, and spawn-time worker selection (same-worker for in-runtime spawns, round-robin for external spawns). It routes. It does not execute and does not own threads.
- **Worker** (core logic, engine-driven): owns its pinned actor pool. Each pass drains its inbox into actor mailboxes and processes non-empty mailboxes up to a fairness budget.
- **Engine** (integrator-supplied — std::thread, tokio, …): decides how many workers to create, hosts each worker's loop, and owns the inbox's consumer side (how the worker idles and how often it drains). **Core only transitions. The engine drives.**

## 4. Delivery regimes (this spec)

| target lives on…                     | delivery                                          |
| ------------------------------------ | ------------------------------------------------- |
| the same worker                      | inline, within the current pass (no queue)        |
| another worker, same process         | pointer-move into that worker's MPSC inbox        |
| another process / isolated / remote  | out of scope — `transport` / `distribution`       |

## 5. Routing and delivery mechanism

A send resolves the target actor to its owning worker and deposits the message. There is **no wake step**. The receiving worker idles on its own inbox, so depositing into it is what makes the next transition runnable.

### 5.1 Send path

For a send of `M` to `addr` from any in-process sender (an actor handler, or an external thread holding a sender handle):

1. Box `M` once → `Box<dyn Any + Send>` (a heap pointer). The payload is never copied again.
2. Look up `addr` in the routing map → `WorkerId`.
3. Branch:
   - **Same worker** as sender → append `(addr, M)` to the worker's local `pending_local` buffer. Delivered within the current pass. No queue, no cross-thread.
   - **Different worker** → wrap as `Envelope { dest: addr, payload: M }` and deposit into that worker's inbox (a pointer-move into shared memory). Return. No signal is sent to the receiver.
   - **Not in the map** → defer to the non-local seam (`transport` / `distribution`). Out of scope here.

### 5.2 The inbox

- One MPSC queue per worker. Many producers (any foreign thread); one consumer (the owning worker).
- **Producer side** (the deposit): non-blocking, unbounded, loss-free, FIFO. Core holds this handle per worker, indexed by `WorkerId`.
- **Consumer side** (the worker's idle point): engine-chosen. A blocking channel under std::thread; an async channel under tokio. Receiving *is* the idle point, so depositing makes the next transition runnable with no separate wake primitive. The engine owns this side and the drain cadence.

### 5.3 The crossing

The message crosses the thread boundary exactly once, inside the inbox queue. The producer writes a pointer into a slot in shared memory; the consumer, blocked or awaiting on that queue, returns it. No serialization, no copy of the payload, no inter-thread signal beyond the queue's own readiness.

## 6. Guarantees

- **Single-writer.** At most one message handled per actor at any instant, across all workers.
- **Per-(sender, target) FIFO.** Messages from one sender to one target are delivered in send order. Cross-sender ordering to the same target is not guaranteed.
- **Loss-free / non-blocking producer.** The inbox never drops and never blocks the sender (unbounded). Mailboxes likewise.
- **Fairness.** No actor processes more than `budget` messages per pass, so one actor cannot starve the others on its worker.
- **Panic isolation.** A panicking actor is poisoned and skipped; it does not take down its worker or other actors. (Existing behavior, retained.)

## 7. Data structures

**Runtime-wide (shared, read-mostly)**

- `address_map`: `RwLock<HashMap<ActorAddress, WorkerId, identity-hash>>` — the routing table; read on send, written at spawn.
- `inbox_txs`: per-worker inbox deposit handles, indexed by `WorkerId`.
- `rr_worker`: `AtomicUsize` round-robin counter, used only for external (out-of-runtime) spawns. In-runtime spawns (`ctx.spawn`) need no counter — the child pins to the caller's worker.

**Per-worker inbox (cross-thread)**

- MPSC queue. Producer = deposit (pointer-move; lock-free ring + overflow). Consumer = the worker's wait point (engine-typed).

**Per-worker, worker-local (single-threaded)**

- `pool`: `HashMap<ActorAddress, ActorSlot>`.
- `ActorSlot { mailbox: VecDeque<Box<dyn Any + Send>>, actor, lifecycle flags }`.
- `pending_local`: `Vec<(ActorAddress, Box<dyn Any + Send>)>` — same-worker buffer.

**Envelope**: `{ dest: ActorAddress, payload: Box<dyn Any + Send> }`.

## 9. Worked example

**Note on `WorkerId` indexing.** `WorkerId` is an opaque internal newtype — minted only by the runtime at spawn and used only to index that same runtime's own `inbox_txs` / `spawn_txs` slices. It never crosses the public API as a raw index, so misuse is bounded to internal code. The per-message cost on the cross-worker path is the routing-map lookup (§11 defers eliminating it via address-encoded routing), not the slice index that follows — the latter is a single pointer-add. The fast path is the same-worker arm (`pending_local`), which bypasses the inbox, the `Envelope`, and the second thread entirely.

X on worker 0 (thread T0) sends `M` to `addr`, which is Y on worker 1 (thread T1):

1. `ctx.send(addr, M)` → `Box::new(M)` (one allocation).
2. `send_any`: `address_map.lookup(addr)` → worker 1; not self → `inbox_txs[1].send(Envelope { addr, M })`. Pointer into worker 1's inbox ring. Return. No signal.
3. T1 was blocked on `inbox.recv()`; the deposit unblocks it and returns the `Envelope`.
4. T1 drains: `pool[addr].mailbox.push_back(M)`.
5. Pass walks the pool, finds Y's mailbox non-empty, pops, `Y.handle(ctx, M)`.

X learned nothing about threads. The only thread-aware steps were the one map read and the queue the pointer sat in.

Had `addr` been on worker 0: step 2 takes the same-worker arm, `M` goes to `pending_local`, and Y handles it later in this same pass — no `Envelope`, no ring, no second thread.

## 10. Changes vs current `src/`

**Removed**

- `Runtime::run()` spawning owned OS threads.
- `thread::park()` / `thread::unpark()` wakeup.
- `notify_worker()` and the `worker_threads: Vec<OnceLock<Thread>>` plumbing (including inside `ExternalSender`).
- `Placement` (the load-aware selector) and its `WorkerStats`-driven `next_worker()` scan; replaced by same-worker pinning for in-runtime spawns and a single round-robin counter (`rr_worker`) for external spawns.

**Changed**

- The per-worker transfer queue becomes the worker **inbox**, and its consumer side becomes the worker's idle point (engine-supplied). Deposit no longer signals the engine.

**Retained unchanged**

- `tick()` / `try_tick()` inline all-workers mode (deterministic, wasm, tests).
- `ActorPool`, `ActorSlot`, mailboxes, `pending_local`, budget, panic isolation, `ExternalSender` / `Inbox` / `Ask` (minus the removed wake).

**Added**

- The engine seam: a way for an integrator to create and register workers, supply each worker's inbox consumer and wait, and drive each worker's loop. Exact API is defined per engine in follow-on integration notes.

## 11. Deferred

- Actor migration, work-stealing, and load balancing beyond spawn-time worker selection.
- Ready-queue optimization.
- Cross-isolation delivery (web workers) and cross-process / WAN delivery (`transport`, `distribution`).
- Address-encoded worker routing (eliminating the routing-map lookup).