From f8fc594b95871813a890b5d60f60dee505ef93bc Mon Sep 17 00:00:00 2001 From: Zachery Aaron Shores-Chmielewski Date: Sun, 9 Aug 2026 15:13:02 +0400 Subject: [PATCH] chore(docs): adopt repo-local spec workflow Specs gain a monotonic Id and relocate by their true status: drafts (WIP/aspirational) to docs/specs/drafts/, and accurate code-behavior references stay in their crate dirs. docs/specs/archive/ is reserved for superseded docs (currently empty). Dispositions were cross-referenced against code, not the specs' own headers. IROH_DRIVER claimed "current-state" but ~30% is unbuilt redesign, so it moves to drafts. DATA_PLANE_ACTOR's central integration claim is unrealized (myelin bypasses its node actor), and ACTOR_PANEL is not a reference; both are dropped rather than reviewed or archived. DATASTREAM and MANAGED_PROCESS stay as references; MYELIN stays in place (stale, flagged for review). From here, commit titles reference a spec by [N] when one applies. This bootstrap commit does not carry one. Signed-off-by: Zachery Aaron Shores-Chmielewski --- apps/myelin/specs/MYELIN_SPEC.md | 4 +- crates/dashboard/ACTOR_PANEL_SPEC.md | 210 ------- crates/data-plane/DATA_PLANE_ACTOR_SPEC.md | 523 ------------------ crates/datastream/DATASTREAM_SPEC.md | 7 +- .../process/SWACTOR_MANAGED_PROCESS_SPEC.md | 4 +- docs/specs/drafts/ENGINE_SPEC.md | 135 +++++ .../specs/drafts}/IROH_DRIVER_SPEC.md | 4 +- docs/specs/drafts/JOB_RUNNER_SPEC.md | 146 +++++ docs/specs/drafts/MULTICORE_SPEC.md | 145 +++++ docs/specs/drafts/RECONCILER_SPEC.md | 303 ++++++++++ 10 files changed, 741 insertions(+), 740 deletions(-) delete mode 100644 crates/dashboard/ACTOR_PANEL_SPEC.md delete mode 100644 crates/data-plane/DATA_PLANE_ACTOR_SPEC.md create mode 100644 docs/specs/drafts/ENGINE_SPEC.md rename {crates/iroh-driver => docs/specs/drafts}/IROH_DRIVER_SPEC.md (99%) create mode 100644 docs/specs/drafts/JOB_RUNNER_SPEC.md create mode 100644 docs/specs/drafts/MULTICORE_SPEC.md create mode 100644 docs/specs/drafts/RECONCILER_SPEC.md diff --git a/apps/myelin/specs/MYELIN_SPEC.md b/apps/myelin/specs/MYELIN_SPEC.md index be27e81..c6994ab 100644 --- a/apps/myelin/specs/MYELIN_SPEC.md +++ b/apps/myelin/specs/MYELIN_SPEC.md @@ -1,7 +1,9 @@ # Myelin System Specification ***STALE! FOR HISTORICAL REFERENCE ONLY*** -**Status:** draft consolidated system specification. +Id: 9 +Last modified: +Last reviewed: This document is the single Myelin reference for the swactor GGUF pipeline system. It folds the system behavior previously split across the orchestration, diff --git a/crates/dashboard/ACTOR_PANEL_SPEC.md b/crates/dashboard/ACTOR_PANEL_SPEC.md deleted file mode 100644 index 3eaf4ac..0000000 --- a/crates/dashboard/ACTOR_PANEL_SPEC.md +++ /dev/null @@ -1,210 +0,0 @@ -# Actor Panel — v1 Spec (DRAFT) - -Status: **DRAFT, v1 scope.** Converged scope for the first menu item of the -swactor dashboard: live observation of actors, their messages, mailbox, and -throughput. North star is `tokio-console`, scoped down to ship fast and iterate. - ---- - -## 1. Goal - -An **actor-first** roster plus a per-actor **dossier**, focused on identity, -lifecycle, mailbox, and message throughput. Streaming-feel, single-runtime in -focus with a runtime selector. Replaces the worker page's worker-centric framing -with actors as the primary axis; workers become a column and a filter. - -One sentence: *a live console over the actor population and what each actor is -doing with its messages.* - -## 2. Non-goals (deferred backlog) - -Explicitly out of v1, queued for later iterations: - -- Tree lens (spawn/supervision hierarchy) — needs `parent`. -- Flow/message graph (who talks to whom). -- All relationship facets: parent, children, senders/receivers, monitors. -- Per-actor **arrival** rate (saturation is derived from mailbox growth instead). -- Death-event capture with `StopReason` + `ExitValue` (dead actors get a stale - badge from the last retained snapshot only). -- Rich lifecycle transition timeline (v1 shows current state + age). -- SSE streaming transport (v1 polls, matching the existing pages). -- Actor state inspection (`GetActorState` — security-gated, tier-3). -- Per-actor busy/poll handler timing. - -## 3. Roster (primary view) - -Sortable, filterable live table. Default sort: **mailbox depth descending** -(hot actors bubble up). Matches the existing worker page's poll cadence (750 ms) -and runtime selector. - -| Column | Source | Notes | -|---------------|-----------------------------------------|-------| -| `name · addr` | `LogicalName` + `ActorAddress` (short) | short hex identity + human label | -| `type` | `actor_type_name` | **new** — the actor's Rust type | -| `state` | derived from lifecycle flags (see §6) | **up from poisoned-only**; colored badge | -| `mailbox` | `mailbox_depth` + growth color | trend color: steady / rising / runaway | -| `msg/s` | derived from `messages_processed` delta | processed rate | -| `processed` | `messages_processed` | lifetime total | -| `last msg` | `last_msg_type` | last handled message type | -| `worker` | `worker_id` | placement; also a filter | -| `age` | from `spawn_time` | **new** — alive duration | - -Filters: by **state** (e.g. "show all poisoned"), by **type**, by **worker**, by -**name/address** substring. - -## 4. Dossier (row click) - -Focus panel with three facets, all about the actor itself — no relationships. - -### 4.1 Lifecycle & identity -Full address, `actor_type`, the `message_type` it accepts, spawn age, current -state. If the actor is dead, show the last-known state as stale (no -`StopReason`/`ExitValue` in v1 — deferred). - -### 4.2 Mailbox dynamics -Depth-over-time chart + growth rate. Saturation signal. Backed by a **per-actor -history ring buffer** the view maintains (bounded `VecDeque`, same pattern as -the existing `HistorySample` in `worker_view.rs`). No core change — the view -folds each incoming snapshot into the buffer. - -### 4.3 Message diet *(signature feature)* -`message_type_counts` rendered as a sorted bar list (top-N) with counts, plus the -last message. Data already exists. This is the column `tokio-console` cannot have -(tasks are opaque); swactor actors are message-typed, so "what does this actor -do" is answered by what it eats. Lean into it visually. - -## 5. Summary strip - -Throughput at the runtime level, above the roster. Same card shape as the worker -view, actor-centric: - -`actors` · `total msg/s` · `total mailbox` · `poisoned` · `uptime` - -## 6. Lifecycle state derivation - -A single display state derived from the four flags, in priority order: - -``` -poisoned → "poisoned" (red) -else stopping → "stopping" (orange) -else suspended → "suspended" (yellow) -else !started → "new" (blue) -else → "running" (green) -``` - -swactor's flags *are* the states — cleaner than `tokio-console`'s running/idle. - -## 7. Data model — the one core prerequisite - -The live feed is `runtime.actors`, emitted by `DatastreamStatsHook` -(`crates/datastream/src/endpoint.rs`) from `ActorSnapshot` (`src/stats.rs:123`), -built in `ActorPool::mailbox_depths_into` (`src/worker.rs:995`). `ActorSlot` -(`src/worker.rs:558`) already holds every field v1 needs; the change is purely -additive in the snapshot, keeping the read-only push contract. - -### Enrich `ActorSnapshot` with - -| Field | Type | Source on `ActorSlot` / actor | -|-----------------|-----------------------|--------------------------------------| -| `started` | `bool` | `slot.started` | -| `suspended` | `bool` | `slot.suspended` | -| `stopping` | `bool` | `slot.stopping` | -| `actor_type` | `&'static str` | `slot.actor.metadata().actor_type_name` | -| `message_type` | `&'static str` | `slot.actor.metadata().message_type_name` | -| `spawn_time` | `Option` | `slot.env` → `SpawnTimestamp` (ms since runtime creation) | - -`poisoned` already present. Propagate through `RuntimeActorSnapshotRecord` -(`endpoint.rs`) so the JSON wire payload carries the new keys. - -**`parent` is excluded** — only needed for the tree lens (deferred). - -### `name` continues via the existing merge - -`name` is a `swactor-std` registry concern, not core. It already flows through the -`runtime.stats` → `actor_details` (`ActorInfo.name`) path and is merged by the -existing worker view. v1 reuses that merge; no new core plumbing for names. - -## 8. View-side state (dashboard) - -New view module, mirroring `SwactorWorkerView`'s structure but actor-centric. - -### Per-actor view state - -```rust -struct ActorState { - address: String, - name: Option, - actor_type: Option, - message_type: Option, - // lifecycle - started: bool, - suspended: bool, - stopping: bool, - poisoned: bool, - spawn_time: Option, - // throughput - mailbox_depth: u32, - mailbox_growth: f64, // depth/s, derived from history - messages_processed: u64, - msg_per_sec: f64, // derived via assign_u64_rate - last_msg_type: Option, - message_type_counts: Vec<(String, u64)>, - worker_id: Option, - history: VecDeque, // per-actor mailbox chart buffer - last_update: Option, -} -``` - -`apply_json` reuses the tolerant multi-alias field helpers already in -`worker_view.rs` (`u32_field`, `string_field`, `assign_u64_rate`, -`parse_message_type_counts`) so the new keys land gracefully across versions. - -### Snapshot JSON contract - -Same envelope as the worker view: - -```jsonc -{ - "runtimes": [ - { - "stream": { "key": "...", "node": "...", "life": 0 }, - "live": true, - "last_seen_ms_ago": 42, - "summary": { "actors": 0, "msg_per_sec": 0.0, "mailbox_depth": 0, "poisoned": 0, "uptime_ms": 0 }, - "actors": [ { /* ActorState fields */ } ] - } - ] -} -``` - -Keyed by `stream_key = "{node}#{life}"` (one entry per runtime stream), matching -the worker view so the runtime selector is shared. - -## 9. Transport & feel (kept cheap for v1) - -- **Poll-first**, 750 ms, matching `worker_page.rs`. SSE is an iteration upgrade. -- **State-badge color** carries the alarm; no flash/animation machinery. -- **Dead actors**: last snapshot retained with a stale badge (`now - last_seen > - LIVE_TTL`), no death-event capture. -- `LIVE_TTL` ~8 s (match the worker view). - -## 10. File map - -| File | Change | -|------|--------| -| `src/stats.rs` | add fields to `ActorSnapshot` (§7) | -| `src/worker.rs` (`mailbox_depths_into`) | populate new fields from `ActorSlot` | -| `crates/datastream/src/endpoint.rs` (`RuntimeActorSnapshotRecord`) | serialize new fields | -| `crates/dashboard/src/swactor/actor_view.rs` | **new** — `ActorPanelView: DashboardView` | -| `crates/dashboard/src/swactor/actor_page.rs` | **new** — `ACTOR_HTML` const | -| `crates/dashboard/src/swactor/mod.rs` | `pub fn actor_view()`, register | -| `crates/dashboard/src/server.rs` / `root_page.rs` | list **first** in the menu | - -## 11. Open decisions - -1. **Menu first**: render the index dynamically from `ViewRegistry::descriptors()` - (today the static root HTML ignores it) vs. hardcode the actor link ahead of - workers. Recommend dynamic — scales as views grow. -2. **Type cardinality display**: with many instances per type, do we also offer a - type-aggregated rollup (OrleansDashboard-style) in v1, or instance-only? - Recommend instance-only for v1; rollup is a fast follow. diff --git a/crates/data-plane/DATA_PLANE_ACTOR_SPEC.md b/crates/data-plane/DATA_PLANE_ACTOR_SPEC.md deleted file mode 100644 index 495ba62..0000000 --- a/crates/data-plane/DATA_PLANE_ACTOR_SPEC.md +++ /dev/null @@ -1,523 +0,0 @@ -# Data Plane Actor Architecture Specification - -**Status:** implemented architecture contract for the `data-plane` crate. - -This document describes the behavioral boundary between reusable data-plane actors -and MVP-specific orchestration/runtime code. It is intentionally architectural: it -names responsibilities, actor roles, message families, and ownership boundaries -without prescribing file layout or migration steps. - ---- - -## 1. Purpose - -`data-plane` owns the behavior required to move model objects between stages and -between a node process and its local GPU worker process. - -The crate defines the actor protocol for: - -- provisioning logical data edges; -- distinguishing network/wire edges from node-local IPC rings; -- leasing, installing, readying, faulting, stopping, quiescing, and releasing - local rings; -- binding logical edges to transport endpoints and local worker rings; -- parsing, validating, sequencing, loading, producing, and reporting objects; -- gating readiness and object visibility on the correct lifecycle transitions; -- translating low-level arena, transport, and worker observations into coarse - data-plane outcomes. - -`mvp-system` uses `data-plane` as a reusable actor subsystem. It provides run -intent, concrete runtime actor addresses, and MVP-specific report sinks. It does -not own the fine-grained data-plane state machine. - ---- - -## 2. Core boundary - -Swactor actors carry control, lifecycle, and identity messages. Payload bytes do -not move through actor mailboxes. - -Payload bytes move through: - -- arena-backed shared-memory rings for node-local IPC; -- transport byte streams for node-to-node data edges; -- GPU-worker-owned device allocations for compute-ready objects. - -The data-plane actors decide when these byte paths are established, readable, -writable, faulted, stopped, and safe to release. Runtime adapter actors execute -concrete effects and report observations back. - ---- - -## 3. Edge and ring model - -### 3.1 Wire edge - -A wire edge is a logical run-plan connection between a producer endpoint and a -consumer endpoint. - -It carries: - -- run identity; -- edge identity; -- producer and consumer node identity; -- edge kind, such as token input, activation, or token output; -- object contract; -- transport contract; -- optional remote endpoint and remote actor identity. - -A wire edge answers: "which logical data stream connects these stage endpoints?" - -It does not answer: "which local arena offset or worker-process ring is being -used on this node?" - -### 3.2 Local IPC ring - -A local IPC ring is a node-local buffer used by a Rust node process and its owned -GPU worker process. - -It carries: - -- ring identity allocated by the local arena manager; -- arena layout and capacity; -- worker-process generation; -- local role port, such as input or output; -- direction relative to the worker process; -- object contract installed into the worker; -- quiescence and release state. - -A local IPC ring answers: "how does this node exchange bytes with its local -worker process for a specific edge endpoint?" - -It does not answer: "which remote node or distributed route owns the other side -of the logical edge?" - -### 3.3 Binding - -A wire edge endpoint may bind to zero or more local resources depending on its -role: - -```text -inbound wire edge endpoint - -> recv transport endpoint - -> local worker ingress ring - -> device object handle - -outbound wire edge endpoint - -> local worker egress ring - -> send transport endpoint - -local-only edge endpoint - -> local producer/consumer binding - -> optional worker ring -``` - -The binding is owned by data-plane state. MVP code may observe the binding only -through coarse reports such as edge ready, object loaded, object produced, edge -faulted, and edge stopped. - ---- - -## 4. Actor topology - -The target topology is actor-oriented. - -### 4.1 Data-plane node actor - -One data-plane node actor owns the data-plane state for one local node within one -active run. - -It owns: - -- local node identity; -- run-scoped edge table; -- mapping from wire edge endpoints to local rings; -- mapping from ring ids to edge endpoints; -- object sequence state; -- device-handle visibility state; -- data-plane child actor addresses; -- MVP report sink addresses. - -It receives provisioning intent from MVP code and observations from runtime -adapters. It emits actor messages to arena, worker, transport, and MVP report -sinks. - -### 4.2 Wire edge actor - -A wire edge actor owns the lifecycle of one logical edge endpoint on the local -node. - -It owns: - -- provisioning state; -- transport establishment state; -- send/receive pump readiness; -- stream faults; -- logical edge readiness; -- stop and fault propagation for that edge endpoint. - -It does not own worker-process state or arena layout details except through a -binding supplied by the data-plane node actor or ring actor. - -### 4.3 Local worker ring actor - -A local worker ring actor owns one local IPC ring lifecycle. - -It owns: - -- arena lease request and result; -- worker ring installation; -- worker ring readable/writable notifications; -- ring fault and quiescence observations; -- release proof collection; -- arena lease release. - -It does not own remote endpoint routing. It can be bound to a wire edge endpoint -by edge id, but the ring lifecycle remains local. - -### 4.4 GPU worker control adapter - -The GPU worker control adapter is the actor-facing boundary to the owned Python -worker process. - -It owns or fronts: - -- worker process generation; -- command serialization to the worker; -- stdout/stderr event parsing; -- device handle generation checks; -- worker stop/crash/restart observations. - -The data plane treats this as an actor endpoint. Worker-process JSON and Python -helper details are not exposed to MVP stage logic. - -### 4.5 Transport adapter actors - -Transport adapter actors own concrete wire byte movement. - -They own or front: - -- accepted edge streams; -- outbound edge streams; -- edge preamble validation; -- byte read/write readiness; -- transport-specific stream faults; -- pump stop observations. - -The data plane treats transport events as observations on a wire edge. Transport -actors do not decide stage readiness or object admission. - -### 4.6 MVP report sink - -The MVP report sink receives coarse data-plane outcomes and maps them to -MVP-specific control messages. - -Examples: - -- inbound edge ready; -- outbound edge ready; -- object loaded for stage execution; -- object produced for downstream transport; -- edge faulted; -- local edges stopped. - -The sink does not inspect ring cursors, arena leases, worker generations, or -transport pump internals. - ---- - -## 5. Actor API surface - -Concrete Rust names are schematic. The contract is the message shape and -ownership boundary. - -### 5.1 Provisioning input - -MVP sends provisioning intent to the data-plane node actor: - -```text -ProvisionDataPlaneRun { - run_id, - local_node_id, - arena_actor, - worker_actor, - transport_actor, - report_sink, -} - -ProvisionWireEdgeEndpoint { - run_id, - edge_id, - direction, - edge_kind, - local_role_port, - local_node_id, - peer_node_id, - peer_endpoint, - object_spec, - transport_spec, - local_ring_spec, -} -``` - -`direction` is relative to the local node's stage role: inbound means the local -stage consumes objects from the edge; outbound means the local stage produces -objects to the edge. - -Provisioning is declarative. MVP describes the intended edge endpoint and the -actors available to execute effects. It does not prescribe lease/install/driver -ordering. - -### 5.2 Runtime observations - -Runtime adapters report observations back to data-plane actors: - -```text -ArenaRingLeased -ArenaRingLeaseRejected -ArenaRingReleased -ArenaRingReleaseRejected - -WorkerReady -WorkerRingInstalled -WorkerRingFaulted -WorkerRingQuiesced -WorkerRingReadable -WorkerRingWritable -WorkerObjectLoaded -WorkerObjectProduced -WorkerObjectFailed -WorkerStopped -WorkerFaulted - -TransportEdgeReady -TransportBytesReceived -TransportBytesSent -TransportStreamClosed -TransportStreamFaulted -TransportPumpStopped -``` - -Observations are facts, not commands. The data plane decides the next state and -any follow-up messages. - -### 5.3 Data-plane effects - -The data plane sends effect requests to runtime adapter actors: - -```text -LeaseArenaRing -CancelArenaRingLease -ReleaseArenaRingLease - -InstallWorkerRing -UninstallWorkerRing -NotifyWorkerRingReadable -NotifyWorkerRingWritable -LoadObjectFromWorkerRing -ExecuteWorkerStep -ReleaseWorkerDeviceObject - -EstablishWireSend -EstablishWireRecv -WriteWireObject -StopWirePump -``` - -Effects are actor messages. The receiving adapter owns the concrete mechanism: -memfd/mmap, JSON stdin/stdout, process supervision, iroh streams, or test doubles. - -### 5.4 Data-plane reports - -The data plane reports only stable semantic outcomes to MVP: - -```text -InboundEdgeReady { edge_id } -OutboundEdgeReady { edge_id } -ObjectLoaded { edge_id, object_id, sequence, device_handle } -ObjectProduced { edge_id, object_id, sequence, extent } -EdgeFaulted { edge_id, reason } -EdgeStopped { edge_id } -LocalEdgesStopped { run_id } -WorkerDataPlaneFaulted { reason } -``` - -Reports are the only data-plane messages MVP stage/orchestrator actors should -need for normal stage progression. - ---- - -## 6. Behavior owned by data-plane - -### 6.1 Edge establishment - -For each provisioned edge endpoint, data-plane actors own the establishment -sequence. - -Inbound endpoint: - -```text -provision endpoint - -> lease local ingress ring - -> install ring into worker input port - -> establish receive transport if the edge is remote - -> report inbound edge ready -``` - -Outbound endpoint: - -```text -provision endpoint - -> lease local egress ring when worker output is required - -> install ring into worker output port - -> establish send transport if the edge is remote - -> report outbound edge ready -``` - -Readiness is reported only after every required local and wire resource for that -endpoint is ready. A local-only endpoint may omit transport establishment. A -wire-only endpoint may omit worker-ring establishment when it terminates outside -the local GPU worker. - -### 6.2 Object ingress - -For inbound data, data-plane actors own object admission. - -The data plane: - -- associates incoming bytes with the correct wire edge and stream; -- validates object framing and object spec constraints; -- preserves sequence ordering required by the edge contract; -- writes or exposes the object through the local ingress ring; -- asks the worker to load the object to device; -- waits for a valid worker object-loaded observation; -- reports object loaded to MVP only after the device handle is current and the - logical object is complete. - -MVP does not parse object headers, track ingress buffers, reload cursors, or gate -object-loaded visibility. - -### 6.3 Object egress - -For outbound data, data-plane actors own object production and forwarding. - -The data plane: - -- receives compute/output observations from the worker; -- binds produced objects to the correct outbound edge and sequence; -- validates object extent and object contract; -- publishes readable/writable state to the worker and transport actors; -- forwards complete object records on the wire when the edge is remote; -- reports object produced or step-visible outcomes to MVP at semantic - boundaries, not cursor boundaries. - -MVP does not decide when a local output ring is readable, when a transport stream -should consume it, or when a produced object is safe to expose downstream. - -### 6.4 Faults - -The data plane owns data movement fault classification and propagation. - -Fault sources include: - -- arena lease rejection or release rejection; -- worker ring installation failure; -- worker ring fault; -- malformed object framing; -- sequence violation; -- worker object load/produce failure; -- transport stream read/write/protocol failure; -- pump stop before quiescence; -- stale worker generation or device handle. - -A fault on one edge endpoint must not silently corrupt another endpoint. The data -plane maps local faults to edge-scoped or worker-scoped reports, starts the -required stop/quiescence path, and emits the appropriate MVP report. - -### 6.5 Stop, quiescence, and release - -The data plane owns teardown ordering for local resources. - -For a bound edge/ring pair, stop requires: - -```text -stop transport pump if present - -> uninstall or quiesce worker ring if installed - -> prove no local reader/writer still uses the ring - -> release arena lease - -> report edge stopped -``` - -Arena release must be gated by quiescence proof. MVP may request run or edge -stop, but it does not supply low-level release proof or decide when a ring is -safe to release. - ---- - -## 7. MVP-system utilization - -`mvp-system` remains responsible for MVP orchestration and stage semantics. - -It owns: - -- run planning and edge assignment; -- stage provisioning authority; -- membership/readiness gates outside the data plane; -- weight loading and role configuration intent; -- stage controller behavior; -- prompt injection and token consumption; -- mapping data-plane reports to MVP lifecycle messages; -- selecting concrete runtime adapters for arena, worker process, and transport. - -For data movement, MVP code acts as a client: - -1. Spawn or obtain actor addresses for the data-plane node actor and required - runtime adapters. -2. Send run and edge provisioning intent to the data-plane node actor. -3. Forward runtime observations from concrete adapter actors when those adapters - are MVP-owned. -4. Receive coarse data-plane reports. -5. Translate those reports into stage-controller or orchestrator messages. - -MVP must not rely on private edge states such as waiting-for-lease, -waiting-for-worker-ring, waiting-for-driver, pump-stopped, ring-quiesced, or -release-ready. Those are data-plane implementation states. - ---- - -## 8. Required invariants - -- `EdgeId` names a logical run-plan edge, not a local ring allocation. -- `RingId` names a local arena-backed IPC ring, not a distributed edge. -- A ring may be bound to an edge endpoint, but the identifiers are not - interchangeable. -- Actor messages carry control and identities, not tensor payload bytes. -- Object-loaded reports are emitted only for complete, validated objects with - current-generation device handles. -- Edge-ready reports are emitted only after required wire and local IPC resources - are ready. -- Arena lease release is gated by local quiescence proof. -- Worker process generation is part of every device-handle validity decision. -- Transport faults and worker faults are classified by data-plane before they - become MVP reports. -- MVP stage logic observes semantic outcomes, never ring cursor mechanics. - ---- - -## 9. Non-goals - -`data-plane` does not own: - -- global run planning; -- placement optimization; -- model layer assignment; -- weight download or weight loading semantics; -- prompt tokenization or output token policy; -- membership convergence; -- provider provisioning; -- concrete iroh endpoint construction; -- concrete Python helper implementation. - -The crate defines reusable actor protocols and data-movement behavior. Concrete -runtime adapters may live beside MVP code, inside reusable support crates, or in -tests, as long as they satisfy the actor contracts above. diff --git a/crates/datastream/DATASTREAM_SPEC.md b/crates/datastream/DATASTREAM_SPEC.md index 35266b8..941b75e 100644 --- a/crates/datastream/DATASTREAM_SPEC.md +++ b/crates/datastream/DATASTREAM_SPEC.md @@ -1,9 +1,8 @@ # The Datastream — Specification -> **Status.** Current-state specification for the `datastream` crate. This keeps -> the original outline, but updates the model to the implementation that now -> exists: stream-local numeric channel ids, a channel catalog, broadcast endpoint -> subscriptions, and catalog-aware transport events. +Id: 7 +Last modified: +Last reviewed: --- diff --git a/crates/process/SWACTOR_MANAGED_PROCESS_SPEC.md b/crates/process/SWACTOR_MANAGED_PROCESS_SPEC.md index c715f93..4b54e8a 100644 --- a/crates/process/SWACTOR_MANAGED_PROCESS_SPEC.md +++ b/crates/process/SWACTOR_MANAGED_PROCESS_SPEC.md @@ -1,6 +1,8 @@ # Swactor Managed Process Specification -**Status:** current contract for `swactor-process`. +Id: 8 +Last modified: +Last reviewed: `swactor-process` provides a Swactor actor interface for launching, supervising, stopping, and observing one operating-system child process per process actor. diff --git a/docs/specs/drafts/ENGINE_SPEC.md b/docs/specs/drafts/ENGINE_SPEC.md new file mode 100644 index 0000000..2551d0d --- /dev/null +++ b/docs/specs/drafts/ENGINE_SPEC.md @@ -0,0 +1,135 @@ +# swactor engine — specification + +Id: 1 +Last modified: +Last reviewed: + +**Scope:** the execution substrate that drives swactor workers and hosts their async side-work, defined as an interface implemented per environment. + +## 1. Purpose + +swactor actors are synchronous, single-writer message handlers. Real systems need work actors cannot do inline: draining byte streams, running retry backoffs, polling on an interval, blocking GPU calls. That work lives in *tasks* on an execution substrate. Today that substrate is Tokio — hardcoded and reinvented per crate (ambient `Handle::try_current()`, silently-owned runtimes, ad-hoc `block_on` sync facades, a mix of tokio tasks and std threads). + +This spec defines the **engine**: a single execution substrate, expressed as an interface, that (a) drives swactor workers and (b) runs the async tasks that back them. Tokio is one implementation; a minimal std-thread engine, a Go engine, a JS-worker engine, and a deterministic test engine are others. Authoring the interface from swactor's needs lets core and each engine implementation be optimized independently on either side of the seam. + +## 2. Scope + +**In scope** + +- The engine interface: what swactor requires of an engine, and what an engine provides. +- The responsibility split between actor-workers and the engine. +- How the engine drives workers (hosting the worker loop, the inbox as the wait seam). +- The capability surface: tasks, timers, async I/O, blocking, time. +- The bridge contract: how a task delivers into an actor mailbox, and the sync/async boundary rules. +- The invariants an engine must uphold. +- Reference instantiations (non-normative). + +**Out of scope** + +- Actor execution semantics — single-writer, per-(sender,target) FIFO, fairness, panic isolation. Those belong to the actor-worker / core. +- Backpressure policy. Producers and consumers share one engine; pressure handling is the application's decision, not swactor's. +- Cancellation and shutdown lifecycle (deferred; nice-to-have). +- Failure / observability propagation, except where it falls out of the bridge contract. +- Cross-process / cross-isolation delivery and serialization. +- Specific protocols and codecs (iroh/QUIC, datastream framing). Those are crate logic built *on* the engine. + +## 3. Model + +- An **actor-worker** owns a disjoint set of actors, processes them one at a time, and is the unit of actor execution. It holds the pool, mailboxes, routing, and a single synchronous entry point: run one **pass** (`tick_once`), which drains its inbox into mailboxes and processes non-empty mailboxes up to a fairness budget. +- The **engine** is the execution substrate. It does exactly two things: + 1. **Drives workers** — hosts each worker's loop: wait until the worker has work, run a pass, repeat. + 2. **Runs tasks** — schedules the async side-work (timers, I/O pumps, blocking calls) that backs the actors. +- **The engine owns all progression.** Actor handlers never `.await`. Every handler is a synchronous transition that returns control immediately. The engine runs the loop that drives them and holds every long-lived flow (a worker idle on its inbox, a task doing I/O, a timer). The actor world is pure transition. Only the engine carries control flow across time. +- Workers and tasks share one substrate and one scheduler. There is no separate "I/O runtime" beside the actor runtime. + +## 4. The engine interface + +The interface is authored from swactor's needs. It is a **contract** — operations plus their semantics and invariants. A Rust trait is its canonical Rust binding; Go, JS, and other hosts implement the same contract natively. This spec defines the contract, not the Rust signature. + +**The engine provides:** +// USER: The `host_worker(id, pass)` fn needs more explanation and justification +// USER: Why are we including a timer as a core function necessary to the engine. Can it not go somewhere else? + +| operation | meaning | +|---|---| +| `host_worker(id, pass) → deposit` | Create the worker's inbox, start its reactive loop (wait on the inbox, call `pass`), and return the **deposit** handle core uses to route messages into it. | +| `spawn(task)` | Schedule an async unit of work on the substrate. | +| `spawn_blocking(work)` | Schedule blocking CPU / syscall work off the async path. | +| `timer(delay)` / `interval(period)` | Schedule future or recurring work. | +| `now()` | The engine's monotonic clock. | + +**Core provides back to the engine and to tasks:** +// USER: Core is fine as it is. We are not modifying core, it was carefully designed and is very pure. The engine is to be abstracted in such a way as to complement the abstractions core gives us. I think we can satisfy these fns through existing core, but we don't say that core provides xyz, as that is not the framing of this spec. + +| surface | meaning | +|---|---| +| `pass` (per worker) | The synchronous entry point `tick_once(&tc) → did_work`, run once per pass. | +| `deliver` | A handle to deposit a message into an actor mailbox by address — the bridge (§7). Cloneable; captured by tasks. | + +The split is deliberate. Core owns actor logic, routing, and the *deposit* side of every inbox. The engine owns the *idle* side and all scheduling. **Core only transitions. The engine drives.** The deposit handle returned by `host_worker` is engine-agnostic (loss-free, non-blocking push) so core's routing can deposit without knowing which engine is in use. + +## 5. Driving workers +// USER: `pass` is stupid when we already have a `tick()` built in. + +- The engine hosts N workers. For each, it runs: call `pass`; if it did work, call it again (a productive pass may have buffered same-worker sends that need draining); if it did no work, idle on the inbox until a deposit makes a pass runnable. There is no separate wake primitive. A deposit into the inbox is what makes the next transition runnable, so the engine drives it. +- The **inbox is the wait seam.** The engine creates each inbox and holds its consumer side, choosing how to wait (a blocking recv under std threads; an async `recv().await` under tokio; an event under JS). Core holds the deposit side for routing. +- **Non-reentrancy.** The engine must never run two passes of the same worker concurrently. A worker's `&mut self` is live only for the duration of a synchronous `pass` call — never held across a wait. +- **Scheduling strategy is the engine's choice.** Whether a pass runs inline on the executor (cooperative) or on a blocking thread is an implementation tradeoff the engine owns; core is agnostic to it. + +## 6. Capability surface +// USER: Maybe just I/O instead of explicitly async? So we can have a blocking I/O if our engine only supports that +// USER: Not sure I want to put time inside the engine. I am open to being convinced, but the added complexity and tying it +// USER: to what I wanted to be a simple task/execution api is worrying me about future compatability. + +The primitives an engine may provide. Capabilities are **per-implementation and discoverable**: each engine reports which it supports, and binding an engine that lacks a required capability fails at construction, never at runtime. + +- **Tasks** — `spawn` of an async unit of work; the substrate's unit of concurrency. +- **Timers** — one-shot delay and recurring interval. +- **Async I/O** — streams, sockets, files. This is where implementations diverge most: a tokio engine offers sockets / QUIC / streams; a JS engine offers fetch / WebSocket; a std-thread engine offers none (only blocking I/O via `spawn_blocking`). +- **Blocking** — `spawn_blocking` for CPU-bound or syscall work that must not stall the executor. +- **Time** — `now()`. In a test engine this is virtual, advanced by the test; this is what makes deterministic testing possible. + +An engine that provides only tasks + blocking + time is still a valid (if unperformant) engine. Crates that need async I/O bind to an engine that provides it. + +## 7. The bridge contract +// USER: Why this contract, why are tasks delivering directly to actors? + +How an engine task gets a result into an actor mailbox. + +- A task captures a **deliver** handle (obtained from core, not from the engine) bound to a destination address, or a runtime-wide `send_to(addr, msg)`. Delivering deposits the message into the owning worker's inbox — a loss-free, non-blocking pointer-move along the same path any sender uses. No serialization, no copy, within one address space. +- Deliver is **fire-and-forget from the task's view**: it returns immediately; the actor handles the message on a later pass of its worker. +- **Boundary rules:** + - Actor handlers are synchronous and single-writer. They never `.await`. + - `&mut Worker` and any actor state is live only during a synchronous `pass`; it is never held across a wait and never sent into a task. + - All `.await` lives in tasks. Tasks never touch actor state directly; they communicate only via the deliver handle and the inbox. +- The inbox a task delivers into is the same FIFO, loss-free, unbounded queue the worker waits on. Mailbox ordering semantics (per-(sender,target) FIFO) are the actor-worker's concern; the engine's only obligation is that the inbox itself is FIFO and loss-free. + +## 8. Invariants + +An engine must uphold: + +- **Non-reentrant passes.** At most one `pass` per worker at any instant. +- **Loss-free, non-blocking delivery.** The inbox never drops and never blocks the sender (unbounded). +- **FIFO inbox.** Messages depart an inbox in deposit order. +- **Progress independence.** A long-running or blocked task must not stall worker passes, and vice versa. The engine provides enough concurrency that workers and tasks progress independently (on a cooperative single-thread host like JS, this is a discipline the engine enforces: no blocking calls in tasks or passes). +- **Actors never await.** No `.await` reaches actor code; the engine owns every wait. + +## 9. Reference instantiations (non-normative) + +Illustrations of how each environment satisfies the contract — not prescription. + +- **tokio.** Workers and tasks are tokio tasks; a worker loop is `loop { inbox.recv().await; pass(); }` with `&mut Worker` live only across the synchronous `pass` (long passes may be moved to `spawn_blocking`; that scheduling choice is the engine's, per §5). Async I/O, `spawn_blocking`, and `now()` are tokio's. This is today's de-facto engine, made explicit. +- **std-thread.** Each worker is an OS thread blocking on its inbox; tasks are OS threads or a small pool; `spawn_blocking` is a thread; there is no async I/O, only blocking I/O. Simple, unperformant, dependency-free — and a valid engine. +- **deterministic test engine.** A single-threaded stepping scheduler: workers and tasks are entries the test advances manually; `now()` is virtual time advanced by the test; async I/O is faked or mocked. It implements the same contract, so crates test against the interface with no real network and no threads, fully deterministic. It falls out of the contract; it is not specified separately. +- **Go / JS-worker (illustrative).** Workers and tasks map to goroutines + channels, or to the JS event loop + `postMessage` / callbacks. Each provides the capability subset its runtime supports. + +## 10. What this spec does not define + +The boundary, stated plainly: + +- Actor execution semantics (single-writer, FIFO, fairness, panic isolation). +- Backpressure. +- Cancellation and shutdown. +- Failure / observability propagation beyond the bridge. +- Cross-process / cross-isolation delivery and serialization. +- Specific protocols and codecs. diff --git a/crates/iroh-driver/IROH_DRIVER_SPEC.md b/docs/specs/drafts/IROH_DRIVER_SPEC.md similarity index 99% rename from crates/iroh-driver/IROH_DRIVER_SPEC.md rename to docs/specs/drafts/IROH_DRIVER_SPEC.md index c9aeaa4..1f0cde8 100644 --- a/crates/iroh-driver/IROH_DRIVER_SPEC.md +++ b/docs/specs/drafts/IROH_DRIVER_SPEC.md @@ -1,6 +1,8 @@ # Iroh Driver Fixed Specification -**Status:** draft current-state specification for `iroh-driver`. +Id: 5 +Last modified: +Last reviewed: > Review checkpoint: reviewed through Section 3.2; resume with Section 3.3 Accepted Connection Output. diff --git a/docs/specs/drafts/JOB_RUNNER_SPEC.md b/docs/specs/drafts/JOB_RUNNER_SPEC.md new file mode 100644 index 0000000..85255bb --- /dev/null +++ b/docs/specs/drafts/JOB_RUNNER_SPEC.md @@ -0,0 +1,146 @@ +# Synaptic Job Runner Specification + +Id: 4 +Last modified: +Last reviewed: + +--- + +## 1. Purpose + +Run an arbitrary command job on a rented GPU node, end to end, through swactor: +provision a node → push the workspace → setup → run → collect outputs → +teardown. swactor is the substrate: provisioning, the node runtime, the data +plane, and observability are reused; the job runner is built on top of them. + +v1 is the single-node command job: one job → one node → one attempt. Multi-node, +sharded, and pipeline jobs are future work-descriptions over the same core. + +--- + +## 2. What swactor grows, what swactor reuses + +**New — the job runner's content:** + +- a node primitive that runs an arbitrary command as a supervised process and + reports exit via datastream; +- a bulk-transfer service on the shared iroh endpoint for workspace push + (operator → node) and output pull (node → operator) — a new service + paralleling the existing edge and datastream ALPN services, not the + arena/tensor data plane; +- a job FSM and the orchestrator↔node protocol that sequence the above. + +**Reused unchanged:** the VastAI provider adapter (provision/teardown), SWIM +membership, the datastream event infrastructure, the shared iroh endpoint (which +already multiplexes the edge and datastream services), and the swactor actor +runtime. + +--- + +## 3. Design principle: job = work, gpu-agnostic + +A job describes *what to do*, not what hardware to run on — no provider, no GPU, +no node. Hardware (provider, GPU type/count/VRAM, disk, image, selection) lives +in a *pool designation* owned by the provider adapter. This keeps a future +sharded-model job the same shape against different placement; sharding is not a +v1 feature. + +--- + +## 4. Job description + +| field | required | meaning | +|-------------|----------|------------------------------------------------------| +| `name` | yes | job identity | +| `run` | yes | the command that does the work | +| `setup` | no | one-time command run before `run` (env install) | +| `workspace` | no | `{ workdir, exclude }` — pushed to the node first | +| `outputs` | no | paths to collect back after run | +| `env` | no | environment variables injected into setup and run | + +Nothing else. No resources, no provider, no GPU, no cleanup policy. + +```toml +[job] +name = "airfrans_smoke_01" +setup = "uv sync --no-dev" +run = "uv run remote-run smoke-train configs/aggressive_smoke.toml" +workspace = { workdir = ".", exclude = ["/artifacts", "/.venv", "__pycache__"] } +outputs = ["metrics.jsonl", "final_metrics.json", "checkpoint_latest.pt"] +env = { HF_TOKEN = "..." } +``` + +A job is submitted with a *pool designation* (capacity) that drives provisioning +via the existing VastAI adapter. The pool is not part of the job and is not +specified further here. + +--- + +## 5. Roles + +- **Orchestrator (job authority):** places the job, drives the lifecycle, owns + the terminal outcome, orders teardown. +- **Node (executor):** runs `setup` and `run` as supervised processes, + materializes the workspace, exposes outputs, reports lifecycle via datastream. +- **Provider adapter:** provisions and tears down nodes per the pool designation. + +--- + +## 6. Lifecycle + +States: `PENDING → RUNNING → COMPLETED | FAILED`. + +| event observed | command emitted | transition | +|-----------------------------------|------------------------------------------|---------------| +| `JobSubmitted{job, pool}` | `ProvisionNode` | → `PENDING` | +| `NodeReady` | `MaterializeWorkspace` | → `RUNNING` | +| `WorkspaceMaterialized` | `RunSetup` | | +| `SetupCompleted` | `RunJob` | | +| `JobExited{0}` | `CollectOutputs` | | +| `OutputsCollected` | `TeardownNode` | → `COMPLETED` | +| `JobExited{non-zero}` | `CollectOutputs` (best-effort), `TeardownNode` | → `FAILED` | +| `NodeFault` / `NodeLost` / `OperatorStop` | `TeardownNode` | → `FAILED` | + +- A job waits for *its one node's* readiness, not pool-wide convergence. +- No `setup` → `WorkspaceMaterialized` goes straight to `RunJob`. No + `workspace` → materialization is skipped. +- The exit code is authoritative: `0` → `COMPLETED`, non-zero → `FAILED`. Declared + outputs are collected either way; collection on failure is best-effort. +- The node is torn down on every terminal state (completed or failed). +- Every transition emits a datastream event. + +`OperatorStop` is the operator kill switch for a running job. + +--- + +## 7. Behavior contracts + +The orchestrator commands these contracts; the node owns the local mechanism. +Control and lifecycle travel as orchestrator↔node messages and datastream +events. Bulk workspace and output bytes travel over a dedicated transfer +service on the shared iroh endpoint, not in actor messages. + +**Supervised command execution (node).** On `RunSetup` / `RunJob`, the node +spawns the command in `workdir` with the declared `env`, supervises it, and +reports `SetupCompleted` / `JobExited{code}` (or `NodeFault`) via datastream. A +failed setup is a failed job. + +**Workspace materialization (operator → node).** On `MaterializeWorkspace`, the +`workdir` tree (with `exclude` applied) is pushed from the operator to the node +over the iroh transfer service, landing at the node's `workdir`. The node reports +`WorkspaceMaterialized`. + +**Output collection (node → operator).** On `CollectOutputs`, declared `outputs` +are pulled from the node to a per-run operator landing directory over the iroh +transfer service. The node reports `OutputsCollected`. Missing outputs do not change +the outcome — the exit code already decided it; whatever exists is gathered. + +--- + +## 8. Relationship to the broader work model + +This spec defines only the job plugin. The intended generalization is a *work +description*: a graph of work-units with typed ports, placed by a planner. The v1 +job resolves to a single work-unit with no edges; multi-unit, sharded, and +pipeline jobs are future work-descriptions over the same core. The gpu-agnostic +job shape is what keeps that path open. diff --git a/docs/specs/drafts/MULTICORE_SPEC.md b/docs/specs/drafts/MULTICORE_SPEC.md new file mode 100644 index 0000000..0491d41 --- /dev/null +++ b/docs/specs/drafts/MULTICORE_SPEC.md @@ -0,0 +1,145 @@ +# swactor process-local multicore runtime — specification + +Id: 2 +Last modified: +Last reviewed: + +**Scope:** multicore (multi-worker) execution and message delivery within a single process (shared address space). + +## 1. Scope + +**In scope** + +- How N workers run concurrently on N cores within one process. +- How a message is routed and delivered between actors on different workers (foreign-thread, shared memory). +- How a message is delivered between actors on the same worker. +- The seam between core and the hosting engine. + +**Out of scope (deferred to separate specs)** + +- Cross-isolation delivery (JS web workers) — no shared heap; requires serialization. +- Cross-process / WAN delivery — owned by the `transport` and `distribution` crates. +- Actor migration, work-stealing, and load balancing beyond spawn-time worker selection. +- A ready-queue / runnable-set optimization. + +**Constraint.** Core (`src/`) stays free of any specific engine: no tokio dependency, no owned thread pool. Any engine that can host a blocking or async receiver can host a worker. + +## 2. Model + +- A **worker** owns a disjoint set of actors and processes them one at a time. It is the unit of parallelism: N workers on N cores run up to N actors concurrently. +- An actor is **pinned**: assigned to one worker at spawn, never moved. Worker selection at spawn is deliberately simple: an actor spawned from within a worker (`ctx.spawn`) pins to that same worker; an actor spawned from outside the runtime (via the runtime handle) is assigned round-robin across workers. A side effect is that a parent and the children it talks to stay co-located, so their traffic stays on the same-worker fast path. +- Multicore parallelizes *different actors*. One actor's work is never split across cores. This preserves the single-writer invariant: at most one message is handled per actor at any instant, across all workers. +- Workers are **autonomous and independent**: each runs its own loop. There is no global tick, no barrier, no per-step cross-worker synchronization. +- Workers are **reactive**: when idle they wait; when work arrives they run a pass. + +## 3. Responsibilities (the core / engine seam) + +- **Runtime** (core): owns the actor address space, the `address → worker` routing map, the per-worker inbox deposit handles, and spawn-time worker selection (same-worker for in-runtime spawns, round-robin for external spawns). It routes. It does not execute and does not own threads. +- **Worker** (core logic, engine-driven): owns its pinned actor pool. Each pass drains its inbox into actor mailboxes and processes non-empty mailboxes up to a fairness budget. +- **Engine** (integrator-supplied — std::thread, tokio, …): decides how many workers to create, hosts each worker's loop, and owns the inbox's consumer side (how the worker idles and how often it drains). **Core only transitions. The engine drives.** + +## 4. Delivery regimes (this spec) + +| target lives on… | delivery | +| ------------------------------------ | ------------------------------------------------- | +| the same worker | inline, within the current pass (no queue) | +| another worker, same process | pointer-move into that worker's MPSC inbox | +| another process / isolated / remote | out of scope — `transport` / `distribution` | + +## 5. Routing and delivery mechanism + +A send resolves the target actor to its owning worker and deposits the message. There is **no wake step**. The receiving worker idles on its own inbox, so depositing into it is what makes the next transition runnable. + +### 5.1 Send path + +For a send of `M` to `addr` from any in-process sender (an actor handler, or an external thread holding a sender handle): + +1. Box `M` once → `Box` (a heap pointer). The payload is never copied again. +2. Look up `addr` in the routing map → `WorkerId`. +3. Branch: + - **Same worker** as sender → append `(addr, M)` to the worker's local `pending_local` buffer. Delivered within the current pass. No queue, no cross-thread. + - **Different worker** → wrap as `Envelope { dest: addr, payload: M }` and deposit into that worker's inbox (a pointer-move into shared memory). Return. No signal is sent to the receiver. + - **Not in the map** → defer to the non-local seam (`transport` / `distribution`). Out of scope here. + +### 5.2 The inbox + +- One MPSC queue per worker. Many producers (any foreign thread); one consumer (the owning worker). +- **Producer side** (the deposit): non-blocking, unbounded, loss-free, FIFO. Core holds this handle per worker, indexed by `WorkerId`. +- **Consumer side** (the worker's idle point): engine-chosen. A blocking channel under std::thread; an async channel under tokio. Receiving *is* the idle point, so depositing makes the next transition runnable with no separate wake primitive. The engine owns this side and the drain cadence. + +### 5.3 The crossing + +The message crosses the thread boundary exactly once, inside the inbox queue. The producer writes a pointer into a slot in shared memory; the consumer, blocked or awaiting on that queue, returns it. No serialization, no copy of the payload, no inter-thread signal beyond the queue's own readiness. + +## 6. Guarantees + +- **Single-writer.** At most one message handled per actor at any instant, across all workers. +- **Per-(sender, target) FIFO.** Messages from one sender to one target are delivered in send order. Cross-sender ordering to the same target is not guaranteed. +- **Loss-free / non-blocking producer.** The inbox never drops and never blocks the sender (unbounded). Mailboxes likewise. +- **Fairness.** No actor processes more than `budget` messages per pass, so one actor cannot starve the others on its worker. +- **Panic isolation.** A panicking actor is poisoned and skipped; it does not take down its worker or other actors. (Existing behavior, retained.) + +## 7. Data structures + +**Runtime-wide (shared, read-mostly)** + +- `address_map`: `RwLock>` — the routing table; read on send, written at spawn. +- `inbox_txs`: per-worker inbox deposit handles, indexed by `WorkerId`. +- `rr_worker`: `AtomicUsize` round-robin counter, used only for external (out-of-runtime) spawns. In-runtime spawns (`ctx.spawn`) need no counter — the child pins to the caller's worker. + +**Per-worker inbox (cross-thread)** + +- MPSC queue. Producer = deposit (pointer-move; lock-free ring + overflow). Consumer = the worker's wait point (engine-typed). + +**Per-worker, worker-local (single-threaded)** + +- `pool`: `HashMap`. +- `ActorSlot { mailbox: VecDeque>, actor, lifecycle flags }`. +- `pending_local`: `Vec<(ActorAddress, Box)>` — same-worker buffer. + +**Envelope**: `{ dest: ActorAddress, payload: Box }`. + +## 9. Worked example + +**Note on `WorkerId` indexing.** `WorkerId` is an opaque internal newtype — minted only by the runtime at spawn and used only to index that same runtime's own `inbox_txs` / `spawn_txs` slices. It never crosses the public API as a raw index, so misuse is bounded to internal code. The per-message cost on the cross-worker path is the routing-map lookup (§11 defers eliminating it via address-encoded routing), not the slice index that follows — the latter is a single pointer-add. The fast path is the same-worker arm (`pending_local`), which bypasses the inbox, the `Envelope`, and the second thread entirely. + +X on worker 0 (thread T0) sends `M` to `addr`, which is Y on worker 1 (thread T1): + +1. `ctx.send(addr, M)` → `Box::new(M)` (one allocation). +2. `send_any`: `address_map.lookup(addr)` → worker 1; not self → `inbox_txs[1].send(Envelope { addr, M })`. Pointer into worker 1's inbox ring. Return. No signal. +3. T1 was blocked on `inbox.recv()`; the deposit unblocks it and returns the `Envelope`. +4. T1 drains: `pool[addr].mailbox.push_back(M)`. +5. Pass walks the pool, finds Y's mailbox non-empty, pops, `Y.handle(ctx, M)`. + +X learned nothing about threads. The only thread-aware steps were the one map read and the queue the pointer sat in. + +Had `addr` been on worker 0: step 2 takes the same-worker arm, `M` goes to `pending_local`, and Y handles it later in this same pass — no `Envelope`, no ring, no second thread. + +## 10. Changes vs current `src/` + +**Removed** + +- `Runtime::run()` spawning owned OS threads. +- `thread::park()` / `thread::unpark()` wakeup. +- `notify_worker()` and the `worker_threads: Vec>` plumbing (including inside `ExternalSender`). +- `Placement` (the load-aware selector) and its `WorkerStats`-driven `next_worker()` scan; replaced by same-worker pinning for in-runtime spawns and a single round-robin counter (`rr_worker`) for external spawns. + +**Changed** + +- The per-worker transfer queue becomes the worker **inbox**, and its consumer side becomes the worker's idle point (engine-supplied). Deposit no longer signals the engine. + +**Retained unchanged** + +- `tick()` / `try_tick()` inline all-workers mode (deterministic, wasm, tests). +- `ActorPool`, `ActorSlot`, mailboxes, `pending_local`, budget, panic isolation, `ExternalSender` / `Inbox` / `Ask` (minus the removed wake). + +**Added** + +- The engine seam: a way for an integrator to create and register workers, supply each worker's inbox consumer and wait, and drive each worker's loop. Exact API is defined per engine in follow-on integration notes. + +## 11. Deferred + +- Actor migration, work-stealing, and load balancing beyond spawn-time worker selection. +- Ready-queue optimization. +- Cross-isolation delivery (web workers) and cross-process / WAN delivery (`transport`, `distribution`). +- Address-encoded worker routing (eliminating the routing-map lookup). diff --git a/docs/specs/drafts/RECONCILER_SPEC.md b/docs/specs/drafts/RECONCILER_SPEC.md new file mode 100644 index 0000000..2015f10 --- /dev/null +++ b/docs/specs/drafts/RECONCILER_SPEC.md @@ -0,0 +1,303 @@ +# cluster reconciler — specification + +Id: 3 +Last modified: +Last reviewed: + +**Scope:** a level-triggered reconciler that drives a declared cluster shape toward +convergence over the existing node lifecycle, living in `crates/provisioning` +alongside `NodeManager`. + +## 1. Purpose + +Today, node lifecycle is **edge-triggered and imperative**: `NodeManager` reacts to +discrete events (`LeaseCreated`, `EndpointKnown`, `BootstrapObserved`…) and emits +commands, and `apps/myelin`'s orchestration module drives those managers by hand, +deciding *when* to start, retry, and tear down each node. There is no object that +owns "the cluster should look like *this*." Scaling, replacement, and recovery are +woven into imperative workflow code. + +This spec introduces a **reconciler**: a pure, level-triggered function that, given +a desired cluster shape and the currently observed cluster state, emits the effects +that move observed → desired. A **driver** calls it repeatedly (periodically and on +events), an **executor** applies the effects against providers, and observations +fold back into state for the next pass. The system converges; it does not execute a +script. + +The mindset shift: **edge-triggered imperative workflow → level-triggered +declarative convergence.** The reconciler never asks "what event just happened?" It +asks "given where this node is and where it must be, what is the next step?" + +## 2. Scope + +**In scope** + +- The reconciler contract: desired state, observed state, the pure `reconcile` + function, and the effect vocabulary. +- Driving semantics: periodic + event triggers, one-step-per-pass convergence, + idempotency, failure/backoff. +- Cluster-topology reconciliation: scale up/down across node groups, replacement on + shape change. +- The responsibility split between the reconciler (pure decider), the driver + (stateful owner of the node fleet), and the executor (applies effects to + providers). +- How the reconciler maps onto the existing `NodeManager` / `NodeStage` state + machine without inventing a parallel lifecycle. + +**Out of scope** + +- Actor workload placement on reconciled nodes (future scope; topology only for v1). +- Data-plane / connectivity reconciliation (iroh mesh, datastream links) as part of + the shape (future scope). +- Specific provider adapters (Vast.ai, Docker). Those implement the existing + `ProviderPlugin` / `ProvisionPlugin` executor seams. +- Backpressure and admission policy across the whole cluster. +- Persistence and leader election (single driver instance assumed for v1). + +## 3. Model + +Three roles, one invariant. + +- **Reconciler** — a pure, deterministic function + `reconcile(observed, desired) → plan`. No I/O, no clocks beyond an injected + `now()`, no mutation of inputs. Given identical inputs it yields an identical + plan. This is the load-bearing seam: everything testable and provider-neutral + lives here. +- **Driver** — the stateful loop. It owns the fleet of per-node state machines + (today: a `NodeManager` per logical node), calls the reconciler each pass, + dispatches the plan to the executor, folds provider/bootstrap observations back + into observed state, and decides *when* to run (periodic tick + event triggers). + The driver is the only writer of observed state. +- **Executor** — applies effects against reality. Maps to the existing seams: + `ProviderPlugin` (lease lifecycle: create / lookup / destroy) and + `ProvisionPlugin` (node process) plus bootstrap sessions. `apps/myelin`'s + orchestration module becomes this layer. + +**Invariant — the reconciler is pure; the driver owns all state and waiting.** +This mirrors the engine split (ENGINE_SPEC §3): core never waits, the engine owns +all waiting. Here, the reconciler never waits or mutates; the driver owns the +node fleet, the clocks, and the retry timers. + +## 4. State + +### Desired state + +Authoritative, supplied by the caller, held immutably between shape edits: + +```rust +// already defined in node.rs — unchanged +pub struct RunNodeGroupSpec { /* run_id, group_id, role, count, provider, shape, boot, swarm_join */ } +pub fn expand_node_group(group: &RunNodeGroupSpec) -> Vec; + +// new +pub struct ClusterShape { + pub run_id: RunId, + pub groups: Vec, +} +impl ClusterShape { + pub fn expand(&self) -> Vec { /* flatMap expand_node_group */ } +} +``` + +`ClusterShape` is a thin bag over the existing group spec; `expand` reuses +`expand_node_group`. The expanded `Vec` is the set of logical nodes +the cluster **should** contain. + +### Observed state + +The set of logical nodes the cluster **does** contain, each with its lifecycle +facts. `NodeRecord` already is per-node observed state. The cluster wraps it: + +```rust +// NodeRecord already holds: desired, stage, ready, lease, connection, bootstrap, +// swactor, failed_reason, destroyed_at. + +pub struct ClusterState { + pub nodes: BTreeMap, +} +``` + +The driver is the sole writer of `ClusterState`. Observations (lease results, +endpoints, bootstrap progress, swactor joins, failures) are folded into `NodeRecord` +between passes — exactly the work `NodeManager::handle` already does internally; +under the reconciler that folding is the driver's job (see §7). + +## 5. The reconciler contract + +```rust +pub struct ReconcilePlan { + /// Desired nodes with no observed record: begin their lifecycle. + pub to_start: Vec, + /// Observed nodes with no desired entry: tear them down. + pub to_destroy: Vec, + /// Live nodes: the next one or more commands to advance each toward desired. + pub per_node: Vec<(LogicalNodeId, Vec)>, +} + +/// Pure. Deterministic. No I/O. +pub fn reconcile( + observed: &ClusterState, + desired: &ClusterShape, + now: SystemTime, +) -> ReconcilePlan; +``` + +`reconcile` is **level-triggered**: it reads only `observed` + `desired` (+ `now` +for backoff; see §8). It does not know which event triggered the pass. It is +**idempotent**: re-running with unchanged inputs yields the same plan, and an effect +whose result is already reflected in observed state is never re-emitted (e.g. a node +whose `record.lease` is `Some` never yields `CreateLease` again). + +### Per-node reconcile + +For each live node, `reconcile` computes the next step from `(NodeRecord, +LogicalNodeSpec)` — a pure function over the existing `NodeStage` machine: + +```rust +/// One pass = at most one lifecycle step per node. Convergence happens across +/// passes, not within one. +fn reconcile_node(record: &NodeRecord, desired: &LogicalNodeSpec, now: SystemTime) + -> Vec; +``` + +The mapping reuses the existing `NodeManagerCommand` vocabulary and the existing +`NodeStage` transitions — it is the **level** reading of the same state machine that +`NodeManager::handle` expresses in **edge** form: + +| observed (`record`) | next effect(s) | +|---|---| +| `New`, no lease | `CreateLease` | +| `LeaseCreated`, lease carries endpoint | `StartBootstrap` | +| `LeaseCreated`, endpoint unknown | `LookupEndpoint` | +| `EndpointKnown` / `BootstrapRunning` | (none — awaiting bootstrap observation) | +| `BootstrapRunning`, swactor joined | `BootstrapConvergenceObserved` | +| `HandedOff` / `Dormant`, `ready` | (none — steady state) | +| `Failed`, within backoff window | (none — waiting; see §8) | +| `Failed`, backoff elapsed | reset to `New` → `CreateLease` (retry) | +| destroy requested | `CancelBootstrap` (if active) + `DestroyLease` | + +**One step per node per pass.** This is the heart of the level-triggered model: +the reconciler never waits within a pass. It emits the step the current observed +state permits; the executor applies it; observation updates the record; the next +pass emits the next step. Ordering across the lease → bootstrap → join chain falls +out of convergence, not from an explicit workflow. + +## 6. Topology reconciliation + +`reconcile` first diffs the expanded desired set against the observed set by +`LogicalNodeId`: + +- **desired, not observed** → `to_start`. The driver instantiates a `NodeManager` + for the spec; the first pass emits `CreateLease`. +- **observed, not desired** → `to_destroy`. The driver runs the destroy path + (`CancelBootstrap` + `DestroyLease`); once `stage == Destroyed` the record is + reaped. +- **both** → `per_node` via `reconcile_node`. + +**Scale policy (v1, deliberately simple):** logical node identity is +`{group_id}-{index}`. Scaling a group up adds higher indices; scaling down removes +the **highest** indices first. A group's `shape` is treated as immutable per node: +changing a field that is not achievable in place (image, gpu, disk) is a +**replacement** — the affected logical nodes move to `to_destroy` and fresh specs to +`to_start` — not an in-place mutation. This is k8s-style immutable-spec rolling +replacement, kept coarse for v1. + +## 7. Driving semantics + +The driver runs a pass when **either** (a) a periodic tick fires, or (b) an event +arrives — a desired-shape edit, or a provider/bootstrap observation that changed +observed state. Each pass: + +1. Snapshot current `ClusterState` and `ClusterShape`. +2. Call `reconcile(&observed, &desired, now)` → `ReconcilePlan`. +3. Apply the plan: create `NodeManager`s for `to_start`, drive destroy for + `to_destroy`, dispatch each `per_node` command to the executor. +4. Fold executor results + pending observations into `NodeRecord`s (the driver's + only write). +5. Repeat. Terminal when every desired node is `ready` and no orphans remain. + +**Observation folding is the bridge from edge to level.** Provider/bootstrap events +arrive as the existing observation types (`CreateLeaseResult`, `SshEndpoint`, +`BootstrapObservation`, swactor-join, failures). The driver folds each into the +node's `NodeRecord` between passes — the same field updates `NodeManager::handle` +performs today (`record.lease = …`, `record.bootstrap.last_stage = …`, etc.). The +reconciler then reads the updated record and emits the next step. The edge-triggered +`NodeManager::handle` and the level-triggered `reconcile_node` are two readings of +one state machine; see §10 for the migration choice. + +**Non-reentrancy.** A pass is synchronous and exclusive: the driver never runs two +passes concurrently. This matches the worker non-reentrancy invariant (ENGINE_SPEC +§5). + +## 8. Failure and backoff + +Failure is **observed state**, not a control-flow signal. A node reaching +`NodeStage::Failed` records `failed_reason` and `failed_at`. The reconciler emits no +command for that node **until its per-node backoff window elapses** (hence `now` in +the signature); after the window it resets the node to `New` and re-emits +`CreateLease`. Backoff is **per-node and isolated** — one failed node never blocks +another (this is the pay-off of the hybrid granularity chosen in §9). Destroyed +nodes that were desired are simply re-started by the topology diff. + +Backoff parameters (initial delay, cap, jitter) are driver configuration, not +reconciler logic — the reconciler only reads `failed_at` + the configured window +and decides "retry now" vs "wait." v1 uses a fixed window; exponential backoff is a +driver-side refinement. + +## 9. Granularity and growth path + +**Hybrid, by design.** v1 ships one top-level `reconcile` whose body is: topology +diff + fan-out to `reconcile_node`. The contract — pure function of (observed, +desired) → effects — is **identical at every level**, so the growth ladder is +internal refactor, never a contract change: + +1. **Now** — one loop, topology diff + per-node reconcile. Simple. +2. **When shapes diversify** — fan out to sub-reconcilers per concern (node-groups, + roles, future: data-plane links), each with the same signature; the top level + merges their effect streams. +3. **If independent backoff/isolation/work-queues are ever needed** — promote a + sub-reconciler to its own loop + driver. Same contract; the migration is + mechanical. + +`NodeManager` is already a sub-reconciler in waiting. The contract is what must not +ossify; the loop count is cheap to grow. + +## 10. Relationship to existing code + +| exists today | role under the reconciler | +|---|---| +| `NodeManager` + `NodeStage` | per-node observed state + lifecycle transitions. Kept. | +| `NodeRecord` | per-node observed state record. Kept; the driver writes it. | +| `NodeManagerCommand` | the effect vocabulary. Reused verbatim by `reconcile_node`. | +| `RunNodeGroupSpec` / `expand_node_group` / `LogicalNodeSpec` | desired state. Wrapped by `ClusterShape`; reused. | +| `ProviderPlugin` / `ProvisionPlugin` | executor seams. Unchanged; the driver calls them. | +| `apps/myelin` orchestration | becomes the driver + executor. Imperative workflow code is replaced by `reconcile` calls. | + +**Open design choice — edge handle vs. level reconcile for `NodeManager`:** +`NodeManager::handle(msg) → Vec` is edge-triggered; the +reconciler needs the level form. Two options, to be settled at implementation: + +- **(A) Add `reconcile_node` alongside `handle`.** `handle` keeps folding streaming + observations into the record (the parts that genuinely need event semantics, e.g. + bootstrap seq numbers); `reconcile_node` reads the record and emits the next step. + Minimal churn; two readings of one machine coexist. *Recommended for v1.* +- **(B) Split into `observe(&mut record, obs)` + `reconcile_node(&record, desired)`** + and retire `handle`. One level-triggered path; more churn, cleaner end state. + +Either way the *contract* in §5 is unchanged; only `NodeManager`'s internal shape +differs. + +## 11. Invariants (normative) + +1. **Purity.** `reconcile` performs no I/O, reads no global state, mutates no input. + `now` is its only non-input dependency. +2. **Level-triggered.** `reconcile` is a function of `(observed, desired, now)`, + never of "which event fired." Safe to call at any time. +3. **Idempotent.** Re-running with unchanged inputs yields the same plan; effects + already reflected in observed state are not re-emitted. +4. **One step per node per pass.** Convergence is across passes, not within one. +5. **Driver is the sole writer of observed state.** The reconciler and executor + never mutate `ClusterState`. +6. **Non-reentrant passes.** The driver never runs two passes concurrently. +7. **Failure isolation.** A failed node's backoff never blocks another node's + progress.