chore(docs): adopt repo-local spec workflow
Specs gain a monotonic Id and relocate by their true status: drafts (WIP/aspirational) to docs/specs/drafts/, and accurate code-behavior references stay in their crate dirs. docs/specs/archive/ is reserved for superseded docs (currently empty). Dispositions were cross-referenced against code, not the specs' own headers. IROH_DRIVER claimed "current-state" but ~30% is unbuilt redesign, so it moves to drafts. DATA_PLANE_ACTOR's central integration claim is unrealized (myelin bypasses its node actor), and ACTOR_PANEL is not a reference; both are dropped rather than reviewed or archived. DATASTREAM and MANAGED_PROCESS stay as references; MYELIN stays in place (stale, flagged for review). From here, commit titles reference a spec by [N] when one applies. This bootstrap commit does not carry one. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
This commit is contained in:
parent
3a13acc0f4
commit
f8fc594b95
10 changed files with 741 additions and 740 deletions
|
|
@ -1,7 +1,9 @@
|
||||||
# Myelin System Specification
|
# Myelin System Specification
|
||||||
***STALE! FOR HISTORICAL REFERENCE ONLY***
|
***STALE! FOR HISTORICAL REFERENCE ONLY***
|
||||||
|
|
||||||
**Status:** draft consolidated system specification.
|
Id: 9
|
||||||
|
Last modified:
|
||||||
|
Last reviewed:
|
||||||
|
|
||||||
This document is the single Myelin reference for the swactor GGUF pipeline system.
|
This document is the single Myelin reference for the swactor GGUF pipeline system.
|
||||||
It folds the system behavior previously split across the orchestration,
|
It folds the system behavior previously split across the orchestration,
|
||||||
|
|
|
||||||
|
|
@ -1,210 +0,0 @@
|
||||||
# Actor Panel — v1 Spec (DRAFT)
|
|
||||||
|
|
||||||
Status: **DRAFT, v1 scope.** Converged scope for the first menu item of the
|
|
||||||
swactor dashboard: live observation of actors, their messages, mailbox, and
|
|
||||||
throughput. North star is `tokio-console`, scoped down to ship fast and iterate.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1. Goal
|
|
||||||
|
|
||||||
An **actor-first** roster plus a per-actor **dossier**, focused on identity,
|
|
||||||
lifecycle, mailbox, and message throughput. Streaming-feel, single-runtime in
|
|
||||||
focus with a runtime selector. Replaces the worker page's worker-centric framing
|
|
||||||
with actors as the primary axis; workers become a column and a filter.
|
|
||||||
|
|
||||||
One sentence: *a live console over the actor population and what each actor is
|
|
||||||
doing with its messages.*
|
|
||||||
|
|
||||||
## 2. Non-goals (deferred backlog)
|
|
||||||
|
|
||||||
Explicitly out of v1, queued for later iterations:
|
|
||||||
|
|
||||||
- Tree lens (spawn/supervision hierarchy) — needs `parent`.
|
|
||||||
- Flow/message graph (who talks to whom).
|
|
||||||
- All relationship facets: parent, children, senders/receivers, monitors.
|
|
||||||
- Per-actor **arrival** rate (saturation is derived from mailbox growth instead).
|
|
||||||
- Death-event capture with `StopReason` + `ExitValue` (dead actors get a stale
|
|
||||||
badge from the last retained snapshot only).
|
|
||||||
- Rich lifecycle transition timeline (v1 shows current state + age).
|
|
||||||
- SSE streaming transport (v1 polls, matching the existing pages).
|
|
||||||
- Actor state inspection (`GetActorState` — security-gated, tier-3).
|
|
||||||
- Per-actor busy/poll handler timing.
|
|
||||||
|
|
||||||
## 3. Roster (primary view)
|
|
||||||
|
|
||||||
Sortable, filterable live table. Default sort: **mailbox depth descending**
|
|
||||||
(hot actors bubble up). Matches the existing worker page's poll cadence (750 ms)
|
|
||||||
and runtime selector.
|
|
||||||
|
|
||||||
| Column | Source | Notes |
|
|
||||||
|---------------|-----------------------------------------|-------|
|
|
||||||
| `name · addr` | `LogicalName` + `ActorAddress` (short) | short hex identity + human label |
|
|
||||||
| `type` | `actor_type_name` | **new** — the actor's Rust type |
|
|
||||||
| `state` | derived from lifecycle flags (see §6) | **up from poisoned-only**; colored badge |
|
|
||||||
| `mailbox` | `mailbox_depth` + growth color | trend color: steady / rising / runaway |
|
|
||||||
| `msg/s` | derived from `messages_processed` delta | processed rate |
|
|
||||||
| `processed` | `messages_processed` | lifetime total |
|
|
||||||
| `last msg` | `last_msg_type` | last handled message type |
|
|
||||||
| `worker` | `worker_id` | placement; also a filter |
|
|
||||||
| `age` | from `spawn_time` | **new** — alive duration |
|
|
||||||
|
|
||||||
Filters: by **state** (e.g. "show all poisoned"), by **type**, by **worker**, by
|
|
||||||
**name/address** substring.
|
|
||||||
|
|
||||||
## 4. Dossier (row click)
|
|
||||||
|
|
||||||
Focus panel with three facets, all about the actor itself — no relationships.
|
|
||||||
|
|
||||||
### 4.1 Lifecycle & identity
|
|
||||||
Full address, `actor_type`, the `message_type` it accepts, spawn age, current
|
|
||||||
state. If the actor is dead, show the last-known state as stale (no
|
|
||||||
`StopReason`/`ExitValue` in v1 — deferred).
|
|
||||||
|
|
||||||
### 4.2 Mailbox dynamics
|
|
||||||
Depth-over-time chart + growth rate. Saturation signal. Backed by a **per-actor
|
|
||||||
history ring buffer** the view maintains (bounded `VecDeque`, same pattern as
|
|
||||||
the existing `HistorySample` in `worker_view.rs`). No core change — the view
|
|
||||||
folds each incoming snapshot into the buffer.
|
|
||||||
|
|
||||||
### 4.3 Message diet *(signature feature)*
|
|
||||||
`message_type_counts` rendered as a sorted bar list (top-N) with counts, plus the
|
|
||||||
last message. Data already exists. This is the column `tokio-console` cannot have
|
|
||||||
(tasks are opaque); swactor actors are message-typed, so "what does this actor
|
|
||||||
do" is answered by what it eats. Lean into it visually.
|
|
||||||
|
|
||||||
## 5. Summary strip
|
|
||||||
|
|
||||||
Throughput at the runtime level, above the roster. Same card shape as the worker
|
|
||||||
view, actor-centric:
|
|
||||||
|
|
||||||
`actors` · `total msg/s` · `total mailbox` · `poisoned` · `uptime`
|
|
||||||
|
|
||||||
## 6. Lifecycle state derivation
|
|
||||||
|
|
||||||
A single display state derived from the four flags, in priority order:
|
|
||||||
|
|
||||||
```
|
|
||||||
poisoned → "poisoned" (red)
|
|
||||||
else stopping → "stopping" (orange)
|
|
||||||
else suspended → "suspended" (yellow)
|
|
||||||
else !started → "new" (blue)
|
|
||||||
else → "running" (green)
|
|
||||||
```
|
|
||||||
|
|
||||||
swactor's flags *are* the states — cleaner than `tokio-console`'s running/idle.
|
|
||||||
|
|
||||||
## 7. Data model — the one core prerequisite
|
|
||||||
|
|
||||||
The live feed is `runtime.actors`, emitted by `DatastreamStatsHook`
|
|
||||||
(`crates/datastream/src/endpoint.rs`) from `ActorSnapshot` (`src/stats.rs:123`),
|
|
||||||
built in `ActorPool::mailbox_depths_into` (`src/worker.rs:995`). `ActorSlot`
|
|
||||||
(`src/worker.rs:558`) already holds every field v1 needs; the change is purely
|
|
||||||
additive in the snapshot, keeping the read-only push contract.
|
|
||||||
|
|
||||||
### Enrich `ActorSnapshot` with
|
|
||||||
|
|
||||||
| Field | Type | Source on `ActorSlot` / actor |
|
|
||||||
|-----------------|-----------------------|--------------------------------------|
|
|
||||||
| `started` | `bool` | `slot.started` |
|
|
||||||
| `suspended` | `bool` | `slot.suspended` |
|
|
||||||
| `stopping` | `bool` | `slot.stopping` |
|
|
||||||
| `actor_type` | `&'static str` | `slot.actor.metadata().actor_type_name` |
|
|
||||||
| `message_type` | `&'static str` | `slot.actor.metadata().message_type_name` |
|
|
||||||
| `spawn_time` | `Option<u64>` | `slot.env` → `SpawnTimestamp` (ms since runtime creation) |
|
|
||||||
|
|
||||||
`poisoned` already present. Propagate through `RuntimeActorSnapshotRecord`
|
|
||||||
(`endpoint.rs`) so the JSON wire payload carries the new keys.
|
|
||||||
|
|
||||||
**`parent` is excluded** — only needed for the tree lens (deferred).
|
|
||||||
|
|
||||||
### `name` continues via the existing merge
|
|
||||||
|
|
||||||
`name` is a `swactor-std` registry concern, not core. It already flows through the
|
|
||||||
`runtime.stats` → `actor_details` (`ActorInfo.name`) path and is merged by the
|
|
||||||
existing worker view. v1 reuses that merge; no new core plumbing for names.
|
|
||||||
|
|
||||||
## 8. View-side state (dashboard)
|
|
||||||
|
|
||||||
New view module, mirroring `SwactorWorkerView`'s structure but actor-centric.
|
|
||||||
|
|
||||||
### Per-actor view state
|
|
||||||
|
|
||||||
```rust
|
|
||||||
struct ActorState {
|
|
||||||
address: String,
|
|
||||||
name: Option<String>,
|
|
||||||
actor_type: Option<String>,
|
|
||||||
message_type: Option<String>,
|
|
||||||
// lifecycle
|
|
||||||
started: bool,
|
|
||||||
suspended: bool,
|
|
||||||
stopping: bool,
|
|
||||||
poisoned: bool,
|
|
||||||
spawn_time: Option<u64>,
|
|
||||||
// throughput
|
|
||||||
mailbox_depth: u32,
|
|
||||||
mailbox_growth: f64, // depth/s, derived from history
|
|
||||||
messages_processed: u64,
|
|
||||||
msg_per_sec: f64, // derived via assign_u64_rate
|
|
||||||
last_msg_type: Option<String>,
|
|
||||||
message_type_counts: Vec<(String, u64)>,
|
|
||||||
worker_id: Option<u32>,
|
|
||||||
history: VecDeque<HistorySample>, // per-actor mailbox chart buffer
|
|
||||||
last_update: Option<Instant>,
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
`apply_json` reuses the tolerant multi-alias field helpers already in
|
|
||||||
`worker_view.rs` (`u32_field`, `string_field`, `assign_u64_rate`,
|
|
||||||
`parse_message_type_counts`) so the new keys land gracefully across versions.
|
|
||||||
|
|
||||||
### Snapshot JSON contract
|
|
||||||
|
|
||||||
Same envelope as the worker view:
|
|
||||||
|
|
||||||
```jsonc
|
|
||||||
{
|
|
||||||
"runtimes": [
|
|
||||||
{
|
|
||||||
"stream": { "key": "...", "node": "...", "life": 0 },
|
|
||||||
"live": true,
|
|
||||||
"last_seen_ms_ago": 42,
|
|
||||||
"summary": { "actors": 0, "msg_per_sec": 0.0, "mailbox_depth": 0, "poisoned": 0, "uptime_ms": 0 },
|
|
||||||
"actors": [ { /* ActorState fields */ } ]
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Keyed by `stream_key = "{node}#{life}"` (one entry per runtime stream), matching
|
|
||||||
the worker view so the runtime selector is shared.
|
|
||||||
|
|
||||||
## 9. Transport & feel (kept cheap for v1)
|
|
||||||
|
|
||||||
- **Poll-first**, 750 ms, matching `worker_page.rs`. SSE is an iteration upgrade.
|
|
||||||
- **State-badge color** carries the alarm; no flash/animation machinery.
|
|
||||||
- **Dead actors**: last snapshot retained with a stale badge (`now - last_seen >
|
|
||||||
LIVE_TTL`), no death-event capture.
|
|
||||||
- `LIVE_TTL` ~8 s (match the worker view).
|
|
||||||
|
|
||||||
## 10. File map
|
|
||||||
|
|
||||||
| File | Change |
|
|
||||||
|------|--------|
|
|
||||||
| `src/stats.rs` | add fields to `ActorSnapshot` (§7) |
|
|
||||||
| `src/worker.rs` (`mailbox_depths_into`) | populate new fields from `ActorSlot` |
|
|
||||||
| `crates/datastream/src/endpoint.rs` (`RuntimeActorSnapshotRecord`) | serialize new fields |
|
|
||||||
| `crates/dashboard/src/swactor/actor_view.rs` | **new** — `ActorPanelView: DashboardView` |
|
|
||||||
| `crates/dashboard/src/swactor/actor_page.rs` | **new** — `ACTOR_HTML` const |
|
|
||||||
| `crates/dashboard/src/swactor/mod.rs` | `pub fn actor_view()`, register |
|
|
||||||
| `crates/dashboard/src/server.rs` / `root_page.rs` | list **first** in the menu |
|
|
||||||
|
|
||||||
## 11. Open decisions
|
|
||||||
|
|
||||||
1. **Menu first**: render the index dynamically from `ViewRegistry::descriptors()`
|
|
||||||
(today the static root HTML ignores it) vs. hardcode the actor link ahead of
|
|
||||||
workers. Recommend dynamic — scales as views grow.
|
|
||||||
2. **Type cardinality display**: with many instances per type, do we also offer a
|
|
||||||
type-aggregated rollup (OrleansDashboard-style) in v1, or instance-only?
|
|
||||||
Recommend instance-only for v1; rollup is a fast follow.
|
|
||||||
|
|
@ -1,523 +0,0 @@
|
||||||
# Data Plane Actor Architecture Specification
|
|
||||||
|
|
||||||
**Status:** implemented architecture contract for the `data-plane` crate.
|
|
||||||
|
|
||||||
This document describes the behavioral boundary between reusable data-plane actors
|
|
||||||
and MVP-specific orchestration/runtime code. It is intentionally architectural: it
|
|
||||||
names responsibilities, actor roles, message families, and ownership boundaries
|
|
||||||
without prescribing file layout or migration steps.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1. Purpose
|
|
||||||
|
|
||||||
`data-plane` owns the behavior required to move model objects between stages and
|
|
||||||
between a node process and its local GPU worker process.
|
|
||||||
|
|
||||||
The crate defines the actor protocol for:
|
|
||||||
|
|
||||||
- provisioning logical data edges;
|
|
||||||
- distinguishing network/wire edges from node-local IPC rings;
|
|
||||||
- leasing, installing, readying, faulting, stopping, quiescing, and releasing
|
|
||||||
local rings;
|
|
||||||
- binding logical edges to transport endpoints and local worker rings;
|
|
||||||
- parsing, validating, sequencing, loading, producing, and reporting objects;
|
|
||||||
- gating readiness and object visibility on the correct lifecycle transitions;
|
|
||||||
- translating low-level arena, transport, and worker observations into coarse
|
|
||||||
data-plane outcomes.
|
|
||||||
|
|
||||||
`mvp-system` uses `data-plane` as a reusable actor subsystem. It provides run
|
|
||||||
intent, concrete runtime actor addresses, and MVP-specific report sinks. It does
|
|
||||||
not own the fine-grained data-plane state machine.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2. Core boundary
|
|
||||||
|
|
||||||
Swactor actors carry control, lifecycle, and identity messages. Payload bytes do
|
|
||||||
not move through actor mailboxes.
|
|
||||||
|
|
||||||
Payload bytes move through:
|
|
||||||
|
|
||||||
- arena-backed shared-memory rings for node-local IPC;
|
|
||||||
- transport byte streams for node-to-node data edges;
|
|
||||||
- GPU-worker-owned device allocations for compute-ready objects.
|
|
||||||
|
|
||||||
The data-plane actors decide when these byte paths are established, readable,
|
|
||||||
writable, faulted, stopped, and safe to release. Runtime adapter actors execute
|
|
||||||
concrete effects and report observations back.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. Edge and ring model
|
|
||||||
|
|
||||||
### 3.1 Wire edge
|
|
||||||
|
|
||||||
A wire edge is a logical run-plan connection between a producer endpoint and a
|
|
||||||
consumer endpoint.
|
|
||||||
|
|
||||||
It carries:
|
|
||||||
|
|
||||||
- run identity;
|
|
||||||
- edge identity;
|
|
||||||
- producer and consumer node identity;
|
|
||||||
- edge kind, such as token input, activation, or token output;
|
|
||||||
- object contract;
|
|
||||||
- transport contract;
|
|
||||||
- optional remote endpoint and remote actor identity.
|
|
||||||
|
|
||||||
A wire edge answers: "which logical data stream connects these stage endpoints?"
|
|
||||||
|
|
||||||
It does not answer: "which local arena offset or worker-process ring is being
|
|
||||||
used on this node?"
|
|
||||||
|
|
||||||
### 3.2 Local IPC ring
|
|
||||||
|
|
||||||
A local IPC ring is a node-local buffer used by a Rust node process and its owned
|
|
||||||
GPU worker process.
|
|
||||||
|
|
||||||
It carries:
|
|
||||||
|
|
||||||
- ring identity allocated by the local arena manager;
|
|
||||||
- arena layout and capacity;
|
|
||||||
- worker-process generation;
|
|
||||||
- local role port, such as input or output;
|
|
||||||
- direction relative to the worker process;
|
|
||||||
- object contract installed into the worker;
|
|
||||||
- quiescence and release state.
|
|
||||||
|
|
||||||
A local IPC ring answers: "how does this node exchange bytes with its local
|
|
||||||
worker process for a specific edge endpoint?"
|
|
||||||
|
|
||||||
It does not answer: "which remote node or distributed route owns the other side
|
|
||||||
of the logical edge?"
|
|
||||||
|
|
||||||
### 3.3 Binding
|
|
||||||
|
|
||||||
A wire edge endpoint may bind to zero or more local resources depending on its
|
|
||||||
role:
|
|
||||||
|
|
||||||
```text
|
|
||||||
inbound wire edge endpoint
|
|
||||||
-> recv transport endpoint
|
|
||||||
-> local worker ingress ring
|
|
||||||
-> device object handle
|
|
||||||
|
|
||||||
outbound wire edge endpoint
|
|
||||||
-> local worker egress ring
|
|
||||||
-> send transport endpoint
|
|
||||||
|
|
||||||
local-only edge endpoint
|
|
||||||
-> local producer/consumer binding
|
|
||||||
-> optional worker ring
|
|
||||||
```
|
|
||||||
|
|
||||||
The binding is owned by data-plane state. MVP code may observe the binding only
|
|
||||||
through coarse reports such as edge ready, object loaded, object produced, edge
|
|
||||||
faulted, and edge stopped.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. Actor topology
|
|
||||||
|
|
||||||
The target topology is actor-oriented.
|
|
||||||
|
|
||||||
### 4.1 Data-plane node actor
|
|
||||||
|
|
||||||
One data-plane node actor owns the data-plane state for one local node within one
|
|
||||||
active run.
|
|
||||||
|
|
||||||
It owns:
|
|
||||||
|
|
||||||
- local node identity;
|
|
||||||
- run-scoped edge table;
|
|
||||||
- mapping from wire edge endpoints to local rings;
|
|
||||||
- mapping from ring ids to edge endpoints;
|
|
||||||
- object sequence state;
|
|
||||||
- device-handle visibility state;
|
|
||||||
- data-plane child actor addresses;
|
|
||||||
- MVP report sink addresses.
|
|
||||||
|
|
||||||
It receives provisioning intent from MVP code and observations from runtime
|
|
||||||
adapters. It emits actor messages to arena, worker, transport, and MVP report
|
|
||||||
sinks.
|
|
||||||
|
|
||||||
### 4.2 Wire edge actor
|
|
||||||
|
|
||||||
A wire edge actor owns the lifecycle of one logical edge endpoint on the local
|
|
||||||
node.
|
|
||||||
|
|
||||||
It owns:
|
|
||||||
|
|
||||||
- provisioning state;
|
|
||||||
- transport establishment state;
|
|
||||||
- send/receive pump readiness;
|
|
||||||
- stream faults;
|
|
||||||
- logical edge readiness;
|
|
||||||
- stop and fault propagation for that edge endpoint.
|
|
||||||
|
|
||||||
It does not own worker-process state or arena layout details except through a
|
|
||||||
binding supplied by the data-plane node actor or ring actor.
|
|
||||||
|
|
||||||
### 4.3 Local worker ring actor
|
|
||||||
|
|
||||||
A local worker ring actor owns one local IPC ring lifecycle.
|
|
||||||
|
|
||||||
It owns:
|
|
||||||
|
|
||||||
- arena lease request and result;
|
|
||||||
- worker ring installation;
|
|
||||||
- worker ring readable/writable notifications;
|
|
||||||
- ring fault and quiescence observations;
|
|
||||||
- release proof collection;
|
|
||||||
- arena lease release.
|
|
||||||
|
|
||||||
It does not own remote endpoint routing. It can be bound to a wire edge endpoint
|
|
||||||
by edge id, but the ring lifecycle remains local.
|
|
||||||
|
|
||||||
### 4.4 GPU worker control adapter
|
|
||||||
|
|
||||||
The GPU worker control adapter is the actor-facing boundary to the owned Python
|
|
||||||
worker process.
|
|
||||||
|
|
||||||
It owns or fronts:
|
|
||||||
|
|
||||||
- worker process generation;
|
|
||||||
- command serialization to the worker;
|
|
||||||
- stdout/stderr event parsing;
|
|
||||||
- device handle generation checks;
|
|
||||||
- worker stop/crash/restart observations.
|
|
||||||
|
|
||||||
The data plane treats this as an actor endpoint. Worker-process JSON and Python
|
|
||||||
helper details are not exposed to MVP stage logic.
|
|
||||||
|
|
||||||
### 4.5 Transport adapter actors
|
|
||||||
|
|
||||||
Transport adapter actors own concrete wire byte movement.
|
|
||||||
|
|
||||||
They own or front:
|
|
||||||
|
|
||||||
- accepted edge streams;
|
|
||||||
- outbound edge streams;
|
|
||||||
- edge preamble validation;
|
|
||||||
- byte read/write readiness;
|
|
||||||
- transport-specific stream faults;
|
|
||||||
- pump stop observations.
|
|
||||||
|
|
||||||
The data plane treats transport events as observations on a wire edge. Transport
|
|
||||||
actors do not decide stage readiness or object admission.
|
|
||||||
|
|
||||||
### 4.6 MVP report sink
|
|
||||||
|
|
||||||
The MVP report sink receives coarse data-plane outcomes and maps them to
|
|
||||||
MVP-specific control messages.
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
|
|
||||||
- inbound edge ready;
|
|
||||||
- outbound edge ready;
|
|
||||||
- object loaded for stage execution;
|
|
||||||
- object produced for downstream transport;
|
|
||||||
- edge faulted;
|
|
||||||
- local edges stopped.
|
|
||||||
|
|
||||||
The sink does not inspect ring cursors, arena leases, worker generations, or
|
|
||||||
transport pump internals.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5. Actor API surface
|
|
||||||
|
|
||||||
Concrete Rust names are schematic. The contract is the message shape and
|
|
||||||
ownership boundary.
|
|
||||||
|
|
||||||
### 5.1 Provisioning input
|
|
||||||
|
|
||||||
MVP sends provisioning intent to the data-plane node actor:
|
|
||||||
|
|
||||||
```text
|
|
||||||
ProvisionDataPlaneRun {
|
|
||||||
run_id,
|
|
||||||
local_node_id,
|
|
||||||
arena_actor,
|
|
||||||
worker_actor,
|
|
||||||
transport_actor,
|
|
||||||
report_sink,
|
|
||||||
}
|
|
||||||
|
|
||||||
ProvisionWireEdgeEndpoint {
|
|
||||||
run_id,
|
|
||||||
edge_id,
|
|
||||||
direction,
|
|
||||||
edge_kind,
|
|
||||||
local_role_port,
|
|
||||||
local_node_id,
|
|
||||||
peer_node_id,
|
|
||||||
peer_endpoint,
|
|
||||||
object_spec,
|
|
||||||
transport_spec,
|
|
||||||
local_ring_spec,
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
`direction` is relative to the local node's stage role: inbound means the local
|
|
||||||
stage consumes objects from the edge; outbound means the local stage produces
|
|
||||||
objects to the edge.
|
|
||||||
|
|
||||||
Provisioning is declarative. MVP describes the intended edge endpoint and the
|
|
||||||
actors available to execute effects. It does not prescribe lease/install/driver
|
|
||||||
ordering.
|
|
||||||
|
|
||||||
### 5.2 Runtime observations
|
|
||||||
|
|
||||||
Runtime adapters report observations back to data-plane actors:
|
|
||||||
|
|
||||||
```text
|
|
||||||
ArenaRingLeased
|
|
||||||
ArenaRingLeaseRejected
|
|
||||||
ArenaRingReleased
|
|
||||||
ArenaRingReleaseRejected
|
|
||||||
|
|
||||||
WorkerReady
|
|
||||||
WorkerRingInstalled
|
|
||||||
WorkerRingFaulted
|
|
||||||
WorkerRingQuiesced
|
|
||||||
WorkerRingReadable
|
|
||||||
WorkerRingWritable
|
|
||||||
WorkerObjectLoaded
|
|
||||||
WorkerObjectProduced
|
|
||||||
WorkerObjectFailed
|
|
||||||
WorkerStopped
|
|
||||||
WorkerFaulted
|
|
||||||
|
|
||||||
TransportEdgeReady
|
|
||||||
TransportBytesReceived
|
|
||||||
TransportBytesSent
|
|
||||||
TransportStreamClosed
|
|
||||||
TransportStreamFaulted
|
|
||||||
TransportPumpStopped
|
|
||||||
```
|
|
||||||
|
|
||||||
Observations are facts, not commands. The data plane decides the next state and
|
|
||||||
any follow-up messages.
|
|
||||||
|
|
||||||
### 5.3 Data-plane effects
|
|
||||||
|
|
||||||
The data plane sends effect requests to runtime adapter actors:
|
|
||||||
|
|
||||||
```text
|
|
||||||
LeaseArenaRing
|
|
||||||
CancelArenaRingLease
|
|
||||||
ReleaseArenaRingLease
|
|
||||||
|
|
||||||
InstallWorkerRing
|
|
||||||
UninstallWorkerRing
|
|
||||||
NotifyWorkerRingReadable
|
|
||||||
NotifyWorkerRingWritable
|
|
||||||
LoadObjectFromWorkerRing
|
|
||||||
ExecuteWorkerStep
|
|
||||||
ReleaseWorkerDeviceObject
|
|
||||||
|
|
||||||
EstablishWireSend
|
|
||||||
EstablishWireRecv
|
|
||||||
WriteWireObject
|
|
||||||
StopWirePump
|
|
||||||
```
|
|
||||||
|
|
||||||
Effects are actor messages. The receiving adapter owns the concrete mechanism:
|
|
||||||
memfd/mmap, JSON stdin/stdout, process supervision, iroh streams, or test doubles.
|
|
||||||
|
|
||||||
### 5.4 Data-plane reports
|
|
||||||
|
|
||||||
The data plane reports only stable semantic outcomes to MVP:
|
|
||||||
|
|
||||||
```text
|
|
||||||
InboundEdgeReady { edge_id }
|
|
||||||
OutboundEdgeReady { edge_id }
|
|
||||||
ObjectLoaded { edge_id, object_id, sequence, device_handle }
|
|
||||||
ObjectProduced { edge_id, object_id, sequence, extent }
|
|
||||||
EdgeFaulted { edge_id, reason }
|
|
||||||
EdgeStopped { edge_id }
|
|
||||||
LocalEdgesStopped { run_id }
|
|
||||||
WorkerDataPlaneFaulted { reason }
|
|
||||||
```
|
|
||||||
|
|
||||||
Reports are the only data-plane messages MVP stage/orchestrator actors should
|
|
||||||
need for normal stage progression.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 6. Behavior owned by data-plane
|
|
||||||
|
|
||||||
### 6.1 Edge establishment
|
|
||||||
|
|
||||||
For each provisioned edge endpoint, data-plane actors own the establishment
|
|
||||||
sequence.
|
|
||||||
|
|
||||||
Inbound endpoint:
|
|
||||||
|
|
||||||
```text
|
|
||||||
provision endpoint
|
|
||||||
-> lease local ingress ring
|
|
||||||
-> install ring into worker input port
|
|
||||||
-> establish receive transport if the edge is remote
|
|
||||||
-> report inbound edge ready
|
|
||||||
```
|
|
||||||
|
|
||||||
Outbound endpoint:
|
|
||||||
|
|
||||||
```text
|
|
||||||
provision endpoint
|
|
||||||
-> lease local egress ring when worker output is required
|
|
||||||
-> install ring into worker output port
|
|
||||||
-> establish send transport if the edge is remote
|
|
||||||
-> report outbound edge ready
|
|
||||||
```
|
|
||||||
|
|
||||||
Readiness is reported only after every required local and wire resource for that
|
|
||||||
endpoint is ready. A local-only endpoint may omit transport establishment. A
|
|
||||||
wire-only endpoint may omit worker-ring establishment when it terminates outside
|
|
||||||
the local GPU worker.
|
|
||||||
|
|
||||||
### 6.2 Object ingress
|
|
||||||
|
|
||||||
For inbound data, data-plane actors own object admission.
|
|
||||||
|
|
||||||
The data plane:
|
|
||||||
|
|
||||||
- associates incoming bytes with the correct wire edge and stream;
|
|
||||||
- validates object framing and object spec constraints;
|
|
||||||
- preserves sequence ordering required by the edge contract;
|
|
||||||
- writes or exposes the object through the local ingress ring;
|
|
||||||
- asks the worker to load the object to device;
|
|
||||||
- waits for a valid worker object-loaded observation;
|
|
||||||
- reports object loaded to MVP only after the device handle is current and the
|
|
||||||
logical object is complete.
|
|
||||||
|
|
||||||
MVP does not parse object headers, track ingress buffers, reload cursors, or gate
|
|
||||||
object-loaded visibility.
|
|
||||||
|
|
||||||
### 6.3 Object egress
|
|
||||||
|
|
||||||
For outbound data, data-plane actors own object production and forwarding.
|
|
||||||
|
|
||||||
The data plane:
|
|
||||||
|
|
||||||
- receives compute/output observations from the worker;
|
|
||||||
- binds produced objects to the correct outbound edge and sequence;
|
|
||||||
- validates object extent and object contract;
|
|
||||||
- publishes readable/writable state to the worker and transport actors;
|
|
||||||
- forwards complete object records on the wire when the edge is remote;
|
|
||||||
- reports object produced or step-visible outcomes to MVP at semantic
|
|
||||||
boundaries, not cursor boundaries.
|
|
||||||
|
|
||||||
MVP does not decide when a local output ring is readable, when a transport stream
|
|
||||||
should consume it, or when a produced object is safe to expose downstream.
|
|
||||||
|
|
||||||
### 6.4 Faults
|
|
||||||
|
|
||||||
The data plane owns data movement fault classification and propagation.
|
|
||||||
|
|
||||||
Fault sources include:
|
|
||||||
|
|
||||||
- arena lease rejection or release rejection;
|
|
||||||
- worker ring installation failure;
|
|
||||||
- worker ring fault;
|
|
||||||
- malformed object framing;
|
|
||||||
- sequence violation;
|
|
||||||
- worker object load/produce failure;
|
|
||||||
- transport stream read/write/protocol failure;
|
|
||||||
- pump stop before quiescence;
|
|
||||||
- stale worker generation or device handle.
|
|
||||||
|
|
||||||
A fault on one edge endpoint must not silently corrupt another endpoint. The data
|
|
||||||
plane maps local faults to edge-scoped or worker-scoped reports, starts the
|
|
||||||
required stop/quiescence path, and emits the appropriate MVP report.
|
|
||||||
|
|
||||||
### 6.5 Stop, quiescence, and release
|
|
||||||
|
|
||||||
The data plane owns teardown ordering for local resources.
|
|
||||||
|
|
||||||
For a bound edge/ring pair, stop requires:
|
|
||||||
|
|
||||||
```text
|
|
||||||
stop transport pump if present
|
|
||||||
-> uninstall or quiesce worker ring if installed
|
|
||||||
-> prove no local reader/writer still uses the ring
|
|
||||||
-> release arena lease
|
|
||||||
-> report edge stopped
|
|
||||||
```
|
|
||||||
|
|
||||||
Arena release must be gated by quiescence proof. MVP may request run or edge
|
|
||||||
stop, but it does not supply low-level release proof or decide when a ring is
|
|
||||||
safe to release.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 7. MVP-system utilization
|
|
||||||
|
|
||||||
`mvp-system` remains responsible for MVP orchestration and stage semantics.
|
|
||||||
|
|
||||||
It owns:
|
|
||||||
|
|
||||||
- run planning and edge assignment;
|
|
||||||
- stage provisioning authority;
|
|
||||||
- membership/readiness gates outside the data plane;
|
|
||||||
- weight loading and role configuration intent;
|
|
||||||
- stage controller behavior;
|
|
||||||
- prompt injection and token consumption;
|
|
||||||
- mapping data-plane reports to MVP lifecycle messages;
|
|
||||||
- selecting concrete runtime adapters for arena, worker process, and transport.
|
|
||||||
|
|
||||||
For data movement, MVP code acts as a client:
|
|
||||||
|
|
||||||
1. Spawn or obtain actor addresses for the data-plane node actor and required
|
|
||||||
runtime adapters.
|
|
||||||
2. Send run and edge provisioning intent to the data-plane node actor.
|
|
||||||
3. Forward runtime observations from concrete adapter actors when those adapters
|
|
||||||
are MVP-owned.
|
|
||||||
4. Receive coarse data-plane reports.
|
|
||||||
5. Translate those reports into stage-controller or orchestrator messages.
|
|
||||||
|
|
||||||
MVP must not rely on private edge states such as waiting-for-lease,
|
|
||||||
waiting-for-worker-ring, waiting-for-driver, pump-stopped, ring-quiesced, or
|
|
||||||
release-ready. Those are data-plane implementation states.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 8. Required invariants
|
|
||||||
|
|
||||||
- `EdgeId` names a logical run-plan edge, not a local ring allocation.
|
|
||||||
- `RingId` names a local arena-backed IPC ring, not a distributed edge.
|
|
||||||
- A ring may be bound to an edge endpoint, but the identifiers are not
|
|
||||||
interchangeable.
|
|
||||||
- Actor messages carry control and identities, not tensor payload bytes.
|
|
||||||
- Object-loaded reports are emitted only for complete, validated objects with
|
|
||||||
current-generation device handles.
|
|
||||||
- Edge-ready reports are emitted only after required wire and local IPC resources
|
|
||||||
are ready.
|
|
||||||
- Arena lease release is gated by local quiescence proof.
|
|
||||||
- Worker process generation is part of every device-handle validity decision.
|
|
||||||
- Transport faults and worker faults are classified by data-plane before they
|
|
||||||
become MVP reports.
|
|
||||||
- MVP stage logic observes semantic outcomes, never ring cursor mechanics.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 9. Non-goals
|
|
||||||
|
|
||||||
`data-plane` does not own:
|
|
||||||
|
|
||||||
- global run planning;
|
|
||||||
- placement optimization;
|
|
||||||
- model layer assignment;
|
|
||||||
- weight download or weight loading semantics;
|
|
||||||
- prompt tokenization or output token policy;
|
|
||||||
- membership convergence;
|
|
||||||
- provider provisioning;
|
|
||||||
- concrete iroh endpoint construction;
|
|
||||||
- concrete Python helper implementation.
|
|
||||||
|
|
||||||
The crate defines reusable actor protocols and data-movement behavior. Concrete
|
|
||||||
runtime adapters may live beside MVP code, inside reusable support crates, or in
|
|
||||||
tests, as long as they satisfy the actor contracts above.
|
|
||||||
|
|
@ -1,9 +1,8 @@
|
||||||
# The Datastream — Specification
|
# The Datastream — Specification
|
||||||
|
|
||||||
> **Status.** Current-state specification for the `datastream` crate. This keeps
|
Id: 7
|
||||||
> the original outline, but updates the model to the implementation that now
|
Last modified:
|
||||||
> exists: stream-local numeric channel ids, a channel catalog, broadcast endpoint
|
Last reviewed:
|
||||||
> subscriptions, and catalog-aware transport events.
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,8 @@
|
||||||
# Swactor Managed Process Specification
|
# Swactor Managed Process Specification
|
||||||
|
|
||||||
**Status:** current contract for `swactor-process`.
|
Id: 8
|
||||||
|
Last modified:
|
||||||
|
Last reviewed:
|
||||||
|
|
||||||
`swactor-process` provides a Swactor actor interface for launching, supervising,
|
`swactor-process` provides a Swactor actor interface for launching, supervising,
|
||||||
stopping, and observing one operating-system child process per process actor.
|
stopping, and observing one operating-system child process per process actor.
|
||||||
|
|
|
||||||
135
docs/specs/drafts/ENGINE_SPEC.md
Normal file
135
docs/specs/drafts/ENGINE_SPEC.md
Normal file
|
|
@ -0,0 +1,135 @@
|
||||||
|
# swactor engine — specification
|
||||||
|
|
||||||
|
Id: 1
|
||||||
|
Last modified:
|
||||||
|
Last reviewed:
|
||||||
|
|
||||||
|
**Scope:** the execution substrate that drives swactor workers and hosts their async side-work, defined as an interface implemented per environment.
|
||||||
|
|
||||||
|
## 1. Purpose
|
||||||
|
|
||||||
|
swactor actors are synchronous, single-writer message handlers. Real systems need work actors cannot do inline: draining byte streams, running retry backoffs, polling on an interval, blocking GPU calls. That work lives in *tasks* on an execution substrate. Today that substrate is Tokio — hardcoded and reinvented per crate (ambient `Handle::try_current()`, silently-owned runtimes, ad-hoc `block_on` sync facades, a mix of tokio tasks and std threads).
|
||||||
|
|
||||||
|
This spec defines the **engine**: a single execution substrate, expressed as an interface, that (a) drives swactor workers and (b) runs the async tasks that back them. Tokio is one implementation; a minimal std-thread engine, a Go engine, a JS-worker engine, and a deterministic test engine are others. Authoring the interface from swactor's needs lets core and each engine implementation be optimized independently on either side of the seam.
|
||||||
|
|
||||||
|
## 2. Scope
|
||||||
|
|
||||||
|
**In scope**
|
||||||
|
|
||||||
|
- The engine interface: what swactor requires of an engine, and what an engine provides.
|
||||||
|
- The responsibility split between actor-workers and the engine.
|
||||||
|
- How the engine drives workers (hosting the worker loop, the inbox as the wait seam).
|
||||||
|
- The capability surface: tasks, timers, async I/O, blocking, time.
|
||||||
|
- The bridge contract: how a task delivers into an actor mailbox, and the sync/async boundary rules.
|
||||||
|
- The invariants an engine must uphold.
|
||||||
|
- Reference instantiations (non-normative).
|
||||||
|
|
||||||
|
**Out of scope**
|
||||||
|
|
||||||
|
- Actor execution semantics — single-writer, per-(sender,target) FIFO, fairness, panic isolation. Those belong to the actor-worker / core.
|
||||||
|
- Backpressure policy. Producers and consumers share one engine; pressure handling is the application's decision, not swactor's.
|
||||||
|
- Cancellation and shutdown lifecycle (deferred; nice-to-have).
|
||||||
|
- Failure / observability propagation, except where it falls out of the bridge contract.
|
||||||
|
- Cross-process / cross-isolation delivery and serialization.
|
||||||
|
- Specific protocols and codecs (iroh/QUIC, datastream framing). Those are crate logic built *on* the engine.
|
||||||
|
|
||||||
|
## 3. Model
|
||||||
|
|
||||||
|
- An **actor-worker** owns a disjoint set of actors, processes them one at a time, and is the unit of actor execution. It holds the pool, mailboxes, routing, and a single synchronous entry point: run one **pass** (`tick_once`), which drains its inbox into mailboxes and processes non-empty mailboxes up to a fairness budget.
|
||||||
|
- The **engine** is the execution substrate. It does exactly two things:
|
||||||
|
1. **Drives workers** — hosts each worker's loop: wait until the worker has work, run a pass, repeat.
|
||||||
|
2. **Runs tasks** — schedules the async side-work (timers, I/O pumps, blocking calls) that backs the actors.
|
||||||
|
- **The engine owns all progression.** Actor handlers never `.await`. Every handler is a synchronous transition that returns control immediately. The engine runs the loop that drives them and holds every long-lived flow (a worker idle on its inbox, a task doing I/O, a timer). The actor world is pure transition. Only the engine carries control flow across time.
|
||||||
|
- Workers and tasks share one substrate and one scheduler. There is no separate "I/O runtime" beside the actor runtime.
|
||||||
|
|
||||||
|
## 4. The engine interface
|
||||||
|
|
||||||
|
The interface is authored from swactor's needs. It is a **contract** — operations plus their semantics and invariants. A Rust trait is its canonical Rust binding; Go, JS, and other hosts implement the same contract natively. This spec defines the contract, not the Rust signature.
|
||||||
|
|
||||||
|
**The engine provides:**
|
||||||
|
// USER: The `host_worker(id, pass)` fn needs more explanation and justification
|
||||||
|
// USER: Why are we including a timer as a core function necessary to the engine. Can it not go somewhere else?
|
||||||
|
|
||||||
|
| operation | meaning |
|
||||||
|
|---|---|
|
||||||
|
| `host_worker(id, pass) → deposit` | Create the worker's inbox, start its reactive loop (wait on the inbox, call `pass`), and return the **deposit** handle core uses to route messages into it. |
|
||||||
|
| `spawn(task)` | Schedule an async unit of work on the substrate. |
|
||||||
|
| `spawn_blocking(work)` | Schedule blocking CPU / syscall work off the async path. |
|
||||||
|
| `timer(delay)` / `interval(period)` | Schedule future or recurring work. |
|
||||||
|
| `now()` | The engine's monotonic clock. |
|
||||||
|
|
||||||
|
**Core provides back to the engine and to tasks:**
|
||||||
|
// USER: Core is fine as it is. We are not modifying core, it was carefully designed and is very pure. The engine is to be abstracted in such a way as to complement the abstractions core gives us. I think we can satisfy these fns through existing core, but we don't say that core provides xyz, as that is not the framing of this spec.
|
||||||
|
|
||||||
|
| surface | meaning |
|
||||||
|
|---|---|
|
||||||
|
| `pass` (per worker) | The synchronous entry point `tick_once(&tc) → did_work`, run once per pass. |
|
||||||
|
| `deliver` | A handle to deposit a message into an actor mailbox by address — the bridge (§7). Cloneable; captured by tasks. |
|
||||||
|
|
||||||
|
The split is deliberate. Core owns actor logic, routing, and the *deposit* side of every inbox. The engine owns the *idle* side and all scheduling. **Core only transitions. The engine drives.** The deposit handle returned by `host_worker` is engine-agnostic (loss-free, non-blocking push) so core's routing can deposit without knowing which engine is in use.
|
||||||
|
|
||||||
|
## 5. Driving workers
|
||||||
|
// USER: `pass` is stupid when we already have a `tick()` built in.
|
||||||
|
|
||||||
|
- The engine hosts N workers. For each, it runs: call `pass`; if it did work, call it again (a productive pass may have buffered same-worker sends that need draining); if it did no work, idle on the inbox until a deposit makes a pass runnable. There is no separate wake primitive. A deposit into the inbox is what makes the next transition runnable, so the engine drives it.
|
||||||
|
- The **inbox is the wait seam.** The engine creates each inbox and holds its consumer side, choosing how to wait (a blocking recv under std threads; an async `recv().await` under tokio; an event under JS). Core holds the deposit side for routing.
|
||||||
|
- **Non-reentrancy.** The engine must never run two passes of the same worker concurrently. A worker's `&mut self` is live only for the duration of a synchronous `pass` call — never held across a wait.
|
||||||
|
- **Scheduling strategy is the engine's choice.** Whether a pass runs inline on the executor (cooperative) or on a blocking thread is an implementation tradeoff the engine owns; core is agnostic to it.
|
||||||
|
|
||||||
|
## 6. Capability surface
|
||||||
|
// USER: Maybe just I/O instead of explicitly async? So we can have a blocking I/O if our engine only supports that
|
||||||
|
// USER: Not sure I want to put time inside the engine. I am open to being convinced, but the added complexity and tying it
|
||||||
|
// USER: to what I wanted to be a simple task/execution api is worrying me about future compatability.
|
||||||
|
|
||||||
|
The primitives an engine may provide. Capabilities are **per-implementation and discoverable**: each engine reports which it supports, and binding an engine that lacks a required capability fails at construction, never at runtime.
|
||||||
|
|
||||||
|
- **Tasks** — `spawn` of an async unit of work; the substrate's unit of concurrency.
|
||||||
|
- **Timers** — one-shot delay and recurring interval.
|
||||||
|
- **Async I/O** — streams, sockets, files. This is where implementations diverge most: a tokio engine offers sockets / QUIC / streams; a JS engine offers fetch / WebSocket; a std-thread engine offers none (only blocking I/O via `spawn_blocking`).
|
||||||
|
- **Blocking** — `spawn_blocking` for CPU-bound or syscall work that must not stall the executor.
|
||||||
|
- **Time** — `now()`. In a test engine this is virtual, advanced by the test; this is what makes deterministic testing possible.
|
||||||
|
|
||||||
|
An engine that provides only tasks + blocking + time is still a valid (if unperformant) engine. Crates that need async I/O bind to an engine that provides it.
|
||||||
|
|
||||||
|
## 7. The bridge contract
|
||||||
|
// USER: Why this contract, why are tasks delivering directly to actors?
|
||||||
|
|
||||||
|
How an engine task gets a result into an actor mailbox.
|
||||||
|
|
||||||
|
- A task captures a **deliver** handle (obtained from core, not from the engine) bound to a destination address, or a runtime-wide `send_to(addr, msg)`. Delivering deposits the message into the owning worker's inbox — a loss-free, non-blocking pointer-move along the same path any sender uses. No serialization, no copy, within one address space.
|
||||||
|
- Deliver is **fire-and-forget from the task's view**: it returns immediately; the actor handles the message on a later pass of its worker.
|
||||||
|
- **Boundary rules:**
|
||||||
|
- Actor handlers are synchronous and single-writer. They never `.await`.
|
||||||
|
- `&mut Worker` and any actor state is live only during a synchronous `pass`; it is never held across a wait and never sent into a task.
|
||||||
|
- All `.await` lives in tasks. Tasks never touch actor state directly; they communicate only via the deliver handle and the inbox.
|
||||||
|
- The inbox a task delivers into is the same FIFO, loss-free, unbounded queue the worker waits on. Mailbox ordering semantics (per-(sender,target) FIFO) are the actor-worker's concern; the engine's only obligation is that the inbox itself is FIFO and loss-free.
|
||||||
|
|
||||||
|
## 8. Invariants
|
||||||
|
|
||||||
|
An engine must uphold:
|
||||||
|
|
||||||
|
- **Non-reentrant passes.** At most one `pass` per worker at any instant.
|
||||||
|
- **Loss-free, non-blocking delivery.** The inbox never drops and never blocks the sender (unbounded).
|
||||||
|
- **FIFO inbox.** Messages depart an inbox in deposit order.
|
||||||
|
- **Progress independence.** A long-running or blocked task must not stall worker passes, and vice versa. The engine provides enough concurrency that workers and tasks progress independently (on a cooperative single-thread host like JS, this is a discipline the engine enforces: no blocking calls in tasks or passes).
|
||||||
|
- **Actors never await.** No `.await` reaches actor code; the engine owns every wait.
|
||||||
|
|
||||||
|
## 9. Reference instantiations (non-normative)
|
||||||
|
|
||||||
|
Illustrations of how each environment satisfies the contract — not prescription.
|
||||||
|
|
||||||
|
- **tokio.** Workers and tasks are tokio tasks; a worker loop is `loop { inbox.recv().await; pass(); }` with `&mut Worker` live only across the synchronous `pass` (long passes may be moved to `spawn_blocking`; that scheduling choice is the engine's, per §5). Async I/O, `spawn_blocking`, and `now()` are tokio's. This is today's de-facto engine, made explicit.
|
||||||
|
- **std-thread.** Each worker is an OS thread blocking on its inbox; tasks are OS threads or a small pool; `spawn_blocking` is a thread; there is no async I/O, only blocking I/O. Simple, unperformant, dependency-free — and a valid engine.
|
||||||
|
- **deterministic test engine.** A single-threaded stepping scheduler: workers and tasks are entries the test advances manually; `now()` is virtual time advanced by the test; async I/O is faked or mocked. It implements the same contract, so crates test against the interface with no real network and no threads, fully deterministic. It falls out of the contract; it is not specified separately.
|
||||||
|
- **Go / JS-worker (illustrative).** Workers and tasks map to goroutines + channels, or to the JS event loop + `postMessage` / callbacks. Each provides the capability subset its runtime supports.
|
||||||
|
|
||||||
|
## 10. What this spec does not define
|
||||||
|
|
||||||
|
The boundary, stated plainly:
|
||||||
|
|
||||||
|
- Actor execution semantics (single-writer, FIFO, fairness, panic isolation).
|
||||||
|
- Backpressure.
|
||||||
|
- Cancellation and shutdown.
|
||||||
|
- Failure / observability propagation beyond the bridge.
|
||||||
|
- Cross-process / cross-isolation delivery and serialization.
|
||||||
|
- Specific protocols and codecs.
|
||||||
|
|
@ -1,6 +1,8 @@
|
||||||
# Iroh Driver Fixed Specification
|
# Iroh Driver Fixed Specification
|
||||||
|
|
||||||
**Status:** draft current-state specification for `iroh-driver`.
|
Id: 5
|
||||||
|
Last modified:
|
||||||
|
Last reviewed:
|
||||||
|
|
||||||
> Review checkpoint: reviewed through Section 3.2; resume with Section 3.3 Accepted Connection Output.
|
> Review checkpoint: reviewed through Section 3.2; resume with Section 3.3 Accepted Connection Output.
|
||||||
|
|
||||||
146
docs/specs/drafts/JOB_RUNNER_SPEC.md
Normal file
146
docs/specs/drafts/JOB_RUNNER_SPEC.md
Normal file
|
|
@ -0,0 +1,146 @@
|
||||||
|
# Synaptic Job Runner Specification
|
||||||
|
|
||||||
|
Id: 4
|
||||||
|
Last modified:
|
||||||
|
Last reviewed:
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Purpose
|
||||||
|
|
||||||
|
Run an arbitrary command job on a rented GPU node, end to end, through swactor:
|
||||||
|
provision a node → push the workspace → setup → run → collect outputs →
|
||||||
|
teardown. swactor is the substrate: provisioning, the node runtime, the data
|
||||||
|
plane, and observability are reused; the job runner is built on top of them.
|
||||||
|
|
||||||
|
v1 is the single-node command job: one job → one node → one attempt. Multi-node,
|
||||||
|
sharded, and pipeline jobs are future work-descriptions over the same core.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. What swactor grows, what swactor reuses
|
||||||
|
|
||||||
|
**New — the job runner's content:**
|
||||||
|
|
||||||
|
- a node primitive that runs an arbitrary command as a supervised process and
|
||||||
|
reports exit via datastream;
|
||||||
|
- a bulk-transfer service on the shared iroh endpoint for workspace push
|
||||||
|
(operator → node) and output pull (node → operator) — a new service
|
||||||
|
paralleling the existing edge and datastream ALPN services, not the
|
||||||
|
arena/tensor data plane;
|
||||||
|
- a job FSM and the orchestrator↔node protocol that sequence the above.
|
||||||
|
|
||||||
|
**Reused unchanged:** the VastAI provider adapter (provision/teardown), SWIM
|
||||||
|
membership, the datastream event infrastructure, the shared iroh endpoint (which
|
||||||
|
already multiplexes the edge and datastream services), and the swactor actor
|
||||||
|
runtime.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Design principle: job = work, gpu-agnostic
|
||||||
|
|
||||||
|
A job describes *what to do*, not what hardware to run on — no provider, no GPU,
|
||||||
|
no node. Hardware (provider, GPU type/count/VRAM, disk, image, selection) lives
|
||||||
|
in a *pool designation* owned by the provider adapter. This keeps a future
|
||||||
|
sharded-model job the same shape against different placement; sharding is not a
|
||||||
|
v1 feature.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Job description
|
||||||
|
|
||||||
|
| field | required | meaning |
|
||||||
|
|-------------|----------|------------------------------------------------------|
|
||||||
|
| `name` | yes | job identity |
|
||||||
|
| `run` | yes | the command that does the work |
|
||||||
|
| `setup` | no | one-time command run before `run` (env install) |
|
||||||
|
| `workspace` | no | `{ workdir, exclude }` — pushed to the node first |
|
||||||
|
| `outputs` | no | paths to collect back after run |
|
||||||
|
| `env` | no | environment variables injected into setup and run |
|
||||||
|
|
||||||
|
Nothing else. No resources, no provider, no GPU, no cleanup policy.
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[job]
|
||||||
|
name = "airfrans_smoke_01"
|
||||||
|
setup = "uv sync --no-dev"
|
||||||
|
run = "uv run remote-run smoke-train configs/aggressive_smoke.toml"
|
||||||
|
workspace = { workdir = ".", exclude = ["/artifacts", "/.venv", "__pycache__"] }
|
||||||
|
outputs = ["metrics.jsonl", "final_metrics.json", "checkpoint_latest.pt"]
|
||||||
|
env = { HF_TOKEN = "..." }
|
||||||
|
```
|
||||||
|
|
||||||
|
A job is submitted with a *pool designation* (capacity) that drives provisioning
|
||||||
|
via the existing VastAI adapter. The pool is not part of the job and is not
|
||||||
|
specified further here.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Roles
|
||||||
|
|
||||||
|
- **Orchestrator (job authority):** places the job, drives the lifecycle, owns
|
||||||
|
the terminal outcome, orders teardown.
|
||||||
|
- **Node (executor):** runs `setup` and `run` as supervised processes,
|
||||||
|
materializes the workspace, exposes outputs, reports lifecycle via datastream.
|
||||||
|
- **Provider adapter:** provisions and tears down nodes per the pool designation.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Lifecycle
|
||||||
|
|
||||||
|
States: `PENDING → RUNNING → COMPLETED | FAILED`.
|
||||||
|
|
||||||
|
| event observed | command emitted | transition |
|
||||||
|
|-----------------------------------|------------------------------------------|---------------|
|
||||||
|
| `JobSubmitted{job, pool}` | `ProvisionNode` | → `PENDING` |
|
||||||
|
| `NodeReady` | `MaterializeWorkspace` | → `RUNNING` |
|
||||||
|
| `WorkspaceMaterialized` | `RunSetup` | |
|
||||||
|
| `SetupCompleted` | `RunJob` | |
|
||||||
|
| `JobExited{0}` | `CollectOutputs` | |
|
||||||
|
| `OutputsCollected` | `TeardownNode` | → `COMPLETED` |
|
||||||
|
| `JobExited{non-zero}` | `CollectOutputs` (best-effort), `TeardownNode` | → `FAILED` |
|
||||||
|
| `NodeFault` / `NodeLost` / `OperatorStop` | `TeardownNode` | → `FAILED` |
|
||||||
|
|
||||||
|
- A job waits for *its one node's* readiness, not pool-wide convergence.
|
||||||
|
- No `setup` → `WorkspaceMaterialized` goes straight to `RunJob`. No
|
||||||
|
`workspace` → materialization is skipped.
|
||||||
|
- The exit code is authoritative: `0` → `COMPLETED`, non-zero → `FAILED`. Declared
|
||||||
|
outputs are collected either way; collection on failure is best-effort.
|
||||||
|
- The node is torn down on every terminal state (completed or failed).
|
||||||
|
- Every transition emits a datastream event.
|
||||||
|
|
||||||
|
`OperatorStop` is the operator kill switch for a running job.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Behavior contracts
|
||||||
|
|
||||||
|
The orchestrator commands these contracts; the node owns the local mechanism.
|
||||||
|
Control and lifecycle travel as orchestrator↔node messages and datastream
|
||||||
|
events. Bulk workspace and output bytes travel over a dedicated transfer
|
||||||
|
service on the shared iroh endpoint, not in actor messages.
|
||||||
|
|
||||||
|
**Supervised command execution (node).** On `RunSetup` / `RunJob`, the node
|
||||||
|
spawns the command in `workdir` with the declared `env`, supervises it, and
|
||||||
|
reports `SetupCompleted` / `JobExited{code}` (or `NodeFault`) via datastream. A
|
||||||
|
failed setup is a failed job.
|
||||||
|
|
||||||
|
**Workspace materialization (operator → node).** On `MaterializeWorkspace`, the
|
||||||
|
`workdir` tree (with `exclude` applied) is pushed from the operator to the node
|
||||||
|
over the iroh transfer service, landing at the node's `workdir`. The node reports
|
||||||
|
`WorkspaceMaterialized`.
|
||||||
|
|
||||||
|
**Output collection (node → operator).** On `CollectOutputs`, declared `outputs`
|
||||||
|
are pulled from the node to a per-run operator landing directory over the iroh
|
||||||
|
transfer service. The node reports `OutputsCollected`. Missing outputs do not change
|
||||||
|
the outcome — the exit code already decided it; whatever exists is gathered.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. Relationship to the broader work model
|
||||||
|
|
||||||
|
This spec defines only the job plugin. The intended generalization is a *work
|
||||||
|
description*: a graph of work-units with typed ports, placed by a planner. The v1
|
||||||
|
job resolves to a single work-unit with no edges; multi-unit, sharded, and
|
||||||
|
pipeline jobs are future work-descriptions over the same core. The gpu-agnostic
|
||||||
|
job shape is what keeps that path open.
|
||||||
145
docs/specs/drafts/MULTICORE_SPEC.md
Normal file
145
docs/specs/drafts/MULTICORE_SPEC.md
Normal file
|
|
@ -0,0 +1,145 @@
|
||||||
|
# swactor process-local multicore runtime — specification
|
||||||
|
|
||||||
|
Id: 2
|
||||||
|
Last modified:
|
||||||
|
Last reviewed:
|
||||||
|
|
||||||
|
**Scope:** multicore (multi-worker) execution and message delivery within a single process (shared address space).
|
||||||
|
|
||||||
|
## 1. Scope
|
||||||
|
|
||||||
|
**In scope**
|
||||||
|
|
||||||
|
- How N workers run concurrently on N cores within one process.
|
||||||
|
- How a message is routed and delivered between actors on different workers (foreign-thread, shared memory).
|
||||||
|
- How a message is delivered between actors on the same worker.
|
||||||
|
- The seam between core and the hosting engine.
|
||||||
|
|
||||||
|
**Out of scope (deferred to separate specs)**
|
||||||
|
|
||||||
|
- Cross-isolation delivery (JS web workers) — no shared heap; requires serialization.
|
||||||
|
- Cross-process / WAN delivery — owned by the `transport` and `distribution` crates.
|
||||||
|
- Actor migration, work-stealing, and load balancing beyond spawn-time worker selection.
|
||||||
|
- A ready-queue / runnable-set optimization.
|
||||||
|
|
||||||
|
**Constraint.** Core (`src/`) stays free of any specific engine: no tokio dependency, no owned thread pool. Any engine that can host a blocking or async receiver can host a worker.
|
||||||
|
|
||||||
|
## 2. Model
|
||||||
|
|
||||||
|
- A **worker** owns a disjoint set of actors and processes them one at a time. It is the unit of parallelism: N workers on N cores run up to N actors concurrently.
|
||||||
|
- An actor is **pinned**: assigned to one worker at spawn, never moved. Worker selection at spawn is deliberately simple: an actor spawned from within a worker (`ctx.spawn`) pins to that same worker; an actor spawned from outside the runtime (via the runtime handle) is assigned round-robin across workers. A side effect is that a parent and the children it talks to stay co-located, so their traffic stays on the same-worker fast path.
|
||||||
|
- Multicore parallelizes *different actors*. One actor's work is never split across cores. This preserves the single-writer invariant: at most one message is handled per actor at any instant, across all workers.
|
||||||
|
- Workers are **autonomous and independent**: each runs its own loop. There is no global tick, no barrier, no per-step cross-worker synchronization.
|
||||||
|
- Workers are **reactive**: when idle they wait; when work arrives they run a pass.
|
||||||
|
|
||||||
|
## 3. Responsibilities (the core / engine seam)
|
||||||
|
|
||||||
|
- **Runtime** (core): owns the actor address space, the `address → worker` routing map, the per-worker inbox deposit handles, and spawn-time worker selection (same-worker for in-runtime spawns, round-robin for external spawns). It routes. It does not execute and does not own threads.
|
||||||
|
- **Worker** (core logic, engine-driven): owns its pinned actor pool. Each pass drains its inbox into actor mailboxes and processes non-empty mailboxes up to a fairness budget.
|
||||||
|
- **Engine** (integrator-supplied — std::thread, tokio, …): decides how many workers to create, hosts each worker's loop, and owns the inbox's consumer side (how the worker idles and how often it drains). **Core only transitions. The engine drives.**
|
||||||
|
|
||||||
|
## 4. Delivery regimes (this spec)
|
||||||
|
|
||||||
|
| target lives on… | delivery |
|
||||||
|
| ------------------------------------ | ------------------------------------------------- |
|
||||||
|
| the same worker | inline, within the current pass (no queue) |
|
||||||
|
| another worker, same process | pointer-move into that worker's MPSC inbox |
|
||||||
|
| another process / isolated / remote | out of scope — `transport` / `distribution` |
|
||||||
|
|
||||||
|
## 5. Routing and delivery mechanism
|
||||||
|
|
||||||
|
A send resolves the target actor to its owning worker and deposits the message. There is **no wake step**. The receiving worker idles on its own inbox, so depositing into it is what makes the next transition runnable.
|
||||||
|
|
||||||
|
### 5.1 Send path
|
||||||
|
|
||||||
|
For a send of `M` to `addr` from any in-process sender (an actor handler, or an external thread holding a sender handle):
|
||||||
|
|
||||||
|
1. Box `M` once → `Box<dyn Any + Send>` (a heap pointer). The payload is never copied again.
|
||||||
|
2. Look up `addr` in the routing map → `WorkerId`.
|
||||||
|
3. Branch:
|
||||||
|
- **Same worker** as sender → append `(addr, M)` to the worker's local `pending_local` buffer. Delivered within the current pass. No queue, no cross-thread.
|
||||||
|
- **Different worker** → wrap as `Envelope { dest: addr, payload: M }` and deposit into that worker's inbox (a pointer-move into shared memory). Return. No signal is sent to the receiver.
|
||||||
|
- **Not in the map** → defer to the non-local seam (`transport` / `distribution`). Out of scope here.
|
||||||
|
|
||||||
|
### 5.2 The inbox
|
||||||
|
|
||||||
|
- One MPSC queue per worker. Many producers (any foreign thread); one consumer (the owning worker).
|
||||||
|
- **Producer side** (the deposit): non-blocking, unbounded, loss-free, FIFO. Core holds this handle per worker, indexed by `WorkerId`.
|
||||||
|
- **Consumer side** (the worker's idle point): engine-chosen. A blocking channel under std::thread; an async channel under tokio. Receiving *is* the idle point, so depositing makes the next transition runnable with no separate wake primitive. The engine owns this side and the drain cadence.
|
||||||
|
|
||||||
|
### 5.3 The crossing
|
||||||
|
|
||||||
|
The message crosses the thread boundary exactly once, inside the inbox queue. The producer writes a pointer into a slot in shared memory; the consumer, blocked or awaiting on that queue, returns it. No serialization, no copy of the payload, no inter-thread signal beyond the queue's own readiness.
|
||||||
|
|
||||||
|
## 6. Guarantees
|
||||||
|
|
||||||
|
- **Single-writer.** At most one message handled per actor at any instant, across all workers.
|
||||||
|
- **Per-(sender, target) FIFO.** Messages from one sender to one target are delivered in send order. Cross-sender ordering to the same target is not guaranteed.
|
||||||
|
- **Loss-free / non-blocking producer.** The inbox never drops and never blocks the sender (unbounded). Mailboxes likewise.
|
||||||
|
- **Fairness.** No actor processes more than `budget` messages per pass, so one actor cannot starve the others on its worker.
|
||||||
|
- **Panic isolation.** A panicking actor is poisoned and skipped; it does not take down its worker or other actors. (Existing behavior, retained.)
|
||||||
|
|
||||||
|
## 7. Data structures
|
||||||
|
|
||||||
|
**Runtime-wide (shared, read-mostly)**
|
||||||
|
|
||||||
|
- `address_map`: `RwLock<HashMap<ActorAddress, WorkerId, identity-hash>>` — the routing table; read on send, written at spawn.
|
||||||
|
- `inbox_txs`: per-worker inbox deposit handles, indexed by `WorkerId`.
|
||||||
|
- `rr_worker`: `AtomicUsize` round-robin counter, used only for external (out-of-runtime) spawns. In-runtime spawns (`ctx.spawn`) need no counter — the child pins to the caller's worker.
|
||||||
|
|
||||||
|
**Per-worker inbox (cross-thread)**
|
||||||
|
|
||||||
|
- MPSC queue. Producer = deposit (pointer-move; lock-free ring + overflow). Consumer = the worker's wait point (engine-typed).
|
||||||
|
|
||||||
|
**Per-worker, worker-local (single-threaded)**
|
||||||
|
|
||||||
|
- `pool`: `HashMap<ActorAddress, ActorSlot>`.
|
||||||
|
- `ActorSlot { mailbox: VecDeque<Box<dyn Any + Send>>, actor, lifecycle flags }`.
|
||||||
|
- `pending_local`: `Vec<(ActorAddress, Box<dyn Any + Send>)>` — same-worker buffer.
|
||||||
|
|
||||||
|
**Envelope**: `{ dest: ActorAddress, payload: Box<dyn Any + Send> }`.
|
||||||
|
|
||||||
|
## 9. Worked example
|
||||||
|
|
||||||
|
**Note on `WorkerId` indexing.** `WorkerId` is an opaque internal newtype — minted only by the runtime at spawn and used only to index that same runtime's own `inbox_txs` / `spawn_txs` slices. It never crosses the public API as a raw index, so misuse is bounded to internal code. The per-message cost on the cross-worker path is the routing-map lookup (§11 defers eliminating it via address-encoded routing), not the slice index that follows — the latter is a single pointer-add. The fast path is the same-worker arm (`pending_local`), which bypasses the inbox, the `Envelope`, and the second thread entirely.
|
||||||
|
|
||||||
|
X on worker 0 (thread T0) sends `M` to `addr`, which is Y on worker 1 (thread T1):
|
||||||
|
|
||||||
|
1. `ctx.send(addr, M)` → `Box::new(M)` (one allocation).
|
||||||
|
2. `send_any`: `address_map.lookup(addr)` → worker 1; not self → `inbox_txs[1].send(Envelope { addr, M })`. Pointer into worker 1's inbox ring. Return. No signal.
|
||||||
|
3. T1 was blocked on `inbox.recv()`; the deposit unblocks it and returns the `Envelope`.
|
||||||
|
4. T1 drains: `pool[addr].mailbox.push_back(M)`.
|
||||||
|
5. Pass walks the pool, finds Y's mailbox non-empty, pops, `Y.handle(ctx, M)`.
|
||||||
|
|
||||||
|
X learned nothing about threads. The only thread-aware steps were the one map read and the queue the pointer sat in.
|
||||||
|
|
||||||
|
Had `addr` been on worker 0: step 2 takes the same-worker arm, `M` goes to `pending_local`, and Y handles it later in this same pass — no `Envelope`, no ring, no second thread.
|
||||||
|
|
||||||
|
## 10. Changes vs current `src/`
|
||||||
|
|
||||||
|
**Removed**
|
||||||
|
|
||||||
|
- `Runtime::run()` spawning owned OS threads.
|
||||||
|
- `thread::park()` / `thread::unpark()` wakeup.
|
||||||
|
- `notify_worker()` and the `worker_threads: Vec<OnceLock<Thread>>` plumbing (including inside `ExternalSender`).
|
||||||
|
- `Placement` (the load-aware selector) and its `WorkerStats`-driven `next_worker()` scan; replaced by same-worker pinning for in-runtime spawns and a single round-robin counter (`rr_worker`) for external spawns.
|
||||||
|
|
||||||
|
**Changed**
|
||||||
|
|
||||||
|
- The per-worker transfer queue becomes the worker **inbox**, and its consumer side becomes the worker's idle point (engine-supplied). Deposit no longer signals the engine.
|
||||||
|
|
||||||
|
**Retained unchanged**
|
||||||
|
|
||||||
|
- `tick()` / `try_tick()` inline all-workers mode (deterministic, wasm, tests).
|
||||||
|
- `ActorPool`, `ActorSlot`, mailboxes, `pending_local`, budget, panic isolation, `ExternalSender` / `Inbox` / `Ask` (minus the removed wake).
|
||||||
|
|
||||||
|
**Added**
|
||||||
|
|
||||||
|
- The engine seam: a way for an integrator to create and register workers, supply each worker's inbox consumer and wait, and drive each worker's loop. Exact API is defined per engine in follow-on integration notes.
|
||||||
|
|
||||||
|
## 11. Deferred
|
||||||
|
|
||||||
|
- Actor migration, work-stealing, and load balancing beyond spawn-time worker selection.
|
||||||
|
- Ready-queue optimization.
|
||||||
|
- Cross-isolation delivery (web workers) and cross-process / WAN delivery (`transport`, `distribution`).
|
||||||
|
- Address-encoded worker routing (eliminating the routing-map lookup).
|
||||||
303
docs/specs/drafts/RECONCILER_SPEC.md
Normal file
303
docs/specs/drafts/RECONCILER_SPEC.md
Normal file
|
|
@ -0,0 +1,303 @@
|
||||||
|
# cluster reconciler — specification
|
||||||
|
|
||||||
|
Id: 3
|
||||||
|
Last modified:
|
||||||
|
Last reviewed:
|
||||||
|
|
||||||
|
**Scope:** a level-triggered reconciler that drives a declared cluster shape toward
|
||||||
|
convergence over the existing node lifecycle, living in `crates/provisioning`
|
||||||
|
alongside `NodeManager`.
|
||||||
|
|
||||||
|
## 1. Purpose
|
||||||
|
|
||||||
|
Today, node lifecycle is **edge-triggered and imperative**: `NodeManager` reacts to
|
||||||
|
discrete events (`LeaseCreated`, `EndpointKnown`, `BootstrapObserved`…) and emits
|
||||||
|
commands, and `apps/myelin`'s orchestration module drives those managers by hand,
|
||||||
|
deciding *when* to start, retry, and tear down each node. There is no object that
|
||||||
|
owns "the cluster should look like *this*." Scaling, replacement, and recovery are
|
||||||
|
woven into imperative workflow code.
|
||||||
|
|
||||||
|
This spec introduces a **reconciler**: a pure, level-triggered function that, given
|
||||||
|
a desired cluster shape and the currently observed cluster state, emits the effects
|
||||||
|
that move observed → desired. A **driver** calls it repeatedly (periodically and on
|
||||||
|
events), an **executor** applies the effects against providers, and observations
|
||||||
|
fold back into state for the next pass. The system converges; it does not execute a
|
||||||
|
script.
|
||||||
|
|
||||||
|
The mindset shift: **edge-triggered imperative workflow → level-triggered
|
||||||
|
declarative convergence.** The reconciler never asks "what event just happened?" It
|
||||||
|
asks "given where this node is and where it must be, what is the next step?"
|
||||||
|
|
||||||
|
## 2. Scope
|
||||||
|
|
||||||
|
**In scope**
|
||||||
|
|
||||||
|
- The reconciler contract: desired state, observed state, the pure `reconcile`
|
||||||
|
function, and the effect vocabulary.
|
||||||
|
- Driving semantics: periodic + event triggers, one-step-per-pass convergence,
|
||||||
|
idempotency, failure/backoff.
|
||||||
|
- Cluster-topology reconciliation: scale up/down across node groups, replacement on
|
||||||
|
shape change.
|
||||||
|
- The responsibility split between the reconciler (pure decider), the driver
|
||||||
|
(stateful owner of the node fleet), and the executor (applies effects to
|
||||||
|
providers).
|
||||||
|
- How the reconciler maps onto the existing `NodeManager` / `NodeStage` state
|
||||||
|
machine without inventing a parallel lifecycle.
|
||||||
|
|
||||||
|
**Out of scope**
|
||||||
|
|
||||||
|
- Actor workload placement on reconciled nodes (future scope; topology only for v1).
|
||||||
|
- Data-plane / connectivity reconciliation (iroh mesh, datastream links) as part of
|
||||||
|
the shape (future scope).
|
||||||
|
- Specific provider adapters (Vast.ai, Docker). Those implement the existing
|
||||||
|
`ProviderPlugin` / `ProvisionPlugin` executor seams.
|
||||||
|
- Backpressure and admission policy across the whole cluster.
|
||||||
|
- Persistence and leader election (single driver instance assumed for v1).
|
||||||
|
|
||||||
|
## 3. Model
|
||||||
|
|
||||||
|
Three roles, one invariant.
|
||||||
|
|
||||||
|
- **Reconciler** — a pure, deterministic function
|
||||||
|
`reconcile(observed, desired) → plan`. No I/O, no clocks beyond an injected
|
||||||
|
`now()`, no mutation of inputs. Given identical inputs it yields an identical
|
||||||
|
plan. This is the load-bearing seam: everything testable and provider-neutral
|
||||||
|
lives here.
|
||||||
|
- **Driver** — the stateful loop. It owns the fleet of per-node state machines
|
||||||
|
(today: a `NodeManager` per logical node), calls the reconciler each pass,
|
||||||
|
dispatches the plan to the executor, folds provider/bootstrap observations back
|
||||||
|
into observed state, and decides *when* to run (periodic tick + event triggers).
|
||||||
|
The driver is the only writer of observed state.
|
||||||
|
- **Executor** — applies effects against reality. Maps to the existing seams:
|
||||||
|
`ProviderPlugin` (lease lifecycle: create / lookup / destroy) and
|
||||||
|
`ProvisionPlugin` (node process) plus bootstrap sessions. `apps/myelin`'s
|
||||||
|
orchestration module becomes this layer.
|
||||||
|
|
||||||
|
**Invariant — the reconciler is pure; the driver owns all state and waiting.**
|
||||||
|
This mirrors the engine split (ENGINE_SPEC §3): core never waits, the engine owns
|
||||||
|
all waiting. Here, the reconciler never waits or mutates; the driver owns the
|
||||||
|
node fleet, the clocks, and the retry timers.
|
||||||
|
|
||||||
|
## 4. State
|
||||||
|
|
||||||
|
### Desired state
|
||||||
|
|
||||||
|
Authoritative, supplied by the caller, held immutably between shape edits:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
// already defined in node.rs — unchanged
|
||||||
|
pub struct RunNodeGroupSpec { /* run_id, group_id, role, count, provider, shape, boot, swarm_join */ }
|
||||||
|
pub fn expand_node_group(group: &RunNodeGroupSpec) -> Vec<LogicalNodeSpec>;
|
||||||
|
|
||||||
|
// new
|
||||||
|
pub struct ClusterShape {
|
||||||
|
pub run_id: RunId,
|
||||||
|
pub groups: Vec<RunNodeGroupSpec>,
|
||||||
|
}
|
||||||
|
impl ClusterShape {
|
||||||
|
pub fn expand(&self) -> Vec<LogicalNodeSpec> { /* flatMap expand_node_group */ }
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`ClusterShape` is a thin bag over the existing group spec; `expand` reuses
|
||||||
|
`expand_node_group`. The expanded `Vec<LogicalNodeSpec>` is the set of logical nodes
|
||||||
|
the cluster **should** contain.
|
||||||
|
|
||||||
|
### Observed state
|
||||||
|
|
||||||
|
The set of logical nodes the cluster **does** contain, each with its lifecycle
|
||||||
|
facts. `NodeRecord` already is per-node observed state. The cluster wraps it:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
// NodeRecord already holds: desired, stage, ready, lease, connection, bootstrap,
|
||||||
|
// swactor, failed_reason, destroyed_at.
|
||||||
|
|
||||||
|
pub struct ClusterState {
|
||||||
|
pub nodes: BTreeMap<LogicalNodeId, NodeRecord>,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The driver is the sole writer of `ClusterState`. Observations (lease results,
|
||||||
|
endpoints, bootstrap progress, swactor joins, failures) are folded into `NodeRecord`
|
||||||
|
between passes — exactly the work `NodeManager::handle` already does internally;
|
||||||
|
under the reconciler that folding is the driver's job (see §7).
|
||||||
|
|
||||||
|
## 5. The reconciler contract
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub struct ReconcilePlan {
|
||||||
|
/// Desired nodes with no observed record: begin their lifecycle.
|
||||||
|
pub to_start: Vec<LogicalNodeSpec>,
|
||||||
|
/// Observed nodes with no desired entry: tear them down.
|
||||||
|
pub to_destroy: Vec<LogicalNodeId>,
|
||||||
|
/// Live nodes: the next one or more commands to advance each toward desired.
|
||||||
|
pub per_node: Vec<(LogicalNodeId, Vec<NodeManagerCommand>)>,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Pure. Deterministic. No I/O.
|
||||||
|
pub fn reconcile(
|
||||||
|
observed: &ClusterState,
|
||||||
|
desired: &ClusterShape,
|
||||||
|
now: SystemTime,
|
||||||
|
) -> ReconcilePlan;
|
||||||
|
```
|
||||||
|
|
||||||
|
`reconcile` is **level-triggered**: it reads only `observed` + `desired` (+ `now`
|
||||||
|
for backoff; see §8). It does not know which event triggered the pass. It is
|
||||||
|
**idempotent**: re-running with unchanged inputs yields the same plan, and an effect
|
||||||
|
whose result is already reflected in observed state is never re-emitted (e.g. a node
|
||||||
|
whose `record.lease` is `Some` never yields `CreateLease` again).
|
||||||
|
|
||||||
|
### Per-node reconcile
|
||||||
|
|
||||||
|
For each live node, `reconcile` computes the next step from `(NodeRecord,
|
||||||
|
LogicalNodeSpec)` — a pure function over the existing `NodeStage` machine:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
/// One pass = at most one lifecycle step per node. Convergence happens across
|
||||||
|
/// passes, not within one.
|
||||||
|
fn reconcile_node(record: &NodeRecord, desired: &LogicalNodeSpec, now: SystemTime)
|
||||||
|
-> Vec<NodeManagerCommand>;
|
||||||
|
```
|
||||||
|
|
||||||
|
The mapping reuses the existing `NodeManagerCommand` vocabulary and the existing
|
||||||
|
`NodeStage` transitions — it is the **level** reading of the same state machine that
|
||||||
|
`NodeManager::handle` expresses in **edge** form:
|
||||||
|
|
||||||
|
| observed (`record`) | next effect(s) |
|
||||||
|
|---|---|
|
||||||
|
| `New`, no lease | `CreateLease` |
|
||||||
|
| `LeaseCreated`, lease carries endpoint | `StartBootstrap` |
|
||||||
|
| `LeaseCreated`, endpoint unknown | `LookupEndpoint` |
|
||||||
|
| `EndpointKnown` / `BootstrapRunning` | (none — awaiting bootstrap observation) |
|
||||||
|
| `BootstrapRunning`, swactor joined | `BootstrapConvergenceObserved` |
|
||||||
|
| `HandedOff` / `Dormant`, `ready` | (none — steady state) |
|
||||||
|
| `Failed`, within backoff window | (none — waiting; see §8) |
|
||||||
|
| `Failed`, backoff elapsed | reset to `New` → `CreateLease` (retry) |
|
||||||
|
| destroy requested | `CancelBootstrap` (if active) + `DestroyLease` |
|
||||||
|
|
||||||
|
**One step per node per pass.** This is the heart of the level-triggered model:
|
||||||
|
the reconciler never waits within a pass. It emits the step the current observed
|
||||||
|
state permits; the executor applies it; observation updates the record; the next
|
||||||
|
pass emits the next step. Ordering across the lease → bootstrap → join chain falls
|
||||||
|
out of convergence, not from an explicit workflow.
|
||||||
|
|
||||||
|
## 6. Topology reconciliation
|
||||||
|
|
||||||
|
`reconcile` first diffs the expanded desired set against the observed set by
|
||||||
|
`LogicalNodeId`:
|
||||||
|
|
||||||
|
- **desired, not observed** → `to_start`. The driver instantiates a `NodeManager`
|
||||||
|
for the spec; the first pass emits `CreateLease`.
|
||||||
|
- **observed, not desired** → `to_destroy`. The driver runs the destroy path
|
||||||
|
(`CancelBootstrap` + `DestroyLease`); once `stage == Destroyed` the record is
|
||||||
|
reaped.
|
||||||
|
- **both** → `per_node` via `reconcile_node`.
|
||||||
|
|
||||||
|
**Scale policy (v1, deliberately simple):** logical node identity is
|
||||||
|
`{group_id}-{index}`. Scaling a group up adds higher indices; scaling down removes
|
||||||
|
the **highest** indices first. A group's `shape` is treated as immutable per node:
|
||||||
|
changing a field that is not achievable in place (image, gpu, disk) is a
|
||||||
|
**replacement** — the affected logical nodes move to `to_destroy` and fresh specs to
|
||||||
|
`to_start` — not an in-place mutation. This is k8s-style immutable-spec rolling
|
||||||
|
replacement, kept coarse for v1.
|
||||||
|
|
||||||
|
## 7. Driving semantics
|
||||||
|
|
||||||
|
The driver runs a pass when **either** (a) a periodic tick fires, or (b) an event
|
||||||
|
arrives — a desired-shape edit, or a provider/bootstrap observation that changed
|
||||||
|
observed state. Each pass:
|
||||||
|
|
||||||
|
1. Snapshot current `ClusterState` and `ClusterShape`.
|
||||||
|
2. Call `reconcile(&observed, &desired, now)` → `ReconcilePlan`.
|
||||||
|
3. Apply the plan: create `NodeManager`s for `to_start`, drive destroy for
|
||||||
|
`to_destroy`, dispatch each `per_node` command to the executor.
|
||||||
|
4. Fold executor results + pending observations into `NodeRecord`s (the driver's
|
||||||
|
only write).
|
||||||
|
5. Repeat. Terminal when every desired node is `ready` and no orphans remain.
|
||||||
|
|
||||||
|
**Observation folding is the bridge from edge to level.** Provider/bootstrap events
|
||||||
|
arrive as the existing observation types (`CreateLeaseResult`, `SshEndpoint`,
|
||||||
|
`BootstrapObservation`, swactor-join, failures). The driver folds each into the
|
||||||
|
node's `NodeRecord` between passes — the same field updates `NodeManager::handle`
|
||||||
|
performs today (`record.lease = …`, `record.bootstrap.last_stage = …`, etc.). The
|
||||||
|
reconciler then reads the updated record and emits the next step. The edge-triggered
|
||||||
|
`NodeManager::handle` and the level-triggered `reconcile_node` are two readings of
|
||||||
|
one state machine; see §10 for the migration choice.
|
||||||
|
|
||||||
|
**Non-reentrancy.** A pass is synchronous and exclusive: the driver never runs two
|
||||||
|
passes concurrently. This matches the worker non-reentrancy invariant (ENGINE_SPEC
|
||||||
|
§5).
|
||||||
|
|
||||||
|
## 8. Failure and backoff
|
||||||
|
|
||||||
|
Failure is **observed state**, not a control-flow signal. A node reaching
|
||||||
|
`NodeStage::Failed` records `failed_reason` and `failed_at`. The reconciler emits no
|
||||||
|
command for that node **until its per-node backoff window elapses** (hence `now` in
|
||||||
|
the signature); after the window it resets the node to `New` and re-emits
|
||||||
|
`CreateLease`. Backoff is **per-node and isolated** — one failed node never blocks
|
||||||
|
another (this is the pay-off of the hybrid granularity chosen in §9). Destroyed
|
||||||
|
nodes that were desired are simply re-started by the topology diff.
|
||||||
|
|
||||||
|
Backoff parameters (initial delay, cap, jitter) are driver configuration, not
|
||||||
|
reconciler logic — the reconciler only reads `failed_at` + the configured window
|
||||||
|
and decides "retry now" vs "wait." v1 uses a fixed window; exponential backoff is a
|
||||||
|
driver-side refinement.
|
||||||
|
|
||||||
|
## 9. Granularity and growth path
|
||||||
|
|
||||||
|
**Hybrid, by design.** v1 ships one top-level `reconcile` whose body is: topology
|
||||||
|
diff + fan-out to `reconcile_node`. The contract — pure function of (observed,
|
||||||
|
desired) → effects — is **identical at every level**, so the growth ladder is
|
||||||
|
internal refactor, never a contract change:
|
||||||
|
|
||||||
|
1. **Now** — one loop, topology diff + per-node reconcile. Simple.
|
||||||
|
2. **When shapes diversify** — fan out to sub-reconcilers per concern (node-groups,
|
||||||
|
roles, future: data-plane links), each with the same signature; the top level
|
||||||
|
merges their effect streams.
|
||||||
|
3. **If independent backoff/isolation/work-queues are ever needed** — promote a
|
||||||
|
sub-reconciler to its own loop + driver. Same contract; the migration is
|
||||||
|
mechanical.
|
||||||
|
|
||||||
|
`NodeManager` is already a sub-reconciler in waiting. The contract is what must not
|
||||||
|
ossify; the loop count is cheap to grow.
|
||||||
|
|
||||||
|
## 10. Relationship to existing code
|
||||||
|
|
||||||
|
| exists today | role under the reconciler |
|
||||||
|
|---|---|
|
||||||
|
| `NodeManager` + `NodeStage` | per-node observed state + lifecycle transitions. Kept. |
|
||||||
|
| `NodeRecord` | per-node observed state record. Kept; the driver writes it. |
|
||||||
|
| `NodeManagerCommand` | the effect vocabulary. Reused verbatim by `reconcile_node`. |
|
||||||
|
| `RunNodeGroupSpec` / `expand_node_group` / `LogicalNodeSpec` | desired state. Wrapped by `ClusterShape`; reused. |
|
||||||
|
| `ProviderPlugin` / `ProvisionPlugin` | executor seams. Unchanged; the driver calls them. |
|
||||||
|
| `apps/myelin` orchestration | becomes the driver + executor. Imperative workflow code is replaced by `reconcile` calls. |
|
||||||
|
|
||||||
|
**Open design choice — edge handle vs. level reconcile for `NodeManager`:**
|
||||||
|
`NodeManager::handle(msg) → Vec<NodeManagerCommand>` is edge-triggered; the
|
||||||
|
reconciler needs the level form. Two options, to be settled at implementation:
|
||||||
|
|
||||||
|
- **(A) Add `reconcile_node` alongside `handle`.** `handle` keeps folding streaming
|
||||||
|
observations into the record (the parts that genuinely need event semantics, e.g.
|
||||||
|
bootstrap seq numbers); `reconcile_node` reads the record and emits the next step.
|
||||||
|
Minimal churn; two readings of one machine coexist. *Recommended for v1.*
|
||||||
|
- **(B) Split into `observe(&mut record, obs)` + `reconcile_node(&record, desired)`**
|
||||||
|
and retire `handle`. One level-triggered path; more churn, cleaner end state.
|
||||||
|
|
||||||
|
Either way the *contract* in §5 is unchanged; only `NodeManager`'s internal shape
|
||||||
|
differs.
|
||||||
|
|
||||||
|
## 11. Invariants (normative)
|
||||||
|
|
||||||
|
1. **Purity.** `reconcile` performs no I/O, reads no global state, mutates no input.
|
||||||
|
`now` is its only non-input dependency.
|
||||||
|
2. **Level-triggered.** `reconcile` is a function of `(observed, desired, now)`,
|
||||||
|
never of "which event fired." Safe to call at any time.
|
||||||
|
3. **Idempotent.** Re-running with unchanged inputs yields the same plan; effects
|
||||||
|
already reflected in observed state are not re-emitted.
|
||||||
|
4. **One step per node per pass.** Convergence is across passes, not within one.
|
||||||
|
5. **Driver is the sole writer of observed state.** The reconciler and executor
|
||||||
|
never mutate `ClusterState`.
|
||||||
|
6. **Non-reentrant passes.** The driver never runs two passes concurrently.
|
||||||
|
7. **Failure isolation.** A failed node's backoff never blocks another node's
|
||||||
|
progress.
|
||||||
Loading…
Reference in a new issue