zacheryasc/swactor
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

draft spec stash

Browse source
This commit is contained in:
Zachery Aaron Shores-Chmielewski 2026-06-21 23:54:58 +04:00
parent 16f337e4ce
commit 91cc8a4b26
4 changed files with 3804 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

367
DESIGN_DIRECTIVES.md Normal file
View file

@ -0,0 +1,367 @@
# Design Directives
Captured design directives for the swactor data-movement / coordination-plane work.
These are the constraints and decisions as directed — not proposals. Resume from here.
## Problem & domain
- Building a **library for neural nets over heterogeneous, WAN-connected, churning hardware**.
- The thing being designed: how nodes **coordinate transfer/streaming of large blobs** — weights, activations, data — as a **reusable primitive**.
- **Nodes are trusted.** All trustless/P2P-style concerns (incentives, adversarial verification, Sybil, DHT discovery) are irrelevant.
- Focus is **ML workloads**, not general P2P.
## Scope discipline
- **Control plane first.** Performance (pre-alloc, buffering, chunked streams) is for later — *we don't know our bottlenecks yet*.
- The problem is **much smaller than what an object store solves** — don't over-engineer toward Ray/Plasma scope.
- Want **a solid set of abstractions that make designing distributed graphs over a WAN easy.**
## The library goal (ought) & lifecycle
- (For directional purposes only, we are only building one piece currently)
- Input: **a workload graph + a pool of hardware.** The library **distributes the work according to the graph across the hardware.**
- The meta lifecycle:
1. **define graph and pool in top-level code**
2. graph is **broken up per optimal placement** onto the extant resource pool
3. nodes **establish edges/networking**
4. nodes **fetch the weights/data they need**
5. **data flows end-to-end**
6. **teardown**, etc.
## Layers
- **Total compute graph** (e.g. inference over the transformer weights in a GGUF).
- **Roles created strategically** from **model shape + available hardware**.
- **Distribution of the model.**
- **Piping the data around.**
## Roles, graph, edges
- **Shape is owned by the graph, decided before roles are assigned.**
- A **role = a portion of the compute graph**: tensors flow in through the network boundary, get processed, new tensors flow back out.
- The graph is **known ahead of time** (pipelined inference). Nodes know ahead of time: they'll **send** activations, the **role they send to**, and that they'll **receive**.
- Each node knows: its **incoming tensors, outgoing tensors, and its own held weights.**
- The control plane specifies **size + shape + abstracted role endpoints** ("I am X sending Y to Z; I am Z receiving Y from X").
- **Graph-specifying code references a role abstractly** — not how the data reaches it or is received.
- Assume a **placement algorithm exists** (hand-defined at first, real later) — assignment/boundaries are already decided.
## swactor's mandate & boundaries
- **swactor is the runtime: the coordination plane for data movement and resource provisioning.**
- swactor actors are **strongly-typed FSMs passing messages**, to allow strict design of what the data looks like.
- The **CP is owned by an orchestrator** that: **runs and observes SWIM**, **handles resource provisioning**, and **stages execution on the nodes**.
- swactor actor paradigm = the **CP/dataflow layer**; **tinygrad = GPU execution.**
- **swactor does NOT handle the stream. iroh handles the streaming** — forget the nuts and bolts of the bytes on the wire.
- swactor is **not** doing flow control / credit / RTS-CTS — that solves a different problem, because **everything but the transport route is known ahead of time.**
## What's known vs. unknown
- **Known ahead of time:** the blob's **shape/dtype family** and its **`capacity`**
(the per-chunk byte ceiling, `max_seq_len × hidden × dtype`), plus the abstracted
endpoint (role).
- **Unknown at runtime:** the **transport** — who role X physically is, how to reach
it — *and* each chunk's actual **`extent`** (e.g. the prompt length: prefill
carries many rows, decode one), a per-step runtime fact `≤ capacity`.
- Runtime job: **resolve the transport problem** and bind the abstract role to a concrete route.
## swactor's concrete job (the gate)
- Assume over-the-wire streams are solved: **bytes travel fast and safe and arrive in a known buffer location.**
- **Receive side:** (zero-)deserialize the already-buffered bytes; **signal that we have the tensors** so they can be piped to the GPU.
- **Send side:** bytes are cooked from the GPU; swactor **populates a send buffer** and **resolves the sink.**
- What swactor tells the process (tinygrad) is **an actual location of the bytes on-device.**
- swactor **handles the stages** of this handoff, **not the actual movement.**
- **swactor actors are simple gates.**
- → Concretely realized in **The process-facing layer (stream & sink)** below.
## The process-facing layer (stream & sink)
Framing: we are designing the **typed send/receive infrastructure**, not the process.
The process view is the motivating lens — start from what the local compute sees, then
build the CP that drives it.
- **The role is a function:** typed chunks come in, get computed, typed chunks go out,
through ports the process exposes. For the CP these ports are an **arbitrary address +
handler space**; the first iteration uses one inbound and one outbound, but nothing
assumes that count. Inbound and outbound ports are **independent** — not a coupled 1:1
pass-through; the source and the sink in a single process are unrelated.
- **The process is driven, not autonomous.** It waits for the CP to stage a chunk,
computes, hands the result back. (We lean toward a blocking pull as the natural shape
for a single straight-line GPU worker, but that's a hedge, not a committed API — the
load-bearing point is that the CP drives.)
- **A chunk is a typed tensor, not bytes** — fully parsed, sanitized, shaped, i.e.
everything except the handoff to tinygrad and the GPU copy, which is the one step the
process performs. Shape/dtype are already determined inside the process by its role;
they aren't designed here, and we stay abstract about the type machinery for now.
- **The control plane handles ordering** and correlation between inbound and outbound
chunks. The process tags and tracks nothing.
- **The sink carries its destination.** The process knows the next address and binds it
to the sink — the sink, as a type/struct, has the *next address* built in. **Resolving
that address, serializing the chunk, and streaming it out is the CP's job.** This is
where we reach for **swactor's distributed address space**: the CP outside the process
receives and parses typed chunks, feeds them into the local process (tinygrad + GPU),
then on the way out resolves the sink's address and streams the outbound chunk.
- **Statefulness** (KV cache, position) is worker-internal and out of scope for this
layer.
- **Handle/payload representation and zero-copy staging are deferred** — performance
lives there, behind an unchanged process-facing surface.
This is an **abstract in-process surface** — a good vantage to design from, not a locked
API:
- an inbound typed-chunk source the process is driven from,
- an outbound typed sink the process pushes to, carrying its next address,
- both shaped by the role, both independent.
The CP's mandate from here: **receive + parse typed chunks into a process, and resolve +
serialize + stream typed chunks out of it.**
## End-to-end single pass (abstract)
Assumes SWIM converged and a start signal received. The pass is one repeating **edge**
(role A → role B) plus two ends; the **orchestrator is just another participant** (sink →
role0, source ← roleN), and **tokens are typed chunks** like activations.
- **Egress (role A, CP):** take the typed chunk from the sink; serialize → wire bytes;
resolve the sink's abstract next-address → concrete route. Serialize and resolve are
independent operations.
- **Transport:** stream bytes A → B (iroh; not swactor's concern).
- **Ingress (role B, CP):** reassemble → deserialize + validate against the role-known
spec → stage → drive the local process.
- **Ends:** orch → role0 ships *tokens*; roleN → orch ships a *token*. Not special cases
— edges whose endpoint is the orch and whose chunk type is tokens.
The open seam, designed next: **addressing** — binding the sink's abstract next-address to
a concrete remote endpoint.
## The edge: addressing, signals, and the byte boundary
How an edge is established and how chunks flow across it. Steady state assumes SWIM
converged and edges established.
**Addressing — orchestrator-direct, no inter-end handshake**
- An edge is a **pair of stream actors** (distinct from the process-driving actor): a
`Tx` (send) and an `Rx` (receive). **Neither end knows the other's actor address.**
The data plane is addressed by **`(node_id, edge_id)`**: the Tx sends to the
consumer's node by its `node_id`, and the `edge_id` at the head of the stream
demuxes it to the right `Rx`.
- **There is no inter-end negotiation** — the two ends never exchange a message; each
is handed everything it needs at provisioning (the orchestrator owns placement).
This still avoids per-chunk negotiation and the RTS-CTS/credit flow-control the
directives rule out ("everything but the route is known ahead of time") — it just
avoids the per-*edge* handshake too.
- The data endpoint is **handed down as the stable `node_id`** (the orchestrator knows
it from placement); iroh resolves the live path from it — so there's no stale
mapping to rot. (Churn is deferred; we design the happy case where both ends are
resolvable.)
- **Byte-level backpressure is pushed into the streaming logic**, not the actor layer.
Actors hold the edge; the streaming layer owns moving the bytes.
**The buffer-ownership baton & signals**
- The sink buffer is owned by either the process or the actor system at any instant;
signals are the handoffs.
- Egress: process **`done`** (buffer filled) → egress actor hands `(buffer, endpoint)` to
streaming → streaming **`released`** returns the slot to the alloc pool.
- Ingress: streaming **`landed`** → ingress actor inspects → **`ready`** drives the
process.
- **`done` is non-blocking.** The process never blocks after signalling it produced a
chunk. It blocks only on **alloc** (acquiring a send slot) and on **recv** (a chunk
arriving). With a single buffer, chunk *k+1*'s alloc blocks until chunk *k*'s
`released`. (Double-buffering deferred.)
**The byte boundary — egress trusts, ingress verifies**
- **Egress:** no actor-level parsing. The buffer is correct by construction, the endpoint
is bound from setup, and the receiver knows how to decode (type known a priori). The
egress actor hands the buffer straight to the streaming layer. **swactor touches zero
bytes on egress.**
- **Ingress:** swactor enters the byte path only to **read/check, never to transform**.
The **`Rx` and its edge service double as the inspector** — no new actor.
- Behind the **stream abstraction (data-plane integrity):** the framed `[extent]`
prefix is read and its `extent` bytes arrive complete (`extent ≤ capacity`).
Size/length lives here; a short or torn frame means no `landed`.
- The **ingress actor (control-plane gate):** given a complete chunk, clears it to
drive the process (belongs to this edge, expected in sequence), then flags `ready`.
The designated home for any sanity/terms check; thin in the happy path, but where
checking lives so the process is never handed an unvetted chunk.
- Net swactor byte-contract, both sides: **never transforms payload bytes; reads them
only to inspect, and only on ingress.**
**Open (not yet decided):** the depth of the ingress check — pure terms/sequence gate vs.
cracking the payload for a content-level (shape/dtype) sanity check before `ready`.
---
## Ingress check depth — decided (resolves "Open" above)
**Optimistic ingress: a chunk is accepted on its framed length alone.** If the
`[extent]` prefix reads cleanly, `extent ≤ capacity`, and that many bytes arrive
complete, they go to the process as-is — no peeking inside, no
deserialize-to-validate, no shape/dtype content check. Correct-by-construction
egress + a clean framed read on ingress is the entire gate. Nodes are trusted;
content trust is total.
## The transport, minimal
- Bytes move over **iroh** (already fixed). One **ordered, reliable stream per
edge** is the whole mechanism. Striping, chunk hashing/verification, and resume
are **deferred**.
- We are **not** building on the existing `crates/datastore/src/streams/` module —
treated as not-ready; design fresh.
## The transfer actors — a reusable primitive
- An edge's ends are **two actor types: `Tx` (send) and `Rx` (receive)**, one per
edge-end. They are a **general blob-moving primitive** — no notion of role,
compute, or the graph.
- Each is **pre-told the blob's `capacity`** (a `BlobSpec`; the per-chunk byte
ceiling — `dtype`/`shape` live in the role layer above, not the transport) and owns
**zero-copy (de)serialization**: Tx views the producer's buffer as bytes (no
transform); Rx views landed bytes back as a typed value. Rx **pre-allocates its
landing buffer** from `capacity` at setup; each chunk's actual `extent ≤ capacity`
varies per step and rides the wire as a length-prefix.
## Edge establishment — orchestrator-direct (contract #1)
- **No derived/hashed addresses, no gossip discovery, no polling.** The
**orchestrator owns placement and wires edges directly** — it hands each end
everything it needs. Addresses stay runtime-assigned (random); identities are
*handed over*, never computed or discovered.
- **The data plane is addressed by `(node_id, edge_id)`.** The orchestrator hands the
`Tx` its consumer's stable `node_id` (known from placement) in the provision
message; iroh resolves the live path from it, so no static transport mapping can
rot. There is no peer-to-peer endpoint exchange.
- **Per-node `Provisioner`** spawns the local `Tx`/`Rx`. The node's **edge service**
(the ALPN-aware `IrohDriver`) demuxes incoming streams to the right `Rx` by the
run-global **`edge_id`** at the head of the stream — not a separate `Listener`
actor; the demux is a tokio task on the existing endpoint.
- **No inter-end handshake; race-free by a single barrier.** The two ends never
exchange a message — each is fully equipped at provisioning, which fans out **in
parallel** (no Tx-before-Rx ordering). Race-freedom is one barrier: a node acks
`Provisioned` only after its `Rx` ends have **registered** their landing, and the
orchestrator injects the prompt only after **every** node has acked — so no stream
can arrive before its `Rx` is registered, with no per-edge ordering.
- **The establishment "exchange" is just the `edge_id` stream preamble (Tx→Rx).** No
`EndpointOffer`, no `Ready`, no `EdgeReady`, no `tx_addr` relay. READY is a local
terminal state: `Tx` is ready on spawn; `Rx` is ready once it has pre-allocated and
registered its landing buffer.
- **Kickoff, not broadcast.** The orchestrator is just another participant; once all
nodes are `Provisioned` it injects the driving prompt into role0 on its own
outbound edge. Every other node derives its own state from arriving data.
- **Asymmetry:** `Tx` never needs `Rx`'s actor address — and now neither end needs
the other's; they are coupled only by the shared `edge_id`.
## Still deferred (unchanged stance)
Churn/failure policy, teardown, the fan-in **join**, and the per-chunk zero-copy
baton (contract #4) remain out of scope. (The **start signal** is no longer here —
it's decided: there is none; see "Kickoff, not broadcast" above.)
---
# Blob streaming & host allocation
Detailing blob streaming + host allocation (contract #4). Full flow in
`BLOB_STREAMING.md` (draft).
## The host substrate — one sparse arena per node
- Blob bytes live on the host in a single `memfd` arena, mapped by **both** the
node process and the Python GPU worker.
- The arena is reserved **big and sparse** (lazy tmpfs backing) and mapped **once**;
the mapping is never moved. Edges are **regions sub-allocated** from it and
returned on teardown, so topology is **dynamic without touching the fd**.
## The slot handoff
- Each edge owns a **ring of N `capacity`-sized slots** (default 2). A slot is owned
at any instant by exactly one of {iroh, GPU worker}; ownership passes by **signal,
two per direction**, over the existing stdin/stdout pipe. Actors never touch a
payload byte; the node process is the sole authority on slot state.
## The GPU boundary
- tinygrad reads/writes slots **in place** via a `memoryview` (`copyin`/`copyout`);
a blob **never enters Python's heap**. The host↔device DMA is the worker's only
copy.
## Streaming
- **One long-lived iroh uni-stream per edge**; `edge_id` preamble once; then
length-prefixed chunks `[extent: u32][extent bytes]`, so the **prefix is the
frame** (each chunk's `extent ≤ capacity` varies per step; the slot is sized once
to `capacity`). Refines establishment's per-call `open_uni` into a persistent
stream, and its single landing buffer into the ring.
## Deferred to their own passes
- Host-pinning + removing tinygrad's CUDA `copyin` bounce (perf).
---
# Activation stream transport — the iroh ↔ swactor boundary
How activation tensors actually cross a READY edge, and how the swactor actors,
the iroh driver, and the GPU worker are wired to move them. Full spec in
`STREAM_TRANSPORT.md`. Scoped to activations (not gossip, not weights).
## The decision in one line
- **One persistent uni-stream per edge; the bytes ride it in place, in the arena;
swactor passes only slot indices, never bytes.**
## Stream shape — persistent, length-framed
- **One long-lived uni-stream per edge**, not a stream per tensor. `edge_id`
preamble once; then back-to-back length-prefixed tensors. Because each tensor's
size varies per step (prefill many rows, decode one), each rides a fixed-width
`u32` `extent` prefix — `[extent][extent bytes]`, `extent ≤ capacity` — and **the
prefix *is* the frame**; the slot is sized once to `capacity`. Stream-per-message
was rejected: it pays a task spawn + alloc + a `max_concurrent_uni_streams` slot
per tensor and buys nothing.
- **QUIC owns reliability.** No app-level fragmentation (the ring already pipelines
a whole-tensor object) and no striping (one connection over one path shares a
single congestion window — striping needs multipath we don't have).
## Zero-copy — bytes never enter an actor
- Bytes live in the **shared arena** from the worker's `copyout` to the far
worker's `copyin`, moved **in place**: the wire `read_exact`/`write_all`s arena
slots directly. Huge tensors are never copied into a `Vec` or an actor message.
- **swactor moves slot indices (`usize`), not bytes.** The only payload-byte
touchers are the **GPU worker** and a per-edge **byte-pump task**.
## Roles — tokio stays behind the driver wall
- **Driver** owns the endpoint, the connection cache, the `edge_id` demux, and
**spawns/owns the byte-pump tasks** — the one place tokio lives, async byte
readers and writers.
- **Reads are tokio-native** — `read_exact`/`write_all` only advance when polled on
the runtime — so a byte-pump *task* is unavoidable while iroh is the transport.
## MVP
- **MVP = one dedicated byte-pump task per edge-end**, driver-owned. The
**actor↔driver contract is slot-indices-in, slot-indices-up**, so the pump
*mechanism* is a driver-internal detail. Toward removing tokio, it can later
collapse to one-task-per-connection or to the node loop polling the stream
futures — a driver refactor that **touches no actor**.
- **No single-threaded-tick assumption:** all cross-thread traffic is `deliver_raw`
+ the slot channels + actor isolation, so the design survives a multi-threaded
runtime.
## Deferred
- The swactor ↔ GPU-worker pipe **mechanism** (async-Python rework) — its
`ready/consumed/filled/drained` signals are fixed here, the transport is not.
---
# Remaining orchestration decisions before implementation
The first concrete workload is **sharded inference of large models across
prosumer GPUs**. Do not prematurely generalize this into a broad graph IR. The
next spec layer should describe only the workload/role shape needed for that use
case: model partitioning, role boundaries, edge object specs, weight/shard
ownership, and the runtime sequence for prefill/decode.
The **orchestrator remains the authority** for resource provisioning, placement,
and run staging. Nodes do not need to advertise capabilities after boot as part
of this design; the orchestrator provisions the pool and already knows the
resource inventory it is placing onto.
The missing spec surface is therefore:
- how the orchestrator decides roles and placement from model shape plus
provisioned hardware,
- what a role provisioning message contains,
- how model weights/shards are assigned, fetched, loaded, and declared ready,
- what execution semantics the first inference path guarantees,
- and what behavioral contracts are required for reliable tests.
Observability should be added when the descriptive specs are converted into
behavioral contracts for testing. The goal is not just prose architecture, but
testable run behavior: provisioned, loaded, ready, object produced/consumed,
completed, faulted, and torn down.

1397
GPU_WORKER_INTERFACE_SPEC.md Normal file
View file

File diff suppressed because it is too large Load diff

518
ORCHESTRATION_SPEC.md Normal file
View file

@ -0,0 +1,518 @@
# GGUF Pipeline Orchestration - MVP Specification
**Status:** draft buildout specification.
**Relationship to other documents.** `DESIGN_DIRECTIVES.md` remains steering
context. `RING_BACKPRESSURE_SPEC.md` defines edge establishment, object records,
rings, pumps, and teardown. `GPU_WORKER_INTERFACE_SPEC.md` defines worker
startup, shard/weight binding, and `ExecuteStep`. This document defines the
missing layer above them: how the orchestrator plans and stages one linear GGUF
pipeline inference run.
This document is intentionally not a general graph specification.
---
## 1. Scope
The MVP workload is pipeline-parallel inference from a GGUF model:
```text
orchestrator --tokens--> stage 0 --activations--> stage 1 --activations-->
... --activations--> stage N-1 --tokens--> orchestrator
```
The orchestrator is a control participant and token endpoint. It does not run GPU
compute. GPU compute happens only inside provisioned stages.
This spec covers:
- building a linear stage plan from a GGUF model and a provisioned GPU pool
- assigning GGUF shard/layer ranges to stages
- assigning run-scoped edge ids
- provisioning each stage with the facts it needs
- defining the readiness barrier before prompt injection
- defining the orchestrator and stage FSMs
- defining security and correctness guarantees for the MVP behavior
This spec does not cover:
- arbitrary graph execution
- automatic placement optimization
- batching, speculative decoding, or continuous serving
- failure recovery by re-placement
- detailed ring, pump, or worker internals already specified elsewhere
- behavioral test contracts; those come after the full system shape is drafted
---
## 2. Core Responsibilities
The orchestrator owns:
- run ids
- stage count and stage order
- GGUF shard/layer assignment
- edge id assignment
- stage provisioning
- the global readiness barrier
- prompt injection
- final token consumption
- EOS and `max_tokens` stop policy
- run-level fault and teardown
Each stage owns:
- loading its assigned GGUF shard/layer range
- configuring its local GPU worker
- establishing its local edge ends
- converting loaded inbound objects into local `ExecuteStep` calls
- producing the next object on its outbound edge
- reporting readiness and faults to the orchestrator
The ring and worker specs own the byte movement and GPU worker command details.
This spec only decides which stages and edges exist and what sequence of control
events makes the run progress.
---
## 3. Run Plan
The orchestrator builds one `RunPlan` before provisioning:
```rust
struct RunPlan {
run_id: RunId,
model: GgufModelPlan,
stages: Vec<StagePlan>,
edges: Vec<EdgePlan>,
max_tokens: u32,
}
struct GgufModelPlan {
model_id: String,
gguf_source: GgufSource,
num_layers: u32,
hidden_dim: u32,
dtype_family: DTypeFamily,
dtype_width_bytes: u32,
max_seq_len: u32,
eos_token_id: u32,
}
```
`gguf_source` may identify a whole GGUF file, a pre-split shard collection, or a
cache key. The orchestration contract is the assigned layer range. Whether the
node reads only part of a whole GGUF file or receives a physically pre-split
artifact is a local loading detail.
Each stage receives a contiguous layer range:
```rust
struct StagePlan {
run_id: RunId,
stage_index: u32,
stage_count: u32,
node_id: NodeId,
gguf_source: GgufSource,
layer_start: u32,
layer_end_exclusive: u32,
inbound_edge: EdgeId,
outbound_edge: EdgeId,
}
```
For stage `0`, `inbound_edge` is the token edge from the orchestrator. For the
last stage, `outbound_edge` is the token edge back to the orchestrator. Interior
edges carry activations.
Each edge has exactly one producer and one consumer:
```rust
struct EdgePlan {
run_id: RunId,
edge_id: EdgeId,
kind: EdgeKind,
producer: EdgeEndpoint,
consumer: EdgeEndpoint,
object_spec: ObjectSpec,
ring_spec: RingSpec,
}
enum EdgeKind {
TokenIn,
Activation,
TokenOut,
}
enum EdgeEndpoint {
Orchestrator { node_id: NodeId },
Stage { node_id: NodeId, stage_index: u32 },
}
```
The orchestrator assigns all `edge_id`s. Stage code never derives edge ids from
names, layer ranges, peer ids, or hashes.
---
## 4. Stage Provisioning Message
The orchestrator sends one provision message to each stage node:
```rust
ProvisionStage {
run_id: RunId,
stage_index: u32,
stage_count: u32,
gguf_source: GgufSource,
layer_start: u32,
layer_end_exclusive: u32,
inbound: InboundEdgeProvision,
outbound: OutboundEdgeProvision,
model: StageModelFacts,
}
struct InboundEdgeProvision {
edge_id: EdgeId,
kind: EdgeKind,
object_spec: ObjectSpec,
ring_spec: RingSpec,
}
struct OutboundEdgeProvision {
edge_id: EdgeId,
kind: EdgeKind,
consumer_node_id: NodeId,
object_spec: ObjectSpec,
ring_spec: RingSpec,
}
struct StageModelFacts {
model_id: String,
hidden_dim: u32,
dtype_family: DTypeFamily,
dtype_width_bytes: u32,
max_seq_len: u32,
}
```
The inbound edge is established locally as a receive edge. The outbound edge is
established locally as a send edge to `consumer_node_id`. For the last stage,
`consumer_node_id` is the orchestrator node.
The node-local stage controller translates this provision message into the lower
level operations:
```text
configure worker for assigned stage
load/bind assigned GGUF shard or layer range
provision receive edge for inbound.edge_id
provision send edge for outbound.edge_id
report StageReady when all required local work is complete
```
---
## 5. Orchestrator FSM
The orchestrator has one run-level FSM:
```text
Planning
build RunPlan
validate layer ranges and edge ids
-> Provisioning
Provisioning
send ProvisionStage to every stage node
create local token producer for token-in edge
create local token consumer for token-out edge
-> WaitingReady
WaitingReady
on StageReady for every stage and local token endpoints ready
-> Running
on StageFault or timeout
-> Faulted
Running
inject prompt token object on token-in edge, sequence 0
consume token objects from token-out edge in sequence order
after token sequence k:
if EOS or max_tokens reached -> Completed
else inject token object sequence k + 1 on token-in edge
on StageFault, edge fault, local token endpoint fault, or timeout
-> Faulted
Completed
stop injecting tokens
finalize output text
-> TearingDown
Faulted
stop injecting tokens
record one run-level failure reason
-> TearingDown
TearingDown
send StopRun to all provisioned stages
tear down local token endpoints
wait for StageStopped from every stage or timeout
-> Done
Done
terminal
```
There is no broadcast start message. The start signal is the first token object
written by the orchestrator after the readiness barrier.
---
## 6. Stage FSM
Each provisioned stage has one node-local stage controller. It is control-path
only: it watches local worker and edge events and issues worker commands. It does
not move payload bytes.
```text
Unprovisioned
on ProvisionStage from authorized orchestrator
validate run_id and stage assignment
-> Preparing
Preparing
configure local worker for assigned GGUF range
start GGUF shard/range load and bind
establish inbound receive edge
establish outbound send edge
when worker configured, shard bound, and both edge ends ready
emit StageReady
-> Ready
on any required setup failure
emit StageFault
-> Faulted
Ready
on inbound ObjectLoaded(sequence = s)
if s is the next expected sequence
issue ExecuteStep for sequence s
-> Executing
else
emit StageFault(sequence violation)
-> Faulted
on StopRun
-> Stopping
Executing
worker runs exactly one step for the loaded inbound object
worker writes the output object to outbound edge with the same sequence
on StepCompleted
release any per-step input handle that is no longer needed
-> Ready
on StepFailed or output fault
emit StageFault
-> Faulted
Faulted
reject new run work
wait for StopRun
-> Stopping
Stopping
stop local edges
release per-run device objects
stop or reset worker according to local policy
emit StageStopped
-> Stopped
Stopped
terminal for this run
```
The controller is the component that decides when `ExecuteStep` is called. The
orchestrator does not issue per-stage execute commands during the run. Once the
prompt object is injected, stage execution is driven by object arrival and local
readiness.
---
## 7. Execution Semantics
Sequence `0` is prefill.
```text
orchestrator writes prompt token object sequence 0
stage 0 executes prefill over prompt tokens
stage 0 writes activation sequence 0
each interior stage executes prefill over activation sequence 0
last stage executes prefill and writes token sequence 0
orchestrator consumes token sequence 0
```
Decode sequences are `1..`:
```text
orchestrator writes one-token object sequence k
stage 0 executes decode for sequence k
each downstream stage executes decode for sequence k
last stage writes token sequence k
orchestrator consumes token sequence k
```
The orchestrator sends sequence `k + 1` only after consuming token sequence `k`
and deciding the run should continue.
For every stage:
- the inbound object sequence is the output object sequence
- one active `ExecuteStep` per stage is allowed in the MVP
- a stage cannot execute before its assigned GGUF shard/range is loaded and bound
- a stage cannot execute before its inbound object is loaded
- a stage cannot produce to an edge that is not ready
The last stage samples or otherwise produces token ids as part of its GPU worker
step. The orchestrator consumes those token ids, accumulates output, applies EOS
and `max_tokens`, and writes the next token object only when continuing.
---
## 8. Control Messages And Events
These are schematic message shapes, not final Rust APIs.
Orchestrator to stage:
```rust
ProvisionStage { ... }
StopRun {
run_id: RunId,
reason: StopReason,
}
```
Stage to orchestrator:
```rust
StageReady {
run_id: RunId,
stage_index: u32,
node_id: NodeId,
}
StageFault {
run_id: RunId,
stage_index: u32,
node_id: NodeId,
reason: StageFaultReason,
}
StageStopped {
run_id: RunId,
stage_index: u32,
node_id: NodeId,
}
```
Optional setup progress events may exist for diagnostics, but `StageReady`,
`StageFault`, and `StageStopped` are the only required run-level events in this
draft.
---
## 9. Object Specs
Token edges carry token objects. The prompt token object may contain multiple
token ids for prefill. Decode token objects contain one token id.
Activation edges carry activation objects with runtime extent bounded by model
shape:
```text
max_extent = max_seq_len * hidden_dim * dtype_width_bytes
```
The object record and ring behavior are defined by `RING_BACKPRESSURE_SPEC.md`.
This orchestration spec only requires that all stage plans for a run agree on the
model facts used to build those object specs.
---
## 10. Security Model
Nodes are trusted. The system does not attempt trustless verification,
adversarial tensor validation, Sybil defense, or incentive enforcement.
The orchestrator is the authority for run topology. A stage accepts run
provisioning only from the authorized orchestrator for its node.
Stages reject:
- unknown `run_id`
- stale `run_id`
- duplicate provisioning for an already-active run unless explicitly stopped
- edge ids not present in the provision message
- peer rewiring requests from another stage
`edge_id`s are run-scoped capabilities for wiring and demux. They are not a
cryptographic trust boundary between trusted nodes, but a stage must still reject
objects and stream setup that do not match its active run plan.
The orchestrator may tear down a run at any time. Stages must treat `StopRun` for
their active `run_id` as authoritative.
---
## 11. Correctness Guarantees
Layer assignment:
- stage layer ranges are contiguous
- stage layer ranges do not overlap
- the union of stage layer ranges covers the intended GGUF block range
- every stage has exactly one assigned range
Edge assignment:
- every `edge_id` is unique within a run
- every edge has exactly one producer and one consumer
- token-in is produced by the orchestrator and consumed by stage `0`
- token-out is produced by stage `N - 1` and consumed by the orchestrator
- activation edge `i` is produced by stage `i` and consumed by stage `i + 1`
Readiness:
- the orchestrator does not inject prompt tokens before every stage reports
`StageReady`
- a stage does not report `StageReady` before its worker, shard/range binding,
inbound edge, and outbound edge are ready
Execution:
- prefill is sequence `0`
- decode sequences are strictly increasing
- a stage executes sequence `s` only after loading inbound object sequence `s`
- a stage output uses the same sequence as its input
- the orchestrator injects sequence `s + 1` only after consuming token sequence
`s`
Termination:
- each run has one terminal outcome: completed, faulted, or torn down
- after a run faults, the orchestrator stops injecting new token objects
- teardown is sent to every stage that was provisioned for the run
---
## 12. Deferred
- placement optimization
- physical GGUF shard format
- multiple concurrent runs on one stage chain
- batching and speculative decoding
- direct stage-to-stage token feedback that bypasses the orchestrator
- warm reuse policy across prompts
- re-placement after node failure
- behavioral test matrix and observability schema

1522
RING_BACKPRESSURE_SPEC.md Normal file
View file

File diff suppressed because it is too large Load diff