Zachery Aaron Shores-Chmielewski 07949b3576 stash cleanup and localized e2e stubbed mock

2026-06-23 19:42:28 +04:00

57 KiB

Raw Blame History

MVP System Specification

Status: draft consolidated system specification.

This document is the single MVP reference for the swactor GGUF pipeline system. It folds the system behavior previously split across the orchestration, ring/backpressure, and GPU-worker drafts into one end-to-end specification.

The focused root specs may remain as design history while this document is being stabilized. For the MVP drafting process, this document is the place where the complete system, its components, guarantees, and systemic behavior are described together.

1. Purpose

The MVP is a trusted distributed ML runtime for one linear GGUF inference pipeline over a rented pool of GPU nodes.

The first deployment shape is:

operator provisions rented GPU nodes
  -> each node runs a barebones Docker image
  -> the image contains swactor runtime + tinygrad/CUDA worker
  -> nodes join membership and reach SWIM convergence
  -> orchestrator assigns pipeline roles and edges
  -> nodes download/load assigned GGUF weights
  -> nodes provision arena-backed data edges
  -> nodes report readiness
  -> orchestrator injects a "hello world" prompt
  -> pipeline returns tokens
  -> run completes or faults and tears down

The document is intentionally about the first complete feature, not a broad graph engine. Later features can generalize from this system after the MVP behavior is stable.

2. Scope

In scope:

trusted node pool boot and readiness
SWIM membership convergence as the planning gate
orchestrator authority, planning, staging, faults, and teardown
one linear GGUF pipeline
stage/layer assignment and edge id assignment
stage-local weight download, load, and bind readiness
arena-backed rings and iroh/QUIC edge transport
Rust node process to Python/tinygrad worker process control
GPU worker object loading, step execution, and object production
prompt injection, prefill, decode, token return
completed, faulted, and operator-stopped terminal outcomes
observability events needed for behavioral contracts and tests

Out of scope:

arbitrary graph IR
automatic placement optimization
batching, speculative decoding, continuous serving
re-placement after churn or node failure
trustless verification, incentives, Sybil defense, adversarial tensor checks
production artifact layout and cache eviction policy
VastAI account automation, bidding, billing, or provider-specific control machinery
high-performance host pinning policy beyond the safety contract

3. Design Commitments

Nodes are trusted. Payload content is trusted after structural checks. The system does not defend against malicious tensor values or adversarial peers.

The orchestrator is the run authority. It owns placement, run planning, provisioning, prompt injection, terminal outcome, and teardown.

swactor owns the control plane. Actors establish, observe, and tear down components. Actors do not move tensor payload bytes.

Payload bytes move through shared-memory rings and iroh/QUIC streams. The hot path is ring cursors, wake hints, and byte-pump tasks, not actor mailboxes.

The GPU worker is a supervised process, not a separate swactor runtime. It maps the shared arena, uses a native ring helper, owns device allocations, and runs tinygrad role code when explicitly commanded.

Compute is explicit. ObjectLoaded means data is ready on device. ExecuteStep is the local compute admission command. Loading an object does not automatically run tinygrad.

Backpressure is absence of writable ring space plus QUIC flow control. There is no actor-level credit protocol, RTS/CTS exchange, or per-range acknowledgement.

One persistent iroh uni-stream carries all objects for one edge. The stream starts with an edge_id preamble and then object records.

The MVP permits transfer and device upload/download to overlap with network streaming. It does not permit compute to observe a logical object until the whole object has loaded.

4. Trust And Authority Model

The orchestrator owns:

intended node pool
resource inventory used by placement
run ids
model and runtime configuration
tokenizer/vocabulary facts
stage count and stage order
stage-to-node assignment
layer range assignment
edge id assignment
object specs and ring specs
weight source assignment
global readiness barrier
prompt tokenization and prompt injection
output token consumption
EOS and max_tokens stop policy
run-level fault and teardown decisions

Each node owns:

stable node identity for the run
node process lifecycle
GPU worker process lifecycle
local arena and arena leases
local iroh endpoint and byte pumps
SWIM participant
local stage controller
local edge establishment records
assigned weight download/load work
local worker/device resources

Nodes do not rewire the graph. A node accepts run provisioning only from the authorized orchestrator and rejects edge data or provisioning that does not match its active run plan.

Other stages cannot redirect a node to a new peer, edge id, or layer range.

Resource inventory is known to the orchestrator before placement in the MVP. A node may report boot health and readiness, but there is no distributed capability negotiation protocol after boot.

5. System Topology

Each GPU node runs one Rust node process and one Python/tinygrad GPU worker process:

GPU node container
------------------
Rust node process
  swactor runtime
  SWIM participant
  iroh endpoint and driver
  ArenaManager
  EdgeEstablisher
  Tx/Rx edge actors
  GpuWorkerCtl
  StageController

Python/tinygrad worker process
  mapped shared arena
  native ring helper
  device bridge
  role code
  device allocations and KV/state

The orchestrator may run in its own container or on one of the provisioned GPU machines. It is a control participant and token endpoint. It does not run model compute for the MVP.

When the orchestrator participates in token edges over the data plane, it has a stable node_id, an iroh endpoint, and local token edge endpoints like any other producer or consumer node. Co-locating the orchestrator with a GPU node is an implementation choice, not a change to edge semantics.

The system has two network-facing planes:

membership/control observation: SWIM provides node liveness/membership state used by the orchestrator as the pool readiness gate
data movement: iroh/QUIC carries persistent uni-streams for data edges, addressed by (node_id, edge_id)

Provisioning, readiness, faults, and teardown are swactor messages. Tensor bytes do not travel in actor messages.

6. Identifiers

Identifier types are schematic. Concrete Rust APIs may wrap these as newtypes.

struct RunId(u64);
struct NodeId(u64);
struct RoleId(u64);
struct StageIndex(u32);
struct EdgeId(u64);
struct RingId(u64);
struct ObjectId(u64);
struct Sequence(u64);
struct StepId(u64);
struct WorkerGeneration(u64);

EdgeId is unique within a run and assigned only by the orchestrator.

RingId is unique for the node lifetime. Arena ranges may be reused after quiescence, but ring ids are not reused.

DeviceObjectHandle is opaque to the Rust node process:

struct DeviceObjectHandle {
    worker_generation: u64,
    id: u64,
}

A device handle is valid only in the worker generation that created it. Worker restart invalidates all prior handles, roles, rings, and in-flight steps.

7. System Lifecycle

The complete MVP lifecycle is:

Deployment
  -> NodeBoot
  -> MembershipConvergence
  -> RunPlanning
  -> StageProvisioning
  -> WeightLoading
  -> EdgeProvisioning
  -> GlobalReadiness
  -> PromptInjection
  -> PipelineExecution
  -> CompletionOrFault
  -> Teardown
  -> Done

Phase ownership:

node boot is local to each node
membership convergence is observed by the orchestrator
run planning is owned by the orchestrator
local provisioning is owned by each node
global readiness is owned by the orchestrator
execution progress is driven by object arrival at stages
terminal outcome and teardown are owned by the orchestrator

There is no broadcast start message. The first prompt object written by the orchestrator after the readiness barrier starts execution.

8. Node Boot

At container start, each node runs:

ContainerStarting
  -> NodeProcessStarting
  -> ArenaReady
  -> WorkerReady
  -> TransportReady
  -> MembershipJoining
  -> NodeAvailable

NodeAvailable means:

Rust node process is alive
swactor runtime can receive control messages
node has a stable node_id known to the orchestrator
arena is created and mapped in the node process
GPU worker has emitted WorkerReady
iroh endpoint is initialized and associated with the node identity
SWIM participant has joined or is joining the intended pool
node can accept run provisioning

NodeAvailable does not mean weights are present, a role is configured, or any run edge is established.

If implementation defers worker startup until run provisioning, the same run-level gate must still hold: a stage cannot report StageReady before its worker is ready and assigned weights are usable.

Boot failure before ArenaReady or WorkerReady means the node is unavailable. No run edge should be provisioned to that node.

9. Membership And Pool Readiness

The orchestrator starts run planning only after the intended candidate pool is ready.

For the MVP, PoolReady means:

every node in the intended candidate pool is known to the orchestrator
every candidate node is live in the SWIM membership view
every candidate node has reported NodeAvailable
every candidate node has data-plane identity/endpoint material the orchestrator can provision into edges
no candidate node is currently suspect or faulted in the orchestrator view
the pool view has remained stable for the configured convergence window

The convergence window is an implementation parameter.

Nodes do not compute placement and do not need to agree on graph state. SWIM is a membership/liveness input to the orchestrator, not a distributed graph protocol.

If pool readiness is lost before a RunPlan is committed, the orchestrator keeps waiting or aborts before provisioning according to local policy.

If pool readiness is lost after provisioning begins, the run faults. The MVP does not re-place an active run after a node disappears.

10. GGUF Pipeline Workload

The MVP workload is one linear pipeline:

orchestrator --tokens--> stage 0 --activations--> stage 1 --activations-->
... --activations--> stage N-1 --tokens--> orchestrator

The orchestrator is a token endpoint and control participant. It does not run GPU compute.

Each stage owns a contiguous GGUF layer range. A stage receives typed input objects, executes its assigned layer range, and writes typed output objects.

Stage 0 consumes token objects from the orchestrator and produces activation objects. Interior stages consume and produce activation objects. The last stage consumes activation objects and produces token objects for the orchestrator.

The last stage samples or otherwise produces token ids as part of its worker step if sampling is delegated to the stage. If sampling is not delegated, the worker output object must contain enough logits/token data for the orchestrator to apply the configured policy. The RunPlan must state which policy is used.

11. Run Plan

The orchestrator builds exactly one RunPlan before provisioning:

struct RunPlan {
    run_id: RunId,
    model: GgufModelPlan,
    runtime: RuntimePlan,
    stages: Vec<StagePlan>,
    edges: Vec<EdgePlan>,
    max_tokens: u32,
}

struct GgufModelPlan {
    model_id: String,
    gguf_source: GgufSource,
    num_layers: u32,
    hidden_dim: u32,
    dtype_family: DTypeFamily,
    dtype_width_bytes: u32,
    max_seq_len: u32,
    eos_token_id: u32,
    tokenizer: TokenizerSource,
}

struct RuntimePlan {
    prompt: PromptSource,
    sampling: SamplingPolicy,
    token_output_policy: TokenOutputPolicy,
}

gguf_source may identify a whole GGUF file, a pre-split shard collection, a cache key, a local path, or an HTTP/object-store artifact. The system contract is the assigned layer range and readiness after that range is usable by the worker.

Each stage receives one contiguous layer range:

struct StagePlan {
    run_id: RunId,
    stage_index: u32,
    stage_count: u32,
    node_id: NodeId,
    gguf_source: GgufSource,
    layer_start: u32,
    layer_end_exclusive: u32,
    inbound_edge: EdgeId,
    outbound_edge: EdgeId,
}

Each edge has exactly one producer and one consumer:

struct EdgePlan {
    run_id: RunId,
    edge_id: EdgeId,
    kind: EdgeKind,
    producer: EdgeEndpoint,
    consumer: EdgeEndpoint,
    object_spec: ObjectSpec,
    ring_spec: RingSpec,
}

enum EdgeKind {
    TokenIn,
    Activation,
    TokenOut,
}

enum EdgeEndpoint {
    Orchestrator { node_id: NodeId },
    Stage { node_id: NodeId, stage_index: u32 },
}

Layer assignment guarantees:

stage layer ranges are contiguous
stage layer ranges do not overlap
the union of stage layer ranges covers the intended GGUF block range
every stage has exactly one assigned range

Edge assignment guarantees:

every edge id is unique within the run
token-in is produced by the orchestrator and consumed by stage 0
token-out is produced by stage N - 1 and consumed by the orchestrator
activation edge i is produced by stage i and consumed by stage i + 1
stages never derive edge ids from names, layer ranges, peer ids, or hashes

12. Provisioning Messages

The orchestrator sends one stage provision message to each stage node:

struct ProvisionStage {
    run_id: RunId,
    stage_index: u32,
    stage_count: u32,
    gguf_source: GgufSource,
    layer_start: u32,
    layer_end_exclusive: u32,
    inbound: InboundEdgeProvision,
    outbound: OutboundEdgeProvision,
    model: StageModelFacts,
    runtime: StageRuntimeFacts,
}

struct InboundEdgeProvision {
    edge_id: EdgeId,
    kind: EdgeKind,
    object_spec: ObjectSpec,
    ring_spec: RingSpec,
}

struct OutboundEdgeProvision {
    edge_id: EdgeId,
    kind: EdgeKind,
    consumer_node_id: NodeId,
    object_spec: ObjectSpec,
    ring_spec: RingSpec,
}

struct StageModelFacts {
    model_id: String,
    hidden_dim: u32,
    dtype_family: DTypeFamily,
    dtype_width_bytes: u32,
    max_seq_len: u32,
}

struct StageRuntimeFacts {
    role_id: RoleId,
    input_port: PortId,
    output_port: PortId,
    sampling: Option<SamplingPolicy>,
}

The inbound edge is established locally as a receive edge. The outbound edge is established locally as a send edge to consumer_node_id.

For the last stage, consumer_node_id is the orchestrator node.

Provisioning fan-out is parallel. There is no Tx-to-Rx actor handshake and no peer-to-peer endpoint exchange.

13. Orchestrator FSM

The orchestrator has one run-level FSM:

Planning
  build RunPlan
  validate layer ranges, edge ids, object specs, node ids
  -> Provisioning

Provisioning
  send ProvisionStage to every stage node
  create local token-in producer
  create local token-out consumer
  -> WaitingReady

WaitingReady
  on StageReady for every stage and local token endpoints ready
    -> Running
  on StageFault, endpoint fault, membership loss, or timeout
    -> Faulted

Running
  inject prompt token object sequence 0
  consume token objects from token-out edge in sequence order
  after token sequence k:
    if EOS or max_tokens reached -> Completed
    else inject token object sequence k + 1
  on StageFault, edge fault, endpoint fault, membership loss, or timeout
    -> Faulted

Completed
  stop injecting tokens
  finalize output text or token stream
  -> TearingDown

Faulted
  stop injecting tokens
  record one run-level failure reason
  -> TearingDown

TearingDown
  send StopRun to all provisioned stages
  tear down local token endpoints
  wait for StageStopped from every stage or timeout
  -> Done

Done
  terminal

The orchestrator does not issue per-stage ExecuteStep during a run. After prompt injection, stage execution is driven by object arrival and local stage readiness.

14. StageController FSM

Each provisioned stage has one node-local StageController. It is control-path only: it watches worker and edge events, issues worker commands, and reports run-level events. It does not move payload bytes.

Unprovisioned
  on ProvisionStage from authorized orchestrator
    validate run_id and stage assignment
    -> Preparing

Preparing
  configure local worker role path
  start GGUF shard/range load and bind
  establish inbound receive edge
  establish outbound send edge
  when worker configured, weights bound, and both edge ends ready
    emit StageReady
    -> Ready
  on required setup failure
    emit StageFault
    -> Faulted

Ready
  on inbound ObjectLoaded(sequence = s)
    if s is next expected sequence and local state is ready
      issue ExecuteStep for sequence s
      -> Executing
    else
      emit StageFault(sequence violation)
      -> Faulted
  on StopRun
    -> Stopping

Executing
  worker runs exactly one step for the loaded input object
  worker writes output object to outbound edge with same sequence
  on StepCompleted
    release per-step input handles according to policy
    -> Ready
  on StepFailed, ObjectFailed, output fault, or worker crash
    emit StageFault
    -> Faulted

Faulted
  reject new run work
  wait for StopRun
  -> Stopping

Stopping
  stop local edges
  release per-run device objects
  stop or reset worker role state according to local policy
  emit StageStopped
  -> Stopped

Stopped
  terminal for this run

The MVP allows one active ExecuteStep per stage.

15. Weight Lifecycle

Weights are stage-local persistent state for the run.

The StageController starts weight work from the assigned GgufSource and layer range. A stage may download:

a whole GGUF and load only its range
one or more physical shards containing its range
a cached artifact that already exists on the node

The physical loading mechanism may be:

worker startup configuration
a ConfigureRole command
a local loader path owned by the StageController
weight objects moved through the same object/ring machinery

The system-visible contract is WeightsReady before StageReady.

WeightsReady means:

assigned artifact bytes are locally available or already cached
assigned layer range has been validated against the run plan
GPU worker has loaded or bound the range needed for execution
failures in download, parse, device allocation, or binding have surfaced as stage faults

Warm model reuse across runs is deferred.

16. Execution Semantics

Sequence 0 is prefill:

orchestrator writes prompt token object sequence 0
stage 0 executes prefill over prompt tokens
stage 0 writes activation sequence 0
each interior stage executes prefill over activation sequence 0
last stage executes prefill and writes token sequence 0
orchestrator consumes token sequence 0

Decode sequences are 1..:

orchestrator writes one-token object sequence k
stage 0 executes decode for sequence k
each downstream stage executes decode for sequence k
last stage writes token sequence k
orchestrator consumes token sequence k

For every stage:

inbound object sequence equals outbound object sequence
a stage cannot execute before weights are loaded and bound
a stage cannot execute before inbound object is loaded
a stage cannot produce to an edge that is not ready
ObjectLoaded is data readiness, not compute completion
ObjectProduced is output object committed to egress ring
StepCompleted is the compute transaction terminal success event

The orchestrator writes sequence k + 1 only after consuming output token sequence k and deciding the run should continue.

17. Object Specs And Object Records

Objects are logical payloads on an edge: token batches, activations, weights, or model shards.

Object specs are role-known validation contracts:

struct ObjectSpec {
    kind: ObjectKind,
    max_extent: u64,
    dtype_family: DTypeFamily,
    dtype_width_bytes: u32,
    shape: ShapeRule,
    layout: LayoutRule,
    alignment: u32,
    sequence_policy: SequencePolicy,
}

The stream carries object records:

ObjectHeader
payload bytes, exactly header.extent bytes

The MVP header is fixed-size:

struct ObjectHeader {
    magic: u32,
    version: u16,
    header_len: u16,
    object_id: u64,
    sequence: u64,
    extent: u64,
    flags: u32,
    reserved: u32,
}

The header supplies runtime facts: object id, sequence, extent, and flags. ObjectSpec supplies dtype, shape/layout family, max extent, alignment, and sequence policy.

Activation maximum extent:

max_extent = max_seq_len * hidden_dim * dtype_width_bytes

For prefill, extent may cover many token rows. For decode, extent may cover one row/token. extent <= ObjectSpec.max_extent.

The worker rejects a record before exposing a device object if:

magic or version is unsupported
header length is malformed
extent > ObjectSpec.max_extent
extent violates alignment/layout rules
sequence violates the edge ordering policy
ring or stream closes before extent bytes arrive

Payload content itself is trusted. The worker does not inspect tensor values.

18. Ring Specs And Layouts

RingSpec is the requested operating shape for a ring:

struct RingSpec {
    data_capacity: u64,
    alignment: u32,
    direction: RingDirection,
    host_pinning: HostPinning,
    wake_coalescing: WakeCoalescing,
}

enum RingDirection {
    Ingress,
    Egress,
}

enum HostPinning {
    Pageable,
    PinnedRequired,
}

enum WakeCoalescing {
    PendingBit,
    ReadySet,
}

RingLayout is the arena-relative layout minted by ArenaManager after a lease:

struct RingLayout {
    ring_id: RingId,
    arena_offset: u64,
    total_len: u64,
    header_offset: u64,
    data_offset: u64,
    data_capacity: u64,
    alignment: u32,
}

RingLayout never contains process-local pointers. Each process derives local pointers from its own arena mapping base plus arena offsets.

The ring header and data region must be aligned for cross-process atomics and backend copy requirements. Host pinning is optional for the MVP, but if a range is pinned, it cannot be unpinned or returned to the free list until ring quiescence.

19. Arena Model

Each node owns one sparse host-memory arena.

The arena is a memfd, created by the Rust node process, truncated to a large sparse ceiling, and mapped once in the Rust process. The GPU worker maps the same fd once. Neither process remaps or resizes the arena during node lifetime.

The arena is address space. It does not define flow control and does not define object ownership.

The ArenaManager owns:

arena fd
node process mapping base pointer
reservation ceiling
arena free list
pending lease queue
table of live ring leases

The ArenaManager does not own:

worker process
QUIC stream
pump task
object parser
payload byte

The ArenaManager only mints stable offsets and releases ranges after proof of quiescence.

ArenaManager FSM:

Booting
  on ConstructArena{ceiling}
    -> Ready if memfd, truncate, mmap succeed
    -> Failed if any boot resource fails

Ready
  on LeaseRing{request_id, requester, edge_id, direction, ring_spec}
    -> RingLeased if a range fits
    -> queue request if satisfiable but no current range fits
    -> RingLeaseRejected if request can never fit in the ceiling

Ready
  on CancelLease{request_id}
    remove queued request if not yet leased

Ready
  on ReleaseRing{ring_id, proof}
    return range to free list
    retry queued leases serially

Ready
  on Shutdown
    -> ShuttingDown

ShuttingDown
  reject new leases

Temporary arena pressure is represented by a queued lease request. Permanent impossibility is explicit through RingLeaseRejected.

Two live leases cannot overlap because lease/release operations pass through one ArenaManager mailbox.

20. Ring Contract

A ring is a bounded single-producer/single-consumer byte stream backed by an arena lease.

Rings define flow control for every payload-moving boundary:

QUIC ingress stream -> host ingress ring -> worker -> GPU memory
GPU memory -> worker -> host egress ring -> QUIC egress stream
disk reader -> host ring -> worker/GPU memory, if used

Each ring has exactly one producer and one consumer.

Ingress:

producer = recv-pump
consumer = GPU worker

Egress:

producer = GPU worker
consumer = send-pump

Fan-in and fan-out are not ring features. They are represented by multiple rings or by a higher-level component that owns one side of a ring.

The shared arena ABI is not a Rust collection. It stores offsets, cursors, state bits, wake bits, and payload bytes. It never stores process-local pointers.

Ring header:

#[repr(C, align(64))]
struct RingHeader {
    magic: u32,
    version: u16,
    header_len: u16,
    ring_id: u64,
    capacity: u64,
    commit: AtomicU64,
    consume: AtomicU64,
    state: AtomicU32,
    wake: AtomicU32,
}

commit is the first byte after the committed readable prefix. Bytes with logical positions < commit are valid for the consumer.

consume is the first byte not yet released by the consumer. Bytes with logical positions < consume are free for producer reuse.

The producer keeps a local write cursor. write is the first byte after the producer's reserved or in-progress write prefix.

Readable interval:

consume .. commit

Reserved but not readable interval:

commit .. write

Physical index:

physical_index = cursor % capacity

The ring is empty when consume == commit. The ring is full when:

write - consume == capacity

Cursor values are monotonic logical byte positions.

21. Ring Producer, Consumer, And Wake Rules

Producer rule:

acquire consume
compute free space as capacity - (write - consume)
reserve a contiguous physical span by advancing local write
write bytes into that span
publish new commit with release ordering after bytes are valid
send or coalesce RingReadable { ring_id }

Consumer rule:

acquire commit
read bytes in consume .. commit
greedily drain all bytes it can use
publish new consume with release ordering after bytes are safe to release
send or coalesce RingWritable { ring_id } if producer may be unblocked

Wake hints are edge-trigger hints, not ownership:

RingReadable { ring_id: RingId }
RingWritable { ring_id: RingId }

The receiver of a wake must reload cursors from shared memory. Wake hints carry no byte ranges, byte counts, host pointers, or free-space counts.

Duplicate wake hints may be coalesced only while a pending bit, ready-set entry, or equivalent durable scheduler state still makes the ring discoverable. Losing the only transition from empty to readable or full to writable is a liveness bug.

If asynchronous DMA reads host ring memory, the consumer cannot advance consume until the DMA no longer depends on those bytes.

Safety:

consumer cannot read unwritten bytes because producer publishes commit only after writing
producer cannot overwrite unread bytes because free space is computed from consumer-owned consume
wraparound cannot alias stale bytes because ownership uses monotonic logical cursors
stale wake/control events cannot alias replacement rings because RingId is unique for node lifetime

22. Native Ring Helper

Both Rust hot-path code and the Python worker access process-crossing rings through the same native implementation.

Python does not implement shared atomics, wrap arithmetic, span calculation, or cursor publication directly.

The helper exposes operations equivalent to:

ring_open(arena_base, RingLayout) -> RingHandle
ring_close(handle)
ring_readable_span(handle) -> ptr, len
ring_advance_consume(handle, len)
ring_writable_span(handle) -> ptr, len
ring_advance_commit(handle, len)
ring_state(handle) -> state

Returned pointers are process-local addresses derived from the caller's mapped arena base plus arena offsets.

Ring cursors must use atomic acquire/release semantics across the process boundary.

23. Edge Stream Protocol

Each edge uses one persistent iroh/QUIC uni-stream from producer node to consumer node.

Wire shape:

[edge_id preamble]
[object record]
[object record]
...

The receiver's edge-demux reader consumes the fixed-width edge_id preamble and hands the stream to the driver rendezvous. After the preamble, the recv-pump is byte-blind. It copies stream bytes into the ingress ring and advances commit. The worker parses object records from the ring.

The stream is persistent for the edge. It is not opened per object.

QUIC owns reliability and stream ordering. The MVP does not add app-level fragment hashes, striping, resume, or per-object streams.

24. Driver And Transport

The driver is the node's swactor-to-iroh boundary. It owns:

one iroh endpoint
connection cache
edge ALPN
edge demux
receive stream rendezvous
send/recv pump tasks

Example edge ALPN:

swactor/edge/1

Connections are cached per (peer_node_id, ALPN). All edges between the same node pair and ALPN reuse the same connection. Each edge has one persistent uni-stream within that connection.

Driver inbound messages:

EstablishSend {
    edge_id: EdgeId,
    rx_node_id: NodeId,
    ring_id: RingId,
    ring_layout: RingLayout,
    tx_addr: ActorAddress,
}

EstablishRecv {
    edge_id: EdgeId,
    ring_id: RingId,
    ring_layout: RingLayout,
    rx_addr: ActorAddress,
}

StreamArrived {
    edge_id: EdgeId,
    stream: RecvStream,
}

StopEdge {
    edge_id: EdgeId,
}

Driver outbound events:

DriverEdgeReady { edge_id: EdgeId }
StreamClosed { edge_id: EdgeId }
StreamFault { edge_id: EdgeId, reason: StreamFaultReason }
PumpStopped { edge_id: EdgeId, ring_id: RingId }

No per-object, per-range, or fragment messages exist in the driver hot path.

25. Receive Demux And Pump FSMs

A recv-pump needs two resources:

local receive establishment state, including ingress ring
arriving QUIC stream

They may arrive in either order. The driver stores both halves:

recv_specs: HashMap<EdgeId, RecvSpec>
pending_streams: HashMap<EdgeId, RecvStream>

On EstablishRecv, if a pending stream exists, the driver spawns the recv-pump. Otherwise it stores the spec.

On StreamArrived, if a recv spec exists, the driver spawns the recv-pump. Otherwise it stores the stream.

This removes the need for an inter-end readiness handshake. If a stream arrives before local receive establishment, it waits in pending_streams; because no recv-pump reads from it, QUIC flow control eventually stalls the sender.

Recv-pump FSM:

WaitingForSpecAndStream
  -> Streaming when ring and stream are both present

Streaming
  read free ring span
  read QUIC bytes into that span
  advance commit
  send/coalesce RingReadable{ring_id} to worker
  repeat

Backpressured
  entered when no ring free space exists
  wait for RingWritable{ring_id}
  return to Streaming

Closed
  entered on stream EOF or edge teardown

Faulted
  entered on read error, protocol edge failure, or ring fault

The recv-pump does not parse ObjectHeader and does not know object boundaries. EOF alignment is classified by the worker parser, not by the pump.

Send-pump FSM:

WaitingForConnection
  ensure or await cached edge-ALPN connection

WaitingForBytes
  wait for RingReadable{ring_id}

OpenStream
  open one uni-stream
  write edge_id preamble
  -> Streaming

Streaming
  acquire commit
  write committed egress bytes to QUIC
  advance consume after bytes are accepted by write_all
  send/coalesce RingWritable{ring_id}
  repeat

Backpressured
  write_all is pending because network/QUIC flow control is slow
  keep ownership of unread ring bytes until write completes

Closed/Faulted
  emit coarse driver event

26. Edge Establishment

Establishment is local actor setup plus transport rendezvous. Remote edge ends do not exchange actor messages.

For each edge, the orchestrator provisions the producer and consumer:

struct ProvisionTx {
    edge_id: EdgeId,
    rx_node_id: NodeId,
    object_spec: ObjectSpec,
    ring_spec: RingSpec,
}

struct ProvisionRx {
    edge_id: EdgeId,
    object_spec: ObjectSpec,
    ring_spec: RingSpec,
}

The producer needs consumer node_id, not consumer actor address. The consumer needs shared edge_id, not producer actor address.

Each node has one EdgeEstablisher actor. It owns per-edge records:

struct EdgeRecord {
    edge_id: EdgeId,
    direction: RingDirection,
    state: EdgeProvisionState,
    lease_request_id: Option<LeaseRequestId>,
    ring_id: Option<RingId>,
    local_edge_actor: ActorAddress,
}

Record FSM:

New
  on ProvisionTx/ProvisionRx
    spawn local Tx/Rx edge actor
    create LeaseRequestId
    send LeaseRing to ArenaManager
    -> WaitingForLease

WaitingForLease
  on RingLeased matching request_id
    record ring_id
    send InstallRing to GpuWorkerCtl or token endpoint
    -> WaitingForWorkerRing
  on RingLeaseRejected matching request_id
    notify local edge actor failure
    -> Failed
  on StopEdge
    send CancelLease
    notify local edge actor stopped
    -> Stopped

WaitingForWorkerRing
  on RingInstalled
    send EstablishSend/EstablishRecv to Driver
    -> WaitingForDriver
  on RingFault or StopEdge
    -> Stopping

WaitingForDriver
  on DriverEdgeReady
    notify local Tx/Rx actor Ready
    -> Ready
  on StreamFault/RingFault/StopEdge
    -> Stopping

Ready
  hot path runs without EdgeEstablisher
  coarse object/fault events may pass through

Stopping
  stop pump if one exists
  uninstall worker ring if installed
  wait for quiescence proofs
  release ring if leased
  -> Stopped

Stale events for stopped records are ignored except that an unused fresh lease granted after cancellation must be released without installing worker or pump state.

27. Tx And Rx Edge Actors

Tx and Rx actors are role-facing lifecycle gates.

Tx actor state:

Provisioning -> Ready -> Producing -> Stopping -> Stopped
                         -> Faulted

Rx actor state:

Provisioning -> Ready -> LoadingObject -> ObjectReady -> Stopping -> Stopped
                         -> Faulted

They receive coarse events:

EdgeReady { edge_id: EdgeId }
ObjectLoaded { edge_id: EdgeId, object_id: ObjectId, device_handle: DeviceObjectHandle }
ObjectProduced { edge_id: EdgeId, object_id: ObjectId }
ObjectFailed { edge_id: EdgeId, object_id: Option<ObjectId>, reason: ObjectFailure }
StreamFault { edge_id: EdgeId, reason: StreamFaultReason }
StopEdge { edge_id: EdgeId }

They do not receive:

bytes
host pointers
per-range readiness
consumed ranges
free-space events
flow-control credits

28. GPU Worker Process Boundary

Each node has one GpuWorkerCtl actor. It owns:

worker process spawn and termination through swactor_process
worker generation numbering
arena fd inheritance or passing setup
process actor address and notification bridge
table of installed rings for current worker generation
routing worker events to EdgeEstablisher, Driver, Tx/Rx actors, and role layer
crash detection and crash fanout
restart policy

GpuWorkerCtl does not own:

arena leases
QUIC streams
ring payload bytes
device allocations
tinygrad execution
graph placement

The worker process owns:

mapped arena view
worker-side ring handles
per-ring parser/producer state
device allocations
device object handles
tinygrad role state
KV cache and worker-internal state
host-to-device and device-to-host copy scheduling
copy completion tracking

The public Rust-side interface is a normal actor message enum:

enum GpuWorkerCtlMsg {
    StartWorker,
    InstallRing(InstallRing),
    UninstallRing(UninstallRing),
    RingReadable { ring_id: RingId },
    RingWritable { ring_id: RingId },
    ConfigureRole(ConfigureRole),
    ExecuteStep(ExecuteStep),
    ReleaseDeviceObject { device_handle: DeviceObjectHandle },
    ShutdownWorker(ShutdownWorker),
    Process(ProcessNotification),
}

Process(ProcessNotification) is delivered by a small ProcessBridge actor. GpuWorkerCtl sends process input through ProcessCommand::WriteStdin.

The process adapter may use newline-delimited JSON for worker commands/events. This is an adapter, not a second distributed protocol.

Adapter rules:

one command/event JSON object per line
stdout is reserved for worker events
stderr is reserved for logs and diagnostics
payload bytes are forbidden in JSON
invalid JSON or unknown event shape is a worker/process fault

Worker environment:

SWACTOR_ARENA_FD      memfd for the shared arena
SWACTOR_ARENA_BYTES   arena reservation ceiling

29. GPU Worker Commands

InitializeWorker:

struct InitializeWorker {
    worker_generation: WorkerGeneration,
    arena_ceiling: u64,
    required_ring_helper_abi: u16,
    backend: JsonValue,
}

Maps the arena, initializes native helper and backend, then emits WorkerReady or WorkerFatal.

ConfigureRole is optional:

struct ConfigureRole {
    role_id: RoleId,
    config: JsonValue,
}

config is low-frequency app metadata and must not carry payload bytes.

InstallRing:

struct InstallRing {
    ring_id: RingId,
    edge_id: EdgeId,
    port_id: PortId,
    direction: RingDirection,
    layout: RingLayout,
    object_spec: ObjectSpec,
}

edge_id and port_id are required so worker events can be reported in graph-facing terms.

UninstallRing:

struct UninstallRing {
    ring_id: RingId,
    reason: UninstallReason,
}

The worker removes ring state, waits for copy lifetimes to end, closes the helper handle, and emits RingQuiesced.

Wake hints:

RingReadable { ring_id: RingId }
RingWritable { ring_id: RingId }

ExecuteStep:

struct ExecuteStep {
    role_id: RoleId,
    step_id: StepId,
    inputs: Vec<InputBinding>,
    outputs: Vec<OutputBinding>,
    runtime: JsonValue,
    release_inputs_after: bool,
}

struct InputBinding {
    port_id: PortId,
    object_id: ObjectId,
    sequence: Sequence,
    device_handle: DeviceObjectHandle,
}

struct OutputBinding {
    port_id: PortId,
    ring_id: RingId,
    object_id: ObjectId,
    sequence: Sequence,
    extent: u64,
    flags: u32,
}

Execution requirements:

Validate role is available.
Validate input handles belong to current worker generation.
Wrap inputs as tinygrad-compatible views.
Run role code.
Validate returned outputs against declared output bindings.
Write each output object to the named egress ring.
Emit ObjectProduced after each full output object is committed.
Emit StepCompleted after all declared outputs are produced and role state updates are complete.

ReleaseDeviceObject frees a device object after no compute or copy event still references it.

ShutdownWorker:

struct ShutdownWorker {
    mode: ShutdownMode,
}

enum ShutdownMode {
    Graceful,
    AbortInFlight,
}

Deferred worker commands:

generic command accepted/rejected acks
BindDeviceObject / UnbindDeviceObject
CancelStep
AbortObject
Ping
role module/factory provisioning

30. GPU Worker Events

Worker lifecycle:

WorkerReady {
    pid: u32,
    worker_generation: WorkerGeneration,
    ring_helper_abi: u16,
    backend: JsonValue,
}

WorkerFatal {
    reason: WorkerFatalReason,
}

WorkerStopped {
    reason: WorkerStoppedReason,
}

Role events, only if ConfigureRole is used:

RoleConfigured { role_id: RoleId }
RoleFailed { role_id: RoleId, reason: RoleFailure }

Ring events:

RingInstalled { ring_id: RingId, edge_id: EdgeId, port_id: PortId }
RingFault { ring_id: RingId, edge_id: EdgeId, port_id: PortId, reason: RingFaultReason }
RingQuiesced { ring_id: RingId }

Object events:

ObjectLoaded {
    ring_id: RingId,
    edge_id: EdgeId,
    port_id: PortId,
    object_id: ObjectId,
    sequence: Sequence,
    extent: u64,
    device_handle: DeviceObjectHandle,
}

ObjectProduced {
    ring_id: RingId,
    edge_id: EdgeId,
    port_id: PortId,
    object_id: ObjectId,
    sequence: Sequence,
    extent: u64,
}

ObjectFailed {
    ring_id: RingId,
    edge_id: EdgeId,
    port_id: PortId,
    object_id: Option<ObjectId>,
    sequence: Option<Sequence>,
    reason: ObjectFailure,
}

Step events:

StepCompleted { role_id: RoleId, step_id: StepId }
StepFailed { role_id: RoleId, step_id: StepId, reason: StepFailure }

Device lifetime:

DeviceObjectReleased { device_handle: DeviceObjectHandle }
ReleaseFailed { device_handle: DeviceObjectHandle, reason: ReleaseFailure }

Wake hints emitted by worker:

ingress rings: RingWritable after advancing consume
egress rings: RingReadable after advancing commit

GpuWorkerCtl may synthesize WorkerCrashed and RingFault after process exit, process error, or stdout control-stream failure.

31. GpuWorkerCtl And Worker FSMs

GpuWorkerCtl FSM:

NotStarted
  on StartWorker -> Spawning

Spawning
  spawn ProcessActor with ProcessSpec
  spawn ProcessBridge
  subscribe bridge to ProcessActor
  wait for ProcessNotification::Started
  send InitializeWorker through ProcessCommand::WriteStdin
  -> Initializing

Initializing
  on WorkerReady -> Running
  on WorkerFatal/process exit/timeout -> Failed

Running
  on actor command -> validate state, write worker command, update tables
  on worker event -> route event
  on ShutdownWorker -> Stopping
  on process exit/error -> Crashed

Stopping
  send ShutdownWorker if process alive
  wait for WorkerStopped and process exit
  on timeout -> Killing

Killing
  close/kill ProcessActor according to policy
  reap process notification
  mark installed rings faulted
  -> Stopped

Crashed
  mark current-generation device handles invalid
  mark installed rings faulted
  ask driver to stop pumps for installed rings
  wait for teardown to release rings
  -> Failed or Restarting

Restarting
  increment worker_generation
  -> Spawning

Worker process FSM:

Booting
  read environment
  wait for InitializeWorker on stdin

Initializing
  map arena
  initialize native helper
  initialize backend/tinygrad
  emit WorkerReady
  -> Running

Running
  on InstallRing -> install ring state
  on RingReadable/RingWritable -> reload cursors and advance ring FSMs
  on ExecuteStep -> run explicit role step subject to backpressure
  on ReleaseDeviceObject -> release handle when safe
  on ShutdownWorker -> Draining

Draining
  reject new work
  finish or abort in-flight work according to mode
  quiesce rings
  emit WorkerStopped
  exit

Fatal
  emit WorkerFatal if possible
  exit non-zero

The worker greedily makes progress after relevant commands or wake hints: drain readable ingress prefixes, advance egress output if writable space exists, observe copy completions, and emit resulting events.

32. Worker Ring FSMs

Ingress ring FSM:

Uninstalled
  on InstallRing(direction = Ingress) -> NeedHeader

NeedHeader
  on committed bytes < header_len -> wait
  on complete header -> validate
    valid -> allocate device object -> NeedPayload
    invalid -> Faulted

NeedPayload
  copy committed payload prefixes to device allocation
  advance consume only after copied bytes are safe to release
  if copied == extent -> ObjectComplete
  on close before copied == extent -> Faulted

ObjectComplete
  wait for copy completion
  emit ObjectLoaded
  -> NeedHeader

Faulted
  emit ObjectFailed or RingFault
  stop consuming until UninstallRing

Uninstalling
  stop consuming
  wait for copy lifetimes to end
  emit RingQuiesced
  -> Uninstalled

Egress ring FSM:

Uninstalled
  on InstallRing(direction = Egress) -> WaitingForOutput

WaitingForOutput
  on ExecuteStep output binding naming this ring -> NeedWritableForHeader

NeedWritableForHeader
  wait for writable span
  write ObjectHeader
  advance commit
  emit RingReadable
  -> NeedWritableForPayload

NeedWritableForPayload
  copy device payload prefixes into writable ring spans
  advance commit only after host bytes are valid
  emit/coalesce RingReadable
  if produced == extent -> ObjectProduced
  -> WaitingForOutput

Faulted
  emit RingFault
  stop producing until UninstallRing

Uninstalling
  stop accepting outputs
  wait for copy lifetimes to end
  emit RingQuiesced
  -> Uninstalled

The egress producer may block an ExecuteStep while waiting for egress ring space. This is the intended backpressure path.

33. Device Bridge

The worker must provide a backend-specific device bridge:

alloc_device(ObjectSpec, extent) -> DeviceAllocation
free_device(DeviceAllocation)
host_to_device(arena_ptr, len, DeviceAllocation, device_offset) -> CopyEvent
device_to_host(DeviceAllocation, device_offset, arena_ptr, len) -> CopyEvent
copy_event_complete(CopyEvent) -> bool
wrap_for_tinygrad(DeviceAllocation, TensorViewSpec) -> tinygrad object

The canonical contract supports range copies so objects may be larger than a ring. Ingress and egress can then stream prefixes through bounded host rings while the logical object lives in device memory.

If an implementation uses a constrained complete-object fallback, that mode must be explicit and must preserve system-visible object, sequence, readiness, and backpressure semantics. It must also constrain object/ring sizing accordingly.

For synchronous copies:

copy returns -> safe to advance relevant cursor

For asynchronous copies:

start copy -> record CopyEvent and host/device ranges
event complete -> advance cursor or release device object

34. Ingress And Egress Flow

Ingress path:

remote worker/GPU -> remote egress ring -> QUIC -> local ingress ring
  -> local worker -> local GPU memory -> ObjectLoaded

Receiving sequence:

Driver demux reads stream edge_id preamble.
Driver pairs stream with local ingress ring.
Recv-pump waits for free ring space.
Recv-pump reads QUIC bytes into free ring span.
Recv-pump advances commit.
Recv-pump sends/coalesces RingReadable.
Worker wakes and reads consume..commit.
Worker parses object headers when enough committed bytes exist.
Worker allocates device object after header validation.
Worker copies committed payload prefixes into device memory.
Worker advances consume after copies are safe to release.
Recv-pump sees free space and continues.
Worker emits ObjectLoaded after full object and copy completion.

Egress path:

local GPU memory -> local worker -> local egress ring -> QUIC
  -> remote ingress ring -> remote worker/GPU

Sending sequence:

Worker receives or creates compute result in GPU memory.
Worker creates ObjectHeader according to ObjectSpec.
Worker waits for free egress ring space.
Worker writes and commits header bytes.
Worker copies payload bytes from device into writable ring spans.
Worker advances commit as host bytes become valid.
Worker sends/coalesces RingReadable.
Send-pump opens persistent uni-stream on first bytes.
Send-pump writes edge_id preamble once.
Send-pump writes committed bytes to QUIC.
Send-pump advances consume after write_all accepts bytes.
Send-pump sends/coalesces RingWritable.

An object may be larger than the ring. The ring is a transfer window, not object storage. Compute visibility still begins only at ObjectLoaded.

35. Backpressure

No credits or per-range acknowledgements exist.

Worker slow on ingress:

worker slow
  -> ingress consume does not advance
  -> recv-pump sees no free ingress space
  -> recv-pump stops reading QUIC
  -> QUIC stream flow control stalls remote sender
  -> sender send-pump stops draining egress ring
  -> sender worker stalls on egress free space

Network slow on egress:

network slow
  -> send-pump write_all remains pending
  -> egress consume does not advance
  -> worker sees no free egress space
  -> worker stops producing more bytes into egress ring

Arena pressure:

arena temporarily exhausted
  -> ArenaManager queues LeaseRing
  -> EdgeEstablisher record remains WaitingForLease
  -> no worker ring is installed
  -> no pump is spawned
  -> no hot-path state exists for that edge

No data is dropped because each layer stops before overwriting unread bytes.

36. Fault Semantics

A run has one terminal outcome:

completed
faulted
operator-stopped before completion

Fault sources include:

node unavailable before or during run
membership loss for a required node
provisioning rejection
arena boot failure
oversized ring request
temporary pressure timeout
weight download/load/bind failure
edge establishment failure
malformed object header
EOF mid-object
stream fault
pump failure
ring fault
worker fatal error
worker crash
device OOM
device copy failure
sequence violation
step failure
teardown timeout

MVP recovery policy is fail-stop at the run level. The orchestrator records the first run-level failure reason, stops injecting tokens, and begins teardown. It does not re-place the run.

After a stage faults, it rejects new run work until stopped.

37. Error Taxonomy

Ring faults:

enum RingFaultReason {
    UnsupportedRingVersion,
    RingLayoutInvalid,
    RingStateInvalid,
    DeviceOutOfMemory,
    DeviceCopyFailed,
    SequenceViolation,
    HeaderMalformed,
    WorkerProcessExited,
    WorkerShuttingDown,
    WorkerInternal,
}

Object failures:

enum ObjectFailure {
    HeaderMalformed,
    UnsupportedObjectVersion,
    ExtentExceedsMax,
    ExtentAlignmentInvalid,
    SequenceViolation,
    DeviceAllocationFailed,
    DeviceCopyFailed,
    EofMidObject,
}

Step failures:

enum StepFailure {
    RoleUnavailable,
    InvalidInputHandle,
    InvalidOutputRing,
    TinygradError,
    DeviceOutOfMemory,
    OutputExtentInvalid,
    OutputCopyFailed,
    WorkerProcessExited,
    WorkerShuttingDown,
}

Worker fatal reasons:

enum WorkerFatalReason {
    ArenaMapFailed,
    RingHelperAbiMismatch,
    BackendInitFailed,
    MalformedControlMessage,
    UnhandledException,
}

Other control reasons:

enum UninstallReason {
    EdgeStopped,
    RingFaulted,
    WorkerShutdown,
}

enum WorkerStoppedReason {
    Graceful,
    AbortInFlight,
    Fatal,
}

38. Teardown

Teardown starts after completion, fault, or operator stop.

Orchestrator teardown:

Stop injecting tokens.
Send StopRun to every provisioned stage.
Stop local token endpoints.
Wait for StageStopped from every stage or timeout.
Record run teardown completion.

Stage teardown:

Stop accepting new run work.
Stop local edges.
Stop driver pumps.
Uninstall worker rings.
Release arena leases after quiescence proof.
Release per-run device objects.
Stop or reset worker role state according to local policy.
Emit StageStopped.

Edge teardown FSM:

Ready
  on StopEdge or fault -> StoppingPump

StoppingPump
  driver stops recv/send pump
  driver emits PumpStopped
  -> StoppingWorkerRing

StoppingWorkerRing
  GpuWorkerCtl sends UninstallRing
  worker removes ring state
  worker waits for in-flight copies or process death
  worker emits RingQuiesced
  -> ReleasingArena

ReleasingArena
  EdgeEstablisher sends ReleaseRing{proof}
  ArenaManager returns lease to free list
  -> Stopped

Arena release is safe only after:

driver pump stopped
worker ring quiesced or worker process reaped
copy lifetimes ended or owning process is gone

Teardown is required after success and fault.

39. System Guarantees

Authority:

orchestrator is the only component that assigns topology, stages, layer ranges, edge ids, and object specs
nodes reject provisioning and edge data that do not match active run plan

Readiness:

orchestrator does not inject prompt before global readiness barrier
a stage does not report StageReady before worker, weights, inbound edge, outbound edge, and StageController are ready

Weight use:

a stage cannot execute before assigned weights are loaded and bound
weight loading failures become stage faults

Edge identity:

every edge id is unique within run
every edge has one producer and one consumer
data plane is addressed by (node_id, edge_id)
remote actor addresses are not needed for data flow

Sequence:

prefill is sequence 0
decode sequences are strictly increasing
a stage executes sequence s only after loading inbound object sequence s
stage output object uses sequence s
orchestrator injects s + 1 only after consuming output token sequence s
workers do not invent graph-visible object ids or sequence numbers

Payload isolation:

payload bytes never travel in actor messages, process control messages, logs, or JSON command/event lines
actors carry identities, lifecycle events, wake hints, and opaque handles

Compute visibility:

compute receives only complete logical objects
ObjectLoaded is emitted only after valid header, exact extent copy, copy completion, and device handle creation

Backpressure:

slow worker, slow network, or downstream stall propagates by ring and QUIC flow control
unread bytes are not overwritten

Terminal outcome:

each run records exactly one terminal outcome
after terminal outcome begins, no new run work is accepted except teardown

Quiescence:

arena memory cannot be reused under a live pump, worker ring, or copy operation
release requires teardown proof

Worker restart:

restart creates a new generation
old device handles, roles, rings, and steps are invalid
roles and rings must be reinstalled

40. Observability Surface

The system must emit stable lifecycle events for behavioral contracts and tests. Event transport and storage are implementation details.

Required event identities:

run_id
node_id
stage_index
edge_id
ring_id
object_id
sequence
step_id
worker_generation

Required lifecycle events:

node_started
node_available
node_faulted
pool_ready
run_planned
stage_provision_started
weights_download_started
weights_downloaded
weights_loaded
edge_provision_started
edge_ready
stage_ready
readiness_barrier_passed
prompt_injected
object_loaded
execute_step_started
object_produced
step_completed
token_received
run_completed
run_faulted
stop_run_sent
stage_stopped
run_torn_down

Fault events must include a stable reason enum and the component that detected the fault. Tests and operators should not need to scrape free-form logs to determine lifecycle progress.

41. Behavioral Contract Inventory

The next specification layer should extract API surfaces and behavioral contracts for these components and required behaviors:

node boot lifecycle
membership/SWIM convergence
resource inventory and run planner
orchestrator run FSM
StageController
stage-local weight loading path
orchestrator token endpoint behavior
ArenaManager
shared ring helper/ABI
EdgeEstablisher
Tx and Rx edge actors
iroh driver and recv/send pumps
GpuWorkerCtl
GPU worker process adapter
GPU worker ingress parser
GPU worker egress producer
device bridge
observability/event surface

Required test layers:

Component behavioral contract tests.
Local end-to-end mock tests without real networking.
Local Docker cluster tests with real ports and real networking.

VastAI smoke/integration validation comes after local Docker cluster behavior is stable.

42. Canonical MVP Scenario

The first end-to-end scenario is one "hello world" prompt over a rented GPU pool:

Operator provisions N GPU nodes with the barebones swactor + tinygrad/CUDA image.
Every node starts the Rust node process and GPU worker process.
Nodes join SWIM membership.
Orchestrator observes PoolReady.
Orchestrator builds a linear GGUF RunPlan.
Orchestrator provisions every stage.
Each stage downloads and loads assigned weights.
Each stage provisions inbound and outbound edges.
Every stage reports StageReady.
Orchestrator observes the global readiness barrier.
Orchestrator tokenizes and injects prompt as token object sequence 0.
Stages execute prefill and return token sequence 0.
Orchestrator continues decode until EOS or max_tokens.
Orchestrator records run_completed.
Orchestrator tears down edges, worker run state, and token endpoints.
Orchestrator records run_torn_down.

Success requires one terminal run outcome and every arena range leased by the run to be quiesced or released by teardown.

43. Deferred

automatic placement optimization
arbitrary graph execution
multi-input joins and fan-out beyond explicit role-layer handling
multiple concurrent runs on one stage chain
batching, speculative decoding, continuous serving
warm model/weight reuse across prompts
production artifact layout and weight cache eviction
re-placement after node failure
trustless verification or adversarial payload defense
host pinning and asynchronous DMA performance policy
removing backend copy limitations
VastAI-specific provisioning automation

57 KiB Raw Blame History