zacheryasc/swactor
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
swactor/GPU_WORKER_INTERFACE_SPEC.md

39 KiB
Raw Blame History

GPU Worker Interface and Control Specification

Status: draft design specification for review.

Relationship to other documents. This document extends RING_BACKPRESSURE_SPEC.md. The ring spec remains the authority for shared memory rings, arena leases, QUIC pumps, ring cursor safety, object records, and backpressure. DESIGN_DIRECTIVES.md remains steering context. This document defines the process-facing control surface: how the Rust node process starts, drives, supervises, and tears down the Python/tinygrad GPU worker.

The design is intentionally a swactor-integrated worker, not a custom runtime. The Rust-side interface is a small actor message FSM. The process boundary uses the existing swactor_process actor pattern: a domain actor owns a ProcessActor, writes process input through ProcessCommand::WriteStdin, and receives lifecycle/output through ProcessNotification.

Current code is pattern guidance for this document, especially the InferenceActor and StageActor process supervision shape. Current payload movement code is not otherwise authoritative.

1. Scope

This document specifies:

  • the node process to GPU worker process boundary
  • the swactor messages used to drive the worker
  • worker boot, ring installation, execution, teardown, and crash handling
  • GPU worker responsibilities and non-responsibilities
  • device object lifetime
  • how tinygrad execution is driven by the control plane
  • correctness and safety invariants

This document does not specify:

  • placement policy
  • model graph partitioning
  • role provisioning beyond the minimal control hook needed by execution
  • QUIC stream implementation details
  • actor runtime internals
  • trustless verification
  • high-performance host pinning policy beyond the required safety contract

2. Design Commitments

The GPU worker is driven by the node control plane. It is not an autonomous graph scheduler.

The Rust-side worker controller is a swactor actor. Other actors send it typed messages; it handles those messages serially according to its current state.

The worker process does not run a separate swactor runtime. It is treated as a process-backed mailbox endpoint owned by the controller actor.

The worker never receives payload bytes in actor/control messages. Payload bytes move through shared-memory rings only.

Actors and the worker controller exchange coarse lifecycle events, terminal events, and wake hints. They do not exchange byte ranges, free byte counts, host pointers, tensor payloads, or fragments.

The worker owns GPU-visible state. The node process owns arena allocation, transport, edge lifecycle, process supervision, and graph ordering.

The role layer drives compute explicitly with ExecuteStep. Loading an ingress object into GPU memory does not automatically run tinygrad.

3. Process Topology

Each node has:

node process (Rust)                         GPU worker process
-------------------                         ------------------
swactor runtime                             Python control loop
GpuWorkerCtl actor                          native ring helper
swactor_process::ProcessActor               tinygrad role code
ArenaManager                                device bridge
Driver and QUIC pumps

shared memfd arena mapped once by both processes
ProcessActor-owned stdin/stdout/stderr pipes for process control and diagnostics

GpuWorkerCtl is the Rust-side owner of the GPU worker process. It is the only domain actor that sends worker commands or interprets worker events.

Driver pumps and edge actors do not talk to the worker process directly. They send worker-bound messages to GpuWorkerCtl. GpuWorkerCtl serializes those commands onto the process actor.

The process actor remains the only code that owns OS pipes, child process lifecycle, and process notifications. This matches the existing InferenceActor/StageActor pattern:

domain actors
  -> GpuWorkerCtlMsg
  -> GpuWorkerCtl actor
  -> ProcessCommand::WriteStdin { data }
  -> ProcessActor
  -> worker stdin

worker stdout/stderr
  -> ProcessNotification::Output
  -> ProcessBridge
  -> GpuWorkerCtlMsg::Process
  -> GpuWorkerCtl actor
  -> domain events

The worker process maps the arena once at startup. The worker uses the native ring helper for all cross-process ring cursor operations. Python code does not implement shared atomics, wrap arithmetic, cursor publication, or ring span calculation.

4. Components and Ownership

4.1 GpuWorkerCtl

GpuWorkerCtl owns:

  • worker process spawn and termination through swactor_process
  • worker generation numbering
  • arena fd inheritance or fd passing setup
  • the process actor address and process notification bridge
  • the Rust-side table of installed rings for the current worker generation
  • routing worker events to EdgeEstablisher, Driver, Tx/Rx actors, and the role layer
  • crash detection and crash fanout
  • restart policy

GpuWorkerCtl does not own:

  • arena leases
  • QUIC streams
  • ring payload bytes
  • device allocations
  • tinygrad execution
  • graph placement

4.2 GPU Worker Process

The worker process owns:

  • its mapped view of the arena
  • worker-side ring handles
  • per-ring parser and producer state
  • device allocations
  • device object handles
  • tinygrad role state
  • worker-internal state such as KV cache
  • host-to-device and device-to-host copy scheduling
  • copy completion tracking

The worker process does not own:

  • arena allocation or lease reuse
  • edge establishment
  • peer routing
  • QUIC stream creation
  • actor addresses
  • graph-level ordering outside the received ExecuteStep commands

4.3 Native Ring Helper

The native ring helper is linked or loaded by the worker process and used by Rust hot-path code. It exposes the operations needed to:

  • open the shared arena and open/close rings by layout
  • read committed spans
  • advance consume after bytes are safe to release
  • read writable spans
  • advance commit after bytes are valid
  • inspect ring state for fault/debug handling

The helper may also expose wake pending-bit helpers, but wake ownership remains outside payload ownership. Ring cursors are the authority.

4.4 Device Bridge

The worker must have a backend-specific device bridge capable of range copies:

alloc_device(ObjectSpec, extent) -> DeviceAllocation
free_device(DeviceAllocation)
host_to_device(arena_ptr, len, DeviceAllocation, device_offset) -> CopyEvent
device_to_host(DeviceAllocation, device_offset, arena_ptr, len) -> CopyEvent
copy_event_complete(CopyEvent) -> bool
wrap_for_tinygrad(DeviceAllocation, TensorViewSpec) -> tinygrad object

A high-level API that can only copy a complete host buffer into a complete tensor is insufficient. Objects may be larger than a ring, so ingress and egress require partial range copies.

For the MVP, copies may be synchronous. If copies are asynchronous, the worker must not advance a ring cursor past bytes still used by DMA.

5. Actor and Process Message Boundary

The public Rust-side interface is a normal swactor actor message enum.

enum GpuWorkerCtlMsg {
    StartWorker,
    InstallRing(InstallRing),
    UninstallRing(UninstallRing),
    RingReadable { ring_id: RingId },
    RingWritable { ring_id: RingId },
    ConfigureRole(ConfigureRole),
    ExecuteStep(ExecuteStep),
    ReleaseDeviceObject { device_handle: DeviceObjectHandle },
    ShutdownWorker(ShutdownWorker),
    Process(ProcessNotification),
}

Process(ProcessNotification) is delivered by a small ProcessBridge actor, exactly like the existing process-backed inference and pipeline actors.

There is no custom length-prefixed frame, no control envelope, no protocol version field on every message, and no generic command id/reply id layer. Correlation uses the domain identifiers already present in the command: ring_id, edge_id, port_id, object_id, sequence, step_id, and device_handle.

5.1 Process Adapter Encoding

The process adapter may encode worker commands and events as one JSON object per line on stdin/stdout. This is an implementation adapter for a Python subprocess, not a separate protocol layer.

Rules:

  • one serialized command or event per line
  • stdout is reserved for worker events
  • stderr is reserved for logs and diagnostics
  • payload bytes are forbidden in command/event JSON
  • invalid JSON or unknown event shape is a worker/process fault from GpuWorkerCtl's perspective

This keeps the Python worker easy to inspect while preserving the actual swactor boundary: actors exchange typed Rust messages with GpuWorkerCtl, and GpuWorkerCtl uses ProcessActor as the subprocess mailbox adapter.

5.2 Process Environment

At spawn, the node makes the arena fd available to the worker and passes the fd number through environment:

SWACTOR_ARENA_FD      memfd for the shared arena
SWACTOR_ARENA_BYTES   arena reservation ceiling

The worker reads commands from stdin and writes events to stdout. It may write logs to stderr. It must not write logs to stdout.

6. Identifiers and Minimal Data Types

struct WorkerGeneration(u64);
struct RoleId(u64);
struct PortId(u64);
struct RingId(u64);
struct EdgeId(u64);
struct ObjectId(u64);
struct Sequence(u64);
struct StepId(u64);

RingId is unique for the node lifetime, as specified by the ring spec.

JsonValue means a serde_json::Value-style opaque configuration value used only for low-frequency app metadata. It must not carry payload bytes.

DeviceObjectHandle is opaque to the node process:

struct DeviceObjectHandle {
    worker_generation: u64,
    id: u64,
}

A device handle is valid only inside the worker generation that created it. A worker restart invalidates every previous handle, even if a later worker maps the same arena.

6.1 Ring Layout and Object Spec

RingLayout, ObjectSpec, and ObjectHeader are defined by RING_BACKPRESSURE_SPEC.md. This document only relies on these facts:

  • RingLayout contains arena-relative offsets and ring capacity; it never contains process-local pointers
  • ObjectHeader supplies runtime facts: object_id, sequence, extent, and flags
  • ObjectSpec supplies the role-known validation contract: object kind, max extent, dtype/shape/layout, alignment, and sequence policy

6.2 Role Configuration

Role provisioning is intentionally not designed here. The current worker may be single-role and may choose its role through ProcessSpec args/env at spawn time.

ConfigureRole exists only as a low-frequency control hook for deployments that need runtime role configuration before execution:

struct ConfigureRole {
    role_id: RoleId,
    config: JsonValue,
}

config is opaque to this spec. It must not carry payload bytes. Detailed module loading, factory selection, versioning, and persistent binding APIs are deferred until role provisioning is actually implemented.

7. Commands Sent To The Worker

These are the worker commands that solve current control problems. Anything not listed here is deferred until a concrete caller needs it.

7.1 InitializeWorker

Sent once after the process starts.

struct InitializeWorker {
    worker_generation: WorkerGeneration,
    arena_ceiling: u64,
    required_ring_helper_abi: u16,
    backend: JsonValue,
}

The worker maps the arena fd, initializes the native helper and backend, and emits WorkerReady or WorkerFatal.

7.2 ConfigureRole

Optional. Sent only when the app needs runtime role configuration.

struct ConfigureRole {
    role_id: RoleId,
    config: JsonValue,
}

The worker records enough role state to execute later ExecuteStep commands. It does not run compute.

7.3 InstallRing

struct InstallRing {
    ring_id: RingId,
    edge_id: EdgeId,
    port_id: PortId,
    direction: RingDirection,
    layout: RingLayout,
    object_spec: ObjectSpec,
}

enum RingDirection {
    Ingress,
    Egress,
}

edge_id and port_id are required. The worker must be able to report ObjectLoaded, ObjectProduced, and RingFault in role/edge terms without guessing from ring_id.

For ingress, the worker creates a parser state machine in NeedHeader.

For egress, the worker creates a producer state machine waiting for an ExecuteStep output that names the ring.

Terminal events:

RingInstalled { ring_id, edge_id, port_id }
RingFault { ring_id, edge_id, port_id, reason }

7.4 UninstallRing

struct UninstallRing {
    ring_id: RingId,
    reason: UninstallReason,
}

The worker removes the ring from active state, stops parsing or producing on the ring, waits for copy lifetimes to end, and closes the native helper handle.

Terminal event:

RingQuiesced { ring_id }

7.5 RingReadable

RingReadable { ring_id: RingId }

Wake hint to the worker. For ingress rings, it tells the worker that committed bytes may be available. The worker must reload ring cursors from shared memory.

Duplicate hints may be coalesced.

7.6 RingWritable

RingWritable { ring_id: RingId }

Wake hint to the worker. For egress rings, it tells the worker that free space may be available. The worker must reload ring cursors from shared memory.

Duplicate hints may be coalesced.

7.7 ExecuteStep

struct ExecuteStep {
    role_id: RoleId,
    step_id: StepId,
    inputs: Vec<InputBinding>,
    outputs: Vec<OutputBinding>,
    runtime: JsonValue,
    release_inputs_after: bool,
}

struct InputBinding {
    port_id: PortId,
    object_id: ObjectId,
    sequence: Sequence,
    device_handle: DeviceObjectHandle,
}

struct OutputBinding {
    port_id: PortId,
    ring_id: RingId,
    object_id: ObjectId,
    sequence: Sequence,
    extent: u64,
    flags: u32,
}

The role layer assigns output object_id and sequence. The worker does not invent graph-visible ordering.

Execution requirements:

  1. Validate the role is available.
  2. Validate input handles belong to the current worker generation.
  3. Wrap input handles as tinygrad-compatible views.
  4. Run the role code.
  5. Validate returned outputs against the declared output bindings.
  6. Write each output object to the named egress ring.
  7. Emit ObjectProduced after each full output object has been committed.
  8. Emit StepCompleted after all declared outputs are produced and role state updates are complete.

Terminal events:

StepCompleted { role_id, step_id }
StepFailed { role_id, step_id, reason }

If release_inputs_after is true, the worker releases each input device object after the step reaches a terminal event and no backend work still references the object.

There is no separate step FSM requirement. The step behavior follows from serial message handling, the per-ring FSMs, and backpressure from writable ring space.

7.8 ReleaseDeviceObject

ReleaseDeviceObject {
    device_handle: DeviceObjectHandle,
}

The worker frees the device allocation after no tinygrad computation or copy event still references it.

Terminal events:

DeviceObjectReleased { device_handle }
ReleaseFailed { device_handle, reason }

7.9 ShutdownWorker

struct ShutdownWorker {
    mode: ShutdownMode,
}

enum ShutdownMode {
    Graceful,
    AbortInFlight,
}

For Graceful, the worker rejects new work, finishes accepted operations if possible, quiesces rings, releases device objects, emits WorkerStopped, and exits.

For AbortInFlight, the worker stops accepting new commands, faults in-flight work, quiesces rings as far as possible, emits WorkerStopped, and exits.

The timeout policy belongs to GpuWorkerCtl and ProcessActor, not to the worker command schema.

7.10 Deferred Commands

These controls are intentionally not part of the MVP:

  • generic command accepted/rejected acknowledgements
  • BindDeviceObject / UnbindDeviceObject
  • CancelStep
  • AbortObject
  • Ping
  • role module/factory provisioning

They can be added when a concrete caller needs them. Until then, the existing domain terminal events carry enough state to route success and failure.

8. Events Sent By The Worker

8.1 Worker Lifecycle Events

WorkerReady {
    pid: u32,
    worker_generation: WorkerGeneration,
    ring_helper_abi: u16,
    backend: JsonValue,
}

WorkerFatal {
    reason: WorkerFatalReason,
}

WorkerStopped {
    reason: WorkerStoppedReason,
}

WorkerReady means the arena is mapped, the native helper ABI is compatible, and the backend can accept ring installation and execution commands. It does not mean any role or ring is installed.

8.2 Role Events

Only needed if ConfigureRole is used:

RoleConfigured { role_id: RoleId }
RoleFailed { role_id: RoleId, reason: RoleFailure }

8.3 Ring Events

RingInstalled {
    ring_id: RingId,
    edge_id: EdgeId,
    port_id: PortId,
}

RingFault {
    ring_id: RingId,
    edge_id: EdgeId,
    port_id: PortId,
    reason: RingFaultReason,
}

RingQuiesced {
    ring_id: RingId,
}

8.4 Object Events

ObjectLoaded {
    ring_id: RingId,
    edge_id: EdgeId,
    port_id: PortId,
    object_id: ObjectId,
    sequence: Sequence,
    extent: u64,
    device_handle: DeviceObjectHandle,
}

ObjectProduced {
    ring_id: RingId,
    edge_id: EdgeId,
    port_id: PortId,
    object_id: ObjectId,
    sequence: Sequence,
    extent: u64,
}

ObjectFailed {
    ring_id: RingId,
    edge_id: EdgeId,
    port_id: PortId,
    object_id: Option<ObjectId>,
    sequence: Option<Sequence>,
    reason: ObjectFailure,
}

ObjectLoaded means a full ingress object has been copied into device memory and all copy events for that object have completed.

ObjectProduced means the full egress object header and payload have been committed to the egress ring. It does not mean the remote node received it.

8.5 Step Events

StepCompleted {
    role_id: RoleId,
    step_id: StepId,
}

StepFailed {
    role_id: RoleId,
    step_id: StepId,
    reason: StepFailure,
}

8.6 Device Lifetime Events

DeviceObjectReleased {
    device_handle: DeviceObjectHandle,
}

ReleaseFailed {
    device_handle: DeviceObjectHandle,
    reason: ReleaseFailure,
}

8.7 Wake Hints Sent By The Worker

RingWritable { ring_id: RingId }
RingReadable { ring_id: RingId }

For ingress rings, the worker sends RingWritable after advancing consume.

For egress rings, the worker sends RingReadable after advancing commit.

Wake hints carry no byte counts, ranges, or ownership.

8.8 GpuWorkerCtl Synthetic Events

Some events are generated by GpuWorkerCtl, not read from the worker:

WorkerCrashed {
    worker_generation: WorkerGeneration,
    status: ExitStatus,
}

RingFault {
    ring_id: RingId,
    edge_id: EdgeId,
    port_id: PortId,
    reason: RingFaultReason,
}

ExitStatus is swactor_process::ExitStatus.

GpuWorkerCtl synthesizes these events after process exit, process error, or stdout control-stream failure. The worker cannot emit them because it is already gone or no longer trustworthy. For process exit, the synthetic ring fault reason is WorkerProcessExited.

9. GpuWorkerCtl Actor FSM

GpuWorkerCtl is a normal swactor actor. Its FSM is advanced by incoming GpuWorkerCtlMsg values and by process notifications delivered through ProcessBridge.

NotStarted
  on StartWorker -> Spawning

Spawning
  spawn ProcessActor with ProcessSpec
  spawn ProcessBridge
  subscribe bridge to ProcessActor
  wait for ProcessNotification::Started
  send InitializeWorker through ProcessCommand::WriteStdin
  -> Initializing

Initializing
  on WorkerReady -> Running
  on WorkerFatal/process exit/timeout -> Failed

Running
  on actor command -> validate state, write worker command, update local tables
  on worker event -> route event to local actors and driver
  on ShutdownWorker -> Stopping
  on process exit/error -> Crashed

Stopping
  send ShutdownWorker if process is still alive
  wait for WorkerStopped and process exit
  on timeout -> Killing

Killing
  send ProcessCommand::Close or stop/kill ProcessActor according to ProcessSpec
  reap process through ProcessActor notification
  mark installed rings faulted
  -> Stopped

Crashed
  mark current-generation device handles invalid
  mark installed rings faulted
  ask driver to stop pumps for all installed rings
  wait for teardown to release rings
  -> Failed or Restarting

Restarting
  increment worker_generation
  -> Spawning

GpuWorkerCtl must not claim a ring is quiesced merely because the worker exited. Quiescence for arena reuse still requires the edge teardown proof from the ring spec: pump stopped, worker process reaped or ring quiesced, and copy lifetimes ended or the process that owns them is gone.

10. Worker Process FSM

The worker process has one state owner and handles process commands serially. It may maintain multiple internal FSMs, especially one per installed ring, but there is no separate worker scheduler runtime requirement.

Booting
  read environment
  wait for InitializeWorker on stdin

Initializing
  map arena
  initialize native helper
  initialize backend/tinygrad
  emit WorkerReady
  -> Running

Running
  on InstallRing -> install ring state
  on RingReadable/RingWritable -> reload cursors and advance affected ring FSMs
  on ExecuteStep -> run explicit role step subject to ring backpressure
  on ReleaseDeviceObject -> release handle when safe
  on ShutdownWorker -> Draining

Draining
  reject new work
  finish or abort in-flight work according to shutdown mode
  quiesce rings
  emit WorkerStopped
  exit

Fatal
  emit WorkerFatal if possible
  exit non-zero

The worker should greedily make progress after any relevant command or wake hint: drain readable ingress prefixes, advance egress output if writable space exists, observe copy completions, and emit resulting events. The concrete internal data structures used to remember ready rings or pending copies are implementation details, provided duplicate wake hints cannot lose liveness.

11. Per-Ring Worker State

11.1 Ingress Ring FSM

Uninstalled
  on InstallRing(direction = Ingress) -> NeedHeader

NeedHeader
  on committed bytes < header_len -> wait
  on complete header -> validate
    valid -> allocate device object -> NeedPayload
    invalid -> Faulted

NeedPayload
  copy committed payload prefixes to device allocation
  advance consume only after copied bytes are safe to release
  if copied == extent -> ObjectComplete
  on close before copied == extent -> Faulted

ObjectComplete
  wait for copy completion
  emit ObjectLoaded
  -> NeedHeader

Faulted
  emit ObjectFailed or RingFault
  stop consuming until UninstallRing

Uninstalling
  stop consuming
  wait for copy lifetimes to end
  emit RingQuiesced
  -> Uninstalled

11.2 Egress Ring FSM

Uninstalled
  on InstallRing(direction = Egress) -> WaitingForOutput

WaitingForOutput
  on ExecuteStep output binding naming this ring -> NeedWritableForHeader

NeedWritableForHeader
  wait for writable span
  write ObjectHeader
  advance commit
  emit RingReadable
  -> NeedWritableForPayload

NeedWritableForPayload
  copy device payload prefixes into writable ring spans
  advance commit only after host bytes are valid
  emit/coalesce RingReadable
  if produced == extent -> ObjectProduced
  -> WaitingForOutput

Faulted
  emit RingFault
  stop producing until UninstallRing

Uninstalling
  stop accepting outputs for this ring
  wait for copy lifetimes to end
  emit RingQuiesced
  -> Uninstalled

The egress producer may block an ExecuteStep while waiting for ring space. That is the intended backpressure path.

12. Role and Step State

Role state is intentionally minimal in this spec.

Unavailable
  on WorkerReady with fixed boot role -> Ready
  on ConfigureRole -> Configuring

Configuring
  load/record app-defined role config
  on success -> Ready
  on failure -> Failed

Ready
  on ExecuteStep -> run one explicit step
  on Shutdown -> Stopping

Stopping
  reject new steps
  release role-local resources

The MVP role concurrency policy is one active ExecuteStep per role. A later worker may support multiple active steps only if the role declares that its state is reentrant or the steps are explicitly independent.

No separate step FSM is specified. The step either reaches StepCompleted or StepFailed, and any partial egress object faults its ring.

13. Message Flows

13.1 Boot

GpuWorkerCtl -> swactor_process: spawn ProcessActor
ProcessActor -> OS: spawn worker with arena fd/env
ProcessActor -> GpuWorkerCtl: ProcessNotification::Started
GpuWorkerCtl -> ProcessActor: WriteStdin(InitializeWorker JSON line)
worker: map arena once
worker: initialize native helper and backend
worker -> stdout: WorkerReady JSON line
ProcessActor -> GpuWorkerCtl: ProcessNotification::Output(stdout)
GpuWorkerCtl: parse WorkerReady; node may now install rings and execute steps

Failure before WorkerReady is a boot failure. No rings are installed, so no arena lease can be corrupted by the worker.

13.2 Install Receive Edge

EdgeEstablisher receives RingLeased
EdgeEstablisher -> GpuWorkerCtl: InstallRing(direction = Ingress, edge_id, port_id)
GpuWorkerCtl -> worker: InstallRing
worker: open native ring handle and create ingress parser
worker -> GpuWorkerCtl: RingInstalled
GpuWorkerCtl -> EdgeEstablisher: RingInstalled
EdgeEstablisher -> Driver: EstablishRecv
Driver: pair recv spec with stream when available
Driver: spawn recv-pump

The driver does not start a recv-pump until the worker has installed the ring.

13.3 Ingress Object Load

recv-pump reads QUIC bytes into free ingress ring span
recv-pump advances commit
recv-pump -> GpuWorkerCtl: RingReadable(ring_id)
GpuWorkerCtl -> worker: RingReadable(ring_id)

worker reloads commit/consume
worker parses ObjectHeader
worker validates extent and sequence
worker allocates device object
worker copies committed payload prefixes to device
worker advances consume after copies are safe
worker -> GpuWorkerCtl: RingWritable(ring_id)
GpuWorkerCtl -> recv-pump: RingWritable(ring_id)

when full object is copied and copy events complete:
worker -> GpuWorkerCtl: ObjectLoaded(..., device_handle)
GpuWorkerCtl routes ObjectLoaded to the role/Rx layer

No actor receives byte ranges. The ring cursors are the only byte ownership state.

13.4 Execute Step and Produce Output

role layer has all required input device handles
role layer chooses output object_id and sequence
role layer -> GpuWorkerCtl: ExecuteStep(inputs, outputs)
GpuWorkerCtl -> worker: ExecuteStep

worker validates handles
worker calls tinygrad role code
worker writes output ObjectHeader to egress ring
worker copies output payload to egress ring as space becomes available
worker advances commit after host bytes are valid
worker -> GpuWorkerCtl: RingReadable(egress_ring)
GpuWorkerCtl -> send-pump: RingReadable

send-pump writes committed bytes to QUIC
send-pump advances consume after write_all accepts bytes
send-pump -> GpuWorkerCtl: RingWritable(egress_ring)
GpuWorkerCtl -> worker: RingWritable

worker -> GpuWorkerCtl: ObjectProduced
worker -> GpuWorkerCtl: StepCompleted

StepCompleted follows ObjectProduced for all declared outputs. For a step with no outputs, StepCompleted follows successful compute and state update.

13.5 Release Device Object

role layer decides object is no longer needed
role layer -> GpuWorkerCtl: ReleaseDeviceObject(device_handle)
GpuWorkerCtl -> worker: ReleaseDeviceObject
worker waits until no copy or compute references the object
worker frees device allocation
worker -> GpuWorkerCtl: DeviceObjectReleased

The node process never dereferences a device handle. It can only pass it back to the worker.

13.6 Stop Edge

EdgeEstablisher -> Driver: StopEdge
Driver stops pump
Driver -> EdgeEstablisher: PumpStopped

EdgeEstablisher -> GpuWorkerCtl: UninstallRing
GpuWorkerCtl -> worker: UninstallRing
worker removes ring from active state
worker waits for copy lifetimes to end
worker -> GpuWorkerCtl: RingQuiesced
GpuWorkerCtl -> EdgeEstablisher: RingQuiesced

EdgeEstablisher -> ArenaManager: ReleaseRing(proof)

The arena lease is released only after both the driver pump and worker ring are quiesced, or after the worker process is dead and reaped.

13.7 Worker Crash

worker process exits or process actor reports error
GpuWorkerCtl marks all current-generation device handles invalid
GpuWorkerCtl emits WorkerCrashed to role layer
GpuWorkerCtl synthesizes RingFault for every installed ring
GpuWorkerCtl asks Driver to stop pumps for those rings
EdgeEstablisher waits for PumpStopped
worker side is considered gone only after process reap
EdgeEstablisher releases arena rings after teardown proof

The arena survives because it is owned by the node process. Device allocations do not survive because they were owned by the worker process/backend context.

Restart creates a new worker generation. Rings and roles must be reinstalled.

14. Error Taxonomy

14.1 RingFaultReason

enum RingFaultReason {
    UnsupportedRingVersion,
    RingLayoutInvalid,
    RingStateInvalid,
    DeviceOutOfMemory,
    DeviceCopyFailed,
    SequenceViolation,
    HeaderMalformed,
    WorkerProcessExited,
    WorkerShuttingDown,
    WorkerInternal,
}

14.2 ObjectFailure

enum ObjectFailure {
    HeaderMalformed,
    UnsupportedObjectVersion,
    ExtentExceedsMax,
    ExtentAlignmentInvalid,
    SequenceViolation,
    DeviceAllocationFailed,
    DeviceCopyFailed,
    EofMidObject,
}

14.3 StepFailure

enum StepFailure {
    RoleUnavailable,
    InvalidInputHandle,
    InvalidOutputRing,
    TinygradError,
    DeviceOutOfMemory,
    OutputExtentInvalid,
    OutputCopyFailed,
    WorkerProcessExited,
    WorkerShuttingDown,
}

14.4 WorkerFatalReason

enum WorkerFatalReason {
    ArenaMapFailed,
    RingHelperAbiMismatch,
    BackendInitFailed,
    MalformedControlMessage,
    UnhandledException,
}

14.5 Role and Release Failures

enum RoleFailure {
    InvalidRoleConfig,
    BackendUnsupported,
    WorkerRejectedRole,
}

enum ReleaseFailure {
    InvalidDeviceHandle,
    ObjectInUse,
    BackendFreeFailed,
}

14.6 Control Reasons

enum UninstallReason {
    EdgeStopped,
    RingFaulted,
    WorkerShutdown,
}

enum WorkerStoppedReason {
    Graceful,
    AbortInFlight,
    Fatal,
}

MVP policy: device OOM during ingress object allocation faults the ring. The worker does not silently skip a payload and continue to later objects. Replanning or retry is a higher-level policy.

15. Correctness and Safety Invariants

Control messages carry no payload bytes. Payload movement occurs only through arena-backed rings. This keeps actors and process stdio out of the byte hot path.

The worker cannot access an unleased arena range. InstallRing is sent only after ArenaManager emits RingLeased; the message contains the only usable ring layout.

The driver cannot pump into a ring the worker has not installed. EdgeEstablisher sends EstablishRecv or EstablishSend only after RingInstalled.

Python does not own ring atomics. All cursor loads, cursor stores, span calculation, and wrap arithmetic use the native ring helper.

Ingress compute visibility starts at ObjectLoaded. The worker emits ObjectLoaded only after header validation, exact extent copy, copy completion, and device handle creation. The role layer cannot pass a partial object to ExecuteStep because no handle exists before then.

The recv-pump cannot overwrite host bytes still needed by DMA. The worker advances ingress consume only after the corresponding host bytes are no longer read by synchronous copy or asynchronous DMA.

The send-pump cannot read unwritten egress bytes. The worker advances egress commit only after header or payload bytes are valid in host memory.

Backpressure is absence of writable ring space. If the worker is slow, ingress consume does not advance and the recv-pump stops reading QUIC. If the network is slow, egress consume does not advance and the worker stops producing more bytes into the egress ring.

Wake hints are not ownership. RingReadable and RingWritable only cause the receiver to reload cursors. Duplicate hints may be coalesced only while the ring remains discoverable.

Device handles are generation-scoped. A handle from an old worker generation is rejected by the new worker. This prevents stale handles from aliasing new device allocations after restart.

Object id and sequence ownership is explicit. Ingress object id and sequence come from the received ObjectHeader and are validated against ObjectSpec. Egress object id and sequence are assigned by the role/control layer in ExecuteStep; the worker serializes them but does not invent graph-visible ordering.

Role execution is explicit. Loading an object does not run compute. The role layer must issue ExecuteStep with concrete input handles and output bindings.

Partial egress objects fault the ring. If the worker commits an output header but cannot commit the full declared payload, the egress ring is faulted. The receiver must not interpret later bytes as the next object.

A released arena range has no live users. ReleaseRing is sent only after the driver pump is stopped and the worker has emitted RingQuiesced, or after the worker process is reaped. Copy lifetimes must have ended or the process that owned them must be gone.

Worker restart does not preserve GPU state. Roles, rings, steps, and device handles are all current-generation state. Restart requires reconfiguration.

16. Implementation Requirements

16.1 Actor Integration

The Rust-side implementation must follow the existing process-actor idiom:

  • GpuWorkerCtl is an ActorInterface implementation with one incoming enum
  • it spawns a ProcessActor through swactor_process
  • it spawns a private bridge actor for ProcessNotification
  • it sends worker input with ProcessCommand::WriteStdin
  • it closes the worker with ProcessCommand::Close and normal actor teardown
  • it never exposes process stdin/stdout handles to driver pumps or edge actors

16.2 Copy Safety

For synchronous copies:

copy returns -> safe to advance relevant cursor

For asynchronous copies:

start copy -> record CopyEvent and host/device ranges
event complete -> advance relevant cursor or release device object

The worker must track host ranges for ring cursor release and device ranges for device object lifetime.

16.3 Output Commit Discipline

For egress:

  1. The worker must know the output extent before writing the object header.
  2. The worker writes and commits the header.
  3. The worker copies payload prefixes into writable spans.
  4. The worker commits each prefix only after host bytes are valid.
  5. The worker emits ObjectProduced only after all extent bytes are committed.

If the worker cannot determine output extent before serialization, it must first materialize the output into a device object with known byte extent. The wire header cannot contain a placeholder extent.

16.4 Logs and Diagnostics

Worker events go to stdout as process-adapter messages. Diagnostic logs go to stderr.

GpuWorkerCtl may keep a stderr tail for crash diagnostics. This is diagnostic only and does not change safety rules.

17. Resolutions To Known Design Issues

17.1 No Custom Control Frames

Decision: remove length-prefixed frames, generic envelopes, per-message protocol versions, command ids, and reply ids. Rust code uses normal swactor messages. The Python subprocess adapter may use newline-delimited JSON because the codebase already uses ProcessActor stdin/stdout for process-backed workers.

Reason: this worker is integrated into swactor. A second local control protocol duplicates actor semantics without solving a current problem.

17.2 Ring Install Needs Edge and Port Identity

Decision: InstallRing includes edge_id and port_id. The worker reports all object and ring events with ring_id, edge_id, and port_id.

Reason: ring_id is a node-local transport identifier. The role and edge layers need graph-facing identity without maintaining an ambiguous reverse lookup in the worker.

17.3 Device Handle Lifetime

Decision: device handles are opaque, generation-scoped, and explicitly released with ReleaseDeviceObject.

Reason: this prevents stale handle aliasing after restart and gives the role layer explicit control over activation and cache lifetimes.

17.4 Compute Driving API

Decision: compute is driven by ExecuteStep; ConfigureRole is optional and opaque until role provisioning exists. ObjectLoaded is a data-ready event, not an execution trigger.

Reason: the control plane owns ordering and correlation.

17.5 Object Id And Sequence Ownership

Decision: ingress validates object id and sequence from the stream header. Egress object id and sequence are assigned by the role/control layer and supplied in ExecuteStep.OutputBinding.

Reason: the worker should not invent graph-visible ordering. The control plane already knows the expected ordering and correlation.

17.6 Weight Loading

Decision: weight/persistent-state specifics are deferred with role provisioning. For now, weights can be loaded as ordinary ingress objects and passed to ExecuteStep, or a fixed-role worker can load them during its own initialization from app-specific config.

Reason: this avoids designing binding APIs before the role provisioning path exists.

17.7 Multi-Port And Fan-In

Decision: rings bind to ports, and ExecuteStep lists all input handles and output rings explicitly. Fan-in is represented at the role layer by waiting for multiple ObjectLoaded events before issuing one ExecuteStep.

Reason: the ring remains SPSC and byte-level simple. Join semantics stay out of the ring and pump layers.

17.8 GPU OOM And Allocation Pressure

Decision: ingress allocation failure faults the ring in the MVP. Step-time OOM fails the step. If an egress object was partially committed, the egress ring is faulted.

Reason: skipping a payload or continuing after a partial object requires more protocol machinery and can violate stream alignment. Replanning or retry belongs above the worker boundary.

17.9 Error Separation

Decision: use separate failure classes for worker fatal errors, ring faults, object failures, release failures, and step failures.

Reason: recovery action differs. A malformed object is not the same as a process crash, and a tinygrad exception is not the same as a corrupt ring cursor.

17.10 tinygrad Range-Copy Gap

Decision: the worker implementation must provide a DeviceBridge with range copy support. If tinygrad does not expose the necessary primitive directly, the bridge must use a native backend helper or tinygrad raw-buffer API. The ring protocol must not be weakened to require extent <= ring_capacity.

Reason: the ring spec explicitly allows objects larger than rings. Partial prefix copy is required for liveness and backpressure.

17.11 Restart Semantics

Decision: restart creates a new worker generation and invalidates all device handles, role instances, ring installs, and in-flight steps. The node may reuse the same arena only after normal ring teardown proves quiescence.

Reason: GPU backend state is process-local. Treating restart as transparent would risk stale handles, leaked DMA, and mismatched role state.