# GPU Worker Interface and Control Specification **Status:** draft design specification for review. **Relationship to other documents.** This document extends `RING_BACKPRESSURE_SPEC.md`. The ring spec remains the authority for shared memory rings, arena leases, QUIC pumps, ring cursor safety, object records, and backpressure. `DESIGN_DIRECTIVES.md` remains steering context. This document defines the process-facing control surface: how the Rust node process starts, drives, supervises, and tears down the Python/tinygrad GPU worker. The design is intentionally a swactor-integrated worker, not a custom runtime. The Rust-side interface is a small actor message FSM. The process boundary uses the existing `swactor_process` actor pattern: a domain actor owns a `ProcessActor`, writes process input through `ProcessCommand::WriteStdin`, and receives lifecycle/output through `ProcessNotification`. Current code is pattern guidance for this document, especially the `InferenceActor` and `StageActor` process supervision shape. Current payload movement code is not otherwise authoritative. --- ## 1. Scope This document specifies: - the node process to GPU worker process boundary - the swactor messages used to drive the worker - worker boot, ring installation, execution, teardown, and crash handling - GPU worker responsibilities and non-responsibilities - device object lifetime - how tinygrad execution is driven by the control plane - correctness and safety invariants This document does not specify: - placement policy - model graph partitioning - role provisioning beyond the minimal control hook needed by execution - QUIC stream implementation details - actor runtime internals - trustless verification - high-performance host pinning policy beyond the required safety contract --- ## 2. Design Commitments The GPU worker is driven by the node control plane. It is not an autonomous graph scheduler. The Rust-side worker controller is a swactor actor. Other actors send it typed messages; it handles those messages serially according to its current state. The worker process does not run a separate swactor runtime. It is treated as a process-backed mailbox endpoint owned by the controller actor. The worker never receives payload bytes in actor/control messages. Payload bytes move through shared-memory rings only. Actors and the worker controller exchange coarse lifecycle events, terminal events, and wake hints. They do not exchange byte ranges, free byte counts, host pointers, tensor payloads, or fragments. The worker owns GPU-visible state. The node process owns arena allocation, transport, edge lifecycle, process supervision, and graph ordering. The role layer drives compute explicitly with `ExecuteStep`. Loading an ingress object into GPU memory does not automatically run tinygrad. --- ## 3. Process Topology Each node has: ```text node process (Rust) GPU worker process ------------------- ------------------ swactor runtime Python control loop GpuWorkerCtl actor native ring helper swactor_process::ProcessActor tinygrad role code ArenaManager device bridge Driver and QUIC pumps shared memfd arena mapped once by both processes ProcessActor-owned stdin/stdout/stderr pipes for process control and diagnostics ``` `GpuWorkerCtl` is the Rust-side owner of the GPU worker process. It is the only domain actor that sends worker commands or interprets worker events. Driver pumps and edge actors do not talk to the worker process directly. They send worker-bound messages to `GpuWorkerCtl`. `GpuWorkerCtl` serializes those commands onto the process actor. The process actor remains the only code that owns OS pipes, child process lifecycle, and process notifications. This matches the existing `InferenceActor`/`StageActor` pattern: ```text domain actors -> GpuWorkerCtlMsg -> GpuWorkerCtl actor -> ProcessCommand::WriteStdin { data } -> ProcessActor -> worker stdin worker stdout/stderr -> ProcessNotification::Output -> ProcessBridge -> GpuWorkerCtlMsg::Process -> GpuWorkerCtl actor -> domain events ``` The worker process maps the arena once at startup. The worker uses the native ring helper for all cross-process ring cursor operations. Python code does not implement shared atomics, wrap arithmetic, cursor publication, or ring span calculation. --- ## 4. Components and Ownership ### 4.1 GpuWorkerCtl `GpuWorkerCtl` owns: - worker process spawn and termination through `swactor_process` - worker generation numbering - arena fd inheritance or fd passing setup - the process actor address and process notification bridge - the Rust-side table of installed rings for the current worker generation - routing worker events to EdgeEstablisher, Driver, Tx/Rx actors, and the role layer - crash detection and crash fanout - restart policy `GpuWorkerCtl` does not own: - arena leases - QUIC streams - ring payload bytes - device allocations - tinygrad execution - graph placement ### 4.2 GPU Worker Process The worker process owns: - its mapped view of the arena - worker-side ring handles - per-ring parser and producer state - device allocations - device object handles - tinygrad role state - worker-internal state such as KV cache - host-to-device and device-to-host copy scheduling - copy completion tracking The worker process does not own: - arena allocation or lease reuse - edge establishment - peer routing - QUIC stream creation - actor addresses - graph-level ordering outside the received `ExecuteStep` commands ### 4.3 Native Ring Helper The native ring helper is linked or loaded by the worker process and used by Rust hot-path code. It exposes the operations needed to: - open the shared arena and open/close rings by layout - read committed spans - advance consume after bytes are safe to release - read writable spans - advance commit after bytes are valid - inspect ring state for fault/debug handling The helper may also expose wake pending-bit helpers, but wake ownership remains outside payload ownership. Ring cursors are the authority. ### 4.4 Device Bridge USER: If it is determined this is an outsized amount of engineering work, we will defer and require objects to be complete before gpu io. The worker must have a backend-specific device bridge capable of range copies: ```text alloc_device(ObjectSpec, extent) -> DeviceAllocation free_device(DeviceAllocation) host_to_device(arena_ptr, len, DeviceAllocation, device_offset) -> CopyEvent device_to_host(DeviceAllocation, device_offset, arena_ptr, len) -> CopyEvent copy_event_complete(CopyEvent) -> bool wrap_for_tinygrad(DeviceAllocation, TensorViewSpec) -> tinygrad object ``` A high-level API that can only copy a complete host buffer into a complete tensor is insufficient. Objects may be larger than a ring, so ingress and egress require partial range copies. For the MVP, copies may be synchronous. If copies are asynchronous, the worker must not advance a ring cursor past bytes still used by DMA. --- ## 5. Actor and Process Message Boundary The public Rust-side interface is a normal swactor actor message enum. ```rust enum GpuWorkerCtlMsg { StartWorker, InstallRing(InstallRing), UninstallRing(UninstallRing), RingReadable { ring_id: RingId }, RingWritable { ring_id: RingId }, ConfigureRole(ConfigureRole), ExecuteStep(ExecuteStep), ReleaseDeviceObject { device_handle: DeviceObjectHandle }, ShutdownWorker(ShutdownWorker), Process(ProcessNotification), } ``` `Process(ProcessNotification)` is delivered by a small `ProcessBridge` actor, exactly like the existing process-backed inference and pipeline actors. There is no custom length-prefixed frame, no control envelope, no protocol version field on every message, and no generic command id/reply id layer. Correlation uses the domain identifiers already present in the command: `ring_id`, `edge_id`, `port_id`, `object_id`, `sequence`, `step_id`, and `device_handle`. ### 5.1 Process Adapter Encoding The process adapter may encode worker commands and events as one JSON object per line on stdin/stdout. This is an implementation adapter for a Python subprocess, not a separate protocol layer. Rules: - one serialized command or event per line - stdout is reserved for worker events - stderr is reserved for logs and diagnostics - payload bytes are forbidden in command/event JSON - invalid JSON or unknown event shape is a worker/process fault from `GpuWorkerCtl`'s perspective This keeps the Python worker easy to inspect while preserving the actual swactor boundary: actors exchange typed Rust messages with `GpuWorkerCtl`, and `GpuWorkerCtl` uses `ProcessActor` as the subprocess mailbox adapter. ### 5.2 Process Environment At spawn, the node makes the arena fd available to the worker and passes the fd number through environment: ```text SWACTOR_ARENA_FD memfd for the shared arena SWACTOR_ARENA_BYTES arena reservation ceiling ``` The worker reads commands from stdin and writes events to stdout. It may write logs to stderr. It must not write logs to stdout. --- ## 6. Identifiers and Minimal Data Types ```rust struct WorkerGeneration(u64); struct RoleId(u64); struct PortId(u64); struct RingId(u64); struct EdgeId(u64); struct ObjectId(u64); struct Sequence(u64); struct StepId(u64); ``` `RingId` is unique for the node lifetime, as specified by the ring spec. `JsonValue` means a `serde_json::Value`-style opaque configuration value used only for low-frequency app metadata. It must not carry payload bytes. `DeviceObjectHandle` is opaque to the node process: ```rust struct DeviceObjectHandle { worker_generation: u64, id: u64, } ``` A device handle is valid only inside the worker generation that created it. A worker restart invalidates every previous handle, even if a later worker maps the same arena. ### 6.1 Ring Layout and Object Spec `RingLayout`, `ObjectSpec`, and `ObjectHeader` are defined by `RING_BACKPRESSURE_SPEC.md`. This document only relies on these facts: - `RingLayout` contains arena-relative offsets and ring capacity; it never contains process-local pointers - `ObjectHeader` supplies runtime facts: `object_id`, `sequence`, `extent`, and flags - `ObjectSpec` supplies the role-known validation contract: object kind, max extent, dtype/shape/layout, alignment, and sequence policy ### 6.2 Role Configuration Role provisioning is intentionally not designed here. The current worker may be single-role and may choose its role through `ProcessSpec` args/env at spawn time. `ConfigureRole` exists only as a low-frequency control hook for deployments that need runtime role configuration before execution: ```rust struct ConfigureRole { role_id: RoleId, config: JsonValue, } ``` `config` is opaque to this spec. It must not carry payload bytes. Detailed module loading, factory selection, versioning, and persistent binding APIs are deferred until role provisioning is actually implemented. --- ## 7. Commands Sent To The Worker These are the worker commands that solve current control problems. Anything not listed here is deferred until a concrete caller needs it. ### 7.1 InitializeWorker Sent once after the process starts. ```rust struct InitializeWorker { worker_generation: WorkerGeneration, arena_ceiling: u64, required_ring_helper_abi: u16, backend: JsonValue, } ``` The worker maps the arena fd, initializes the native helper and backend, and emits `WorkerReady` or `WorkerFatal`. ### 7.2 ConfigureRole Optional. Sent only when the app needs runtime role configuration. ```rust struct ConfigureRole { role_id: RoleId, config: JsonValue, } ``` The worker records enough role state to execute later `ExecuteStep` commands. It does not run compute. ### 7.3 InstallRing ```rust struct InstallRing { ring_id: RingId, edge_id: EdgeId, port_id: PortId, direction: RingDirection, layout: RingLayout, object_spec: ObjectSpec, } enum RingDirection { Ingress, Egress, } ``` `edge_id` and `port_id` are required. The worker must be able to report `ObjectLoaded`, `ObjectProduced`, and `RingFault` in role/edge terms without guessing from `ring_id`. For ingress, the worker creates a parser state machine in `NeedHeader`. For egress, the worker creates a producer state machine waiting for an `ExecuteStep` output that names the ring. Terminal events: ```rust RingInstalled { ring_id, edge_id, port_id } RingFault { ring_id, edge_id, port_id, reason } ``` ### 7.4 UninstallRing ```rust struct UninstallRing { ring_id: RingId, reason: UninstallReason, } ``` The worker removes the ring from active state, stops parsing or producing on the ring, waits for copy lifetimes to end, and closes the native helper handle. Terminal event: ```rust RingQuiesced { ring_id } ``` ### 7.5 RingReadable ```rust RingReadable { ring_id: RingId } ``` Wake hint to the worker. For ingress rings, it tells the worker that committed bytes may be available. The worker must reload ring cursors from shared memory. Duplicate hints may be coalesced. ### 7.6 RingWritable ```rust RingWritable { ring_id: RingId } ``` Wake hint to the worker. For egress rings, it tells the worker that free space may be available. The worker must reload ring cursors from shared memory. Duplicate hints may be coalesced. ### 7.7 ExecuteStep USER: Why do we need this explicit? Why doesnt it fall out naturally from the I/O? This whole thing is async message driven, so why do we need a start message? ```rust struct ExecuteStep { role_id: RoleId, step_id: StepId, inputs: Vec, outputs: Vec, runtime: JsonValue, release_inputs_after: bool, } struct InputBinding { port_id: PortId, object_id: ObjectId, sequence: Sequence, device_handle: DeviceObjectHandle, } struct OutputBinding { port_id: PortId, ring_id: RingId, object_id: ObjectId, sequence: Sequence, extent: u64, flags: u32, } ``` The role layer assigns output `object_id` and `sequence`. The worker does not invent graph-visible ordering. Execution requirements: 1. Validate the role is available. 2. Validate input handles belong to the current worker generation. 3. Wrap input handles as tinygrad-compatible views. 4. Run the role code. 5. Validate returned outputs against the declared output bindings. 6. Write each output object to the named egress ring. 7. Emit `ObjectProduced` after each full output object has been committed. 8. Emit `StepCompleted` after all declared outputs are produced and role state updates are complete. Terminal events: ```rust StepCompleted { role_id, step_id } StepFailed { role_id, step_id, reason } ``` If `release_inputs_after` is true, the worker releases each input device object after the step reaches a terminal event and no backend work still references the object. There is no separate step FSM requirement. The step behavior follows from serial message handling, the per-ring FSMs, and backpressure from writable ring space. ### 7.8 ReleaseDeviceObject ```rust ReleaseDeviceObject { device_handle: DeviceObjectHandle, } ``` The worker frees the device allocation after no tinygrad computation or copy event still references it. Terminal events: ```rust DeviceObjectReleased { device_handle } ReleaseFailed { device_handle, reason } ``` ### 7.9 ShutdownWorker ```rust struct ShutdownWorker { mode: ShutdownMode, } enum ShutdownMode { Graceful, AbortInFlight, } ``` For `Graceful`, the worker rejects new work, finishes accepted operations if possible, quiesces rings, releases device objects, emits `WorkerStopped`, and exits. For `AbortInFlight`, the worker stops accepting new commands, faults in-flight work, quiesces rings as far as possible, emits `WorkerStopped`, and exits. The timeout policy belongs to `GpuWorkerCtl` and `ProcessActor`, not to the worker command schema. ### 7.10 Deferred Commands These controls are intentionally not part of the MVP: - generic command accepted/rejected acknowledgements - `BindDeviceObject` / `UnbindDeviceObject` - `CancelStep` - `AbortObject` - `Ping` - role module/factory provisioning They can be added when a concrete caller needs them. Until then, the existing domain terminal events carry enough state to route success and failure. --- ## 8. Events Sent By The Worker ### 8.1 Worker Lifecycle Events ```rust WorkerReady { pid: u32, worker_generation: WorkerGeneration, ring_helper_abi: u16, backend: JsonValue, } WorkerFatal { reason: WorkerFatalReason, } WorkerStopped { reason: WorkerStoppedReason, } ``` `WorkerReady` means the arena is mapped, the native helper ABI is compatible, and the backend can accept ring installation and execution commands. It does not mean any role or ring is installed. ### 8.2 Role Events Only needed if `ConfigureRole` is used: ```rust RoleConfigured { role_id: RoleId } RoleFailed { role_id: RoleId, reason: RoleFailure } ``` ### 8.3 Ring Events ```rust RingInstalled { ring_id: RingId, edge_id: EdgeId, port_id: PortId, } RingFault { ring_id: RingId, edge_id: EdgeId, port_id: PortId, reason: RingFaultReason, } RingQuiesced { ring_id: RingId, } ``` ### 8.4 Object Events ```rust ObjectLoaded { ring_id: RingId, edge_id: EdgeId, port_id: PortId, object_id: ObjectId, sequence: Sequence, extent: u64, device_handle: DeviceObjectHandle, } ObjectProduced { ring_id: RingId, edge_id: EdgeId, port_id: PortId, object_id: ObjectId, sequence: Sequence, extent: u64, } ObjectFailed { ring_id: RingId, edge_id: EdgeId, port_id: PortId, object_id: Option, sequence: Option, reason: ObjectFailure, } ``` `ObjectLoaded` means a full ingress object has been copied into device memory and all copy events for that object have completed. `ObjectProduced` means the full egress object header and payload have been committed to the egress ring. It does not mean the remote node received it. ### 8.5 Step Events USER: Why is this here? It seems useless. ```rust StepCompleted { role_id: RoleId, step_id: StepId, } StepFailed { role_id: RoleId, step_id: StepId, reason: StepFailure, } ``` ### 8.6 Device Lifetime Events ```rust DeviceObjectReleased { device_handle: DeviceObjectHandle, } ReleaseFailed { device_handle: DeviceObjectHandle, reason: ReleaseFailure, } ``` ### 8.7 Wake Hints Sent By The Worker ```rust RingWritable { ring_id: RingId } RingReadable { ring_id: RingId } ``` For ingress rings, the worker sends `RingWritable` after advancing `consume`. For egress rings, the worker sends `RingReadable` after advancing `commit`. Wake hints carry no byte counts, ranges, or ownership. ### 8.8 GpuWorkerCtl Synthetic Events Some events are generated by `GpuWorkerCtl`, not read from the worker: ```rust WorkerCrashed { worker_generation: WorkerGeneration, status: ExitStatus, } RingFault { ring_id: RingId, edge_id: EdgeId, port_id: PortId, reason: RingFaultReason, } ``` `ExitStatus` is `swactor_process::ExitStatus`. `GpuWorkerCtl` synthesizes these events after process exit, process error, or stdout control-stream failure. The worker cannot emit them because it is already gone or no longer trustworthy. For process exit, the synthetic ring fault reason is `WorkerProcessExited`. --- ## 9. GpuWorkerCtl Actor FSM `GpuWorkerCtl` is a normal swactor actor. Its FSM is advanced by incoming `GpuWorkerCtlMsg` values and by process notifications delivered through `ProcessBridge`. ```text NotStarted on StartWorker -> Spawning Spawning spawn ProcessActor with ProcessSpec spawn ProcessBridge subscribe bridge to ProcessActor wait for ProcessNotification::Started send InitializeWorker through ProcessCommand::WriteStdin -> Initializing Initializing on WorkerReady -> Running on WorkerFatal/process exit/timeout -> Failed Running on actor command -> validate state, write worker command, update local tables on worker event -> route event to local actors and driver on ShutdownWorker -> Stopping on process exit/error -> Crashed Stopping send ShutdownWorker if process is still alive wait for WorkerStopped and process exit on timeout -> Killing Killing send ProcessCommand::Close or stop/kill ProcessActor according to ProcessSpec reap process through ProcessActor notification mark installed rings faulted -> Stopped Crashed mark current-generation device handles invalid mark installed rings faulted ask driver to stop pumps for all installed rings wait for teardown to release rings -> Failed or Restarting Restarting increment worker_generation -> Spawning ``` `GpuWorkerCtl` must not claim a ring is quiesced merely because the worker exited. Quiescence for arena reuse still requires the edge teardown proof from the ring spec: pump stopped, worker process reaped or ring quiesced, and copy lifetimes ended or the process that owns them is gone. --- ## 10. Worker Process FSM The worker process has one state owner and handles process commands serially. It may maintain multiple internal FSMs, especially one per installed ring, but there is no separate worker scheduler runtime requirement. ```text Booting read environment wait for InitializeWorker on stdin Initializing map arena initialize native helper initialize backend/tinygrad emit WorkerReady -> Running Running on InstallRing -> install ring state on RingReadable/RingWritable -> reload cursors and advance affected ring FSMs on ExecuteStep -> run explicit role step subject to ring backpressure on ReleaseDeviceObject -> release handle when safe on ShutdownWorker -> Draining Draining reject new work finish or abort in-flight work according to shutdown mode quiesce rings emit WorkerStopped exit Fatal emit WorkerFatal if possible exit non-zero ``` The worker should greedily make progress after any relevant command or wake hint: drain readable ingress prefixes, advance egress output if writable space exists, observe copy completions, and emit resulting events. The concrete internal data structures used to remember ready rings or pending copies are implementation details, provided duplicate wake hints cannot lose liveness. --- ## 11. Per-Ring Worker State ### 11.1 Ingress Ring FSM ```text Uninstalled on InstallRing(direction = Ingress) -> NeedHeader NeedHeader on committed bytes < header_len -> wait on complete header -> validate valid -> allocate device object -> NeedPayload invalid -> Faulted NeedPayload copy committed payload prefixes to device allocation advance consume only after copied bytes are safe to release if copied == extent -> ObjectComplete on close before copied == extent -> Faulted ObjectComplete wait for copy completion emit ObjectLoaded -> NeedHeader Faulted emit ObjectFailed or RingFault stop consuming until UninstallRing Uninstalling stop consuming wait for copy lifetimes to end emit RingQuiesced -> Uninstalled ``` ### 11.2 Egress Ring FSM ```text Uninstalled on InstallRing(direction = Egress) -> WaitingForOutput WaitingForOutput on ExecuteStep output binding naming this ring -> NeedWritableForHeader NeedWritableForHeader wait for writable span write ObjectHeader advance commit emit RingReadable -> NeedWritableForPayload NeedWritableForPayload copy device payload prefixes into writable ring spans advance commit only after host bytes are valid emit/coalesce RingReadable if produced == extent -> ObjectProduced -> WaitingForOutput Faulted emit RingFault stop producing until UninstallRing Uninstalling stop accepting outputs for this ring wait for copy lifetimes to end emit RingQuiesced -> Uninstalled ``` The egress producer may block an `ExecuteStep` while waiting for ring space. That is the intended backpressure path. --- ## 12. Role and Step State Role state is intentionally minimal in this spec. ```text Unavailable on WorkerReady with fixed boot role -> Ready on ConfigureRole -> Configuring Configuring load/record app-defined role config on success -> Ready on failure -> Failed Ready on ExecuteStep -> run one explicit step on Shutdown -> Stopping Stopping reject new steps release role-local resources ``` The MVP role concurrency policy is one active `ExecuteStep` per role. A later worker may support multiple active steps only if the role declares that its state is reentrant or the steps are explicitly independent. No separate step FSM is specified. The step either reaches `StepCompleted` or `StepFailed`, and any partial egress object faults its ring. --- ## 13. Message Flows ### 13.1 Boot ```text GpuWorkerCtl -> swactor_process: spawn ProcessActor ProcessActor -> OS: spawn worker with arena fd/env ProcessActor -> GpuWorkerCtl: ProcessNotification::Started GpuWorkerCtl -> ProcessActor: WriteStdin(InitializeWorker JSON line) worker: map arena once worker: initialize native helper and backend worker -> stdout: WorkerReady JSON line ProcessActor -> GpuWorkerCtl: ProcessNotification::Output(stdout) GpuWorkerCtl: parse WorkerReady; node may now install rings and execute steps ``` Failure before `WorkerReady` is a boot failure. No rings are installed, so no arena lease can be corrupted by the worker. ### 13.2 Install Receive Edge ```text EdgeEstablisher receives RingLeased EdgeEstablisher -> GpuWorkerCtl: InstallRing(direction = Ingress, edge_id, port_id) GpuWorkerCtl -> worker: InstallRing worker: open native ring handle and create ingress parser worker -> GpuWorkerCtl: RingInstalled GpuWorkerCtl -> EdgeEstablisher: RingInstalled EdgeEstablisher -> Driver: EstablishRecv Driver: pair recv spec with stream when available Driver: spawn recv-pump ``` The driver does not start a recv-pump until the worker has installed the ring. ### 13.3 Ingress Object Load ```text recv-pump reads QUIC bytes into free ingress ring span recv-pump advances commit recv-pump -> GpuWorkerCtl: RingReadable(ring_id) GpuWorkerCtl -> worker: RingReadable(ring_id) worker reloads commit/consume worker parses ObjectHeader worker validates extent and sequence worker allocates device object worker copies committed payload prefixes to device worker advances consume after copies are safe worker -> GpuWorkerCtl: RingWritable(ring_id) GpuWorkerCtl -> recv-pump: RingWritable(ring_id) when full object is copied and copy events complete: worker -> GpuWorkerCtl: ObjectLoaded(..., device_handle) GpuWorkerCtl routes ObjectLoaded to the role/Rx layer ``` No actor receives byte ranges. The ring cursors are the only byte ownership state. ### 13.4 Execute Step and Produce Output ```text role layer has all required input device handles role layer chooses output object_id and sequence role layer -> GpuWorkerCtl: ExecuteStep(inputs, outputs) GpuWorkerCtl -> worker: ExecuteStep worker validates handles worker calls tinygrad role code worker writes output ObjectHeader to egress ring worker copies output payload to egress ring as space becomes available worker advances commit after host bytes are valid worker -> GpuWorkerCtl: RingReadable(egress_ring) GpuWorkerCtl -> send-pump: RingReadable send-pump writes committed bytes to QUIC send-pump advances consume after write_all accepts bytes send-pump -> GpuWorkerCtl: RingWritable(egress_ring) GpuWorkerCtl -> worker: RingWritable worker -> GpuWorkerCtl: ObjectProduced worker -> GpuWorkerCtl: StepCompleted ``` `StepCompleted` follows `ObjectProduced` for all declared outputs. For a step with no outputs, `StepCompleted` follows successful compute and state update. ### 13.5 Release Device Object ```text role layer decides object is no longer needed role layer -> GpuWorkerCtl: ReleaseDeviceObject(device_handle) GpuWorkerCtl -> worker: ReleaseDeviceObject worker waits until no copy or compute references the object worker frees device allocation worker -> GpuWorkerCtl: DeviceObjectReleased ``` The node process never dereferences a device handle. It can only pass it back to the worker. ### 13.6 Stop Edge ```text EdgeEstablisher -> Driver: StopEdge Driver stops pump Driver -> EdgeEstablisher: PumpStopped EdgeEstablisher -> GpuWorkerCtl: UninstallRing GpuWorkerCtl -> worker: UninstallRing worker removes ring from active state worker waits for copy lifetimes to end worker -> GpuWorkerCtl: RingQuiesced GpuWorkerCtl -> EdgeEstablisher: RingQuiesced EdgeEstablisher -> ArenaManager: ReleaseRing(proof) ``` The arena lease is released only after both the driver pump and worker ring are quiesced, or after the worker process is dead and reaped. ### 13.7 Worker Crash ```text worker process exits or process actor reports error GpuWorkerCtl marks all current-generation device handles invalid GpuWorkerCtl emits WorkerCrashed to role layer GpuWorkerCtl synthesizes RingFault for every installed ring GpuWorkerCtl asks Driver to stop pumps for those rings EdgeEstablisher waits for PumpStopped worker side is considered gone only after process reap EdgeEstablisher releases arena rings after teardown proof ``` The arena survives because it is owned by the node process. Device allocations do not survive because they were owned by the worker process/backend context. Restart creates a new worker generation. Rings and roles must be reinstalled. --- ## 14. Error Taxonomy ### 14.1 RingFaultReason ```rust enum RingFaultReason { UnsupportedRingVersion, RingLayoutInvalid, RingStateInvalid, DeviceOutOfMemory, DeviceCopyFailed, SequenceViolation, HeaderMalformed, WorkerProcessExited, WorkerShuttingDown, WorkerInternal, } ``` ### 14.2 ObjectFailure ```rust enum ObjectFailure { HeaderMalformed, UnsupportedObjectVersion, ExtentExceedsMax, ExtentAlignmentInvalid, SequenceViolation, DeviceAllocationFailed, DeviceCopyFailed, EofMidObject, } ``` ### 14.3 StepFailure ```rust enum StepFailure { RoleUnavailable, InvalidInputHandle, InvalidOutputRing, TinygradError, DeviceOutOfMemory, OutputExtentInvalid, OutputCopyFailed, WorkerProcessExited, WorkerShuttingDown, } ``` ### 14.4 WorkerFatalReason ```rust enum WorkerFatalReason { ArenaMapFailed, RingHelperAbiMismatch, BackendInitFailed, MalformedControlMessage, UnhandledException, } ``` ### 14.5 Role and Release Failures ```rust enum RoleFailure { InvalidRoleConfig, BackendUnsupported, WorkerRejectedRole, } enum ReleaseFailure { InvalidDeviceHandle, ObjectInUse, BackendFreeFailed, } ``` ### 14.6 Control Reasons ```rust enum UninstallReason { EdgeStopped, RingFaulted, WorkerShutdown, } enum WorkerStoppedReason { Graceful, AbortInFlight, Fatal, } ``` MVP policy: device OOM during ingress object allocation faults the ring. The worker does not silently skip a payload and continue to later objects. Replanning or retry is a higher-level policy. --- ## 15. Correctness and Safety Invariants **Control messages carry no payload bytes.** Payload movement occurs only through arena-backed rings. This keeps actors and process stdio out of the byte hot path. **The worker cannot access an unleased arena range.** `InstallRing` is sent only after ArenaManager emits `RingLeased`; the message contains the only usable ring layout. **The driver cannot pump into a ring the worker has not installed.** EdgeEstablisher sends `EstablishRecv` or `EstablishSend` only after `RingInstalled`. **Python does not own ring atomics.** All cursor loads, cursor stores, span calculation, and wrap arithmetic use the native ring helper. **Ingress compute visibility starts at `ObjectLoaded`.** The worker emits `ObjectLoaded` only after header validation, exact extent copy, copy completion, and device handle creation. The role layer cannot pass a partial object to `ExecuteStep` because no handle exists before then. **The recv-pump cannot overwrite host bytes still needed by DMA.** The worker advances ingress `consume` only after the corresponding host bytes are no longer read by synchronous copy or asynchronous DMA. **The send-pump cannot read unwritten egress bytes.** The worker advances egress `commit` only after header or payload bytes are valid in host memory. **Backpressure is absence of writable ring space.** If the worker is slow, ingress `consume` does not advance and the recv-pump stops reading QUIC. If the network is slow, egress `consume` does not advance and the worker stops producing more bytes into the egress ring. **Wake hints are not ownership.** `RingReadable` and `RingWritable` only cause the receiver to reload cursors. Duplicate hints may be coalesced only while the ring remains discoverable. **Device handles are generation-scoped.** A handle from an old worker generation is rejected by the new worker. This prevents stale handles from aliasing new device allocations after restart. **Object id and sequence ownership is explicit.** Ingress object id and sequence come from the received `ObjectHeader` and are validated against `ObjectSpec`. Egress object id and sequence are assigned by the role/control layer in `ExecuteStep`; the worker serializes them but does not invent graph-visible ordering. **Role execution is explicit.** Loading an object does not run compute. The role layer must issue `ExecuteStep` with concrete input handles and output bindings. **Partial egress objects fault the ring.** If the worker commits an output header but cannot commit the full declared payload, the egress ring is faulted. The receiver must not interpret later bytes as the next object. **A released arena range has no live users.** `ReleaseRing` is sent only after the driver pump is stopped and the worker has emitted `RingQuiesced`, or after the worker process is reaped. Copy lifetimes must have ended or the process that owned them must be gone. **Worker restart does not preserve GPU state.** Roles, rings, steps, and device handles are all current-generation state. Restart requires reconfiguration. --- ## 16. Implementation Requirements ### 16.1 Actor Integration The Rust-side implementation must follow the existing process-actor idiom: - `GpuWorkerCtl` is an `ActorInterface` implementation with one incoming enum - it spawns a `ProcessActor` through `swactor_process` - it spawns a private bridge actor for `ProcessNotification` - it sends worker input with `ProcessCommand::WriteStdin` - it closes the worker with `ProcessCommand::Close` and normal actor teardown - it never exposes process stdin/stdout handles to driver pumps or edge actors ### 16.2 Copy Safety For synchronous copies: ```text copy returns -> safe to advance relevant cursor ``` For asynchronous copies: ```text start copy -> record CopyEvent and host/device ranges event complete -> advance relevant cursor or release device object ``` The worker must track host ranges for ring cursor release and device ranges for device object lifetime. ### 16.3 Output Commit Discipline For egress: 1. The worker must know the output extent before writing the object header. 2. The worker writes and commits the header. 3. The worker copies payload prefixes into writable spans. 4. The worker commits each prefix only after host bytes are valid. 5. The worker emits `ObjectProduced` only after all `extent` bytes are committed. If the worker cannot determine output extent before serialization, it must first materialize the output into a device object with known byte extent. The wire header cannot contain a placeholder extent. ### 16.4 Logs and Diagnostics Worker events go to stdout as process-adapter messages. Diagnostic logs go to stderr. `GpuWorkerCtl` may keep a stderr tail for crash diagnostics. This is diagnostic only and does not change safety rules. --- ## 17. Resolutions To Known Design Issues ### 17.1 No Custom Control Frames Decision: remove length-prefixed frames, generic envelopes, per-message protocol versions, command ids, and reply ids. Rust code uses normal swactor messages. The Python subprocess adapter may use newline-delimited JSON because the codebase already uses `ProcessActor` stdin/stdout for process-backed workers. Reason: this worker is integrated into swactor. A second local control protocol duplicates actor semantics without solving a current problem. ### 17.2 Ring Install Needs Edge and Port Identity Decision: `InstallRing` includes `edge_id` and `port_id`. The worker reports all object and ring events with `ring_id`, `edge_id`, and `port_id`. Reason: `ring_id` is a node-local transport identifier. The role and edge layers need graph-facing identity without maintaining an ambiguous reverse lookup in the worker. ### 17.3 Device Handle Lifetime Decision: device handles are opaque, generation-scoped, and explicitly released with `ReleaseDeviceObject`. Reason: this prevents stale handle aliasing after restart and gives the role layer explicit control over activation and cache lifetimes. ### 17.4 Compute Driving API Decision: compute is driven by `ExecuteStep`; `ConfigureRole` is optional and opaque until role provisioning exists. `ObjectLoaded` is a data-ready event, not an execution trigger. Reason: the control plane owns ordering and correlation. ### 17.5 Object Id And Sequence Ownership Decision: ingress validates object id and sequence from the stream header. Egress object id and sequence are assigned by the role/control layer and supplied in `ExecuteStep.OutputBinding`. Reason: the worker should not invent graph-visible ordering. The control plane already knows the expected ordering and correlation. ### 17.6 Weight Loading Decision: weight/persistent-state specifics are deferred with role provisioning. For now, weights can be loaded as ordinary ingress objects and passed to `ExecuteStep`, or a fixed-role worker can load them during its own initialization from app-specific config. Reason: this avoids designing binding APIs before the role provisioning path exists. ### 17.7 Multi-Port And Fan-In Decision: rings bind to ports, and `ExecuteStep` lists all input handles and output rings explicitly. Fan-in is represented at the role layer by waiting for multiple `ObjectLoaded` events before issuing one `ExecuteStep`. Reason: the ring remains SPSC and byte-level simple. Join semantics stay out of the ring and pump layers. ### 17.8 GPU OOM And Allocation Pressure Decision: ingress allocation failure faults the ring in the MVP. Step-time OOM fails the step. If an egress object was partially committed, the egress ring is faulted. Reason: skipping a payload or continuing after a partial object requires more protocol machinery and can violate stream alignment. Replanning or retry belongs above the worker boundary. ### 17.9 Error Separation Decision: use separate failure classes for worker fatal errors, ring faults, object failures, release failures, and step failures. Reason: recovery action differs. A malformed object is not the same as a process crash, and a tinygrad exception is not the same as a corrupt ring cursor. ### 17.10 tinygrad Range-Copy Gap Decision: the worker implementation must provide a `DeviceBridge` with range copy support. If tinygrad does not expose the necessary primitive directly, the bridge must use a native backend helper or tinygrad raw-buffer API. The ring protocol must not be weakened to require `extent <= ring_capacity`. Reason: the ring spec explicitly allows objects larger than rings. Partial prefix copy is required for liveness and backpressure. ### 17.11 Restart Semantics Decision: restart creates a new worker generation and invalidates all device handles, role instances, ring installs, and in-flight steps. The node may reuse the same arena only after normal ring teardown proves quiescence. Reason: GPU backend state is process-local. Treating restart as transparent would risk stale handles, leaked DMA, and mismatched role state.