swactor/examples/single-gpu-inference/BEHAVIOR.md

single-gpu-inference — Behavioral Spec

This document describes what the single-gpu-inference example actually does at runtime. It is built around the actor graph — what each actor's job is, who it talks to, and what flows along each edge.

Audience: an engineer who has cloned the repo and wants to debug a failure or extend the deployment. Scope: the vast.ai deployment path. The localhost path is the same actors with relay disabled and is covered briefly at the end.

Status: As of cb789c3 the deployment runs end-to-end. The original SPEC.md was the design document; this file describes what was built.

---

1. Topology

Two machines, two control planes between them.

┌─────────────────────────┐                                  ┌─────────────────────────┐
│  Operator laptop        │ ─── vast.ai REST (HTTPS) ──────► │  vast.ai container      │
│  single-gpu-inference   │     find / create / poll /       │  gpu-node               │
│                         │     logs / destroy               │  + tinygrad_worker.py   │
│                         │                                  │                         │
│                         │ ◄══ iroh QUIC (via relay) ══════►│                         │
└─────────────────────────┘     • SWIM gossip                └─────────────────────────┘
                                • actor envelopes

• vast.ai REST is laptop-only. Provisioning and teardown.
• iroh QUIC is bidirectional and carries everything else: SWIM membership gossip (cluster join, name registration) and actor envelopes (the InferenceRequest/InferenceResponse pair).
• Both ends are typically behind NAT, so iroh relay infrastructure is required to bootstrap connectivity; iroh upgrades to a direct QUIC path opportunistically.

---

2. Actor topology

2.1 The graph

There is one swactor::Runtime per machine. The laptop runtime is almost empty — the laptop is a client, so its only actor is the response inbox. The remote runtime contains the inference actor tree.

   ╔═══════════════════════ Laptop runtime ═══════════════════════╗            ╔═══════════════════════════ gpu-node runtime ════════════════════════════╗
   ║                                                              ║            ║                                                                         ║
   ║   ┌─────────────┐                                            ║            ║          ╔═══════════════════╗                                          ║
   ║   │ main thread │                                            ║            ║          ║   RequestBridge   ║                                          ║
   ║   └──────┬──────┘                                            ║            ║          ║  name="inference" ║                                          ║
   ║          │ send_to(bridge_addr,                              ║            ║          ╚═════════╤═════════╝                                          ║
   ║          │   InferenceRequest{reply_to=inbox_addr})          ║   wire     ║                    │ InferenceActorMsg::Request(req)                    ║
   ║          ▼                                                   ║   QUIC     ║                    ▼                                                    ║
   ║   ┌──────────────────┐                                       ║            ║          ╔═════════════════════════════╗     InferenceActorStatus       ║
   ║   │ TransportRouter  │ ─► IrohActorTransport ────────────────╫═══════════►║          ║                             ║ ───────────────────►┌────────┐ ║
   ║   │  bridge_addr →   │    (target: remote endpoint)          ║ Inference- ║          ║       InferenceActor        ║   (Started, Ready,  │ Inbox< │ ║
   ║   │  remote          │                                       ║ Request    ║          ║                             ║    Exited)          │ Status>│ ║
   ║   └──────────────────┘                                       ║            ║          ║   state:                    ║                     └────┬───┘ ║
   ║                                                              ║            ║          ║   • ready : bool            ║                          │try_ ║
   ║                                                              ║            ║          ║   • process_alive : bool    ║                          │recv ║
   ║                                                              ║            ║          ║   • pending_replies (FIFO)  ║                          ▼     ║
   ║                                                              ║            ║          ║   • output_buffer : String  ║                  ┌─────────────┐║
   ║                                                              ║            ║          ║                             ║                  │ main thread │║
   ║                                                              ║            ║          ╚════╤═════════════════════╤══╝                  └─────────────┘║
   ║                                                              ║            ║               │                     │                                   ║
   ║                                                              ║            ║   Inference-  │                     │ ProcessCommand::                  ║
   ║                                                              ║            ║   Response    │                     │   WriteStdin{json}                ║
   ║                                                              ║            ║   (to         │                     ▼                                   ║
   ║                                                              ║            ║   reply_to)   │             ╔════════════════╗                          ║
   ║                                                              ║   wire     ║               │             ║  ProcessActor  ║                          ║
   ║   ┌──────────────────────┐                                   ║   QUIC     ║               │             ║ (swactor_      ║                          ║
   ║   │ Inbox<               │ ◄─────────────────────────────────╫════════════╫───────────────┘             ║  process)      ║                          ║
   ║   │   InferenceResponse> │   codec decodes →                 ║ Inference- ║                             ╚════╤═════════╤═╝                          ║
   ║   └──────────┬───────────┘   rt.deliver_raw                  ║ Response   ║                                  │         │                            ║
   ║              │ try_recv                                      ║            ║                                  │         │ ProcessNotification        ║
   ║              ▼                                               ║            ║                                  │         │ (Started/Output/Exited)    ║
   ║   ┌─────────────┐                                            ║            ║                                  │         ▼                            ║
   ║   │ main thread │                                            ║            ║                                  │ ╔════════════════╗                   ║
   ║   └─────────────┘                                            ║            ║                                  │ ║ ProcessBridge  ║                   ║
   ║                                                              ║            ║                                  │ ╚════════╤═══════╝                   ║
   ║                                                              ║            ║                                  │          │ InferenceActorMsg::       ║
   ║                                                              ║            ║                                  │          │   Process(notif)          ║
   ║                                                              ║            ║                                  │          └─► (back to InferenceActor)║
   ║                                                              ║            ║                                  │                                      ║
   ║                                                              ║            ║   stdin/stdout pipes             ▼                                      ║
   ║                                                              ║            ║   (not actor traffic)        ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐               ║
   ║                                                              ║            ║                                tinygrad_worker.py (Python)              ║
   ║                                                              ║            ║                              └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘               ║
   ╚══════════════════════════════════════════════════════════════╝            ╚═════════════════════════════════════════════════════════════════════════╝

Legend:  ╔═══╗ actor   ┌───┐ inbox / runtime infra   ┌─ ─ ─┐ non-actor (OS pipes)
         ────► actor send   ════►  wire (iroh QUIC)   ─ ─ ─►  pipes

2.2 At a glance

Actor	Runtime	Created by	Role
RequestBridge	gpu-node	gpu-node main	Inbound network adapter; entry point for all InferenceRequests
InferenceActor	gpu-node	gpu-node main	The brain: supervises the worker, queues replies, owns response routing
ProcessActor	gpu-node	InferenceActor::on_start	Owns the Python child process and its pipes
ProcessBridge	gpu-node	InferenceActor::on_start	Type adapter; rewraps ProcessNotification for the inference actor
Inbox<InferenceActorStatus>	gpu-node	gpu-node main	Lifecycle observer (used by main thread to wait for WorkerReady)
Inbox<InferenceResponse>	laptop	single-gpu-inference main	Where the final response lands

The laptop spawns no business actors. It is a client.

---

3. Actor roles

Each subsection follows the same pattern: role · runtime · inputs · outputs · state · why it exists.

3.1 RequestBridge

• Role: entry point. Receives InferenceRequests from the network and forwards them to InferenceActor.
• Runtime: gpu-node.
• Inputs: InferenceRequest (delivered by the codec registry from inbound QUIC streams).
• Outputs: InferenceActorMsg::Request(req) → InferenceActor.
• State: none, beyond the target address.
• Why it exists: swactor actors have a single Incoming type. InferenceActor needs to accept both requests from the network and notifications from the process. Its Incoming is a union enum (InferenceActorMsg); RequestBridge is the thin adapter that converts the raw network type into the right variant. bridge_addr is what's registered under the cluster name "inference" — it's the network's public-facing handle.

3.2 InferenceActor

• Role: the brain. Supervises the worker process, dispatches prompts, and routes responses back to the original caller.
• Runtime: gpu-node.
• Inputs: InferenceActorMsg, which is a union of:
  • Request(InferenceRequest) — a prompt to run, carrying a reply_to address.
  • Process(ProcessNotification) — lifecycle/IO events from the Python child.
• Outputs:
  • ProcessCommand::WriteStdin{data} → ProcessActor (JSON prompt + \n).
  • InferenceResponse{text} → the request's reply_to (cross-runtime, travels back over iroh).
  • InferenceActorStatus events → optional status_addr (the lifecycle inbox).
• State:
  • process_addr, bridge_addr — handles to the actors it spawned.
  • ready : bool — has the worker emitted {"status":"ready"}?
  • process_alive : bool — has the worker exited?
  • pending_replies : VecDeque<ActorAddress> — FIFO queue of reply_to addresses for in-flight prompts.
  • output_buffer : String — accumulates stdout bytes until a full \n-delimited line.
  • worker_pid : Option<u32> — for diagnostics.
• Why it exists: the actor that means something domain-wise. It is the reified "inference service" — everything else exists to feed it or relay for it.

The FIFO queue is the only mechanism for matching responses to requests. The worker produces output strictly in order, so popping the front of pending_replies for each {"response": …} line is sufficient. There is no request id. This is fine because the worker is single-threaded and replies in arrival order.

3.3 ProcessActor

• Role: owns the Python child process and its OS pipes.
• Runtime: gpu-node.
• Inputs: ProcessCommand (Subscribe, WriteStdin, Close, …).
• Outputs: ProcessNotification (Started, Output, Exited, Error) to its subscribers.
• State: the child PID, the stdin/stdout/stderr pipes, a buffer of pending notifications.
• Why it exists: it's a generic actor provided by the swactor_process crate. Not specific to this example. It is the only thing in the system that actually fork-execs the Python interpreter and reads its stdout.

3.4 ProcessBridge

• Role: type adapter from ProcessNotification → InferenceActorMsg.
• Runtime: gpu-node.
• Inputs: ProcessNotification (subscribed via ProcessCommand::Subscribe).
• Outputs: InferenceActorMsg::Process(notif) → InferenceActor.
• State: target address only.
• Why it exists: same reason as RequestBridge — the inference actor has one Incoming type. ProcessBridge is the thin adapter for the other source. Note: ProcessBridge is created by InferenceActor::on_start and subscribed to the ProcessActor in the same step. It is private to the inference actor's lifetime.

3.5 Inbox<InferenceActorStatus>

• Role: out-of-band lifecycle channel.
• Runtime: gpu-node.
• Inputs: InferenceActorStatus variants (ProcessStarted, WorkerReady{pid}, ProcessExited{status}).
• Outputs: none; drained by gpu-node's main thread via try_recv.
• Why it exists: the gpu-node main thread needs to block until the worker is actually ready before entering its serve loop, and wants to log worker-death events. An inbox is the simplest cross-thread observer primitive the runtime provides.

3.6 Inbox<InferenceResponse> (laptop side)

• Role: receiving end of the call.
• Runtime: laptop.
• Inputs: InferenceResponse (delivered by the codec from inbound QUIC streams).
• Outputs: none; the laptop's main thread reads via try_recv.
• Why it exists: its ActorAddress is what the laptop puts in req.reply_to. The remote uses that address to direct the response back — no out-of-band reply channel needed.

---

4. Dataflows

Each scenario below is a message-sequence walk through the actor graph. Steps that cross machines are flagged with ── wire ──. Steps that touch the OS (fork, pipe IO) are flagged with ── OS ──.

4.1 Worker boot

When gpu-node starts, the inference actor tree is spawned and the Python worker comes up. This typically takes 3–10 minutes due to model download + CUDA kernel compilation.

Step  Actor / source         Event
────  ─────────────────────  ─────────────────────────────────────────────────────────
  1   gpu-node main          rt.spawn(InferenceActor)
  2   InferenceActor         on_start runs:
                                a. spawn_local_process(spec) ─► ProcessActor
                                b. ctx.spawn(ProcessBridge{target: self_addr})
                                c. ctx.send(ProcessActor, Subscribe{address: bridge_addr})
                                d. store process_addr, bridge_addr
  3   gpu-node main          ctx.spawn(RequestBridge{target: inference_addr})
                                rt.node.register_name("inference", bridge_addr)
── OS ───────────────────────────────────────────────────────────────────────
  4   ProcessActor           OS-fork+exec: python3 tinygrad_worker.py
  5   ProcessActor           emits ProcessNotification::Started
  6   ProcessBridge          wraps ─► InferenceActorMsg::Process(Started)
  7   InferenceActor         process_alive = true
                                ctx.send(status_addr, ProcessStarted)
... (tinygrad imports, fetches GGUF, loads model into VRAM — minutes) ...
  8   tinygrad_worker.py     prints {"status":"ready","pid":N} to stdout
── OS ───────────────────────────────────────────────────────────────────────
  9   ProcessActor           reads line; emits ProcessNotification::Output{data}
 10   ProcessBridge          wraps ─► InferenceActorMsg::Process(Output)
 11   InferenceActor         output_buffer += data; finds '\n'; parses JSON;
                                ready = true; worker_pid = N;
                                ctx.send(status_addr, WorkerReady{pid:N})
 12   gpu-node main          status_inbox.try_recv() ─► WorkerReady; exits startup wait

4.2 Request → Response (happy path)

The end-to-end RPC. This is the scenario the example was built to prove.

Step  Actor / source         Event
────  ─────────────────────  ─────────────────────────────────────────────────────────
  1   laptop main            rt.send_to(bridge_addr,
                                InferenceRequest{prompt, ..., reply_to: inbox_addr})
  2   laptop TransportRouter resolves bridge_addr ─► IrohActorTransport
                                codec encodes; opens QUIC uni stream; writes wire
                                envelope (32B dest + tag + payload); finishes stream
── wire ────────────────────────────────────────────────────────────────────
  3   gpu-node main loop     drain_and_collect_reply_addrs():
                                reads stream → decode_wire()
                                envelope.type_tag = "smoke::InferenceRequest"
                                deserialize → extract req.reply_to
                                codecs.receive() → rt.deliver_raw(bridge_addr,
                                                                  InferenceRequest)
  4   RequestBridge          ctx.send(target, InferenceActorMsg::Request(req))
  5   InferenceActor         handle:
                                guard: ready && process_alive (else: empty reply, return)
                                pending_replies.push_back(req.reply_to)
                                ctx.send(process_addr,
                                  WriteStdin{json {prompt, max_tokens, temperature} + \n})
  6   gpu-node main loop     for each reply_to collected in step 3:
                                look up the only alive SWIM peer (the laptop)
                                build IrohActorTransport(laptop endpoint)
                                router.add_route(reply_to, transport)
── OS ───────────────────────────────────────────────────────────────────────
  7   ProcessActor           writes bytes to worker stdin
  8   tinygrad_worker.py     forward pass on CUDA; prints {"response":"..."}
  9   ProcessActor           emits ProcessNotification::Output{data}
 10   ProcessBridge          wraps ─► InferenceActorMsg::Process(Output)
 11   InferenceActor         output_buffer += data; '\n' delimits a line;
                                parses JSON; pending_replies.pop_front() → reply_to
                                ctx.send(reply_to, InferenceResponse{text})
 12   gpu-node TransportRouter  resolves reply_to (route added in step 6) ─►
                                IrohActorTransport (laptop endpoint)
                                codec encodes; QUIC uni stream to laptop
── wire ────────────────────────────────────────────────────────────────────
 13   laptop main loop       drain_actor_messages():
                                decode_wire() → codecs.receive()
                                → rt.deliver_raw(inbox_addr, InferenceResponse)
 14   laptop main            response_inbox.try_recv() → text; print; loop exits

The interesting moment is step 6: the response route is built lazily. The remote has no idea what the laptop's inbox address is until it sees the inbound request, at which point it cracks the envelope open and registers a route from the response's reply_to to the only alive peer.

4.3 Concurrent requests

The actor graph supports overlapping requests naturally:

Step  Actor / source         Event
────  ─────────────────────  ─────────────────────────────────────────────────────────
  …   laptop                 sends Req#1{reply_to:A}, then Req#2{reply_to:B}
  …   InferenceActor         pending_replies = [A, B]
                                writes prompt#1 to stdin, then prompt#2 to stdin
  …   tinygrad_worker.py     replies in order: {"response":"r1"}, {"response":"r2"}
  …   InferenceActor         line#1 → pop_front=A → send InferenceResponse{r1} to A
                                line#2 → pop_front=B → send InferenceResponse{r2} to B

The worker is single-threaded and writes complete responses sequentially. The queue's FIFO discipline plus the worker's ordering invariant is what keeps responses paired with their callers.

If the worker emits a {"error": …} line, InferenceActor logs it but does not pop the queue — that pending caller will never get a response. (See §7.)

4.4 Worker dies after ready

Best-effort cleanup, then stay alive for diagnostics.

Step  Actor / source         Event
────  ─────────────────────  ─────────────────────────────────────────────────────────
  1   tinygrad_worker.py     crashes (segfault / OOM / exception)
── OS ───────────────────────────────────────────────────────────────────────
  2   ProcessActor           detects child exit;
                                emits ProcessNotification::Exited{status}
  3   ProcessBridge          wraps ─► InferenceActorMsg::Process(Exited)
  4   InferenceActor         process_alive = false; ready = false
                                drain pending_replies: send InferenceResponse{text:""}
                                to each (so the laptop sees an empty, not a hang)
                                ctx.send(status_addr, ProcessExited{status})
  5   gpu-node main          status_inbox.try_recv() → ProcessExited
                                logs "worker exited, keeping main loop alive
                                       for diagnostics"
                                main loop does NOT exit — SWIM gossip continues
  6   laptop                 sees empty responses (or times out at 300s);
                                fetches remote logs via vast.ai REST;
                                destroys instance; exits 1

The "don't exit" choice in step 5 is deliberate. If gpu-node exited, the container would die, SWIM would mark it dead, and the laptop would have no way to fetch logs to figure out why the worker crashed. Keeping the process alive lets vast.ai's log endpoint capture the worker's stderr.

4.5 Request arrives before worker is ready

This can happen if the laptop somehow sends a request before SWIM reports the peer alive. In practice the laptop's pump loop won't send until name resolution succeeds, but the InferenceActor is defensive about it.

Step  Actor / source         Event
────  ─────────────────────  ─────────────────────────────────────────────────────────
  1   InferenceActor         handle Request: !ready
                                ctx.send(req.reply_to, InferenceResponse{text:""})
                                return (do NOT enqueue or write stdin)
  2   laptop                 try_recv sees empty text; pump loop logs "retrying"
                                and continues — there is no automatic resend

So an empty-string InferenceResponse is the in-band "not ready" signal.

---

5. Machines, processes, environment

The actor graph is hosted by a small number of OS processes:

Machine	Process	Source	What it owns
Laptop	single-gpu-inference	src/bin/single_gpu_inference.rs	tokio runtime, iroh Endpoint (RelayMode::Default), IrohDriver, swactor Runtime, the response inbox
Container	gpu-node	src/bin/gpu_node.rs	tokio runtime, iroh Endpoint (RelayMode::Default), IrohDriver, swactor Runtime, the inference actor tree
Container	tinygrad_worker.py	tinygrad_worker.py	Python interpreter with tinygrad 0.12.0; child of gpu-node; no network of its own (except the initial GGUF fetch)

Container image: nvidia/cuda:12.6.3-devel-ubuntu24.04, with Python 3 and tinygrad pre-installed. Built from Dockerfile.

Environment provided by the orchestrator at container boot:

Var	Set by	Purpose
SEED_ADDR	create_instance env	64-char hex node id of the laptop's iroh endpoint
SEED_RELAY	create_instance env	Laptop's home relay URL (needed for WAN NAT traversal)
CUDA=1	Dockerfile	tinygrad uses the CUDA backend
WORKER_SCRIPT	Dockerfile	Path gpu-node uses to launch the Python worker

Neither machine needs a public IP or open inbound ports. All inbound traffic arrives via the iroh relay infrastructure.

---

6. Wire protocols

Three distinct protocols carry data; one carries pipe traffic inside the container.

6.1 Actor envelopes — ALPN swactor/actor/1

Application-level messages between actors on different runtimes. Format (from iroh_transport.rs::encode_wire):

┌─────────────────┬───────────────┬──────────────┬─────────────────┐
│ dest_addr 32B   │ tag_len u32   │ type_tag UTF8│ payload bytes   │
└─────────────────┴───────────────┴──────────────┴─────────────────┘

One envelope per QUIC uni stream. Connections are cached (IrohActorTransport.conn) so subsequent sends reuse the connection; streams are per-message and finished immediately.

type_tag	Direction	Payload (JSON)
smoke::InferenceRequest	laptop → remote	{prompt, max_tokens, temperature, reply_to}
smoke::InferenceResponse	remote → laptop	{text}

6.2 SWIM gossip

Membership and metadata, owned by the distribution crate (separate ALPN). Carries probes between members (probe_interval = 10, probe_timeout = 15), indirect probes, and piggybacked metadata gossip. Name registrations propagate via this metadata channel — the laptop's driver.node().resolve_name("inference") is reading state that was gossiped from the remote.

6.3 vast.ai REST — HTTPS to cloud.vast.ai

Laptop-only.

Operation	Method	URL
Find offer	GET	/api/v0/bundles/?q=<json>
Create instance	PUT	/api/v0/asks/{offer_id}/
Poll status	GET	/api/v0/instances/{contract_id}/
Request logs	PUT	/api/v0/instances/request_logs/{contract_id}/
Fetch logs	GET	S3 URL returned by request_logs
Destroy	DELETE	/api/v0/instances/{contract_id}/

The create_instance body sets the Docker image, the env (SEED_ADDR, SEED_RELAY), an onstart command (exec /usr/local/bin/gpu-node 2>&1), and disk size.

6.4 Worker IPC — OS pipes inside the container

ProcessActor owns these pipes; nothing else in the system touches them. Newline-delimited JSON.

Direction	Shape
worker → parent	{"status": "ready", "pid": <int>} — emitted once after model load.
parent → worker	{"prompt": ..., "max_tokens": ..., "temperature": ...}
worker → parent	{"response": "..."}
worker → parent	{"error": "..."} on bad JSON or generation failure

InferenceActor::process_output_line parses these and maps them to the right actor message (either a status event or a response to the queue's front).

---

7. Failure modes and recovery

Failures are documented per-stage. "Recovery" means automated behavior in the current code — if no recovery is listed, the failure is fatal after destroying any allocated instance.

7.1 Provisioning

Failure	Detected by	Behavior
No vast.ai offers match the filter	find_offer returns empty	Exit 1. Nothing to clean up.
create_instance returns non-success	HTTP status check	Exclude offer id, retry with next-cheapest (up to 3 attempts).
Instance never reaches running	wait_for_running exhausts 60 polls, sees exited/error, or intended_status=stopped. Host-side OCI/CDI errors surface as status_msg containing "Error" or "failed".	Destroy instance, exclude offer, retry.

7.2 Cluster join

Failure	Detected by	Behavior
SWIM never converges (no peer goes to alive within 120s)	driver.snapshot().members loop on the laptop	Fetch last 40 lines of instance logs, destroy instance, exclude offer, retry. The usual cause: broken host (image pull failure, CDI errors, blocked outbound UDP).
Name "inference" never resolves (60s)	resolve_name returns None	Destroy instance, exit 1. No retry: convergence already happened so the host is healthy; this means the bridge crashed before registering.

7.3 Inference

Failure	Detected by	Behavior
Worker never reports ready (600s)	gpu-node startup loop on the remote	gpu-node exits 1. Container exits; SWIM marks it dead; laptop times out on convergence and treats it like §7.2.
Worker exits after reporting ready	See §4.4	Pending replies drained with empty text; gpu-node main loop stays alive for diagnostics; laptop times out and fetches logs.
Worker emits {"error": ...}	process_output_line	Logged to stderr. Pending queue is not popped — the caller will time out. (Known gap; a real service would pop with an explicit error response.)
Empty response (request arrived before worker was ready)	See §4.5	Laptop ignores empty responses in its pump loop. No automatic resend.
Inference timeout (300s)	Laptop pump loop	Fetch last 60 lines of remote logs, destroy instance, exit 1.

7.4 Teardown

Failure	Detected by	Behavior
destroy_instance errors	Reqwest error	Error is dropped (let _ = …). Instance keeps costing money until manually destroyed. Known gap.

7.5 Invariants worth knowing

• Every code path that creates an instance also destroys it. Search destroy_instance in single_gpu_inference.rs — three call sites cover all three post-provisioning failure points.
• pending_replies is the only request↔response correlation. There is no request id on the wire. This is safe given the worker's single-threaded, in-order behavior; it is not safe if you ever swap in a multi-worker backend.
• ActorAddress is opaque to iroh and to vast.ai. It's a swactor-level identifier. Routing is done by the swactor runtime's TransportRouter.
• No per-request authentication. Anything that can reach the remote's iroh endpoint and knows bridge_addr can submit work. This relies on the address being unguessable. Don't reuse the pattern in production without layering auth on top.

---

8. Localhost mode (brief)

single-gpu-inference --seed <hex> and a locally-running gpu-node use the same actor graph and same wire protocols as the vast.ai path, with two changes:

• Both ends use RelayMode::Disabled — no relay needed on loopback.
• No vast.ai REST traffic. The operator runs gpu-node directly, copies its GPU_NODE_ADDR hex out of the log, and passes it to single-gpu-inference --seed.

Everything else — the bridges, the pending_replies queue, the InferenceRequest/InferenceResponse flow, the worker stdin/stdout JSON protocol — is identical. This is the mode used by tests/t_binary.rs for end-to-end testing without a GPU or an API key.