swactor/docs/specs/drafts/JOB_RUNNER_SPEC.md

# Synaptic Job Runner Specification

Id: 4
Last modified: b887e941cbe6f1e209339abd0375507aca9bfe52
Last reviewed:
> Any edit to this spec must update `Last modified` above to the current `git HEAD` commit.

---

## 1. Purpose

Run an arbitrary command job on a rented GPU node, end to end, through swactor:
provision a node → push the workspace → setup → run → collect outputs →
teardown. swactor is the substrate: provisioning, the node runtime, the data
plane, and observability are reused; the job runner is built on top of them.

v1 is the single-node command job: one job → one node → one attempt. Multi-node,
sharded, and pipeline jobs are future work-descriptions over the same core.

---

## 2. What swactor grows, what swactor reuses

**New — the job runner's content:**

- a node primitive that runs an arbitrary command as a supervised process and
  reports exit via datastream;
- a bulk-transfer service on the shared iroh endpoint for workspace push
  (operator → node) and output pull (node → operator) — a new service
  paralleling the existing edge and datastream ALPN services, not the
  arena/tensor data plane;
- a job FSM and the orchestrator↔node protocol that sequence the above.

**Reused unchanged:** the VastAI provider adapter (provision/teardown), SWIM
membership, the datastream event infrastructure, the shared iroh endpoint (which
already multiplexes the edge and datastream services), and the swactor actor
runtime.

---

## 3. Design principle: job = work, gpu-agnostic

A job describes *what to do*, not what hardware to run on — no provider, no GPU,
no node. Hardware (provider, GPU type/count/VRAM, disk, image, selection) lives
in a *pool designation* owned by the provider adapter. This keeps a future
sharded-model job the same shape against different placement; sharding is not a
v1 feature.

---

## 4. Job description

| field       | required | meaning                                              |
|-------------|----------|------------------------------------------------------|
| `name`      | yes      | job identity                                         |
| `run`       | yes      | the command that does the work                       |
| `setup`     | no       | one-time command run before `run` (env install)      |
| `workspace` | no       | `{ workdir, exclude }` — pushed to the node first    |
| `outputs`   | no       | paths to collect back after run                      |
| `env`       | no       | environment variables injected into setup and run    |

Nothing else. No resources, no provider, no GPU, no cleanup policy.

```toml
[job]
name = "airfrans_smoke_01"
setup = "uv sync --no-dev"
run = "uv run remote-run smoke-train configs/aggressive_smoke.toml"
workspace = { workdir = ".", exclude = ["/artifacts", "/.venv", "__pycache__"] }
outputs = ["metrics.jsonl", "final_metrics.json", "checkpoint_latest.pt"]
env = { HF_TOKEN = "..." }
```

A job is submitted with a *pool designation* (capacity) that drives provisioning
via the existing VastAI adapter. The pool is not part of the job and is not
specified further here.

---

## 5. Roles

- **Orchestrator (job authority):** places the job, drives the lifecycle, owns
  the terminal outcome, orders teardown.
- **Node (executor):** runs `setup` and `run` as supervised processes,
  materializes the workspace, exposes outputs, reports lifecycle via datastream.
- **Provider adapter:** provisions and tears down nodes per the pool designation.

---

## 6. Lifecycle

States: `PENDING → RUNNING → COMPLETED | FAILED`.

| event observed                    | command emitted                          | transition    |
|-----------------------------------|------------------------------------------|---------------|
| `JobSubmitted{job, pool}`         | `ProvisionNode`                          | → `PENDING`   |
| `NodeReady`                       | `MaterializeWorkspace`                   | → `RUNNING`   |
| `WorkspaceMaterialized`           | `RunSetup`                               |               |
| `SetupCompleted`                  | `RunJob`                                 |               |
| `JobExited{0}`                    | `CollectOutputs`                         |               |
| `OutputsCollected`                | `TeardownNode`                           | → `COMPLETED` |
| `JobExited{non-zero}`             | `CollectOutputs` (best-effort), `TeardownNode` | → `FAILED` |
| `NodeFault` / `NodeLost` / `OperatorStop` | `TeardownNode`                   | → `FAILED`    |

- A job waits for *its one node's* readiness, not pool-wide convergence.
- No `setup` → `WorkspaceMaterialized` goes straight to `RunJob`. No
  `workspace` → materialization is skipped.
- The exit code is authoritative: `0` → `COMPLETED`, non-zero → `FAILED`. Declared
  outputs are collected either way; collection on failure is best-effort.
- The node is torn down on every terminal state (completed or failed).
- Every transition emits a datastream event.

`OperatorStop` is the operator kill switch for a running job.

---

## 7. Behavior contracts

The orchestrator commands these contracts; the node owns the local mechanism.
Control and lifecycle travel as orchestrator↔node messages and datastream
events. Bulk workspace and output bytes travel over a dedicated transfer
service on the shared iroh endpoint, not in actor messages.

**Supervised command execution (node).** On `RunSetup` / `RunJob`, the node
spawns the command in `workdir` with the declared `env`, supervises it, and
reports `SetupCompleted` / `JobExited{code}` (or `NodeFault`) via datastream. A
failed setup is a failed job.

**Workspace materialization (operator → node).** On `MaterializeWorkspace`, the
`workdir` tree (with `exclude` applied) is pushed from the operator to the node
over the iroh transfer service, landing at the node's `workdir`. The node reports
`WorkspaceMaterialized`.

**Output collection (node → operator).** On `CollectOutputs`, declared `outputs`
are pulled from the node to a per-run operator landing directory over the iroh
transfer service. The node reports `OutputsCollected`. Missing outputs do not change
the outcome — the exit code already decided it; whatever exists is gathered.

---

## 8. Relationship to the broader work model

This spec defines only the job plugin. The intended generalization is a *work
description*: a graph of work-units with typed ports, placed by a planner. The v1
job resolves to a single work-unit with no edges; multi-unit, sharded, and
pipeline jobs are future work-descriptions over the same core. The gpu-agnostic
job shape is what keeps that path open.