518 lines
13 KiB
Markdown
518 lines
13 KiB
Markdown
# GGUF Pipeline Orchestration - MVP Specification
|
|
|
|
**Status:** draft buildout specification.
|
|
|
|
**Relationship to other documents.** `DESIGN_DIRECTIVES.md` remains steering
|
|
context. `RING_BACKPRESSURE_SPEC.md` defines edge establishment, object records,
|
|
rings, pumps, and teardown. `GPU_WORKER_INTERFACE_SPEC.md` defines worker
|
|
startup, shard/weight binding, and `ExecuteStep`. This document defines the
|
|
missing layer above them: how the orchestrator plans and stages one linear GGUF
|
|
pipeline inference run.
|
|
|
|
This document is intentionally not a general graph specification.
|
|
|
|
---
|
|
|
|
## 1. Scope
|
|
|
|
The MVP workload is pipeline-parallel inference from a GGUF model:
|
|
|
|
```text
|
|
orchestrator --tokens--> stage 0 --activations--> stage 1 --activations-->
|
|
... --activations--> stage N-1 --tokens--> orchestrator
|
|
```
|
|
|
|
The orchestrator is a control participant and token endpoint. It does not run GPU
|
|
compute. GPU compute happens only inside provisioned stages.
|
|
|
|
This spec covers:
|
|
|
|
- building a linear stage plan from a GGUF model and a provisioned GPU pool
|
|
- assigning GGUF shard/layer ranges to stages
|
|
- assigning run-scoped edge ids
|
|
- provisioning each stage with the facts it needs
|
|
- defining the readiness barrier before prompt injection
|
|
- defining the orchestrator and stage FSMs
|
|
- defining security and correctness guarantees for the MVP behavior
|
|
|
|
This spec does not cover:
|
|
|
|
- arbitrary graph execution
|
|
- automatic placement optimization
|
|
- batching, speculative decoding, or continuous serving
|
|
- failure recovery by re-placement
|
|
- detailed ring, pump, or worker internals already specified elsewhere
|
|
- behavioral test contracts; those come after the full system shape is drafted
|
|
|
|
---
|
|
|
|
## 2. Core Responsibilities
|
|
|
|
The orchestrator owns:
|
|
|
|
- run ids
|
|
- stage count and stage order
|
|
- GGUF shard/layer assignment
|
|
- edge id assignment
|
|
- stage provisioning
|
|
- the global readiness barrier
|
|
- prompt injection
|
|
- final token consumption
|
|
- EOS and `max_tokens` stop policy
|
|
- run-level fault and teardown
|
|
|
|
Each stage owns:
|
|
|
|
- loading its assigned GGUF shard/layer range
|
|
- configuring its local GPU worker
|
|
- establishing its local edge ends
|
|
- converting loaded inbound objects into local `ExecuteStep` calls
|
|
- producing the next object on its outbound edge
|
|
- reporting readiness and faults to the orchestrator
|
|
|
|
The ring and worker specs own the byte movement and GPU worker command details.
|
|
This spec only decides which stages and edges exist and what sequence of control
|
|
events makes the run progress.
|
|
|
|
---
|
|
|
|
## 3. Run Plan
|
|
|
|
The orchestrator builds one `RunPlan` before provisioning:
|
|
|
|
```rust
|
|
struct RunPlan {
|
|
run_id: RunId,
|
|
model: GgufModelPlan,
|
|
stages: Vec<StagePlan>,
|
|
edges: Vec<EdgePlan>,
|
|
max_tokens: u32,
|
|
}
|
|
|
|
struct GgufModelPlan {
|
|
model_id: String,
|
|
gguf_source: GgufSource,
|
|
num_layers: u32,
|
|
hidden_dim: u32,
|
|
dtype_family: DTypeFamily,
|
|
dtype_width_bytes: u32,
|
|
max_seq_len: u32,
|
|
eos_token_id: u32,
|
|
}
|
|
```
|
|
|
|
`gguf_source` may identify a whole GGUF file, a pre-split shard collection, or a
|
|
cache key. The orchestration contract is the assigned layer range. Whether the
|
|
node reads only part of a whole GGUF file or receives a physically pre-split
|
|
artifact is a local loading detail.
|
|
|
|
Each stage receives a contiguous layer range:
|
|
|
|
```rust
|
|
struct StagePlan {
|
|
run_id: RunId,
|
|
stage_index: u32,
|
|
stage_count: u32,
|
|
node_id: NodeId,
|
|
gguf_source: GgufSource,
|
|
layer_start: u32,
|
|
layer_end_exclusive: u32,
|
|
inbound_edge: EdgeId,
|
|
outbound_edge: EdgeId,
|
|
}
|
|
```
|
|
|
|
For stage `0`, `inbound_edge` is the token edge from the orchestrator. For the
|
|
last stage, `outbound_edge` is the token edge back to the orchestrator. Interior
|
|
edges carry activations.
|
|
|
|
Each edge has exactly one producer and one consumer:
|
|
|
|
```rust
|
|
struct EdgePlan {
|
|
run_id: RunId,
|
|
edge_id: EdgeId,
|
|
kind: EdgeKind,
|
|
producer: EdgeEndpoint,
|
|
consumer: EdgeEndpoint,
|
|
object_spec: ObjectSpec,
|
|
ring_spec: RingSpec,
|
|
}
|
|
|
|
enum EdgeKind {
|
|
TokenIn,
|
|
Activation,
|
|
TokenOut,
|
|
}
|
|
|
|
enum EdgeEndpoint {
|
|
Orchestrator { node_id: NodeId },
|
|
Stage { node_id: NodeId, stage_index: u32 },
|
|
}
|
|
```
|
|
|
|
The orchestrator assigns all `edge_id`s. Stage code never derives edge ids from
|
|
names, layer ranges, peer ids, or hashes.
|
|
|
|
---
|
|
|
|
## 4. Stage Provisioning Message
|
|
|
|
The orchestrator sends one provision message to each stage node:
|
|
|
|
```rust
|
|
ProvisionStage {
|
|
run_id: RunId,
|
|
stage_index: u32,
|
|
stage_count: u32,
|
|
gguf_source: GgufSource,
|
|
layer_start: u32,
|
|
layer_end_exclusive: u32,
|
|
inbound: InboundEdgeProvision,
|
|
outbound: OutboundEdgeProvision,
|
|
model: StageModelFacts,
|
|
}
|
|
|
|
struct InboundEdgeProvision {
|
|
edge_id: EdgeId,
|
|
kind: EdgeKind,
|
|
object_spec: ObjectSpec,
|
|
ring_spec: RingSpec,
|
|
}
|
|
|
|
struct OutboundEdgeProvision {
|
|
edge_id: EdgeId,
|
|
kind: EdgeKind,
|
|
consumer_node_id: NodeId,
|
|
object_spec: ObjectSpec,
|
|
ring_spec: RingSpec,
|
|
}
|
|
|
|
struct StageModelFacts {
|
|
model_id: String,
|
|
hidden_dim: u32,
|
|
dtype_family: DTypeFamily,
|
|
dtype_width_bytes: u32,
|
|
max_seq_len: u32,
|
|
}
|
|
```
|
|
|
|
The inbound edge is established locally as a receive edge. The outbound edge is
|
|
established locally as a send edge to `consumer_node_id`. For the last stage,
|
|
`consumer_node_id` is the orchestrator node.
|
|
|
|
The node-local stage controller translates this provision message into the lower
|
|
level operations:
|
|
|
|
```text
|
|
configure worker for assigned stage
|
|
load/bind assigned GGUF shard or layer range
|
|
provision receive edge for inbound.edge_id
|
|
provision send edge for outbound.edge_id
|
|
report StageReady when all required local work is complete
|
|
```
|
|
|
|
---
|
|
|
|
## 5. Orchestrator FSM
|
|
|
|
The orchestrator has one run-level FSM:
|
|
|
|
```text
|
|
Planning
|
|
build RunPlan
|
|
validate layer ranges and edge ids
|
|
-> Provisioning
|
|
|
|
Provisioning
|
|
send ProvisionStage to every stage node
|
|
create local token producer for token-in edge
|
|
create local token consumer for token-out edge
|
|
-> WaitingReady
|
|
|
|
WaitingReady
|
|
on StageReady for every stage and local token endpoints ready
|
|
-> Running
|
|
on StageFault or timeout
|
|
-> Faulted
|
|
|
|
Running
|
|
inject prompt token object on token-in edge, sequence 0
|
|
consume token objects from token-out edge in sequence order
|
|
after token sequence k:
|
|
if EOS or max_tokens reached -> Completed
|
|
else inject token object sequence k + 1 on token-in edge
|
|
on StageFault, edge fault, local token endpoint fault, or timeout
|
|
-> Faulted
|
|
|
|
Completed
|
|
stop injecting tokens
|
|
finalize output text
|
|
-> TearingDown
|
|
|
|
Faulted
|
|
stop injecting tokens
|
|
record one run-level failure reason
|
|
-> TearingDown
|
|
|
|
TearingDown
|
|
send StopRun to all provisioned stages
|
|
tear down local token endpoints
|
|
wait for StageStopped from every stage or timeout
|
|
-> Done
|
|
|
|
Done
|
|
terminal
|
|
```
|
|
|
|
There is no broadcast start message. The start signal is the first token object
|
|
written by the orchestrator after the readiness barrier.
|
|
|
|
---
|
|
|
|
## 6. Stage FSM
|
|
|
|
Each provisioned stage has one node-local stage controller. It is control-path
|
|
only: it watches local worker and edge events and issues worker commands. It does
|
|
not move payload bytes.
|
|
|
|
```text
|
|
Unprovisioned
|
|
on ProvisionStage from authorized orchestrator
|
|
validate run_id and stage assignment
|
|
-> Preparing
|
|
|
|
Preparing
|
|
configure local worker for assigned GGUF range
|
|
start GGUF shard/range load and bind
|
|
establish inbound receive edge
|
|
establish outbound send edge
|
|
when worker configured, shard bound, and both edge ends ready
|
|
emit StageReady
|
|
-> Ready
|
|
on any required setup failure
|
|
emit StageFault
|
|
-> Faulted
|
|
|
|
Ready
|
|
on inbound ObjectLoaded(sequence = s)
|
|
if s is the next expected sequence
|
|
issue ExecuteStep for sequence s
|
|
-> Executing
|
|
else
|
|
emit StageFault(sequence violation)
|
|
-> Faulted
|
|
on StopRun
|
|
-> Stopping
|
|
|
|
Executing
|
|
worker runs exactly one step for the loaded inbound object
|
|
worker writes the output object to outbound edge with the same sequence
|
|
on StepCompleted
|
|
release any per-step input handle that is no longer needed
|
|
-> Ready
|
|
on StepFailed or output fault
|
|
emit StageFault
|
|
-> Faulted
|
|
|
|
Faulted
|
|
reject new run work
|
|
wait for StopRun
|
|
-> Stopping
|
|
|
|
Stopping
|
|
stop local edges
|
|
release per-run device objects
|
|
stop or reset worker according to local policy
|
|
emit StageStopped
|
|
-> Stopped
|
|
|
|
Stopped
|
|
terminal for this run
|
|
```
|
|
|
|
The controller is the component that decides when `ExecuteStep` is called. The
|
|
orchestrator does not issue per-stage execute commands during the run. Once the
|
|
prompt object is injected, stage execution is driven by object arrival and local
|
|
readiness.
|
|
|
|
---
|
|
|
|
## 7. Execution Semantics
|
|
|
|
Sequence `0` is prefill.
|
|
|
|
```text
|
|
orchestrator writes prompt token object sequence 0
|
|
stage 0 executes prefill over prompt tokens
|
|
stage 0 writes activation sequence 0
|
|
each interior stage executes prefill over activation sequence 0
|
|
last stage executes prefill and writes token sequence 0
|
|
orchestrator consumes token sequence 0
|
|
```
|
|
|
|
Decode sequences are `1..`:
|
|
|
|
```text
|
|
orchestrator writes one-token object sequence k
|
|
stage 0 executes decode for sequence k
|
|
each downstream stage executes decode for sequence k
|
|
last stage writes token sequence k
|
|
orchestrator consumes token sequence k
|
|
```
|
|
|
|
The orchestrator sends sequence `k + 1` only after consuming token sequence `k`
|
|
and deciding the run should continue.
|
|
|
|
For every stage:
|
|
|
|
- the inbound object sequence is the output object sequence
|
|
- one active `ExecuteStep` per stage is allowed in the MVP
|
|
- a stage cannot execute before its assigned GGUF shard/range is loaded and bound
|
|
- a stage cannot execute before its inbound object is loaded
|
|
- a stage cannot produce to an edge that is not ready
|
|
|
|
The last stage samples or otherwise produces token ids as part of its GPU worker
|
|
step. The orchestrator consumes those token ids, accumulates output, applies EOS
|
|
and `max_tokens`, and writes the next token object only when continuing.
|
|
|
|
---
|
|
|
|
## 8. Control Messages And Events
|
|
|
|
These are schematic message shapes, not final Rust APIs.
|
|
|
|
Orchestrator to stage:
|
|
|
|
```rust
|
|
ProvisionStage { ... }
|
|
|
|
StopRun {
|
|
run_id: RunId,
|
|
reason: StopReason,
|
|
}
|
|
```
|
|
|
|
Stage to orchestrator:
|
|
|
|
```rust
|
|
StageReady {
|
|
run_id: RunId,
|
|
stage_index: u32,
|
|
node_id: NodeId,
|
|
}
|
|
|
|
StageFault {
|
|
run_id: RunId,
|
|
stage_index: u32,
|
|
node_id: NodeId,
|
|
reason: StageFaultReason,
|
|
}
|
|
|
|
StageStopped {
|
|
run_id: RunId,
|
|
stage_index: u32,
|
|
node_id: NodeId,
|
|
}
|
|
```
|
|
|
|
Optional setup progress events may exist for diagnostics, but `StageReady`,
|
|
`StageFault`, and `StageStopped` are the only required run-level events in this
|
|
draft.
|
|
|
|
---
|
|
|
|
## 9. Object Specs
|
|
|
|
Token edges carry token objects. The prompt token object may contain multiple
|
|
token ids for prefill. Decode token objects contain one token id.
|
|
|
|
Activation edges carry activation objects with runtime extent bounded by model
|
|
shape:
|
|
|
|
```text
|
|
max_extent = max_seq_len * hidden_dim * dtype_width_bytes
|
|
```
|
|
|
|
The object record and ring behavior are defined by `RING_BACKPRESSURE_SPEC.md`.
|
|
This orchestration spec only requires that all stage plans for a run agree on the
|
|
model facts used to build those object specs.
|
|
|
|
---
|
|
|
|
## 10. Security Model
|
|
|
|
Nodes are trusted. The system does not attempt trustless verification,
|
|
adversarial tensor validation, Sybil defense, or incentive enforcement.
|
|
|
|
The orchestrator is the authority for run topology. A stage accepts run
|
|
provisioning only from the authorized orchestrator for its node.
|
|
|
|
Stages reject:
|
|
|
|
- unknown `run_id`
|
|
- stale `run_id`
|
|
- duplicate provisioning for an already-active run unless explicitly stopped
|
|
- edge ids not present in the provision message
|
|
- peer rewiring requests from another stage
|
|
|
|
`edge_id`s are run-scoped capabilities for wiring and demux. They are not a
|
|
cryptographic trust boundary between trusted nodes, but a stage must still reject
|
|
objects and stream setup that do not match its active run plan.
|
|
|
|
The orchestrator may tear down a run at any time. Stages must treat `StopRun` for
|
|
their active `run_id` as authoritative.
|
|
|
|
---
|
|
|
|
## 11. Correctness Guarantees
|
|
|
|
Layer assignment:
|
|
|
|
- stage layer ranges are contiguous
|
|
- stage layer ranges do not overlap
|
|
- the union of stage layer ranges covers the intended GGUF block range
|
|
- every stage has exactly one assigned range
|
|
|
|
Edge assignment:
|
|
|
|
- every `edge_id` is unique within a run
|
|
- every edge has exactly one producer and one consumer
|
|
- token-in is produced by the orchestrator and consumed by stage `0`
|
|
- token-out is produced by stage `N - 1` and consumed by the orchestrator
|
|
- activation edge `i` is produced by stage `i` and consumed by stage `i + 1`
|
|
|
|
Readiness:
|
|
|
|
- the orchestrator does not inject prompt tokens before every stage reports
|
|
`StageReady`
|
|
- a stage does not report `StageReady` before its worker, shard/range binding,
|
|
inbound edge, and outbound edge are ready
|
|
|
|
Execution:
|
|
|
|
- prefill is sequence `0`
|
|
- decode sequences are strictly increasing
|
|
- a stage executes sequence `s` only after loading inbound object sequence `s`
|
|
- a stage output uses the same sequence as its input
|
|
- the orchestrator injects sequence `s + 1` only after consuming token sequence
|
|
`s`
|
|
|
|
Termination:
|
|
|
|
- each run has one terminal outcome: completed, faulted, or torn down
|
|
- after a run faults, the orchestrator stops injecting new token objects
|
|
- teardown is sent to every stage that was provisioned for the run
|
|
|
|
---
|
|
|
|
## 12. Deferred
|
|
|
|
- placement optimization
|
|
- physical GGUF shard format
|
|
- multiple concurrent runs on one stage chain
|
|
- batching and speculative decoding
|
|
- direct stage-to-stage token feedback that bypasses the orchestrator
|
|
- warm reuse policy across prompts
|
|
- re-placement after node failure
|
|
- behavioral test matrix and observability schema
|