swactor/mvp-node-image-gguf-shape.md

MVP node image + GGUF deployment shape

Goal

Prepare the first deployment-test node image for one rented GPU node.

The image should be minimal: swactor MVP node binary plus tinygrad with CUDA support. The node should stay at the swactor level; Python is only the compute helper, not the top-level application runtime.

The first GGUF target is a small ~1B parameter model fetched whole from HuggingFace and cached on the node.

Existing reference

The closest existing image is in apps/old-pipeline-parallel-inference:

• Dockerfile.base

  • CUDA runtime/base image layering;
  • Python, tinygrad, numpy;
  • NVRTC runtime;
  • CUDA headers copied from a builder stage;
  • sshd;
  • PID-1 entrypoint.

• Dockerfile

  • thin code layer over the base image;
  • copies Rust binaries and worker script.

• pp_entrypoint.sh

  • starts sshd deterministically;
  • accepts VastAI-injected public keys;
  • runs the worker as a child;
  • keeps the container alive after worker exit for postmortem.

• pp_tinygrad_worker.py

  • contains practical GGUF/tinygrad loading lessons;
  • fetches model artifacts;
  • uses tinygrad tokenizer/model code;
  • old runtime enters the Python worker protocol directly.

The new image should reuse the packaging/runtime lessons, not the old pipeline-parallel application shape.

Runtime architecture

VastAI or Docker provisioner
  -> container entrypoint starts sshd
  -> entrypoint starts mvp node binary
  -> mvp node joins orchestrator over iroh/swactor
  -> mvp node registers node actor(s)
  -> mvp node starts tinygrad helper as a child process
  -> helper fetches/caches GGUF
  -> helper loads model on CUDA
  -> orchestrator drives prompt/inference via swactor messages

Binary shape

Add or identify a dedicated deployment node binary, for example:

crates/mvp-system/src/bin/mvp-node.rs

Responsibilities:

• read deployment env/config;
• start iroh/swactor runtime;
• join the orchestrator;
• register the MVP node actor path;
• start and supervise the tinygrad helper process;
• emit a ready JSON line on stdout once the swactor node is reachable;
• continue pumping runtime until shutdown or fault.

This should be the deployment form of the existing local node role, not coupled to local-only TCP/test assumptions.

Image shape

Layering should follow the old image pattern:

mvp-node-base
  CUDA runtime
  NVRTC runtime library
  CUDA headers required by tinygrad runtime JIT
  python3
  tinygrad + numpy
  openssh-server
  mvp entrypoint

mvp-node
  /usr/local/bin/mvp-node
  /usr/local/share/mvp/tinygrad_worker.py

NVRTC is needed because tinygrad's CUDA backend JIT-compiles kernels at runtime. The old image installs the runtime library in the final image and copies CUDA headers from a builder stage; keep that lesson unless a better tinygrad-compatible base is chosen.

Entrypoint shape

The entrypoint should:

• install/accept SSH public keys from VastAI-compatible env;
• start sshd before the node binary;
• print bootstrap progress to stdout/stderr for SSH datastream capture;
• run /usr/local/bin/mvp-node as a child;
• tee node output to a log file readable over SSH;
• keep PID 1 alive after node crash for postmortem.

This mirrors the old pp_entrypoint.sh behavior, with MVP names and paths.

GGUF shape

First deployment target:

• whole-file HuggingFace GGUF fetch;
• small ~1B parameter model;
• node-local cache path;
• progress/fault observations surfaced through the node/datastream path;
• tokenizer from GGUF or adjacent tokenizer source, depending on what the chosen model path supports.

Existing MVP planning code already has concepts for this:

• run_plan.rs::GgufSource::{LocalPath, HuggingFaceGguf};
• TokenizerSource::{EmbeddedGguf, LocalPath}.

Known gap:

• actor/runtime weight commands currently collapse to test artifacts in the node-agent/stage-controller path.
• real GgufSource must be carried through to the node/worker load command before this deployment path is real.

Tinygrad helper shape

Python helper remains subordinate to swactor.

Responsibilities:

• import tinygrad lazily;
• verify CUDA with a small tensor realization;
• fetch/cache the requested GGUF;
• load the model/tokenizer;
• execute inference operations requested by the swactor node;
• emit structured lifecycle/progress/fault events.

Do not make Python the deployment app entrypoint. The app entrypoint is the swactor node binary.

Local verification target

This workstation has a GPU and Docker CUDA should work. The deployment image should be verified locally before VastAI.

Target E2E:

1. Build base image.
2. Build thin MVP node image.
3. Run with docker run --gpus all.
4. Confirm bootstrap stdout/stderr are capturable.
5. Confirm mvp-node starts and joins the orchestrator path.
6. Confirm native datastream frame collection from the container.
7. Confirm tinygrad imports and realizes a CUDA tensor.
8. Confirm the chosen GGUF is fetched/cached from HuggingFace.
9. Confirm one prompt/inference request completes.

The Docker E2E should use the same image shape expected by VastAI; only provisioning differs.

Non-goals for this doc

• designing the full prompt loop schema;
• full VastAI provisioning policy;
• sharded/range GGUF fetching;
• multi-node pipeline-parallel layout.