- Split arena/ring/object-record into a new data-plane crate and node/plugin contracts
into a provisioning crate.
- Reorganize mvp-system into orchestration, staging, node, chat, and worker modules;
extract binaries into chat/runtime and node/worker_node_runtime.
- Add MVP_SYSTEM_MODULE_BOUNDARY_SPEC.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Land end-to-end canonical benchmark observability and a synthetic datastream-connectivity preflight across the orchestrator, chat, worker-node, and Python tinygrad worker, plus an xtask validator, so a 4-stage 7B pipeline run is fully diagnosable.
- benchmark_observability: expand stamp() with canonical producer fields (producer_component/instance_id/process_id/sequence, wall_clock_unix_ms, monotonic_ms, clock_source) and schema_version so every event shares one envelope shape
- orchestrator_app + bin/{mvp_chat,worker_node}: stamp OrchBootstrap/OrchPromptEvent/ChatProgress/NodeEvent/SamplerHealth with the canonical fields plus span_id/parent_span_id, and add a 4-phase synthetic datastream preflight (ProducerConfigured/Connected/SyntheticEventSent/Observed) plus an endpoint_config_snapshot event on each process
- apps/mvp-node/tinygrad_worker: add apply_canonical_envelope()/datastream_endpoint_snapshot() and emit_python_datastream_preflight() mirroring the Rust preflight, and enrich benchmark_stamp() with the same producer fields
- bin/worker_node: pass MVP_DATASTREAM_ENDPOINT_ID/MVP_BENCHMARK_PRODUCER_INSTANCE/MVP_IROH_ENDPOINT_ADDR_MASK/MVP_IROH_RELAY_MODE env to the spawned tinygrad worker so its stamps identify the stage
- bin/mvp_chat: add --pipeline-parallel as an alias for --pipeline-stages (with a duplicate-guard) and bump recursion_limit
- xtask: add a benchmark-observability validator (ValidatorFinding/BenchmarkValidation, validate_benchmark_observability, canonical-stamp and stage/edge checks, evidence + gap-report builders) with tests for missing python datastream connectivity, wrong run_id, and missing span_id
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Complete an 8-stage pipeline-parallel run over VastAI by provisioning stages high-to-low, adding per-stage/per-step metrics, host anti-colocation, and provider state-timeout guardrails.
- orchestrator_app: select the next weight-load stage by max index (provision stages high-to-low for parallel spread), add a throttled "loaded N of M; waiting on stage X" stage_provision_wait headline, and surface min_compute_cap/state_timeout_secs in the config dump.
- orchestrator_app: enrich pipeline_token_in/out and tokenizer_decode events with token_count/token_ids/generated_index.
- worker_node: add timing metrics across the data path (helper_execute_ms, egress_ring_read_ms, send_ms, ingress_ring_write_ms, object_load_ms), refactor take_complete_ingress_record into IngressRecordBytes (object_id/sequence/extent/flags), and emit a new object_loaded event.
- vastai_provisioning: track leased host_ids and blacklist already-leased hosts in later ProvisionRequests so stages don't co-locate, and tag SSH-bootstrap retry logs with the attempt number.
- tools/vastai: add min_compute_cap (PP_MIN_COMPUTE_CAP) filter/search query and a LifecyclePolicy state_timeout (PP_STATE_TIMEOUT_SECS) that fails instances stuck in a non-running status instead of polling forever.
- xtask: raise the check timeout to 1800s/30s grace, drop --skip-rebuild for VastAI, aggregate per-stage StepExecuted metrics, add a vastai summary section, and write failure artifacts on abort.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Lift the VastAI two-stage cap and let mvp-chat-check drive an arbitrary N-stage deployment through a new --pipeline-stages flag.
- orchestrator_app: remove the provider=vastai >2 stage cap and replace the two-stage plan test with an eight-stage plan test (8 specs, no mounts, remote GGUF, max-context).
- xtask: refactor scenario parsing into MvpChatCheckInvocation carrying an optional pipeline_stages, parse a `--pipeline-stages n` flag (rejecting 0/missing values), and default multinode/docker to 2 while VastAI uses the explicit count.
- xtask: thread the invocation through run_mvp_chat_check/run_mvp_chat_check_process and update the usage text and scenario tests (including `--vastai --pipeline-stages 8`).
- gitignore: ignore .deployment-notes/.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Land the first working two-stage pipeline-parallel run over VastAI, wiring a real inter-stage data path with observability, a max-price offer cap, and remote-image reuse.
- orchestrator_app: raise the VastAI pipeline-stage cap from 1 to 2 and let VastAI pipeline planning resolve the HuggingFace GGUF from the default local cached-model metadata path instead of requiring host mounts; add --vastai-max-dph-total (CLI/env/TOML) config.
- vastai_provisioning: make complete_bootstrap a no-op so the SSH bootstrap log tail stays alive past runtime-ready until node stop, preserving post-ready worker logs; add a test asserting the tail is only stopped on NodeStop.
- worker_node: emit data-path NodeEvents across the pipeline (iroh_edge_stream_arrived/bytes_read/bytes_sent, egress_ring_read, ingress_ring_write) with edge/byte metadata.
- tools/vastai: add max_dph_total (PP_MAX_DPH_TOTAL) to SelectionPolicy, the reachable-offer filter, and the search query, and improve the empty-pool error message.
- xtask: pass --skip-rebuild for the VastAI scenario and gate it on a new require_vastai_data_path_facts plus GPU facts (ring install, activation object load/step, interstage handoff, iroh edge read/sent).
- mvp_chat: add ChatModelConfig (model id/gguf/tokenizer/max-context) forwarded to the orchestrator; for VastAI + skip-rebuild, emit skip events and reuse the remote node image without a local build.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add a `--vastai` acceptance scenario to mvp-chat-check that provisions real VastAI nodes and verifies the remote provider data path, and standardize the VastAI API key on VAST_API_KEY.
- xtask: add the VastAi variant and `--vastai` flag to the mvp-chat-check scenario, passing `--vastai --yes --endpoint-addr-mask relay-only` (no --cached-model) and gating it on a new require_vastai_network_facts check (node_spec workers, ProvisionStart, provider_start, iroh_driver ready).
- xtask: track VastAI dump-log facts (vastai_node_spec_worker_count, vastai_provision_start_nodes, vastai_provider_start_nodes) via record_vastai_provision_dump_log_event and skip the local ChatProgress span assertion for the remote scenario.
- config: rename the VastAI key env var to VAST_API_KEY in ResolvedVastAiConfig validation while keeping MVP_VASTAI_API_KEY/VASTAI_API_KEY fallbacks.
- orchestrator_app: resolve the api key with VAST_API_KEY first, then MVP_VASTAI_API_KEY/VASTAI_API_KEY, and update the missing-key error messages.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Stop the mvp-chat process on Ctrl-C/SIGTERM so it tears down cleanly instead of leaking past signal delivery.
- orchestrator_app: spawn_stop_listener now spawns a Linux SIGINT/SIGTERM handler (signal_hook) that sends the shutdown signal alongside the existing stdin "stop"/"shutdown"/"quit" listener; on non-Linux the spare sender is dropped.
- orchestrator_app: split the channel sender into a stdin_tx clone so the stdin thread and the signal thread each own a sender without moving it out of scope.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add endpoint-address masking and a relay-only advertisement path so the multinode Docker mvp-chat scenario can run with direct addresses stripped.
- endpoint_advertisement: add EndpointAddrMask (Full/RelayOnly) parsed from --endpoint-addr-mask/MVP_IROH_ENDPOINT_ADDR_MASK, and advertised_endpoint that rebuilds an EndpointAddr from relay URLs only, rejecting relay-only without a relay URL
- orchestrator_app: mask the coordinator endpoint before advertising it, thread the masked collector endpoint into datastream subscribe/runtime-ready acks, surface endpoint_addr_mask/has_relay/direct_addr_count in iroh_driver and node_spec events, forward the mask env to workers, and add a 60s RUNTIME_READY_TIMEOUT to the runtime-ready barriers
- worker_node: advertise the masked self endpoint in the iroh_driver ready and coordinator_join events and propagate it through runtime_ready_local and PendingRuntimeReady
- mvp-chat: add --relay-mode/--relay-url/--endpoint-addr-mask plus a [relay] toml section, require (with a Vast.ai fallback) a relay URL when relay-only, and forward all three to the orchestrator CLI
- node_image: resolve the worker binary to a workspace-relative path for the Docker COPY via docker_build_context_path, rejecting paths outside the build context
- xtask/specs: run MultinodeDocker with --relay-mode default --endpoint-addr-mask relay-only, add dump-log fact checks for relay-masked orchestrator/node/coordinator advertisement, and document the mask/relay flags in mvp_chat.md
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
- Pull ~7.4k lines out of the orchestrator bin into a new orchestrator_app library
module.
- Wire a local single-node GPU prompt loop into the mvp_chat bin; touch
gpu_worker_ingress_parser.
- Grow xtask and the mvp-node tinygrad worker.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
