Speed up VastAI provisioning by creating instances directly from a cached offer pool, and replace best-effort SSH readiness with classified, post-grace bootstrap-failure detection plus a dedicated per-node provider-status monitor.
- vastai_provisioning: provision_one now iterates a cached candidate_pool of offers calling create_instance directly (with host blacklist and failed-host dedup) instead of re-running client.provision; plan_first_wave_offers caches planned_offer_pool/planned_offer_ids for reuse
- vastai_provisioning: add VastAiProviderMonitor (background thread + AtomicBool stop + Drop) spawned per node via the new spawn_provider_monitor trait method, polling instance_status and emitting VastAiProviderStatusObserved/PollRetry/StatusFailure and terminal-start failures
- vastai_provisioning: add classify_ssh_observation (auth_denied/refused/timeout) with spawn_classifying_stderr_reader; spawn_retrying_ssh_bootstrap aborts after POST_GRACE_BOOTSTRAP_FAILURE_LIMIT repeated classified failures past the grace window instead of retrying forever
- vastai_provisioning: ssh_endpoint delegates to client.wait_for_ssh_endpoint; VastAiNode carries run_id/node_id/label/sink and emits structured VastAiLeaseReady/SshEndpointDiscoveryStarted/SshEndpointReady/RuntimeReadyAccepted/ContractCleanup events; LifecyclePolicy is threaded into start_bootstrap
- tools/vastai: add fetch_instance_status and wait_for_ssh_endpoint_with_policy, refactor wait_for_running onto fetch_instance_status, and export both plus ProviderInstanceStatus from lib.rs
- tools/vastai/types: add ProviderInstanceStatus (actual/intended status, status_msg, public_ipaddr, ssh_port, disk_usage) with ssh_endpoint() and From<InstanceStatus>
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Move iroh_driver and the relay binary out of distribution into a dedicated
crates/iroh-driver (lib re-exports IrohDriver; relay bin renamed). Remove the node crate
and the single-gpu-inference example; drop the docker/datastream demo. Slim
pipeline-parallel vastai.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
