swactor/crates/mvp-system/src
Zachery Aaron Shores-Chmielewski 49747a03d6 fix: faster provisioning, better ssh checks
Speed up VastAI provisioning by creating instances directly from a cached offer pool, and replace best-effort SSH readiness with classified, post-grace bootstrap-failure detection plus a dedicated per-node provider-status monitor.

- vastai_provisioning: provision_one now iterates a cached candidate_pool of offers calling create_instance directly (with host blacklist and failed-host dedup) instead of re-running client.provision; plan_first_wave_offers caches planned_offer_pool/planned_offer_ids for reuse
- vastai_provisioning: add VastAiProviderMonitor (background thread + AtomicBool stop + Drop) spawned per node via the new spawn_provider_monitor trait method, polling instance_status and emitting VastAiProviderStatusObserved/PollRetry/StatusFailure and terminal-start failures
- vastai_provisioning: add classify_ssh_observation (auth_denied/refused/timeout) with spawn_classifying_stderr_reader; spawn_retrying_ssh_bootstrap aborts after POST_GRACE_BOOTSTRAP_FAILURE_LIMIT repeated classified failures past the grace window instead of retrying forever
- vastai_provisioning: ssh_endpoint delegates to client.wait_for_ssh_endpoint; VastAiNode carries run_id/node_id/label/sink and emits structured VastAiLeaseReady/SshEndpointDiscoveryStarted/SshEndpointReady/RuntimeReadyAccepted/ContractCleanup events; LifecyclePolicy is threaded into start_bootstrap
- tools/vastai: add fetch_instance_status and wait_for_ssh_endpoint_with_policy, refactor wait_for_running onto fetch_instance_status, and export both plus ProviderInstanceStatus from lib.rs
- tools/vastai/types: add ProviderInstanceStatus (actual/intended status, status_msg, public_ipaddr, ssh_port, disk_usage) with ssh_endpoint() and From<InstanceStatus>

Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-27 12:30:49 +04:00
..
actors feat: per-stage GGUF weight sharding and deploy hardening 2026-07-26 13:01:18 +04:00
bin feat: working 7B inferenced over 4 pipeline stages 2026-07-27 10:47:17 +04:00
engine_builder refactor: retire old pipeline-parallel app, restructure mvp-system 2026-07-07 14:40:02 +04:00
tests fix: faster provisioning, better ssh checks 2026-07-27 12:30:49 +04:00
arena_manager.rs refactor: retire old pipeline-parallel app, restructure mvp-system 2026-07-07 14:40:02 +04:00
benchmark_observability.rs feat: working 7B inferenced over 4 pipeline stages 2026-07-27 10:47:17 +04:00
bootstrap_datastream.rs feat: rework datastream into catalog events, add gguf metadata 2026-07-12 10:14:34 +04:00
config.rs feat: add swim probe telemetry, vastai planning cache 2026-07-26 00:05:44 +04:00
dashboard_view.rs feat: rework datastream into catalog events, add gguf metadata 2026-07-12 10:14:34 +04:00
device_bridge.rs feat(mvp): implement mvp-system modules and in-crate tests 2026-06-23 17:51:34 +04:00
distribution_stack.rs feat: add swim probe telemetry, vastai planning cache 2026-07-26 00:05:44 +04:00
docker_cluster_provisioning.rs refactor: retire old pipeline-parallel app, restructure mvp-system 2026-07-07 14:40:02 +04:00
driver_pumps.rs feat: add mvp provisioning subsystem and datastream transport 2026-06-25 16:30:18 +04:00
edge_establisher.rs feat: add mvp provisioning subsystem and datastream transport 2026-06-25 16:30:18 +04:00
endpoint_advertisement.rs feat: mvp-chat multinode docker test with network masking 2026-07-23 14:58:03 +04:00
gguf_metadata.rs feat: rework datastream into catalog events, add gguf metadata 2026-07-12 10:14:34 +04:00
gguf_shard.rs feat: per-stage GGUF weight sharding and deploy hardening 2026-07-26 13:01:18 +04:00
gpu_worker_ctl.rs refactor: retire old pipeline-parallel app, restructure mvp-system 2026-07-07 14:40:02 +04:00
gpu_worker_egress_producer.rs feat: add mvp provisioning subsystem and datastream transport 2026-06-25 16:30:18 +04:00
gpu_worker_ingress_parser.rs refactor(mvp-system): extract orchestrator_app, add gpu prompt loop 2026-07-23 13:39:17 +04:00
gpu_worker_process_adapter.rs refactor: consolidate dist tests, add mvp actors and e2e 2026-06-24 13:30:29 +04:00
lib.rs feat: per-stage GGUF weight sharding and deploy hardening 2026-07-26 13:01:18 +04:00
membership_pool_readiness.rs feat: docker provisioning 2026-06-28 22:54:02 +04:00
mvp_chat.rs feat(mvp-system): extract mvp_chat library module 2026-07-18 13:40:09 +04:00
node_boot_lifecycle.rs feat(mvp): implement mvp-system modules and in-crate tests 2026-06-23 17:51:34 +04:00
node_image.rs fix: faster provisioning, better ssh checks 2026-07-27 12:30:49 +04:00
node_provisioning.rs feat: rework datastream into catalog events, add gguf metadata 2026-07-12 10:14:34 +04:00
observability_surface.rs refactor: retire old pipeline-parallel app, restructure mvp-system 2026-07-07 14:40:02 +04:00
orchestrator_app.rs feat: working 7B inferenced over 4 pipeline stages 2026-07-27 10:47:17 +04:00
orchestrator_run_fsm.rs refactor: retire old pipeline-parallel app, restructure mvp-system 2026-07-07 14:40:02 +04:00
orchestrator_token_endpoint.rs feat(mvp): implement mvp-system modules and in-crate tests 2026-06-23 17:51:34 +04:00
prompt_rpc.rs feat: rework datastream into catalog events, add gguf metadata 2026-07-12 10:14:34 +04:00
provisioning.rs feat: per-stage GGUF weight sharding and deploy hardening 2026-07-26 13:01:18 +04:00
relay_provisioning.rs feat(mvp-system): add toml config overlay and orchestrator cli args 2026-07-08 13:14:49 +04:00
resource_inventory.rs feat(mvp): implement mvp-system modules and in-crate tests 2026-06-23 17:51:34 +04:00
run_plan.rs feat: rework datastream into catalog events, add gguf metadata 2026-07-12 10:14:34 +04:00
shard_fetch.rs feat: docker provisioning 2026-06-28 22:54:02 +04:00
shard_weight_lifecycle.rs feat: docker provisioning 2026-06-28 22:54:02 +04:00
shared_ring_helper_abi.rs feat(mvp): implement mvp-system modules and in-crate tests 2026-06-23 17:51:34 +04:00
stage_controller.rs feat: per-stage GGUF weight sharding and deploy hardening 2026-07-26 13:01:18 +04:00
telemetry.rs feat: rework datastream into catalog events, add gguf metadata 2026-07-12 10:14:34 +04:00
tx_rx_edge_actor.rs refactor(dashboard): rebuild around swactor worker view 2026-06-25 11:29:16 +04:00
vastai_offer_preview.rs feat: successful 8 stage pipeline parallel run, more metrics 2026-07-25 20:04:57 +04:00
vastai_provisioning.rs fix: faster provisioning, better ssh checks 2026-07-27 12:30:49 +04:00
weight_lifecycle.rs feat(mvp): implement mvp-system modules and in-crate tests 2026-06-23 17:51:34 +04:00
weight_shards.rs feat: docker provisioning 2026-06-28 22:54:02 +04:00