Zachery Aaron Shores-Chmielewski
c6af8e0a5d
Complete an 8-stage pipeline-parallel run over VastAI by provisioning stages high-to-low, adding per-stage/per-step metrics, host anti-colocation, and provider state-timeout guardrails. - orchestrator_app: select the next weight-load stage by max index (provision stages high-to-low for parallel spread), add a throttled "loaded N of M; waiting on stage X" stage_provision_wait headline, and surface min_compute_cap/state_timeout_secs in the config dump. - orchestrator_app: enrich pipeline_token_in/out and tokenizer_decode events with token_count/token_ids/generated_index. - worker_node: add timing metrics across the data path (helper_execute_ms, egress_ring_read_ms, send_ms, ingress_ring_write_ms, object_load_ms), refactor take_complete_ingress_record into IngressRecordBytes (object_id/sequence/extent/flags), and emit a new object_loaded event. - vastai_provisioning: track leased host_ids and blacklist already-leased hosts in later ProvisionRequests so stages don't co-locate, and tag SSH-bootstrap retry logs with the attempt number. - tools/vastai: add min_compute_cap (PP_MIN_COMPUTE_CAP) filter/search query and a LifecyclePolicy state_timeout (PP_STATE_TIMEOUT_SECS) that fails instances stuck in a non-running status instead of polling forever. - xtask: raise the check timeout to 1800s/30s grace, drop --skip-rebuild for VastAI, aggregate per-stage StepExecuted metrics, add a vastai summary section, and write failure artifacts on abort. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
15 lines
337 B
TOML
15 lines
337 B
TOML
[package]
|
|
name = "swactor-vastai"
|
|
version = "0.1.0"
|
|
edition = "2024"
|
|
publish = false
|
|
|
|
[dependencies]
|
|
reqwest = { version = "0.12", features = ["json"] }
|
|
serde = { version = "1", features = ["derive"] }
|
|
serde_json = "1"
|
|
tokio = { version = "1", features = ["macros", "rt", "time"] }
|
|
urlencoding = "2"
|
|
|
|
[dev-dependencies]
|
|
wiremock = "0.6"
|