Zachery Aaron Shores-Chmielewski
a5c1c1e68b
Turn mvp-chat-check into a benchmarking harness with per-stage latency capture, a run envelope, summary artifacts, and a mvp-chat-compare command for delta analysis. - xtask: add mvp-chat-compare <baseline> <candidate> that checks comparability (schema/scenario/workload/model/provider/pipeline_stages) and prints deltas for total, prepare, standup-to-RPC, and per-prompt roundtrip/first-token/decode/text-decode ms. - xtask: build_benchmark_summary now writes stdout/stderr/prompts/redacted-config/summary artifacts with per-artifact byte counts and a vastai summary section; rename the dump log to datastream.ndjson. - xtask: add write_failure_artifacts so failed checks still emit a failure summary with the prompt-corpus blake3 and artifact sizes. - mvp_chat: add the mvp.chat.benchmark channel and emit_benchmark_envelope (BenchmarkRunEnvelope with model/runtime/provider/workload detail), and tag prompt events with prompt_index and a blake3 prompt_hash. - tinygrad_worker: add per-phase latency metrics (encode/decode/text-decode/first-token elapsed_ms; stage_execution_ms/record_write_ms on execute_step; ring_readable/encode_prompt/decode_tokens elapsed_ms plus payload sizes). - vastai_provisioning: emit VastAiLeaseReady and VastAiSshEndpointReady provider lines (contract/offer/host/gpu/dph, ssh host/port/user) for observability. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
11 lines
169 B
TOML
11 lines
169 B
TOML
[package]
|
|
name = "xtask"
|
|
version = "0.1.0"
|
|
edition = "2024"
|
|
|
|
[dependencies]
|
|
serde_json = "1"
|
|
blake3 = "1"
|
|
|
|
[target.'cfg(target_os = "linux")'.dependencies]
|
|
libc = "0.2"
|