Zachery Aaron Shores-Chmielewski
03ae518fb9
Commit the E2E plan and two linked checkpoint handoffs. Describe the frozen-checkpoint remote diagnostic track, working deployment/runtime/campaign paths, evidence limits, and the prerequisites for later paid VastAI execution. Prioritize complete workflow timing, remaining safety verification, and fresh identity-bound qualification on the local track. Remove the internal workload/cleanup budget split as a standalone blocker without changing runtime limits or paid admission guards. Replace the stale active-stream failure status with newer Gate A/campaign evidence and the 186 passing focused review tests. Validate relative handoff links. Historical target artifacts remain outside Git and must be preserved separately; no remote or paid execution was performed.
6.2 KiB
VastAI checkpoint: remote-ready work
Reviewed 2026-09-14. This is the remote-work handoff, not a paid-run authorization. See local blockers for the parallel local track and the E2E plan for the acceptance contract.
Decision
The deployment, runtime, and campaign machinery is implemented far enough to move a frozen checkpoint onto an existing remote development host and test it there. Do not wait for all local qualification work before collecting useful remote evidence.
There are two different milestones:
- Ready now: remote-host diagnostics. Run the existing Docker/SSH fixture and real-binary workloads on infrastructure already available for development.
- Not yet authorized: paid VastAI execution. This still requires successful, fresh ordered qualification and the existing paid admission/cleanup guards. Moving a run to another machine does not bypass those requirements.
What is ready to exercise
| Area | Remote work | Existing evidence |
|---|---|---|
| Deployment and retained-node redeployment | Install new bundles, replace prior worker state, interrupt SSH/orchestrator boundaries, and verify fresh membership without replacing nodes. | Five-node Gate A passed all 12 rounds, all-pairs behavior, and fixture cleanup. |
| Contextual processes and data paths | Exercise public Python bindings, process launch/stop, namespace/blob/stream behavior, and resource reclamation. | Complete local campaign plus current focused process-lifecycle tests. |
| Prepared-fixture reuse | Keep the fixture while running fresh cases, orchestrator recovery, and exact four-node/three-node survivor phases. | 128 normal, 16 recovery, 32 four-node, and 32 three-node cases completed: 208 logical cases / 224 workload segments. |
| Telemetry transport | Observe startup records, reconnect/replay, cancellation, and consistency between worker identity and collected observations. | Current transport tests passed catalog/frame delivery, compressed-frame validation, cancellation, and replay without duplicates. |
| Failure handling | Exercise the existing failure-case workload and inspect terminal observations and cleanup. | A separate local failure-case run records success; it is not complete ordered Gate B evidence. |
These are working test candidates, not claims of stability on real VastAI hosts. Provider offer selection, real network conditions, and provider-side cleanup still need live-provider evidence after admission is authorized.
Remote track: execution outline
1. Freeze and prepare
- Use the committed checkpoint in a separate checkout from ongoing local edits.
- Build and record exact source, executable, deployment-bundle, and image identities. Do not assume the development machine's cached binaries belong to the new checkout.
- Use an existing development host with the required Docker, SSH, image, and networking capabilities. The static-SSH provider manages pre-created Docker nodes; it is not a generic adapter for arbitrary SSH machines.
- Keep the existing fixture ownership and cleanup machinery. Do not introduce a second deployment harness or manually install binaries inside tested nodes.
2. Exercise the established paths
Start with the five-node retained-node redeployment gate. After it passes, exercise the complete campaign and focused failure cases. Preserve exact node sets, public-binding workloads, and cleanup checks.
Measure preparation, redeployment rounds, campaign phases, and cleanup separately. Useful findings include remote-only connectivity failures, stale membership after redeployment, missing telemetry, process-lifetime problems, and stages that dominate elapsed time.
An isolated diagnostic run is useful even when it is not ordered acceptance. Label it as diagnostic; do not merge its coverage into a later accepted run. Do not use another full campaign as the debugging loop after a failure: retain and reproduce the exact failing case or deployment round.
3. Hand off failures without moving the baseline
For each result, record:
- checkpoint/source, executable, image, and deployment identities;
- host environment, gate or case, exact node set, and artifact location;
- PASS/FAIL, stage timings, and cleanup outcome;
- for failure, the first failing observation and smallest known reproduction.
Continue local fixes in the separate local checkout. Promote them to the remote track only as a new checkpoint, with fresh identities and evidence. Do not silently update binaries in the middle of an accepted campaign.
4. Advance to real VastAI only after qualification
The paid path requires fresh successful ordered_acceptance.py evidence,
matching artifacts and immutable image provenance, eligible distinct-host
offers, explicit cost/lifetime limits, credentials supplied through the existing
private configuration, and active durable cleanup ownership.
The current checkpoint has no such successful attestation. Leave the gate checks intact. Do not substitute the static-SSH fixture, a scripted-provider pass, or the historical local campaign for paid authorization.
Evidence and limits
Development-machine artifacts inspected for this checkpoint:
target/ordered-final-3/ordered-acceptance.json: Gate A passed in 331.13 s; campaign passed in 274.08 s; the complete workflow failed at 605.23 s before later Gate B checks.target/ordered-final-3/warm-1/gate-a/deployment-e2e/gate-a-evidence.json: five nodes, 12 rounds, successful behavior and complete cleanup.target/ordered-final-3/warm-1/campaign/vastai-e2e-000000000135282e/: coverage ledgers and checkpoint account for all campaign phases.target/gate-b-failure-cases/timing-runs/1789241279248-532.json: separate failure-case run passed in 14.39 s.
At review time all three release executable hashes matched the ordered-run record, but the current source digest did not. These are historical execution results, not current-tree qualification. The artifact paths are local build outputs and are not included in Git; retain or transfer the evidence explicitly when handing off a run.
Current-source verification during review: 169 harness library tests, 5 shared contract tests, 4 contextual-process tests, 4 telemetry transport tests, and 4 attestation-guard tests passed. No new remote or paid run was performed.