zacheryasc/swactor
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
swactor/examples/pipeline-parallel-inference/N3_COVERAGE_EXTENSION_SPEC.md

24 KiB
Raw Blame History

N=3 collection coverage extension — behavioral spec

Companion to N3_POSTMORTEM_2026-05-25_1779733878.md, N3_SIM_TEST_BATTERY_SPEC.md, and the simulator's SIM_SPEC.md. This document is the contract for a separate coding agent to extend diagnostic collection coverage along three layers — production diagnostics, simulator emit/model, and simulator test verification — for the gaps the 1779733878 run surfaced.

This is a behavioral spec. It names the gap, the contract the collected data must satisfy, and the layer(s) the contract threads through. It does not prescribe field names, file layout, or implementation choices.

0. Motivation

The 1779733878 run validated the prior observability upgrade — A+B+C tiers were load-bearing, the bundle attributed the failure to "dials to orchestrator fail by timeout 7/11 while every inter-stage dial succeeds 19/19" in one table — and surfaced five residual collection gaps the upgrade either left as carry-forward (gaps 1, 5, 8 from the original scorecard) or that this run exposed for the first time (response-leg event absence, bundle-serve behavior under run-id reuse).

A gap whose collection landed only in production but not in the sim is a gap that the sim test battery can never guard — the next regression in that field will be caught only by another live deploy. A gap whose sim model exists but is not exercised by a test is dead code. The coverage in this spec is required to thread through every layer where it can — and the spec is explicit when a layer does not apply.

Five coverages, each threaded through up to three layers. Each coverage may close one of {Pass, Mixed, Fail} against the current source, and each names the close criterion.

1. Cross-cutting requirements

These hold for every coverage in §2.

1.1 Three-layer threading

For each coverage, the spec names which of the three layers it threads through:

  • D — Diagnostics: the production bundle gains a field, event, or section that closes the gap the postmortem named.
  • S — Sim: the simulator's relevant component (host kind, network, relay vertex, bundle writer) emits the same field / event / section under the same conditions, with bundle-shape parity per SIM_SPEC.md §5 (cross-cutting "Bundle-shape parity with prod") and §9 (bundle schema).
  • T — Tests: the sim test battery gains a scenario or property test asserting the bundle carries the new data when the triggering condition holds, and gains a discriminator assertion when absence is meaningful (per the honesty-under-absence pattern the prior upgrade established).

A coverage that threads through fewer than three layers is honest about which it skips and why. Skipping S because "the simulator does not model this surface" is acceptable; skipping it because "this is not interesting" is not.

1.2 Honesty-under-absence carries forward

The prior upgrade's status_source discriminator pattern (a status field always paired with a field naming how that status was derived — "iroh" for native, "derived" for inferred) is the model. Any new field whose value might be absent or derived must carry an adjacent discriminator. A bundle reader must never be left guessing "unknown means the thing is unknown" vs "we couldn't ask."

1.3 Additive evolution

Every new field on SnapshotBody, every new event variant, every new section in the post-processor output is additive. An old bundle reader on a new bundle still parses; a new bundle reader on an old bundle reports the new field absent rather than erroring. The prior upgrade established this contract; coverage 2.x preserves it.

1.4 Sim/prod schema parity

Per SIM_SPEC.md §9.2: the event payload schema is exactly the production diagnostics schema for that kind. The sim invents no new event kinds. A coverage that lands an event in prod and in sim uses the same schema in both, verified by the existing parity tests under crates/simulation/tests/sim_cross_pollination.rs. A schema added to sim ahead of prod is a deliberate amendment and declares so explicitly.

1.5 Verdict-first per layer

Every coverage in §2 declares, per layer, its expected status on the current source: landed (the layer satisfies the contract; the work is verification / regression-guard), partial (the layer has structure but not data flow), absent (the layer has nothing today). The implementing agent's work is to bring each layer to "landed" against this spec or to file a structural reason why a layer cannot land.

2. The coverages

Five, ordered by the postmortem's own ranking of residual gaps.

2.1 Orchestrator-side host provider metadata forwarding

Source: postmortem §"Observability upgrade scorecard" row "gap 5 host metadata" (◐); postmortem §"Data-collection / deployment gaps surfaced by this run" item 1.

Gap: The orchestrator has each rental's public IP, datacenter, country, and contract id at lease_chain return time. The container can read these from SWACTOR_DIAG_* env vars. The container env is never set. The boot record's host_ip_public/datacenter_id/host_country/vastai_contract_id/ home_relay_url_at_boot fields are still null in every bundle. The contract id arrives but in container_id, not vastai_contract_id — the naming is currently load-bearing-but-wrong.

Contract — what closing the gap looks like:

  • D: the orchestrator's per-rental env payload, at the point it creates each container, carries every field the boot record can consume — public IP, datacenter id, host country, vast.ai contract id, the home relay URL the container will use. The boot record reflects every field as a concrete value, not null, whenever the orchestrator had the data. The fields that name cloud-provider state stay absent only on hosts where they genuinely do not apply (e.g., local development), and the bundle's ## Hosts section renders ? for absent fields (already implemented per S-A2).
  • S: scenarios declare per-peer host context as part of the peer's kind_config. The sim's stage host populates its boot record / HostContext from the scenario declaration the same way prod populates from env. A scenario without declared host context produces a bundle whose ## Hosts section is all-? for that peer — same absence shape as a local-dev prod bundle.
  • T: a scenario declaring heterogeneous host context across three peers (e.g., two datacenters, two countries) produces a bundle whose ## Hosts section renders the declared fields verbatim. A scenario that declares no context for one peer and full context for the others produces a bundle distinguishable from "no context declared for any peer" by the ? placement.

Expected status:

  • D: partial. The container reads the env vars (per S-A2); the orchestrator does not set them. The misnaming of contract id → container_id is a separate cleanup.
  • S: absent. The sim's stage host carries no host context in its current scenario schema.
  • T: absent. No test exercises this discriminator.

Close criterion: a deployed bundle's ## Hosts section names the datacenter, country, public IP, and contract id of every vast.ai rental, and the docker container_id field carries the docker container id, not the vast.ai contract id. A sim bundle with declared host context produces the matching shape.

2.2 Relay-port reachability probe

Source: postmortem §"Observability upgrade scorecard" row "gap 8 relay-port probe" (✗); postmortem §"Data-collection / deployment gaps surfaced by this run" item 3.

Gap: Stage probe arrays carry only collector_udp_echo (:9081). No probe targets the relay's actual port (:7843). Whether a stage retained transport-level reachability to the relay at the moment its peer-connection died is currently inferable only from a different port on the same host. The S-E1 work is documented as landed (per the prior iteration log) but the 1779733878 bundle shows no relay-port probe records. The wiring is in place; the data is not.

Contract — what closing the gap looks like:

  • D: every stage's snapshot carries a probe outcome for the relay's UDP listener (host + port resolved from the home relay URL). The outcome is one of the five-discriminator set the prior upgrade established: ok / timeout / refused / unresolved / error. A snapshot taken when the relay is reachable carries ok with an RTT; a snapshot taken when the relay is unreachable carries the appropriate failure discriminator with no silent fallback to "absent."
  • S: the sim's stage host emits the same probe record on every snapshot, sourced from a query the network answers about the stage→relay edge. The relay vertex's RelayKill / RelayCapacityChange mutations are reflected in the probe's outcome distribution.
  • T: a scenario that issues a RelayKill mutation mid-run produces a bundle whose every stage's relay-port probe outcome flips from ok to unresolved (or timeout, per the network's policy) at the mutation's at_ns and remains there through RelayBoot. The probe-outcome timeline is the test's discriminator between "tunnel down" and "tunnel up but peer conn down" — coverage 2.x.A from the battery spec consumes this signal.

Expected status:

  • D: partial. Probe scheduler wires the target; emission to the bundle is unverified by this run's evidence.
  • S: absent. The sim's network has no probe-query surface today.
  • T: absent.

Close criterion: the next deployment's bundle has a relay-port probe outcome on every stage's snapshots. A sim scenario with RelayKill produces the probe-outcome flip in the bundle.

2.3 Relay session lifecycle on the relay side

Source: postmortem §"Observability upgrade scorecard" row "gap 1 relay observability" (◐); postmortem §"Data-collection / deployment gaps surfaced by this run" item 4.

Gap: The relay reports identity and 186 snapshots into the bundle but cannot answer "who closed session X and why" — the per-session lifecycle hooks are the documented skeleton with active=0 opens=0 closes=0. iroh_relay::server exposes no session hooks. Until it does, a relay-side eviction is unanswerable from the relay's own data; the postmortem fell back to node-side dial outcomes.

Contract — what closing the gap looks like:

  • D: the relay's bundle contribution names, per peer session, the open time, close time, close-initiator discriminator (relay / peer / transport / unknown), close reason string (relay-specific or transport-specific), bytes transferred per direction, and duration. The mechanism is free — middleware around the relay binary, kernel-layer observation, a forked relay, or upstream hooks when iroh exposes them. The contract is the shape, not the source. When the source is unavailable, the relay's bundle contribution still emits the gap-1 absence-line the prior upgrade introduced in summary.md (the post-processor's acceptance branch for "no relay-role node has session data").
  • S: the sim's relay vertex emits RelaySessionOpened / RelaySessionClosed records when it accepts and releases per-peer queues. The records carry the same shape D requires. A RelayKill mutation produces a RelaySessionClosed { initiator: "relay", reason: "killed", ... } for every session active at the mutation time.
  • T: a scenario where the relay accepts three peer sessions, runs to steady state, then receives a RelayKill mutation, produces a bundle whose relay contribution names three RelaySessionOpened events at the convergence boundary and three RelaySessionClosed { initiator: "relay" } events at the mutation time. A scenario where a peer voluntarily disconnects produces a session-closed event with initiator: "peer". The discriminator must hold.

Expected status:

  • D: skeleton — wired call sites, no data flow. Whether the unblock path is upstream hooks, middleware, or kernel observation is implementer's call.
  • S: partial. RelayObservability exists on the host side per the prior upgrade (S-B1); the sim's relay vertex itself does not emit lifecycle events as engine-synthesized records.
  • T: absent.

Close criterion: a deployed bundle from a run that included a peer dial failure attributable to a relay-side close names the close-initiator and reason in the relay's bundle contribution. A sim RelayKill scenario produces the matching event stream.

2.4 Inference response-leg instrumentation

Source: postmortem §"Data-collection / deployment gaps surfaced by this run" item 5.

Gap: The 1779733878 postmortem's conclusion — "last stage could not deliver the response" — was inferred from dial timeouts plus the absence of an inbound InferenceResponse, not from a typed event on the last stage saying "I tried to send the response and the send outcome was X." The chain stage-(N-1) → InferenceResponse → orchestrator's inbox has no event on the sending side. A typed event makes attribution a one-line read rather than a triangulation.

Contract — what closing the gap looks like:

  • D: the production stage actor, on attempting to send an InferenceResponse upstream, emits a typed event naming the target peer, the request id the response corresponds to, the byte size, and the send outcome. The outcome discriminator is the iroh-level result the transport returns (succeed / timeout / connection-closed / refused / unresolved / queued-but-not- acked-in-budget). The post-processor surfaces these in summary.md under a section that names which inference request was answered by which stage's send and how that send resolved.
  • S: the sim's stage host kind grows a minimal inference message surface (InferenceRequest inbound to stage-0, InferenceResponse outbound from stage-(N-1), forwarded between adjacent stages as opaque payload in the MVP). The stage host emits the same typed response-send event when it attempts the outbound to the orchestrator. The codec contract (SIM_SPEC.md §3.3) carries the inference messages with byte-equality between sim and prod encoding.
  • T: a scenario where the orchestrator's inbound path is broken via RelayPeerConnDown on the last leg (stage-(N-1) → orch) while every other leg works produces a bundle whose last stage emits exactly one InferenceResponseSent event with send_outcome in the failure-discriminator set. A scenario where every leg works produces an InferenceResponseSent with send_outcome=success and a matching InferenceResponseReceived (or equivalent) on the orchestrator's side.

Expected status:

  • D: absent. The current stage actor's send call is not wrapped in a typed diagnostic event for the response leg.
  • S: absent. The sim's stage host kind today produces no Send actions during its lifecycle (SIM_SPEC.md §6A.5 notes this explicitly and defers inter-stage traffic to a later revision). Closing this coverage moves that deferral forward.
  • T: absent.

Close criterion: a deployed bundle from any run where the response did not return names the send outcome of the last stage's response attempt in a single event. A sim scenario modeling the same failure produces the same shape.

2.5 Bundle serve hardening under run-id reuse

Source: postmortem §"Bundle recovery" caveat; postmortem §"Data-collection / deployment gaps surfaced by this run" item 2.

Gap: When a run id is reused across the failed-first-lease / successful-second-lease shape the 1779733878 run exhibited, a finalize record from the first phase pins a stale canonical bundle in the collector's cache. A subsequent GET serves the stale 5.3 KB bundle instead of synthesizing the rich 9.3 MB one from current staging. Two adjacent quirks: finalize_received stays true after the on-disk finalize-*.json is deleted, and the synthesized manifest still lists a removed node directory.

Contract — what closing the gap looks like:

  • D: the collector's download_bundle handler prefers the richer of {canonical-cached, synthesized-from-current-staging} by a size or node-count heuristic, or rebuilds canonical when staging has grown past the cached bundle's manifest. Deleting a node directory from staging clears the corresponding finalize record from in-memory state. The synthesized manifest reflects the current on-disk state, never a stale in-memory record. The finalize_received boolean is sourced from the same place the serve decision is sourced from — a single source of truth, not two diverging caches.
  • S: not applicable. The sim writes bundles directly to a destination directory; there is no serve logic, no finalize cache, no run-id reuse semantics. The coverage threads through D only.
  • T: not applicable as a sim test. The discriminator (stale vs fresh serve on a finalize-then-staging-growth sequence) is a collector unit-test concern living under crates/distribution/tests/, not a scenario the sim engine can express. The implementing agent should land the collector test alongside the D-layer change; it is named here so that the coverage's verification surface is honest about where it lives.

Expected status:

  • D: absent. Current serve logic prefers cached canonical unconditionally when finalize_received is true.
  • S: not applicable.
  • T: collector unit test absent.

Close criterion: a collector unit test writes two phases of staging with an intervening finalize, deletes the first-phase node, and verifies the second GET serves the richer bundle and that the cleared node does not appear in the manifest.

2.6 Per-SWIM-probe RTT and observed latency distribution

Source: postmortem §"SWIM churn and relay events" (1701 SwimTransitions over ~7 min, all with conn_type=Relay); postmortem §"UDP echo probes" (tier-2 RTTs spread 181–405 ms; SWIM probes ride a relay-mediated path on top of these); SWIM_TUNING_REPORT.md §6 limit 3 ("SWIM host adapter does not emit probe_sent / probe_received / probe_timed_out events").

Gap: The bundle has tier-2 UDP-echo RTT to docean:9081 — a host-level surface that does not represent the latency SWIM actually sees. SWIM rides a relay-mediated peer connection whose RTT is at least one extra hop and is subject to relay-side HOL queueing under load. The bundle currently exposes:

  • per-snapshot iroh counters (cumulative MessageSent / MessageReceived),
  • aggregate SwimTransition counts,
  • per-peer dial outcomes (Timeout / Success rollup),

but it does not expose per-probe RTT, per-peer RTT distribution over the run window, or correlation between probe_timed_out-class outcomes and observed RTT spikes. Without this surface, SWIM tuning is a guess against the deploy's actual latency distribution rather than a measurement.

This gap also mirrors the simulator's own limit per SWIM_TUNING_REPORT.md §6.3: the SWIM host adapter does not emit the probe lifecycle events, so the §10 evaluator's no_flap_while_probes_ok is structurally Inconclusive. Closing the gap on both sides closes the assertion's precondition.

Contract — what closing the gap looks like:

  • D: each SWIM ping/ack pair emits a typed event naming the observer, target, virtual-or-wall send time, virtual-or-wall receive time, the resulting RTT, and the discriminator (success / timeout / connection-closed / etc.). The post-processor surfaces a ## Probe RTT distribution section with median, p95, p99 per (observer, target) pair, plus per five-second bucket so degradation over time is visible. A probe_timed_out outcome carries the configured timeout budget alongside the observed RTT (where one exists) so a reader sees "probe missed a 3 s budget by 200 ms" vs "no response within 3 s, never arrived."
  • S: the simulator's SWIM host adapter emits the same probe lifecycle events. Per SIM_SPEC.md §9.2 parity, the schema is identical to D's. This is the §6.3 limit from SWIM_TUNING_REPORT.md closing simultaneously with D — the bundle reader cannot tell a sim run from a prod run by this surface.
  • T: a scenario with a declared per-link latency distribution (heavy-tailed, peer-symmetric) produces a bundle whose postproc RTT section's median, p95, p99 fall within stated tolerance of the scenario's declared distribution. A scenario with a LatencySpike mutation produces a bundle whose RTT section shows the spike at the mutation time. The precondition for no_flap_while_probes_ok is now satisfied; the assertion moves off Inconclusive for every scenario using a SWIM-host kind.

Expected status:

  • D: absent. No per-probe event today.
  • S: absent. SWIM_TUNING_REPORT.md §6.3 names this explicitly.
  • T: absent.

Close criterion: a deployed bundle's postproc summary names the median / p99 RTT per (observer, target) and a sim bundle produces the matching surface. no_flap_while_probes_ok resolves to Pass or Fail (not Inconclusive) on every SWIM scenario in the calibration library.

Downstream: this coverage is the data surface N3_SWIM_TUNING_SPEC.md consumes. SWIM tuning itself is downstream of collection and lives in that sibling document.

3. Out of scope

  • Inference protocol surface beyond the response leg. Coverage 2.4 instruments the response-send event. A full inference- protocol event stream (microbatch routing, KV cache, per-stage worker activity) is broader than what the 1779733878 postmortem could not answer; it belongs in a separate spec when a postmortem demands it.
  • Post-processor summary enhancements. SWIM transition distributions, per-(observer, target, reason) breakdowns, cross-node temporal alignment around the moment of failure — these are renderer concerns, not collection concerns. They presuppose the data is in the bundle; this spec is about the data.
  • Orchestrator-topology fixes. The 1779733878 postmortem's item 6 names the root cause as a NAT'd local orchestrator with no reachable port. That is a deployment-shape question for the runbook, not a collection-coverage question.
  • Runbook fixes. The --gpu RTX_4090 vs RTX 4090 line in DEPLOYMENT_TEST.md (postmortem item 7) is a runbook bug, not a collection gap.
  • Sim coverage of upstream-blocked surfaces. If iroh_relay::server continues to expose no session hooks, the sim's relay vertex can model the lifecycle events the contract requires, but the production D layer of coverage 2.3 may remain partial. That partiality is a structural blind spot to file per the established blind-spot discipline; this spec does not resolve it.

4. References

  • N3_POSTMORTEM_2026-05-25_1779733878.md — the second 2026-05-25 deployment's postmortem. §"Observability upgrade scorecard" is the source for coverages 2.1, 2.2, 2.3; §"Data- collection / deployment gaps surfaced by this run" items 1–5 map to coverages 2.1, 2.5, 2.2, 2.3, 2.4 respectively.
  • N3_SIM_TEST_BATTERY_SPEC.md — the sim-test battery spec. The battery's families A (relay peer-conn down) and the discriminator it builds against the relay-port probe (coverage 2.2) and the relay session lifecycle (coverage 2.3) consume the data this spec lands.
  • crates/simulation/SIM_SPEC.md — the simulator's behavioral surface. §3.3 codec contract, §5A relay vertex, §6A stage host kind, §9 bundle layout are the load-bearing references for the S-layer contracts.
  • crates/simulation/SWIM_TUNING_REPORT.md — the prior tuning pass against simulated 60 ms latency. §6 limits (especially §6.3 "SWIM host adapter does not emit probe_sent / probe_received / probe_timed_out events") are the source for the S-layer of coverage 2.6.
  • N3_SWIM_TUNING_SPEC.md — the downstream spec that consumes coverage 2.6's data surface to retune SWIM against the observed 1779733878 latency distribution. Sibling document.