swactor/examples/pipeline-parallel-inference/N3_DATA_GAPS.md

N=3 data-coverage gaps

Companion to N3_POSTMORTEM_2026-05-25.md. Where the postmortem documents what we do know about the failure, this doc is about the things we don't — and why we should care. Input for the data-collection upgrade.

The framing is investigator-first: each gap is named for the question we couldn't answer, not the file that doesn't emit the field.

The investigation we couldn't finish

Walking back from the symptom — orchestrator's relay-mediated path to stage-2 died at ~5 s, never recovered, stage-2 went silent — the chain of questions we'd want to answer is roughly:

1. Did stage-2's underlying relay tunnel to docean stay up, or did it drop too?
2. If the tunnel stayed up, why didn't iroh re-establish the peer-to-peer path?
3. If the tunnel dropped, who closed it (relay vs. stage-2's iroh vs. the OS), and why?
4. Was stage-2's host network actually broken at that moment, or was this a software-level failure on a working network?
5. Independent of all of the above: why did stage-2 never start its Python worker, when stage-0 and stage-1 both did within seconds?

We could not answer any of these from the bundle. Each one is blocked by a specific missing data source.

The gaps, ranked by how much they hurt this investigation

1. The relay is a black box

The biggest single hole. swactor-iroh-relay on docean produced nothing that ended up in the bundle: no session log, no metrics scrape, no log tail, no record of which node connected, when, how long, and what closed each session.

The orchestrator's local cache says last_failure_reason: "connection-closed". That string is iroh's report of what iroh observed at the application layer. It doesn't tell us whether the relay terminated the session, whether the QUIC stack on either end did, or whether a NAT mapping expired and the relay noticed first.

What this blocks: distinguishing a relay-side eviction from an endpoint-side close from a path-level timeout. Three very different root causes, indistinguishable in the bundle.

2. Relay session and peer connection are conflated

body.iroh.metrics.socket.relay_home_change is a counter that increments when a node changes its home relay. num_conns_opened and num_conns_closed are counters for iroh peer connections. None of these tell us, per moment, whether a given node's tunnel to its relay is up.

This matters because of the asymmetry we hit: from stage-2's view nothing closed (counters quiescent, relay_home_change: 1 for the whole run), but the orchestrator-side cache shows the connection through the relay dying after 5 s. We have no way, from stage-2's data alone, to say whether its relay tunnel was actually still alive when the peer connection died.

What this blocks: answering "did stage-2's tunnel survive?" — the question that decides whether we're looking at a network problem or an iroh state-machine problem.

3. No event when a relay path is established, lost, or replaced

We have snapshot counters but no event stream for relay-path transitions. RelayChanged event count across all four nodes for the whole run: zero. If iroh internally noticed and recovered a relay session inside one snapshot interval, we'd never see it. If iroh didn't notice a dead session, we equally can't see that.

This is the "no log line for the interesting moment" problem. The counter says the final state; we want the transitions.

What this blocks: correlating the moment of failure with what iroh thought was happening. Right now the only event-stream evidence is the orchestrator's connect-timeout retries, which is a downstream symptom.

4. The Python worker subprocess is invisible until it emits

Stage-2 emitted zero worker_starting and zero worker_ready events. Stage-0 and stage-1 emitted both within seconds of boot. Whatever happened to stage-2's worker — never spawned, spawned and crashed before its first event, spawned but blocked — left no trace in our bundle. Stage-2's node process was clearly alive (23 snapshots, 38 event batches), so it isn't a node-process crash.

We don't capture:

• the moment the stage actor decides to spawn the worker
• the subprocess pid, exit code, or stderr tail
• whether the stage actor was gating worker spawn on something (cluster membership? a peer dial?) that never happened

This is a separate failure from the relay flap, possibly with a common upstream cause, possibly not. We can't tell.

What this blocks: deciding whether to focus the fix on transport, on the stage actor's startup ordering, or on worker launch itself.

5. We don't know what host stage-2 was on

boot.json carries container_id, datacenter_id, host_country, host_ip_public, hostname, home_relay_url_at_boot, git_sha, iroh_version — all null except hostname, which is a Docker short id. The orchestrator already has the public IP, datacenter id, and country for each rental at the point lease_chain returns. None of that is forwarded into the container or persisted into the boot snapshot.

So when we say "stage-2's vast.ai rental had a hostile NAT," we literally cannot point at the machine. We can't re-rent the same host to reproduce, we can't compare it against the hosts that did work, we can't even tell you which country it was in.

What this blocks: any kind of fleet-level statistics across runs ("which datacenters fail more often"), and the ability to reproduce the bad rental.

6. Iroh introspection is computed against the wrong API version

The iroh_api_missing event reports iroh_version: "0.96" as a literal string. The lockfile is iroh 0.98.2. The list of "missing" fields is whatever was missing in 0.96 — we have no idea what 0.98 actually exposes, because we never checked.

So when stage-2's snapshot reports observed_conn_type_at_last_use: "None", we don't know whether that's "iroh told us None" or "we couldn't read the field because we're holding a 0.96 shape against a 0.98 struct."

What this blocks: trusting any of the per-peer iroh state in the bundle. This is corrosive — it undermines the whole iroh tier of evidence.

7. Bundle assembly is finalize-or-nothing

The collector only writes MANIFEST.json and the tarball when the orchestrator sends a finalize record. SIGKILL skipped that, so GET /diag/bundle/<run_id> returned 404. The bundle we analyzed was hand-reconstructed from staging files we got to before the collector's TTL cleaned them up.

A real operator hitting a real production incident is going to kill things ungracefully. The "we got lucky" failure mode here is bad enough that we should treat the staging directory as the source of truth and have finalize be an optimization, not a precondition.

What this blocks: any incident bundle from a hard-killed run.

8. Reachability probes only cover one port

We probe UDP echo to :9081 on docean. Stage-2 timed out 1 of 12. We don't probe :7843 (the relay's actual port). So when the relay session dies, we can't say "but the host could still reach the relay port at that moment" — only "but the host could still reach a different port on the same machine."

What this blocks: ruling out transport-level reachability as the cause of relay session death.

9. No event-level breakdown of dials by peer

We have DialStarted: 83 and DialOutcome: 80 as raw event counts. The 3-event drift is not attributed to a specific peer in summary.md. With three peers it's easy enough to grep manually, but the summary should be doing this for us, especially at higher N where per-peer asymmetry is the whole story.

What this blocks: at-a-glance answer to "which peer was hard to reach," which is the first question for any cluster failure.

10. No gossip-arrival evidence on the silent node

Stage-2's peers[] contained only the orchestrator. We don't know whether stage-2 received NameRegistry gossip about its siblings and failed to dial, or never received the gossip at all. The bundle has MessageReceived: 72 for stage-2 but the breakdown isn't recorded.

What this blocks: distinguishing a control-plane failure (gossip didn't arrive) from a data-plane failure (dials based on gossip didn't connect).

11. No kernel-level network counters

/proc/net/snmp, /proc/net/udp, per-interface drop counts — none captured. For stage-2, with 537 holepunch attempts and 5 reported mapping failures, we can't tell "iroh sent and the OS dropped it" from "iroh sent and the OS accepted it and the path silently lost it." These are at the edge of what's worth collecting — modest cost per snapshot, but the cases where they matter are real.

What this blocks: distinguishing iroh-layer pathology from host-network pathology when the two look identical from above.

What this looks like in priority order

If we only get to fix a few of these for the next deployment:

Must-have to investigate another N=3 failure:

• gap 1 (relay-side data)
• gap 4 (worker subprocess visibility)
• gap 5 (host metadata forwarding)
• gap 7 (bundle assembly without finalize)
• gap 6 (iroh API version sanity check)

Strong-have:

• gap 3 (relay-path transition events)
• gap 2 (relay-tunnel-state field, separable from peer state)
• gap 10 (gossip-receipt event)

Nice-to-have:

• gap 8 (relay-port probe)
• gap 9 (per-peer dial rollup in summary)
• gap 11 (kernel counters)

The "must-haves" are the ones where, looking back at this bundle, the absence actually prevented a conclusion. The rest would have made the investigation faster but weren't strictly load-bearing.

What this implies for the sim

A separate concern that overlaps: most of these gaps are real-network gaps that the sim doesn't model at all. The sim doesn't have a relay, doesn't model NAT-mapping behavior, doesn't model relay-session-up-but-peer-connection-down asymmetry, and doesn't distinguish kernel-level packet loss from iroh-level path failure.

If we want the sim to reproduce a failure like this one, the data model the sim exposes has to be at least as rich as the data the postmortem needed to read — otherwise "we reproduced it in sim" won't actually mean we understand it. Whatever fields we add to the bundle should land in the sim's per-tick state too.