swactor/examples/pipeline-parallel-inference/N3_OBSERVABILITY_UPGRADE_SPEC.md

# N=3 observability upgrade — behavioral spec

Sister doc to `N3_DATA_GAPS.md`. The gaps doc says *what's missing
and why we care*. This doc says *what the system must do once the
gaps are closed.*

Each section is a behavior contract: requirements the running
system has to satisfy after the work is done. Implementation
strategy — which crate, which file, which trait — is left to the
person picking up the work, except where a pattern is load-bearing
to the contract itself (the subprocess introspector is the one
explicit pattern requirement, called out below at the user's
direction).

Throughout: every "the bundle contains X" claim is testable. A
post-deployment run that doesn't satisfy these is a failed upgrade.

## Cross-cutting requirements

1. **Additive evolution.** A node running new code emits bundles
   that a post-processor built against old code can still parse —
   missing fields are absent, not malformed. Symmetrically, a
   post-processor built against new code reads an old bundle by
   showing the new fields as "absent" rather than erroring.

2. **Separation of lifecycle from state.** Anything that has a
   "moment it happened" is an event on the event stream. Anything
   that has a "current value" is a snapshot field. The same fact
   should not be reported both ways unless one is a counter and
   the other is a transition.

3. **Schema-version honesty.** Any version string the bundle
   carries about a dependency must reflect the dependency actually
   linked at build time. The bundle never contains a version
   string that disagrees with the lockfile.

4. **Generic over the use case.** Tier-3 capture surfaces (process,
   subprocess, host, etc.) are wired the same way as the existing
   `ProcessIntrospector`: a trait on the aggregator with a default
   production implementation and the ability to install a test
   fake without going through production paths. A new caller of
   `swactor` should be able to opt into the new surfaces with no
   knowledge of how data flows out.

5. **Boundary stays where it is today.** Generic observability
   primitives live in the distribution crate's diagnostics module.
   Role-specific decisions (which PIDs to register, which probes
   to install, which labels to use) live in the calling crate
   (`examples/pipeline-parallel-inference/...` for this codebase).

---

## 1. Relay observability (gap 1)

After this work, the bundle answers, for every relay-mediated
peer connection that died during a run:

- Who initiated the close: the relay, the remote node, or an idle
  timeout.
- What the close reason was, in a short string the relay assigned.
- How long the session had been open and how many bytes had
  crossed in each direction.
- The relay's own count of active sessions, opens, closes, and
  bytes transferred at end-of-run, broken down by close reason.

The bundle reader can answer "was this a relay-side eviction"
without consulting any external system, by reading the relay's
report and correlating it against the node-side
`connection_cache[peer].last_failure_reason` already in the
bundle.

The post-processor's summary surfaces this correlation per peer
in a "relay sessions" section. When the relay was not observed
(legacy run, relay observability not configured), the section
renders one line explaining that and pointing at this gap.

Acceptance: replay the 2026-05-25 incident with a new bundle.
The summary tells you who closed stage-2's session and why,
without further digging.

---

## 2. Relay-session vs. peer-connection separation (gap 2)

After this work, every snapshot a node emits carries an explicit
answer to "is my tunnel to my relay healthy right now," separate
from "do my peer connections through that tunnel work."

The field carries:
- The relay URL the node is currently using.
- A status (connected / connecting / disconnected / unknown).
- Wall-clock millis of the last status change and the moment the
  current status was entered.
- The last moment the node successfully sent over the tunnel and
  the last moment it received over it.
- Lifetime byte counters in each direction.

When the underlying transport library does not expose enough state
to populate the field truthfully, the snapshot must say so
explicitly: the status is `unknown`, a discriminator field
identifies the value as derived rather than reported, and the
existing `iroh_api_missing` event pattern records the gap by name.
A bundle reader must never have to guess whether `unknown` means
"the tunnel is unknown" vs. "we couldn't ask."

Acceptance: in the 2026-05-25 bundle's stage-2 snapshots, this
field reports either a real status ("disconnected" or "connected")
or `unknown` with `status_source: derived`. The investigator can
distinguish "tunnel alive but peer connection dead" from "tunnel
itself died" without speculation.

---

## 3. Per-transition relay events (gap 3)

After this work, every relay-related state flip produces an event
on the event stream, in addition to whatever counter increments.

Two kinds of flips are observable:
- **Relay session state changed**: the tunnel status field from
  section 2 moved between values. Event carries the relay URL,
  from-status, to-status, and a short reason string when one is
  available.
- **Relay home changed**: the node switched which relay it
  considers home. Event carries the from-URL and the to-URL.

Counters (e.g. `relay_home_change`) are retained for sanity-check
totals, but the per-transition event is the authoritative source.
A bundle reader can reconstruct the relay-state timeline of a
node by replaying the event stream, with no need to derive
transitions from counter deltas across snapshots.

Acceptance: in any run where a node experiences a relay flap, the
event stream contains at least one `RelaySessionStateChanged`
record. A grep for that event kind across the bundle tells you
which nodes flapped and when, with no other inputs.

---

## 4. Subprocess introspector (gap 4) — generic, through swactor

This is the largest section. The user's explicit requirement:
**the Python worker introspection must flow through swactor in a
generic way, like the existing process crate does** — meaning it
is not specific to "the Python worker" or "this example crate,"
but a reusable surface that any future user of `swactor_process`
can opt into.

### Behavior contract

After this work, every subprocess that a node owns via
`swactor_process` is reflected in the bundle on two channels,
identically to how the parent process is reflected today:

- **As snapshot state**: each periodic snapshot carries a
  per-subprocess entry with the subprocess's caller-supplied
  label, PID, parent PID, status (running / exited / unknown),
  spawn time, exit time and code/signal when applicable, RSS,
  virtual size, open FD count, CPU time, and a truncated
  command line.
- **As lifecycle events**: a `SubprocessSpawned` event fires when
  the subprocess starts, and a `SubprocessExited` event fires when
  it ends. Both carry the caller's label, the PID, the command,
  and (for exit) the exit code or terminating signal and uptime
  in millis.

Subprocess capture is a tier-3 surface alongside the existing
process-stats one. It is installed via an introspector trait on
the aggregator, with the same install pattern as today's
`ProcessIntrospector`, `HostIntrospector`, etc. A test can wire a
fake introspector without going through any production code path.

The capture surface is **stage-agnostic** and **worker-agnostic**:
it knows about a PID, a label, and a parent. The fact that "the
Python worker" is one such subprocess is a decision made at the
calling site, not in the introspector.

### Wiring contract — the swactor side

The `swactor_process` driver, when it spawns a child, must
publish the child's PID through its existing notification
channel. The data flow looks like:

1. The owning actor calls into `swactor_process` to spawn.
2. `swactor_process` reports the spawn outcome back through its
   existing notification mechanism, with the PID included.
3. The owning actor forwards "this PID, this label" into the
   subprocess introspector it owns.
4. The owning actor forwards "this PID has exited with this
   status" into the introspector on exit.

The actor's role in step 3-4 is intentionally minimal — a handful
of lines wrapping notifications it already receives. The
introspector does the actual `/proc` reading, lifecycle-event
emission, and snapshot population. A future swactor user gets
subprocess observability by installing the introspector at boot
and forwarding two notification kinds; nothing else.

### Lifecycle event coverage

The pre-existing ad-hoc `Custom { kind: "worker_starting" }` and
`Custom { kind: "worker_exited" }` strings in the example crate
are replaced by the typed `SubprocessSpawned` and
`SubprocessExited` events. The role-specific signal "the
subprocess has produced its first protocol output and is
functioning" (currently `worker_ready`) stays a `Custom` event
because functioning-as-a-pipeline-worker is not a generic
subprocess concept.

### What this gives us for the next investigation

For a stage that didn't start its worker, the bundle now tells us
unambiguously which of three things happened:

- The actor never reached its `on_start` and the subprocess was
  never asked to spawn. No `SubprocessSpawned`. The bug is in
  actor scheduling.
- The subprocess spawned and exited immediately. Both events
  present, with exit code and the existing stderr tail available.
  The bug is in the subprocess itself.
- The subprocess spawned and stayed alive but never produced
  protocol output. `SubprocessSpawned` present, no
  `SubprocessExited`, no `worker_ready` Custom event, and the
  per-snapshot RSS/CPU on the subprocess show whether it's stuck
  or thrashing. The bug is in the subprocess's startup logic
  before its first protocol line.

These three were indistinguishable in the 2026-05-25 bundle.
They are immediately distinguishable after this work.

Acceptance: in any future deployment, a stage that fails to
produce inference output can be classified into one of those
three buckets by reading the bundle alone.

---

## 5. Host metadata forwarding (gap 5)

After this work, every node's boot record carries the physical
host context the node is running on:

- Public IP of the rental.
- Datacenter id and country reported by the cloud provider.
- The provider's identifier for the rental (e.g. vast.ai instance
  id) — enough to re-rent or correlate against provider-side
  logs.
- The hostname as the container sees it.
- The relay URL the node was configured with at boot.
- The git SHA the binary was built from.
- The version string of the underlying transport library, taken
  from what is actually linked (see gap 6).

When a node runs outside the orchestrator's lease flow (e.g. a
locally-launched node for development), the cloud-provider fields
are absent rather than blank or wrong. The bundle reader can tell
"this node was not on vast.ai" from "this node was on vast.ai but
metadata wasn't forwarded" — the former leaves fields absent, the
latter is no longer a possible state.

The post-processor's summary lists each node's host context one
line per node, so "which rental was stage-2" is answerable
without grep.

Acceptance: replay the 2026-05-25 incident's recovery process.
Identifying stage-2's host requires reading one line of the
summary, not cross-referencing provider records.

---

## 6. Iroh API version sanity (gap 6)

After this work:

- The `iroh_api_missing` event reports the version of the
  transport library actually linked into the binary. The version
  string is sourced from the build, not a literal.
- Every tier-2 transport snapshot carries the same version string
  as a field, so a bundle reader does not need to scan the event
  stream to know what version the node ran.
- The list of "API gaps" — fields the bundle reader should treat
  as "we couldn't ask" rather than "we asked and got zero" —
  reflects what the linked version actually omits. Upgrading to a
  version that exposes a previously-missing field causes the gap
  to disappear from the bundle automatically; no code change is
  needed to recompute the list.

Acceptance: bumping the iroh dependency to a version that exposes
`conn_type` produces a bundle whose `api_gaps` no longer mentions
`conn_type`, without any other change.

---

## 7. Bundle assembly without finalize (gap 7)

After this work:

- A bundle is retrievable for any run that has at least one boot
  record in staging, regardless of whether the orchestrator sent
  a finalize record. `GET /diag/bundle/<run_id>` succeeds in both
  cases.
- The retrieved bundle's manifest explicitly states whether
  finalize was received. Bundle readers must not have to guess.
- When finalize was received, the bundle is the canonical one and
  serving it is cheap. When it wasn't, the bundle is synthesized
  at request time from staging files; the latency is fine because
  unfinalized bundles are by definition retrieved during incident
  response.
- Staging files for runs that never finalized are retained at
  least until the operator has had a reasonable window to
  retrieve them (default: 30 days), bounded by a hard
  disk-space cap that trims oldest-first when exceeded.

The hand-rolled recovery process used for the 2026-05-25 incident
(tar staging from the collector, scp it down, reshape, retar) is
no longer needed for any future incident, regardless of how the
orchestrator died.

Acceptance: kill an orchestrator with SIGKILL mid-run. A subsequent
`GET /diag/bundle/<run_id>` returns a usable bundle with
`finalize_received: false` in its manifest.

---

## 8. Relay-port reachability probe (gap 8)

After this work, every node periodically attempts a transport-level
reachability check against the relay's actual port, and reports
the outcome in the same snapshot probe array as the existing UDP
echo. The probe's existence does not require operator
configuration: when the node has been told a relay URL, the relay
probe is automatically registered.

The probe's outcome distinguishes:
- Reached and responded ("ok").
- Reached, no response within deadline ("timeout").
- Host reachable, port closed ("refused").
- Could not resolve target ("unresolved").
- Other error ("error").

A bundle reader can answer "could stage-2 reach the relay port at
moment T" by reading stage-2's probe array around T, without
inferring reachability from a different probe to a different port
on the same host.

Acceptance: a node placed behind a firewall that blocks the relay
port but not the existing UDP echo port produces a bundle in
which the relay probe consistently reports `refused` or `timeout`
while the UDP echo continues to report `ok`.

---

## 9. Per-peer dial rollup in summary (gap 9)

After this work, the post-processor's summary contains, per peer
in the run, a row listing:

- Total dials started against that peer.
- Total successful dials.
- Total failed dials.
- The last dial outcome (string) and its wall-clock millis.

The 3-event drift in the 2026-05-25 bundle (`DialStarted: 83`,
`DialOutcome: 80`) is attributable to specific peers in the
table; the reader can immediately tell which peers' dials never
completed.

This is a pure post-processor change — the raw events are already
in the bundle. No new fields, no new events.

Acceptance: re-run the post-processor against the existing
2026-05-25 bundle. The summary contains a per-peer dial table
that accounts for all 83 `DialStarted` events.

---

## 10. Gossip-receipt event (gap 10)

After this work, every time a node receives a payload through the
gossip / dissemination layer — name-registry update, SWIM
membership piggyback, anything similar — it emits a typed event
on its event stream. The event carries the source peer, the
payload kind (string, extensible), the payload size in bytes, and
the number of items inside.

The existing coarse `MessageReceived` counter remains for backward
compatibility, but the new event is the authoritative source for
"did node X ever hear about name Y from peer Z."

The post-processor's summary, per node, reports the total receipt
counts broken down by payload kind. "Stage-2 never received any
name-registry gossip from anyone" is a one-line answer.

Acceptance: in any run where one node fails to learn about
another node's registered name, the bundle distinguishes
unambiguously whether the gossip was never received vs. received
and ignored.

---

## 11. Kernel network counters (gap 11)

After this work, every host-scrape snapshot carries kernel-level
UDP and per-interface counters:

- UDP-side: aggregate packets in/out, drops attributable to
  no-listening-port, packets discarded due to errors, packets
  lost to socket buffer overflow.
- Per-interface: rx/tx bytes, rx/tx dropped, rx/tx errors.

A bundle reader can compute deltas across consecutive snapshots
to attribute packet loss to one of three layers:
- "Iroh sent and the OS dropped it" — UDP send error counters
  rise on the sender.
- "OS sent it and the path silently lost it" — sender counters
  clean, receiver counters clean.
- "It arrived and got dropped at the receiver's NIC" — receiver
  interface drop counters rise.

All counters are best-effort: absent on non-Linux hosts, absent
when the file can't be read, never silently zero. The
post-processor's summary surfaces any node whose UDP-drop or
interface-drop deltas are non-zero across the run window, so the
reader doesn't have to inspect every snapshot.

Acceptance: a node deliberately subjected to UDP-drop-rate
injection produces a bundle whose summary highlights it with the
correct counter rising.

---

## Sim cross-pollination

The behavioral contracts above also constrain the simulator. A
node simulated by the sim should produce snapshots and events
that conform to the same shape as a real node — the bundle reader
should not be able to tell from the data shape alone whether a
given snapshot came from a real deployment or the sim.

Three areas where today's sim lags this contract and must catch up
as part of the same upgrade:

- The sim must model a relay actor whose behavior produces the
  same tunnel-status field (gap 2) on simulated nodes. Without
  this, sim runs of cluster scenarios are not bundle-shape
  compatible with real ones.
- The sim must support installing a subprocess introspector fake
  (gap 4). Scenarios that want to model "a stage's worker never
  came up" wire this fake to produce a `SubprocessSpawned` with
  no following `worker_ready` Custom event.
- The sim's network failure model must allow "tunnel up,
  peer-connection-via-tunnel down" as a distinct failure case
  from "tunnel down." Without it the sim cannot reproduce the
  exact 2026-05-25 failure even after the observability lands.

These are sim-side work, not data-collection work, but they
share the data model defined here.

---

## Implementation order

Grouped by independence. Within a group, work is parallel-safe;
across groups, later groups don't depend on earlier groups
*finishing*, only on earlier groups' contracts being agreed.

**Group A — small, independent, unblock confidence elsewhere**
- 5 (host metadata) — small and pure-mechanical
- 6 (iroh version sanity) — small, but until it lands, every
  iroh-side field in the bundle has a credibility asterisk
- 9 (per-peer dial rollup) — pure post-processor
- 11 (kernel counters) — additive host-scrape extension

**Group B — relay tier**
- 1 (relay observability) — the largest single info gain
- 2 (relay-session field) — depends on having something to
  populate it from, ideally the work in 1
- 3 (relay events) — depends on 2's status field existing

**Group C — subprocess tier**
- 4 (subprocess introspector + events) — independent of B,
  parallel-safe with it

**Group D — collector robustness**
- 7 (bundle without finalize) — independent of all the above;
  land last to avoid churning the collector while other tiers
  are still moving

**Group E — polish**
- 8 (relay-port probe) — small, independent
- 10 (gossip-receipt event) — small, independent

The 2026-05-25 investigation would have been closeable with
A + B + C alone. D + E reduce future investigation cost but
weren't load-bearing for the failure we hit.