Zachery Aaron Shores-Chmielewski
92d62daa33
Adds the pipeline-stage host kind wrapping the production stage-supervisor lifecycle (Cold/Registering/Running/Halted per RELAY_SPEC), N3 relay calibration scenarios (own-relay and canary-relay, real-worker and stub), relay invariant and scenario-validation tests, and an adversarial judge test suite hunting spec/code gaps. Expands SIM_SPEC to a three-layer failure decomposition. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
76 lines
3.9 KiB
Markdown
76 lines
3.9 KiB
Markdown
# Hole-punching never succeeded — N3 bundle re-analysis
|
||
|
||
Date: 2026-05-22. Bundles: `vastai-N3-1`, `vastai-N3-2`, `vastai-N3-stub` from
|
||
`docean:/var/lib/swactor-diag/bundles/`.
|
||
|
||
## Finding
|
||
|
||
**No node ever held a stable direct path.** Across the 430 s N3-stub run,
|
||
every directed edge was `Relay` in every snapshot. The single non-Relay
|
||
sighting in all three bundles was stage-0→orchestrator in N3-1 going
|
||
`Relay → Mixed → Relay` over a ~24 s window, then collapsing; the reverse
|
||
direction never saw Mixed, so it was likely a transient asymmetric reading,
|
||
not a real two-way direct path. **Stage↔stage was 100% Relay in every run.**
|
||
|
||
## Suspected cause: symmetric (endpoint-dependent) NAT on vast.ai
|
||
|
||
Last-snapshot iroh socket counters, N3-stub:
|
||
|
||
| node | holepunch_attempts | paths_direct | send_ipv4 | **recv_data_ipv4** |
|
||
|--------------|-------------------:|-------------:|----------:|-------------------:|
|
||
| orchestrator | 0 | 0 | 199 072 | **0** |
|
||
| stage-0 | 3 220 | 0 | 255 282 | **0** |
|
||
| stage-1 | 1 072 | 0 | 152 515 | **0** |
|
||
| stage-2 | 3 068 | 0 | 241 752 | **0** |
|
||
|
||
Cluster-wide: ~7 360 hole-punch attempts, ~850 K direct-path datagrams sent,
|
||
**zero received**. `actor_tick_direct_addr_heartbeat = 0` everywhere (the
|
||
direct-path keepalive never ticked, because no direct path ever validated).
|
||
`portmap.upnp_available = 0`, `pcp_available = 0`, `mapping_failures ≈
|
||
mapping_attempts` on every stage — no NAT control protocol reachable inside
|
||
vast.ai containers, so iroh can't request a stable external port.
|
||
|
||
Ruled out as causes:
|
||
|
||
- **Address discovery worked.** `net_report.reports_full ≥ 1` on every node;
|
||
thousands of hole-punch attempts means stages had remote candidates from
|
||
the relay's signalling path.
|
||
- **UDP isn't blocked.** Every node's `udp_echo` probes to the VPS at
|
||
146.190.110.128:9081 had 100% reply success. Outbound delivers; return on a
|
||
pre-opened mapping delivers.
|
||
- **Relay path is fine.** All 8 `DialOutcome` events in N3-stub are `Success`
|
||
in 1.2–1.9 s. iroh just never upgrades from Relay to Direct.
|
||
|
||
The signature — outbound delivers, return-on-existing-mapping delivers,
|
||
return-on-newly-punched-port never delivers — is the classic
|
||
endpoint-dependent-mapping fingerprint. vast.ai's container egress NAT
|
||
appears to pick a different external source port per destination, so the
|
||
candidate stage-A learned about stage-B (the port B used reflecting off the
|
||
VPS) is not the port B uses sending to A.
|
||
|
||
## Implication
|
||
|
||
Between vast.ai stages, Direct is structurally unreliable on this provider.
|
||
The relay is the path, not a fallback. Provision relay bandwidth/headroom
|
||
accordingly and size SWIM timeouts around relay RTT.
|
||
|
||
## Data gaps — what would convert inference to proof
|
||
|
||
1. **`Tier2Peer.direct_addresses` in snapshots.** Today the peer object
|
||
carries only `conn_type` and `relay_urls`. Adding iroh's
|
||
`direct_addresses` list (from `Endpoint::remote_info()`) would let us see
|
||
which candidates each side learned for each peer, instead of inferring
|
||
"they had some" from `holepunch_attempts > 0`.
|
||
|
||
2. **UDP-echo source-port reflection.** Have the collector's UDP echo include
|
||
the observed `srcAddr:srcPort` in its reply (currently opaque). Probe from
|
||
each node to two collector destinations and compare external ports — same
|
||
port = endpoint-independent, different = symmetric. One number per node
|
||
would answer the NAT-type question definitively rather than by signature.
|
||
|
||
3. **Per-edge bandwidth as a packaged metric.** Today reconstructed post-hoc
|
||
by summing `MessageSent.size`/`MessageReceived.size` per `(local, peer)`.
|
||
Either ship `Tier2Peer { bytes_sent_total, bytes_received_total }` (small
|
||
per-peer accumulator on the aggregator; hooks exist in `iroh_driver.rs`),
|
||
or document the post-hoc derivation so future analyses don't re-discover
|
||
it.
|