77 lines
3.9 KiB
Markdown
77 lines
3.9 KiB
Markdown
|
# Hole-punching never succeeded — N3 bundle re-analysis
|
|||
|
|
|
|||
|
|
Date: 2026-05-22. Bundles: `vastai-N3-1`, `vastai-N3-2`, `vastai-N3-stub` from
|
|||
|
|
`docean:/var/lib/swactor-diag/bundles/`.
|
|||
|
|
|
|||
|
|
## Finding
|
|||
|
|
|
|||
|
|
**No node ever held a stable direct path.** Across the 430 s N3-stub run,
|
|||
|
|
every directed edge was `Relay` in every snapshot. The single non-Relay
|
|||
|
|
sighting in all three bundles was stage-0→orchestrator in N3-1 going
|
|||
|
|
`Relay → Mixed → Relay` over a ~24 s window, then collapsing; the reverse
|
|||
|
|
direction never saw Mixed, so it was likely a transient asymmetric reading,
|
|||
|
|
not a real two-way direct path. **Stage↔stage was 100% Relay in every run.**
|
|||
|
|
|
|||
|
|
## Suspected cause: symmetric (endpoint-dependent) NAT on vast.ai
|
|||
|
|
|
|||
|
|
Last-snapshot iroh socket counters, N3-stub:
|
|||
|
|
|
|||
|
|
| node | holepunch_attempts | paths_direct | send_ipv4 | **recv_data_ipv4** |
|
|||
|
|
|--------------|-------------------:|-------------:|----------:|-------------------:|
|
|||
|
|
| orchestrator | 0 | 0 | 199 072 | **0** |
|
|||
|
|
| stage-0 | 3 220 | 0 | 255 282 | **0** |
|
|||
|
|
| stage-1 | 1 072 | 0 | 152 515 | **0** |
|
|||
|
|
| stage-2 | 3 068 | 0 | 241 752 | **0** |
|
|||
|
|
|
|||
|
|
Cluster-wide: ~7 360 hole-punch attempts, ~850 K direct-path datagrams sent,
|
|||
|
|
**zero received**. `actor_tick_direct_addr_heartbeat = 0` everywhere (the
|
|||
|
|
direct-path keepalive never ticked, because no direct path ever validated).
|
|||
|
|
`portmap.upnp_available = 0`, `pcp_available = 0`, `mapping_failures ≈
|
|||
|
|
mapping_attempts` on every stage — no NAT control protocol reachable inside
|
|||
|
|
vast.ai containers, so iroh can't request a stable external port.
|
|||
|
|
|
|||
|
|
Ruled out as causes:
|
|||
|
|
|
|||
|
|
- **Address discovery worked.** `net_report.reports_full ≥ 1` on every node;
|
|||
|
|
thousands of hole-punch attempts means stages had remote candidates from
|
|||
|
|
the relay's signalling path.
|
|||
|
|
- **UDP isn't blocked.** Every node's `udp_echo` probes to the VPS at
|
|||
|
|
146.190.110.128:9081 had 100% reply success. Outbound delivers; return on a
|
|||
|
|
pre-opened mapping delivers.
|
|||
|
|
- **Relay path is fine.** All 8 `DialOutcome` events in N3-stub are `Success`
|
|||
|
|
in 1.2–1.9 s. iroh just never upgrades from Relay to Direct.
|
|||
|
|
|
|||
|
|
The signature — outbound delivers, return-on-existing-mapping delivers,
|
|||
|
|
return-on-newly-punched-port never delivers — is the classic
|
|||
|
|
endpoint-dependent-mapping fingerprint. vast.ai's container egress NAT
|
|||
|
|
appears to pick a different external source port per destination, so the
|
|||
|
|
candidate stage-A learned about stage-B (the port B used reflecting off the
|
|||
|
|
VPS) is not the port B uses sending to A.
|
|||
|
|
|
|||
|
|
## Implication
|
|||
|
|
|
|||
|
|
Between vast.ai stages, Direct is structurally unreliable on this provider.
|
|||
|
|
The relay is the path, not a fallback. Provision relay bandwidth/headroom
|
|||
|
|
accordingly and size SWIM timeouts around relay RTT.
|
|||
|
|
|
|||
|
|
## Data gaps — what would convert inference to proof
|
|||
|
|
|
|||
|
|
1. **`Tier2Peer.direct_addresses` in snapshots.** Today the peer object
|
|||
|
|
carries only `conn_type` and `relay_urls`. Adding iroh's
|
|||
|
|
`direct_addresses` list (from `Endpoint::remote_info()`) would let us see
|
|||
|
|
which candidates each side learned for each peer, instead of inferring
|
|||
|
|
"they had some" from `holepunch_attempts > 0`.
|
|||
|
|
|
|||
|
|
2. **UDP-echo source-port reflection.** Have the collector's UDP echo include
|
|||
|
|
the observed `srcAddr:srcPort` in its reply (currently opaque). Probe from
|
|||
|
|
each node to two collector destinations and compare external ports — same
|
|||
|
|
port = endpoint-independent, different = symmetric. One number per node
|
|||
|
|
would answer the NAT-type question definitively rather than by signature.
|
|||
|
|
|
|||
|
|
3. **Per-edge bandwidth as a packaged metric.** Today reconstructed post-hoc
|
|||
|
|
by summing `MessageSent.size`/`MessageReceived.size` per `(local, peer)`.
|
|||
|
|
Either ship `Tier2Peer { bytes_sent_total, bytes_received_total }` (small
|
|||
|
|
per-peer accumulator on the aggregator; hooks exist in `iroh_driver.rs`),
|
|||
|
|
or document the post-hoc derivation so future analyses don't re-discover
|
|||
|
|
it.
|