zacheryasc/swactor
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
swactor/crates/simulation/HOLE_PUNCH_FINDINGS.md
Zachery Aaron Shores-Chmielewski 92d62daa33 feat(sim): add stage host kind and relay scenarios 
Adds the pipeline-stage host kind wrapping the production stage-supervisor lifecycle
(Cold/Registering/Running/Halted per RELAY_SPEC), N3 relay calibration scenarios
(own-relay and canary-relay, real-worker and stub), relay invariant and
scenario-validation tests, and an adversarial judge test suite hunting spec/code gaps.
Expands SIM_SPEC to a three-layer failure decomposition.


Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-05-25 14:33:35 +04:00

3.9 KiB
Raw Blame History

Hole-punching never succeeded — N3 bundle re-analysis

Date: 2026-05-22. Bundles: vastai-N3-1, vastai-N3-2, vastai-N3-stub from docean:/var/lib/swactor-diag/bundles/.

Finding

No node ever held a stable direct path. Across the 430 s N3-stub run, every directed edge was Relay in every snapshot. The single non-Relay sighting in all three bundles was stage-0→orchestrator in N3-1 going Relay → Mixed → Relay over a ~24 s window, then collapsing; the reverse direction never saw Mixed, so it was likely a transient asymmetric reading, not a real two-way direct path. Stage↔stage was 100% Relay in every run.

Suspected cause: symmetric (endpoint-dependent) NAT on vast.ai

Last-snapshot iroh socket counters, N3-stub:

node holepunch_attempts paths_direct send_ipv4 recv_data_ipv4
orchestrator 0 0 199 072 0
stage-0 3 220 0 255 282 0
stage-1 1 072 0 152 515 0
stage-2 3 068 0 241 752 0

Cluster-wide: ~7 360 hole-punch attempts, ~850 K direct-path datagrams sent, zero received. actor_tick_direct_addr_heartbeat = 0 everywhere (the direct-path keepalive never ticked, because no direct path ever validated). portmap.upnp_available = 0, pcp_available = 0, mapping_failures ≈ mapping_attempts on every stage — no NAT control protocol reachable inside vast.ai containers, so iroh can't request a stable external port.

Ruled out as causes:

  • Address discovery worked. net_report.reports_full ≥ 1 on every node; thousands of hole-punch attempts means stages had remote candidates from the relay's signalling path.
  • UDP isn't blocked. Every node's udp_echo probes to the VPS at 146.190.110.128:9081 had 100% reply success. Outbound delivers; return on a pre-opened mapping delivers.
  • Relay path is fine. All 8 DialOutcome events in N3-stub are Success in 1.2–1.9 s. iroh just never upgrades from Relay to Direct.

The signature — outbound delivers, return-on-existing-mapping delivers, return-on-newly-punched-port never delivers — is the classic endpoint-dependent-mapping fingerprint. vast.ai's container egress NAT appears to pick a different external source port per destination, so the candidate stage-A learned about stage-B (the port B used reflecting off the VPS) is not the port B uses sending to A.

Implication

Between vast.ai stages, Direct is structurally unreliable on this provider. The relay is the path, not a fallback. Provision relay bandwidth/headroom accordingly and size SWIM timeouts around relay RTT.

Data gaps — what would convert inference to proof

  1. Tier2Peer.direct_addresses in snapshots. Today the peer object carries only conn_type and relay_urls. Adding iroh's direct_addresses list (from Endpoint::remote_info()) would let us see which candidates each side learned for each peer, instead of inferring "they had some" from holepunch_attempts > 0.

  2. UDP-echo source-port reflection. Have the collector's UDP echo include the observed srcAddr:srcPort in its reply (currently opaque). Probe from each node to two collector destinations and compare external ports — same port = endpoint-independent, different = symmetric. One number per node would answer the NAT-type question definitively rather than by signature.

  3. Per-edge bandwidth as a packaged metric. Today reconstructed post-hoc by summing MessageSent.size/MessageReceived.size per (local, peer). Either ship Tier2Peer { bytes_sent_total, bytes_received_total } (small per-peer accumulator on the aggregator; hooks exist in iroh_driver.rs), or document the post-hoc derivation so future analyses don't re-discover it.