Replace the single-purpose chat runtime with a persistent fleet daemon that provisions, adopts, and controls nodes through the dashboard.
Add distributed job-runner actors and provider-backed deployment so jobs can materialize workspaces, execute remotely, and return outputs over iroh.
Rename `cargo xtask provisioning-reconciler-demo` to `cargo xtask demo`
(CLI dispatch, help, child re-exec argv, launch spec strings, module dir
xtask/src/provisioning_demo -> xtask/src/demo).
Add iteration-1 data-plane edges, established from Fleet Control:
- Fleet Control "edge" button -> POST /control/edge (new
ControlCommand::EstablishEdge) -> supervisor actor resolves the node's
advertised EndpointAddr (stashed in NodeRuntime by AnnounceActor) and
provisions a real outbound EdgeRuntime (arena ring lease, recorder
WorkerPort, EDGE_ALPN send pump) in a new edge pump thread.
- Node gains EDGE_ALPN, an actor bridge decoding EdgeProvision gossip,
and a NodeEdgeAgent that provisions its (single) inbound edge, polls
it, mirrors observations onto the node.edge telemetry channel
(render-only), and answers EdgeAck gossip which terminates the
supervisor's provision retries. Node teardown replaces its inbound on
re-provision; supervisor replaces sessions per node and tears them
down on node exit/replacement/shutdown.
- The edge pump runs on the engine's blocking pool with sole session
ownership (commands in, state mirror + feed lines out): the connect
handshake blocks its thread and must not run on a Tokio worker or
share a lock with the actor. Connects are bounded (10s) so a dead
node faults its session instead of wedging edge polling.
- iroh-driver: retain_telemetry_connections() opts an application out
of the driver-owned TELEMETRY_ALPN ingress so the node's pull server
can drain those connections itself (the actor-bridge pump would
otherwise claim them).
- Dashboard: edges array in the reconciler snapshot, per-node edge
badges and edge button in Fleet Control, node_edges render mirror.
`cargo xtask provisioning-reconciler-demo [--port n] [--nodes n]` boots a
lightweight orchestrator for visual, human-checked E2E confirmation of the
provisioning reconciler: swactor engine + real ClusterDriver + demo
provider, with node children re-exec'ing the same xtask binary in node
role and joining the supervisor over real iroh connections.
- supervisor actor owns driver/provider/shape on a 250ms wall-clock tick,
mirroring the production ClusterReconciler poll semantics; emits
prov.reconciler.events/snapshot plus per-node lifecycle/status streams
- k8s-styled reconciler view: ready/desired header, node stage cards,
commands-out and events-in feeds
- dashboard `demo-control` feature: POST /control/{kill,provision,remove}
+ Fleet Control view; regular builds compile none of it (symbol-verified)
- fleet cards fold proc.<node>.lifecycle and node.status heartbeats into
per-node pid/state pills that stay live
- hardening: exe resolution survives binary replacement by rebuilds,
spawn failures feed back as BootstrapFailed so the reconciler retries
instead of wedging at SshReady, teardown skips exit waits for
never-started children
Verified in-browser: boot 3/3 converged with real joins; dashboard kill
dips and fully recovers with a replacement; provision +1 → 4/4; remove −2
graceful teardown → 2/2; child process count matches reconciler nodes.
