Add actor-backed manual node provisioning, control-plane endpoints, and fleet UI assets with durable provider lifecycle handling.
Simplify Myelin orchestration, node runtime, staging, and telemetry paths while removing obsolete engine-builder, dashboard-view, and local-mock implementations.
Align runtime delivery, data-plane, distribution, job-runner, process, telemetry, dashboard, Vast.ai integrations, and their tests with the revised actor and transport contracts.
Subscribe to each worker telemetry pull endpoint during runtime readiness, retry interrupted streams, and remove the obsolete telemetry actor-address path.
Make shutdown state sticky, clean local Docker resources on Ctrl+C, validate fleet controls, and keep dashboard node state honest under failure. Wait for bidirectional iroh routes before job submission and render the orchestrator role as a card header.
Replace the single-purpose chat runtime with a persistent fleet daemon that provisions, adopts, and controls nodes through the dashboard.
Add distributed job-runner actors and provider-backed deployment so jobs can materialize workspaces, execute remotely, and return outputs over iroh.
Rename `cargo xtask provisioning-reconciler-demo` to `cargo xtask demo`
(CLI dispatch, help, child re-exec argv, launch spec strings, module dir
xtask/src/provisioning_demo -> xtask/src/demo).
Add iteration-1 data-plane edges, established from Fleet Control:
- Fleet Control "edge" button -> POST /control/edge (new
ControlCommand::EstablishEdge) -> supervisor actor resolves the node's
advertised EndpointAddr (stashed in NodeRuntime by AnnounceActor) and
provisions a real outbound EdgeRuntime (arena ring lease, recorder
WorkerPort, EDGE_ALPN send pump) in a new edge pump thread.
- Node gains EDGE_ALPN, an actor bridge decoding EdgeProvision gossip,
and a NodeEdgeAgent that provisions its (single) inbound edge, polls
it, mirrors observations onto the node.edge telemetry channel
(render-only), and answers EdgeAck gossip which terminates the
supervisor's provision retries. Node teardown replaces its inbound on
re-provision; supervisor replaces sessions per node and tears them
down on node exit/replacement/shutdown.
- The edge pump runs on the engine's blocking pool with sole session
ownership (commands in, state mirror + feed lines out): the connect
handshake blocks its thread and must not run on a Tokio worker or
share a lock with the actor. Connects are bounded (10s) so a dead
node faults its session instead of wedging edge polling.
- iroh-driver: retain_telemetry_connections() opts an application out
of the driver-owned TELEMETRY_ALPN ingress so the node's pull server
can drain those connections itself (the actor-bridge pump would
otherwise claim them).
- Dashboard: edges array in the reconciler snapshot, per-node edge
badges and edge button in Fleet Control, node_edges render mirror.
Fleet page: drop the aggregate totals row and bar skeletons — cards carry
identity, liveness, and runtime summary; hardware detail stays one click
down. Stream descriptors (origin/label) now ride FrameEvents into the
fleet view, so the orchestrator renders as a full-width amber-framed
module pinned above the grid. Cards are real links; roster rows are
keyboard-operable.
Fleet Control: /view/reconciler folds into /view/demo-control as one
control bench — ghost-segment ready/desired counter, generation digit,
unified node rows (reconciler stage + pid + state + kill), activity feeds
demoted to a collapsed tail. The standalone reconciler page is retired;
its API stays live to feed the merge.
Visual world (both themes, nav toggle, persisted, prefers-color-scheme
default): dark = Bloomberg night housing (black ground, navy panels,
amber data register); light = Atom One Light. 2px corners, monospace
data, outline chips for states, cyan as the only interactive voice,
ghost-eight segments for counters, blink reserved for unresolved states,
depressing controls, reduced-motion collapse.
Replace fleet/actor-overview/actor-dossier/workers views and the root link
list with one ControlPlaneView serving / and /view/fleet: node cards fuse
machine stats with actor rollup, node click opens machine detail + roster,
actor click opens an in-page dossier (identity, message diet, sparkline,
sampled message history via /api/view/fleet/detail).
- Message history folds view-side from messages_processed deltas: 16-receipt
ring, 250ms spacing, sampled-out counters — noisy actors cannot flood the
page and producers stay untouched.
- Stale streams render in a separate collapsed pool; superseded life
generations are evicted immediately; stale pool hard-capped at 50.
- Unified navbar injected server-side from the view registry; pages opt in
with a <!--swactor:nav--> placeholder so app-registered views appear
automatically. Hardcoded per-page navs stripped.
- Rust type names are the actor display name; address is the unique key.
Stale doc comment claiming types are not on the frame fixed.
`cargo xtask provisioning-reconciler-demo [--port n] [--nodes n]` boots a
lightweight orchestrator for visual, human-checked E2E confirmation of the
provisioning reconciler: swactor engine + real ClusterDriver + demo
provider, with node children re-exec'ing the same xtask binary in node
role and joining the supervisor over real iroh connections.
- supervisor actor owns driver/provider/shape on a 250ms wall-clock tick,
mirroring the production ClusterReconciler poll semantics; emits
prov.reconciler.events/snapshot plus per-node lifecycle/status streams
- k8s-styled reconciler view: ready/desired header, node stage cards,
commands-out and events-in feeds
- dashboard `demo-control` feature: POST /control/{kill,provision,remove}
+ Fleet Control view; regular builds compile none of it (symbol-verified)
- fleet cards fold proc.<node>.lifecycle and node.status heartbeats into
per-node pid/state pills that stay live
- hardening: exe resolution survives binary replacement by rebuilds,
spawn failures feed back as BootstrapFailed so the reconciler retries
instead of wedging at SshReady, teardown skips exit waits for
never-started children
Verified in-browser: boot 3/3 converged with real joins; dashboard kill
dips and fully recovers with a replacement; provision +1 → 4/4; remove −2
graceful teardown → 2/2; child process count matches reconciler nodes.
