Subscribe to each worker telemetry pull endpoint during runtime readiness, retry interrupted streams, and remove the obsolete telemetry actor-address path.
Make shutdown state sticky, clean local Docker resources on Ctrl+C, validate fleet controls, and keep dashboard node state honest under failure. Wait for bidirectional iroh routes before job submission and render the orchestrator role as a card header.
Replace the single-purpose chat runtime with a persistent fleet daemon that provisions, adopts, and controls nodes through the dashboard.
Add distributed job-runner actors and provider-backed deployment so jobs can materialize workspaces, execute remotely, and return outputs over iroh.
Rename `cargo xtask provisioning-reconciler-demo` to `cargo xtask demo`
(CLI dispatch, help, child re-exec argv, launch spec strings, module dir
xtask/src/provisioning_demo -> xtask/src/demo).
Add iteration-1 data-plane edges, established from Fleet Control:
- Fleet Control "edge" button -> POST /control/edge (new
ControlCommand::EstablishEdge) -> supervisor actor resolves the node's
advertised EndpointAddr (stashed in NodeRuntime by AnnounceActor) and
provisions a real outbound EdgeRuntime (arena ring lease, recorder
WorkerPort, EDGE_ALPN send pump) in a new edge pump thread.
- Node gains EDGE_ALPN, an actor bridge decoding EdgeProvision gossip,
and a NodeEdgeAgent that provisions its (single) inbound edge, polls
it, mirrors observations onto the node.edge telemetry channel
(render-only), and answers EdgeAck gossip which terminates the
supervisor's provision retries. Node teardown replaces its inbound on
re-provision; supervisor replaces sessions per node and tears them
down on node exit/replacement/shutdown.
- The edge pump runs on the engine's blocking pool with sole session
ownership (commands in, state mirror + feed lines out): the connect
handshake blocks its thread and must not run on a Tokio worker or
share a lock with the actor. Connects are bounded (10s) so a dead
node faults its session instead of wedging edge polling.
- iroh-driver: retain_telemetry_connections() opts an application out
of the driver-owned TELEMETRY_ALPN ingress so the node's pull server
can drain those connections itself (the actor-bridge pump would
otherwise claim them).
- Dashboard: edges array in the reconciler snapshot, per-node edge
badges and edge button in Fleet Control, node_edges render mirror.
Duty mixing between iroh-driver and data-plane is resolved: the transport
crate now owns only byte pumping, and the data-plane owns every edge
semantic.
data-plane:
- ids.rs: single EdgeId/RingId/StreamId/NodeId/RunId/LeaseRequestId/
ActorAddress definitions; arena, edge_lifecycle, and ring re-export them
(previously duplicated per module)
- edge_wire.rs: the whole transport contract — WireEvent, EdgeWriter, and
the EdgeTransport port (associated Writer/PeerAddr types)
- edge_runtime.rs: EdgeRuntime composition engine absorbing iroh-driver's
driver_pumps bookkeeping, the EdgeEstablisher lifecycle drive, arena
leasing, ingress stream buffering with object-record parsing, and ring
writes; effects go through a WorkerPort trait; progress surfaces as
structured Observations the application maps to telemetry/agent messages
- delete superseded test-only layers: actor.rs (DataPlaneNodeActor),
edge_actor.rs, ingress.rs, egress.rs and their guarantee tests
- fold ObjectIdAllocator into object_record (now edge-free, starts at 1)
iroh-driver:
- edge_transport speaks pure data_plane::edge_wire vocabulary; EdgeSendHandle
implements EdgeWriter; IrohDriver implements EdgeTransport (PeerAddr =
EndpointAddr) — the entire edge surface is open_writer + drain_events
- delete driver_pumps.rs; new dependency on data-plane (no cycle)
- IROH_DRIVER_SPEC §5 updated for the new module set and edge boundary
myelin:
- WorkerEdgeRuntime shrinks from ~830 lines of hand glue to an EdgeRuntime
holder plus a tinygrad WorkerPort impl and observation reporting; the
driver-event/edge-event translation layers and newtype re-wrapping are
gone
- orchestration/app.rs and job edge drains consume WireEvent
Tests: data-plane 31 (5 new EdgeRuntime contract tests), iroh-driver 13,
myelin 65 — all green.
Replace the per-attempt key-file side channel (DEMO_NODE_KEY_FILE,
read_key_report, JoinCheck) with a control-plane announce: node roles
send a tagged gossip frame {attempt, logical_node, key_hex,
endpoint_addr_json} after joining and every heartbeat thereafter.
- iroh-driver: IrohDriver::send_tagged_gossip — fire-and-forget
tag-routed gossip egress for bridge-less clients (reuses cached/join
connections, dials with backoff).
- provisioning: BootstrapMsg::Announce — first delivery while
bootstrapping completes the attempt (collector + exactly-once
Bootstrapped report); duplicates, misrouted attempts, and
terminal-phase announces drop. Unit-tested.
- xtask demo: AnnounceActor decodes the tag-routed frame and forwards
by attempt to the owning bootstrap actor; last_announce_ms is the
wire heartbeat. LocalProcessLogic keeps only process lifecycle.
- --docker: DockerProcessLogic (kind "docker") — attached
"docker run --rm" child on a per-run labeled bridge network
(foreign-node masking: per-container IPs, gateway-dialed
supervisor). Standalone scratch image from the static-musl xtask
binary (37MB), staged one-file build context. The container is
force-removed on every terminal path so a SIGKILLed docker CLI
cannot orphan a running container.
- Cleanup: no volumes/mounts; label-filtered exit sweep (SIGINT and
SIGTERM both drain first) + startup sweep of stale demo resources;
images persist per run token.
Verified live: process kind (kill -> replacement in 3.4s, provision/
remove/kill waves, zero orphans) and docker kind (8-node abuse waves
across docker kill, mid-provision control kills, CLI SIGKILL orphans
force-removed, SIGKILL-crash leftovers swept on restart, clean exits
leave zero containers/networks/CLIs). provisioning 22 + iroh-driver
13 tests pass.
The crate is the per-node metrics/logging pipe with a universal
subscriber endpoint, but "datastream" kept getting misread as a general
messaging plane. Rename crate, module paths, and public API
(`DatastreamEndpoint` → `TelemetryEndpoint`, etc.) so misuse is visible
on sight.
Renamed contracts (all in-repo producers/consumers migrated):
- env vars `MYELIN_DATASTREAM_*` → `MYELIN_TELEMETRY_*`
- artifact `datastream.ndjson` → `telemetry.ndjson`
- actor names `telemetry-publisher` / `telemetry-sink`
- wire ALPN `swactor/telemetry/0`
- `DATASTREAM_SPEC.md` → `TELEMETRY_SPEC.md`
Also fixes two latent test breaks: `process` and `iroh-driver` tests
imported `DatastreamEvent` from the crate root, which was never
re-exported; they now use the observer path `telemetry::frame::`.
The datastream is metrics/logging only; control decisions must never branch
on a frame. This was a recurring cultural problem with no structural
enforcement. This change makes it a compile-time and CI-enforced fact.
datastream crate (lib.rs):
- Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root.
is now a compile error (E0425). These types live
only in datastream::frame::* and are documented as the observer surface.
- Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain
re-exported at root for producer-side callers.
orchestration/app.rs:
- Extracted all frame-touching code (CollectedDatastreamFrame,
drain_datastream_connections, update_load_progress_from_frame,
drain_frames, archive_collected_frame, pump, OrchDatastream,
DashboardSupport) into two new observability modules:
frame_collector.rs and orch_datastream.rs.
- The orchestrator now interacts through a FrameCollector whose
drain/drain_with_progress methods take closures; it never names Frame,
DatastreamEvent, or CollectedDatastreamFrame.
- StageLoadProgress (the one control-relevant signal previously scraped
from frame payloads) is extracted inside FrameCollector and handed to
the control loop as plain data.
xtask:
- New check-telemetry-isolation command scans control-plane modules
(orchestration/, distribution/, data-plane/, provisioning/) for
forbidden frame-type references and fails the build if any are found.
Verified: workspace builds (myelin + dashboard feature), datastream 29
tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes
clean.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
Specs gain a monotonic Id and relocate by their true status: drafts
(WIP/aspirational) to docs/specs/drafts/, and accurate code-behavior
references stay in their crate dirs. docs/specs/archive/ is reserved for
superseded docs (currently empty).
Dispositions were cross-referenced against code, not the specs' own headers.
IROH_DRIVER claimed "current-state" but ~30% is unbuilt redesign, so it moves
to drafts. DATA_PLANE_ACTOR's central integration claim is unrealized (myelin
bypasses its node actor), and ACTOR_PANEL is not a reference; both are dropped
rather than reviewed or archived. DATASTREAM and MANAGED_PROCESS stay as
references; MYELIN stays in place (stale, flagged for review).
From here, commit titles reference a spec by [N] when one applies. This
bootstrap commit does not carry one.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add project licensing and a full README, plus a minimal WebAssembly ping-pong example that exercises the actor runtime.
- `LICENSE`: add the full GNU AGPL-3.0 text and set `license = "AGPL-3.0-only"` on every package (`swactor`, each crate, `apps/myelin`, `tools/vastai`, `xtask`)
- `README.md`: rewrite from a stub into a full project overview, covering features (actor_id routing, WASM, iroh QUIC/SWIM, OTP-style std, process manager, zero-copy objects, datastream metrics, dashboard), architecture, examples, developing, status, and license
- `examples/ping-pong`: new standalone workspace (`pingpong` cdylib) where two actors volley on a single-threaded `wasm` runtime driven by a Node host via `tick()`, demonstrating spawning, message passing, and death monitoring (`watching`)
- `examples/ping-pong`: add a host/run harness -- `run.sh` (wasm-pack build + `run.mjs`), `serve.sh` (dashboard + static demo), `index.html`, and a pinned `Cargo.lock`
- `.gitignore`: stop ignoring `.loop/`, `.deployment-notes/`, and `.omp/`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add read-only actor overview and per-actor dossier pages to the dashboard, fed by enriched per-actor runtime snapshots.
- `dashboard/swactor/actor_view`: new `ActorPanelView`, a tolerant frame consumer over `runtime.actors`/`runtime.stats` that folds per-actor snapshots and serves `/view/swactor/actor-overview` (roster) and `/view/swactor/actor-dossier` (per-actor detail), each backed by an embedded HTML template (`actor_overview.html`, `actor_dossier.html`)
- `dashboard`: register both views in `DashboardHandle` and export `actor_overview_view()`/`actor_dossier_view()` from the swactor module
- `swactor` core: enrich `ActorSnapshot` with `actor_type` and `message_type` (populated from `slot.actor.metadata()` in `ActorPool`) and add `ActorAddress::to_full_hex()` for untruncated display
- `myelin/orchestration`: publish actor stats to the dashboard via a `runtime.actors` channel producer (`stats_hook_on`) threaded through the distribution stack, and carry the orchestrator actor address into readiness signaling
- workspace `Cargo.toml`: add `default-members` for native iteration and a centralized `[workspace.dependencies] tokio` so members share one feature set; `dashboard/Cargo.toml` switches to `tokio.workspace = true`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Consolidate the duplicated JSON codec into `transport` and relocate the iroh edge-transport pieces into `iroh-driver`, dissolving the `mvp-system` transport shim.
- `transport`: add a canonical `json_codec::JsonCodec<M>` (re-exported from the crate root) as the single JSON codec for serde message types
- `distribution`/`datastream`: drop the per-crate `JsonCodec` copies and the `impl_json_codec!` macro; register SWIM/gossip and publisher messages against the shared `swactor_transport::JsonCodec`
- `iroh-driver`: move `driver_pumps` and `endpoint_advertisement` out of `mvp-system/src/transport/`, re-exporting `EndpointAddrMask`/`advertised_endpoint`/`MVP_IROH_ENDPOINT_ADDR_MASK_ENV`, and relocate the endpoint guarantee test to `iroh-driver/tests/endpoint_advertisement.rs`
- `mvp-system`: delete the `transport/` module and keep codec aggregation in a new `codecs.rs` (`register_mvp_actor_codecs`)
- `mvp-system/node`: shrink `worker_node_runtime.rs` (~260 lines) by adopting the relocated modules and collapsing verbose `emit_stdio_node_event` calls into local `boot()`/`worker_evt()` closures
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add the iroh-driver spec; trim and restructure the mvp-system orchestrator and
mvp-chat specs. Reframe prompt RPC as the prompt-engine contract in mvp_chat.md; add an
iroh-driver dependency; minor datastream_transport cleanup.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Move iroh_driver and the relay binary out of distribution into a dedicated
crates/iroh-driver (lib re-exports IrohDriver; relay bin renamed). Remove the node crate
and the single-gpu-inference example; drop the docker/datastream demo. Slim
pipeline-parallel vastai.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
