Refine fleet and provisioning interactions, expand control-plane projections, and add CPU, memory, pressure, and storage telemetry for the local dashboard surfaces.
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
Add actor-backed manual node provisioning, control-plane endpoints, and fleet UI assets with durable provider lifecycle handling.
Simplify Myelin orchestration, node runtime, staging, and telemetry paths while removing obsolete engine-builder, dashboard-view, and local-mock implementations.
Align runtime delivery, data-plane, distribution, job-runner, process, telemetry, dashboard, Vast.ai integrations, and their tests with the revised actor and transport contracts.
Replace the single-purpose chat runtime with a persistent fleet daemon that provisions, adopts, and controls nodes through the dashboard.
Add distributed job-runner actors and provider-backed deployment so jobs can materialize workspaces, execute remotely, and return outputs over iroh.
Replace fleet/actor-overview/actor-dossier/workers views and the root link
list with one ControlPlaneView serving / and /view/fleet: node cards fuse
machine stats with actor rollup, node click opens machine detail + roster,
actor click opens an in-page dossier (identity, message diet, sparkline,
sampled message history via /api/view/fleet/detail).
- Message history folds view-side from messages_processed deltas: 16-receipt
ring, 250ms spacing, sampled-out counters — noisy actors cannot flood the
page and producers stay untouched.
- Stale streams render in a separate collapsed pool; superseded life
generations are evicted immediately; stale pool hard-capped at 50.
- Unified navbar injected server-side from the view registry; pages opt in
with a <!--swactor:nav--> placeholder so app-registered views appear
automatically. Hardcoded per-page navs stripped.
- Rust type names are the actor display name; address is the unique key.
Stale doc comment claiming types are not on the frame fixed.
The crate is the per-node metrics/logging pipe with a universal
subscriber endpoint, but "datastream" kept getting misread as a general
messaging plane. Rename crate, module paths, and public API
(`DatastreamEndpoint` → `TelemetryEndpoint`, etc.) so misuse is visible
on sight.
Renamed contracts (all in-repo producers/consumers migrated):
- env vars `MYELIN_DATASTREAM_*` → `MYELIN_TELEMETRY_*`
- artifact `datastream.ndjson` → `telemetry.ndjson`
- actor names `telemetry-publisher` / `telemetry-sink`
- wire ALPN `swactor/telemetry/0`
- `DATASTREAM_SPEC.md` → `TELEMETRY_SPEC.md`
Also fixes two latent test breaks: `process` and `iroh-driver` tests
imported `DatastreamEvent` from the crate root, which was never
re-exported; they now use the observer path `telemetry::frame::`.
The datastream is metrics/logging only; control decisions must never branch
on a frame. This was a recurring cultural problem with no structural
enforcement. This change makes it a compile-time and CI-enforced fact.
datastream crate (lib.rs):
- Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root.
is now a compile error (E0425). These types live
only in datastream::frame::* and are documented as the observer surface.
- Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain
re-exported at root for producer-side callers.
orchestration/app.rs:
- Extracted all frame-touching code (CollectedDatastreamFrame,
drain_datastream_connections, update_load_progress_from_frame,
drain_frames, archive_collected_frame, pump, OrchDatastream,
DashboardSupport) into two new observability modules:
frame_collector.rs and orch_datastream.rs.
- The orchestrator now interacts through a FrameCollector whose
drain/drain_with_progress methods take closures; it never names Frame,
DatastreamEvent, or CollectedDatastreamFrame.
- StageLoadProgress (the one control-relevant signal previously scraped
from frame payloads) is extracted inside FrameCollector and handed to
the control loop as plain data.
xtask:
- New check-telemetry-isolation command scans control-plane modules
(orchestration/, distribution/, data-plane/, provisioning/) for
forbidden frame-type references and fails the build if any are found.
Verified: workspace builds (myelin + dashboard feature), datastream 29
tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes
clean.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Introduce a pure, level-triggered reconciler in `crates/provisioning`
that drives a declared cluster shape toward convergence over the
existing node lifecycle, replacing the edge-triggered imperative node
orchestration in `apps/myelin`.
- `reconcile`/`reconcile_node`/`observe`: pure decider and observation
folder with stable logical-node identity, per-attempt operation
identity, and deterministic retry backoff; `ClusterDriver` is the sole
writer of observed state, coalescing triggers, recording operations as
pending before dispatch, and scheduling timed requeues.
- `IdempotentEffectExecutor`: deduplicates submissions by
`(run_id, logical_node_id, attempt)` and runs provider work on the
engine-hosted blocking substrate, never blocking a reconcile pass.
- Myelin integration: `MyelinEffectBackend` bridges `ProvisionPlugin` to
the executor contract; `LocalProcessPlugin`/`LocalDockerPlugin`
provider adapters; `ProvisionedClusterGuard` pumps triggers,
observations, and due operations.
- Retire the imperative acquire/bootstrap/teardown sequencing across
`apps/myelin` orchestration, staging, observability, and provider
adapters in favor of the declarative driver.
- Move the reconciler specification to `docs/specs/archive`.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
