Deny workspace warnings and lint suppressions, consolidate Rust and Python coverage under cargo xtask test with 60-second per-test limits, and remove stale flaky, stateful, Docker, and orphaned test artifacts.
Replace the eventfd/ring job bootstrap with one inherited arena descriptor, actor-owned sessions, sealed blob leases, awaitable inbox wakeups, and zero-copy Python mappings. Route VastAI mock provisioning through image-backed local Docker workers and preserve pinned child and controller routes across directory updates.
Add TOML job-file submission to the Fleet UI with generic started, running, and completed feedback, reusable remote job controller routing, cancellation and kill invariants, Tinygrad fixture and image support, and comprehensive Rust, Python, CUDA, and lifecycle-ordering coverage.
Add actor-backed manual node provisioning, control-plane endpoints, and fleet UI assets with durable provider lifecycle handling.
Simplify Myelin orchestration, node runtime, staging, and telemetry paths while removing obsolete engine-builder, dashboard-view, and local-mock implementations.
Align runtime delivery, data-plane, distribution, job-runner, process, telemetry, dashboard, Vast.ai integrations, and their tests with the revised actor and transport contracts.
Core no longer owns or drives OS threads. The runtime is now a single
tick-driven worker whose loop an external engine hosts and advances.
This is the cutover required before the engine seam is introduced.
Removed from core:
- Runtime::run() and its owned thread pool (spawn, park/unpark, join)
- notify_worker() and worker_threads: Vec<OnceLock<Thread>> plumbing
- Placement load-aware selector and its WorkerStats-driven next_worker()
- WorkerId newtype and the address->worker routing map; AddressMap is
now a plain AddrSet membership set
- num_threads from RuntimeConfig
Rewired for the single-worker tick API:
- python/wasm bindings, dashboard dummy node (deleted), myelin vastai
adapter, and the runtime/test suites
Cleanup folded in during review:
- prune three never-written WorkerStats counters (cross_sends,
messages_dropped, restarts)
- collapse the redundant tick_all params onto the WorkerContext handle
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add a RuntimeAdmin control plane for out-of-band inspection and mutation of live actors, bypassing their normal message handlers.
- src/admin.rs: add RuntimeAdmin + Admin<T> reply handle, the AdminCommand enum (ListActors, InspectActor, GetActorState, ReplaceActorState, StopActor, SuspendActor, ResumeActor), typed result types (ActorSummary/ActorStatus/ActorStateSnapshot/ListActorsResponse), and AdminError (ActorNotFound/AddressMismatch/TypeMismatch/Timeout)
- src/actor.rs: expose type-erased accessors on AnyActor — metadata() returning ActorTypeMetadata plus as_any()/as_any_mut() for downcasting — and Actor::inner()/replace_inner() so admin can snapshot or swap concrete actor state
- src/runtime.rs: wire a per-worker admin channel (admin_txs), expose Runtime::admin(), and implement the RuntimeAdmin ops that resolve an address to its owning worker, send the command, notify that worker, and return an Admin<T> handle; Admin::recv_ticking drives ticks and collects the reply
- src/worker.rs: add an admin_rx receiver and a drain_admin tick phase run before actor handlers, plus ActorPool helpers (actor_summary, get_actor_erased[_mut], suspend/resume/stop_actor_admin) that apply commands on the worker thread; fold admin_rx into the fast-idle has_work/tick_once checks
- tests/runtime_admin.rs: add 608 lines of end-to-end coverage for list/inspect/get/replace/stop/suspend/resume and the type-mismatch and not-found error paths
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Introduce pool/planner/launcher/runtime_stack/model/roles primitives for topology
construction and cluster launch. Drop the core guarantees module entirely; rework
worker bootstrap.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Move iroh_driver and the relay binary out of distribution into a dedicated
crates/iroh-driver (lib re-exports IrohDriver; relay bin renamed). Remove the node crate
and the single-gpu-inference example; drop the docker/datastream demo. Slim
pipeline-parallel vastai.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Remove the datastore crate, the top-level design/orchestration/ring specs, the
benches, and the ci config. Add the dashboard host telemetry sampler
(cpu/disk/net/gpu/mem). Localize the pipeline-parallel e2e stub/mock paths.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Promote pipeline-parallel-inference to a first-class app and consolidate observability on the datastream wire, decoupling the dashboard crate from `distribution`.
- apps/pipeline-parallel-inference: move the example out of `examples/` into `apps/` as its own workspace, rename binaries to `pp-worker`/`pp-orchestrator`, and strip release binaries
- cluster: add `ClusterNode`, a synchronous facade over the actorized distribution protocol (IrohDriver + per-node Runtime hosting Swim/Registry/Metadata/Directory actors with a `MembershipFanout`), replacing ad-hoc `driver.node()`/`tick()` call sites
- fleet: add per-node fleet telemetry that ships identity/resource records as `DatastreamFrame`s over the cluster transport to the orchestrator's `DatastreamSink`, folded into a `FleetView` on a 3s tick
- provision: add best-effort, opt-in SSH boot-phase telemetry (`PP_DEPLOY_KEY`) that streams rented-node boot logs onto the orchestrator's datastream as `proc.boot.<stage>.*`
- dashboard: rewire the crate dependency from `distribution` to `datastream`, drop the standalone `swactor-datastream-dashboard` binary, and rewrite `datastream_source.rs` to demux per-node frames into Overview/Distribution/Fleet views with live-node TTL filtering
- distribution: refresh dist/netmap plugin copy and README from "Kademlia routing" to gossip-directory terminology
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Strip the collector/aggregator/postproc/snapshot, vastai sampler+shipper,
host/iroh/subprocess/swim introspection, relay observability, sink/spool, and the diag
binaries; drop the t_diag_* tests. Remove DiagEvent emission from iroh_driver. Add
datastream emit/wire (mux + NoopSink/UdpFrameSink/ClusterFrameSink) and rewire the
dashboard onto datastream_source.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Upgrades iroh/iroh-relay across datastore, distribution, node, and integration,
adapts iroh_driver to the new Endpoint::builder(Minimal).relay_mode(...) API, and
removes the vendored ed25519-dalek patch now that 0.98 resolves the upstream compile
errors. Adds the SIM_SPEC.md simulator MVP spec under pipeline-parallel-inference.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Using a bounded model checker to provide absolute guarantees on runtime behavior.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Remove the unused TCP transport and dead CLI/simulation scaffolding and collapse scattered single-purpose modules into consolidated files across the dashboard, process, and simulation crates.
- crates/transport: drop the TCP transport (src/tcp.rs and its tcp feature), leaving only ed25519 identity and encoding utilities behind the iroh transport
- crates/dashboard: collapse actor_detail_html/actors_html/dashboard_html/topology_html into a single html.rs, drop command/parse.rs, and inline the trace types into lib.rs
- crates/process: fold driver, pipeline_types, queue, subscriber, waker, and local pipes/signal/wait into local/mod.rs, pipeline.rs, and a unified types.rs, consolidating the public re-exports
- crates/simulation: remove the ci subdirectory (local_sim, sim) and dead node/config/trace modules, and flatten the distribution subdirectory into top-level files
- crates/datastore: remove the unused store_cli, cli, and in-memory storage, and deduplicate crypto.rs across datastore and distribution (about 190 lines of shared code removed)
- crates/distribution: drop dead codec code and trim the messages module
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Reduce idle cpu usage on my main machine from 17% to 1%. Made SWIM gossip more lazy.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Remove co-dependencies for different modules found in `crates` and migrate the development history to a new repository. The docs were stale, and largely not getting used, so simply deleted for now. When code stabilizes more, they will become useful again.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Introduce a gossip-converged pooled datastore protocol layered on SWIM piggybacking, backed by a reusable gossip-channel abstraction, plus a Docker-free multi-process cluster test runner.
- crates/distribution/src/gossip_channel.rs: add the GossipChannel trait (piggyback on SWIM messages) and a budget-limited DisseminationBuffer<T> that replaces the four duplicated Lambda*ceil(log2(n)) dissemination copies
- crates/datastore/src/pool: add PoolDisseminator (CRDT state for membership/capacity/content-location/ACL with join/leave/announce) and PoolCoordinator (placement-aware CRUD actor delegating to the co-located DatastoreNode)
- crates/shared-types/src/pool.rs: add shared pool protocol types (PoolId plus member/capacity/content-location/ACL entries and PoolConfig) consumed by both distribution and datastore
- crates/dashboard/src/pool_html.rs: add a live pool dashboard page (membership, capacity, content locations) and add pool_tests integration coverage
- xtask/src/sim_cluster.rs: add the sim-cluster runner that spawns N swactor nodes over iroh through a local relay server, reusing the docker cluster scenarios without Docker
- crates/distribution/src/iroh_driver.rs: add relay-URL resolution (cache, then SWIM gossip, then home relay) with a 2s connect timeout to back the relay-based connections
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Make distribution and deployment more stable. Consolidate the logic for a generic swactor node.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Extract the timer wheel and watch registry from the worker into a reusable std-extension crate, generalize the worker around a RuntimeExtension factory, and decompose the monolithic runtime test file into focused suites.
- crates/std: extract TimerWheel (deterministic tick-counted one-shot/interval timers) and WatchRegistry (target-to-watcher death-notification index) out of src/worker.rs into reusable modules
- crates/std: add Ctx extension traits (CtxMonitoring, CtxNaming, CtxWatching, CtxTimers) and Runtime extension traits (RuntimeNaming, RuntimeWatching, RuntimeGroups) wiring monitor/name/watch/timer/group support
- src/worker.rs: replace the hard-coded timer/watch fields with a generic RuntimeExtension factory and add route_to_pool_or_remote for message routing (local pool, then cross-worker address map, then external inboxes)
- tests: split the 4622-line tests/runtime_api.rs into focused suites (actor_lifecycle, message_delivery, runtime_stress, std_extension) plus a shared tests/common/mod.rs harness, and drop watch_api.rs
- benches/fuzz: add runtime_benchmarks and adjust the runtime fuzz target
- tools/docs: add fn_complexity.py and loc_analysis.py analysis scripts and refresh the runtime and worker-thread docs
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Stand up a runnable distribution stack on top of the core node logic.
- distribution: add NodeDriver bridging DistributedNode to real TCP I/O
(TcpTransport/TcpAcceptor), translating NodeActions to/from wire messages;
refine swim probe timing and transport wiring.
- node: new swactor-node binary (crates/node) hosting a real node over TCP.
- tests/docker: multi-host LAN cluster harness (compose, run-lan-cluster.sh,
cluster + lan_cluster integration tests) exercising the full SWIM path.
- simulation: cluster_scenarios integration + distribution property coverage.
- docs: reorganize into distribution/, runtime/, diagrams/, connectome/; add
DOCKER_REALIZATION + SIMULATION_TESTING realization notes.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Lay down the wasm-actor host, a frontend-agnostic command layer, and a
distribution registry.
- command (new crate): CommandRouter dispatching to built-in inspection handlers
(overview/workers/actors) plus user-registered handlers, with line and
query-param parsers; built for REPL/REST/TUI/WebSocket frontends.
- wasm-actor (new crate): skeleton host — WasmActor, Builder, Engine, error
types — with echo/double/silent guest fixtures and integration tests.
- distribution: add Registry (member catalog + lookups) and Snapshot, with tests.
- core: extend the worker watch API; add watch_api integration tests.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Adds the std crate on top of the core runtime: Supervisor with RestartPolicy
(Permanent/Transient/Temporary) and SupervisorStrategy (OneForOne/OneForAll/
RestForOne), Router with RoutingStrategy, name/monitor/group registries, StdExtension,
and Ctx/Runtime extension traits. Also extends core (worker, actor, delivery identity
hashing, config, stats), adds a fuzz target, a proptest suite, expands runtime_api
tests, and adds cfuzz cycle notes + benchmarks.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Make mailbox push infallible and replace the locked tick-timing buffer with a
lock-free ring.
- channel: HybridChannel::push and Sender::send now return () — overflow always
absorbs, never rejects — dropping the Result<(), T> surface and its callers.
- stats: tick_timings moves from Mutex<VecDeque> to a lock-free crossbeam
ArrayQueue (drop-oldest-on-full), removing the per-tick lock.
- ripple the signature change through worker/runtime/config; drop worker_benchmarks.
- expand runtime_api tests around the new channel/stats shapes.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Trim runtime/worker/channel per coverage-fuzz findings; expand runtime_api tests; drop worker_benchmarks.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Refactoring to logically separate component modules in order to make it easier to develop tests, metrics, tracing, etc.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Adds some basic benchmarking, stress tests. They still need to be properly examined to ensure they are testing the correct properties, but fit for "good enough". Implements the HybridChannel type, which features a channel buffer that can withstand overflows. It does so by providing a dequeue behind a mutex. Without overflow, will push messages into the lock free ArrayQueue implemented by crossbeam_queue; when that buffer fills, will use the locking portion provided by the Mutex<VecDequeue>.
In the future we can even further optimize this, perhaps with some linked list implementations of lock-free channels, but, like the benchmarks, this fits the "good enough" bar for now.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
