Adds the pipeline-stage host kind wrapping the production stage-supervisor lifecycle
(Cold/Registering/Running/Halted per RELAY_SPEC), N3 relay calibration scenarios
(own-relay and canary-relay, real-worker and stub), relay invariant and
scenario-validation tests, and an adversarial judge test suite hunting spec/code gaps.
Expands SIM_SPEC to a three-layer failure decomposition.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Restructures the crate to the SIM_SPEC six-component model (engine, network, host,
bundle, evaluator, scenario) and relocates SIM_SPEC.md into the crate. Adds the SWIM
host adapter wrapping production distribution::swim::SwimNode, the swim codec, a
virtual network model, the scenario loader, an assertion evaluator, deterministic RNG,
property-based tests, and reproduction/smoke/topology scenario TOMLs; drops the
detector/divergence/lint/postproc/spec/runtime modules and prior spec docs.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Appends section 7.7 clarifying that the determinism contracts bind only the
simulator's own components: hidden entropy inside a wrapped production dependency
(allocator state, HashMap RandomState, process-local entropy) is out of scope, and
property tests comparing two independently-allocated wrapped state machines should
not be written.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Six per-component property-test subsections (§4.10 engine, §5.8 network,
§6.4 hosting, §8.4 scenario loader, §9.6 bundle writer, §10.5 assertion
evaluator) written in property-language form. Each names the property the
component must have; how the property is verified is the test author's
call.
These were drafted in conversation but never committed, causing the
boundary check to flag SIM_SPEC.md on iteration 1 of the implementation
loop. Committing now puts them at HEAD so future loop iterations diff
cleanly.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Upgrades iroh/iroh-relay across datastore, distribution, node, and integration,
adapts iroh_driver to the new Endpoint::builder(Minimal).relay_mode(...) API, and
removes the vendored ed25519-dalek patch now that 0.98 resolves the upstream compile
errors. Adds the SIM_SPEC.md simulator MVP spec under pipeline-parallel-inference.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Replaces the generic SimNode/gossip/dashboard framework with a virtual-time
discrete-event engine (priority queue ordered by time/node/fiber/seq), a TOML spec
parser, bundle writer, replay, divergence detector, lint, and postproc, plus the
parity-bar test harness with fixtures and xtask parity-lock tooling. Rewrites
transport identity/crypto and adds the SPEC/TESTING_SPEC/OBSERVABILITY/NORTH_STAR docs.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Structured observability for the iroh/SWIM layer: Aggregator, typed Event/Snapshot
types, Sink (NoopSink default), ProbeScheduler, process stats, and host/iroh/swim
introspection, plus the swactor-diag-collector, -postproc, and -iroh-relay binaries
that assemble and render per-run bundles. Generalizes the pipeline-parallel-inference
example to N stages and adds the topology-planner spec.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Extend the single-GPU example into a two-node pipeline-parallel run that splits llama3.2:1b across two rented vast.ai GPUs and closes the autoregressive decode loop over iroh.
- topology: add linear-chain helpers where each stage derives its neighbours locally from `STAGE`/`NUM_STAGES`, registering `pp-entry`/`pp-exit`/`pp-stage-{i}` SWIM names
- messages: add `StageActivation` (bf16 hidden-state hand-off carrying position/seq_len/is_prefill) and `NextToken` (sampled-token feedback with a `done` flag) that close the autoregressive loop between stage 0 and stage 1
- stage_actor: add `Stage0Actor` (tokenize -> embed_and_forward -> prefill activation; decode_step on each NextToken) and `Stage1Actor` (forward_and_sample -> NextToken back; emit InferenceResponse on EOS/max_tokens)
- vastai: fork the client and add `create_pipeline_instances` (rents one instance per stage, threading `STAGE`/`NUM_STAGES`, best-effort destroys on partial failure) and `destroy_all_instances`
- pp_tinygrad_worker.py: per-stage worker slicing `model.blk[start:end]` in stub and real (GGUF) modes, plus new `pp_gpu_node`/`pp_smoke_run` binaries and ROADMAP/SPEC/TEST_SPEC docs
- reuse: build on the single-GPU example's iroh transport and process bridge unchanged; add actor/codec/topology/integration test suites
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add a complete single-GPU distributed-inference example that rents a vast.ai GPU, boots a worker container, and runs a prompt end-to-end over iroh/SWIM.
- examples/single-gpu-inference: add the `single_gpu_inference` orchestrator binary that starts a local iroh node, waits for the remote gpu-node to register the `"inference"` SWIM name, then sends an `InferenceRequest` and prints the response
- examples/single-gpu-inference: add the `gpu_node` binary that joins the cluster via `SEED_ADDR`, spawns an `InferenceActor` over `tinygrad_worker.py`, and registers the `"inference"` bridge
- inference_actor: bridge swactor messaging to a Python child process via stdin/stdout JSON, with `ProcessBridge`/`RequestBridge` adapters that satisfy the single-`Incoming` actor constraint
- iroh_transport: add `IrohActorTransport` that sends `WireEnvelope`s over iroh QUIC uni-streams (connection-cached against early close), plus wire encode/decode and an inbound drain helper
- vastai: add a vast.ai REST client (`find_offer` with reliability/cuda/geo filters excluding CN, `create_instance`, `wait_for_running`, `destroy_instance`) parameterised by a mockable `base_url`
- worker/docs/tests: ship `tinygrad_worker.py`/`echo_worker.py` (newline-JSON, `--stub`/`--model` defaulting to llama3.2:1b), a Dockerfile, Makefile, SPEC, and actor/codec/cluster/integration/vastai test suites
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Using a bounded model checker to provide absolute guarantees on runtime behavior.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Remove the unused TCP transport and dead CLI/simulation scaffolding and collapse scattered single-purpose modules into consolidated files across the dashboard, process, and simulation crates.
- crates/transport: drop the TCP transport (src/tcp.rs and its tcp feature), leaving only ed25519 identity and encoding utilities behind the iroh transport
- crates/dashboard: collapse actor_detail_html/actors_html/dashboard_html/topology_html into a single html.rs, drop command/parse.rs, and inline the trace types into lib.rs
- crates/process: fold driver, pipeline_types, queue, subscriber, waker, and local pipes/signal/wait into local/mod.rs, pipeline.rs, and a unified types.rs, consolidating the public re-exports
- crates/simulation: remove the ci subdirectory (local_sim, sim) and dead node/config/trace modules, and flatten the distribution subdirectory into top-level files
- crates/datastore: remove the unused store_cli, cli, and in-memory storage, and deduplicate crypto.rs across datastore and distribution (about 190 lines of shared code removed)
- crates/distribution: drop dead codec code and trim the messages module
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Reduce idle cpu usage on my main machine from 17% to 1%. Made SWIM gossip more lazy.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Remove co-dependencies for different modules found in `crates` and migrate the development history to a new repository. The docs were stale, and largely not getting used, so simply deleted for now. When code stabilizes more, they will become useful again.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Initial experiments in 'formal verification' of authorization tooling. Current state is not to be trusted, but we are not in a security critical situation, so that is fine.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Introduce a gossip-converged pooled datastore protocol layered on SWIM piggybacking, backed by a reusable gossip-channel abstraction, plus a Docker-free multi-process cluster test runner.
- crates/distribution/src/gossip_channel.rs: add the GossipChannel trait (piggyback on SWIM messages) and a budget-limited DisseminationBuffer<T> that replaces the four duplicated Lambda*ceil(log2(n)) dissemination copies
- crates/datastore/src/pool: add PoolDisseminator (CRDT state for membership/capacity/content-location/ACL with join/leave/announce) and PoolCoordinator (placement-aware CRUD actor delegating to the co-located DatastoreNode)
- crates/shared-types/src/pool.rs: add shared pool protocol types (PoolId plus member/capacity/content-location/ACL entries and PoolConfig) consumed by both distribution and datastore
- crates/dashboard/src/pool_html.rs: add a live pool dashboard page (membership, capacity, content locations) and add pool_tests integration coverage
- xtask/src/sim_cluster.rs: add the sim-cluster runner that spawns N swactor nodes over iroh through a local relay server, reusing the docker cluster scenarios without Docker
- crates/distribution/src/iroh_driver.rs: add relay-URL resolution (cache, then SWIM gossip, then home relay) with a 2s connect timeout to back the relay-based connections
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Make distribution and deployment more stable. Consolidate the logic for a generic swactor node.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Not working too well and difficult to track, but has a skeleton there to work from.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Content addressable datastore. Allows you to configure a node to store and stream large blobs of data, and retrieve them from any swactor-connected node.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Crates continued to grow in number, but most are still quite small and feature specific. This refactor consolidates them.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Extract the timer wheel and watch registry from the worker into a reusable std-extension crate, generalize the worker around a RuntimeExtension factory, and decompose the monolithic runtime test file into focused suites.
- crates/std: extract TimerWheel (deterministic tick-counted one-shot/interval timers) and WatchRegistry (target-to-watcher death-notification index) out of src/worker.rs into reusable modules
- crates/std: add Ctx extension traits (CtxMonitoring, CtxNaming, CtxWatching, CtxTimers) and Runtime extension traits (RuntimeNaming, RuntimeWatching, RuntimeGroups) wiring monitor/name/watch/timer/group support
- src/worker.rs: replace the hard-coded timer/watch fields with a generic RuntimeExtension factory and add route_to_pool_or_remote for message routing (local pool, then cross-worker address map, then external inboxes)
- tests: split the 4622-line tests/runtime_api.rs into focused suites (actor_lifecycle, message_delivery, runtime_stress, std_extension) plus a shared tests/common/mod.rs harness, and drop watch_api.rs
- benches/fuzz: add runtime_benchmarks and adjust the runtime fuzz target
- tools/docs: add fn_complexity.py and loc_analysis.py analysis scripts and refresh the runtime and worker-thread docs
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Adds some more polish to the dashboard. Further modification will have to wait until swactor is used for more complex projects.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Harden the WASM actor host against malformed guest modules and add a property-test-backed suite covering builder validation, message handling, allocation, trap recovery, and native interop.
- crates/wasm-actor/src/actor.rs: bounds-check the guest memory write against the alloc-returned pointer (saturating_add plus OOB guard) so an out-of-bounds allocation drops the message instead of panicking the host
- crates/wasm-actor/src/actor.rs: clear the host outbox when the guest handle traps, discarding partial sends from the incomplete operation while keeping the actor alive
- crates/wasm-actor/tests/wasm_actor.rs: add a comprehensive suite covering builder validation (missing exports, invalid wasm, disabled features), echo round-trip, alloc trap/exhaustion recovery, send boundary conditions, full lifecycle, and wasm<->native interop/relay/watch scenarios
- crates/wasm-actor/tests/wasm_actor.rs: add proptest cases (builder fuzz, arbitrary-payload echo, arbitrary alloc returns, send-arg fuzz, lifecycle fuzz) and assert Send/Sync of the error types
- crates/wasm-actor/Cargo.toml: pull in proptest as a dev-dependency
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Skeleton up for a web browser swactor engine that is capable of connecting with a cluster.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Stand up a runnable distribution stack on top of the core node logic.
- distribution: add NodeDriver bridging DistributedNode to real TCP I/O
(TcpTransport/TcpAcceptor), translating NodeActions to/from wire messages;
refine swim probe timing and transport wiring.
- node: new swactor-node binary (crates/node) hosting a real node over TCP.
- tests/docker: multi-host LAN cluster harness (compose, run-lan-cluster.sh,
cluster + lan_cluster integration tests) exercising the full SWIM path.
- simulation: cluster_scenarios integration + distribution property coverage.
- docs: reorganize into distribution/, runtime/, diagrams/, connectome/; add
DOCKER_REALIZATION + SIMULATION_TESTING realization notes.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Lay down the wasm-actor host, a frontend-agnostic command layer, and a
distribution registry.
- command (new crate): CommandRouter dispatching to built-in inspection handlers
(overview/workers/actors) plus user-registered handlers, with line and
query-param parsers; built for REPL/REST/TUI/WebSocket frontends.
- wasm-actor (new crate): skeleton host — WasmActor, Builder, Engine, error
types — with echo/double/silent guest fixtures and integration tests.
- distribution: add Registry (member catalog + lookups) and Snapshot, with tests.
- core: extend the worker watch API; add watch_api integration tests.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Adds the std crate on top of the core runtime: Supervisor with RestartPolicy
(Permanent/Transient/Temporary) and SupervisorStrategy (OneForOne/OneForAll/
RestForOne), Router with RoutingStrategy, name/monitor/group registries, StdExtension,
and Ctx/Runtime extension traits. Also extends core (worker, actor, delivery identity
hashing, config, stats), adds a fuzz target, a proptest suite, expands runtime_api
tests, and adds cfuzz cycle notes + benchmarks.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Major feature addition. For full details read `./docs/development_history/DISTRIBUTION.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Runtime dashboard now features a TUI option and an interface for LLM tool use. Removed some bloat from stats collecting and replaced with a hook function to dump runtime stats into.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Make mailbox push infallible and replace the locked tick-timing buffer with a
lock-free ring.
- channel: HybridChannel::push and Sender::send now return () — overflow always
absorbs, never rejects — dropping the Result<(), T> surface and its callers.
- stats: tick_timings moves from Mutex<VecDeque> to a lock-free crossbeam
ArrayQueue (drop-oldest-on-full), removing the per-tick lock.
- ripple the signature change through worker/runtime/config; drop worker_benchmarks.
- expand runtime_api tests around the new channel/stats shapes.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add a data-parallel MNIST training example where swactor actors coordinate gradient averaging across workers.
- crates/swactor-dp-mnist/worker.py: add `MnistNet` MLP and `MnistWorker` actor handling `train_batch`/`update`/`evaluate`/`save_model` over a sharded MNIST split with SGD
- crates/swactor-dp-mnist/aggregator.py: add `Aggregator` actor that buffers per-worker gradients, averages them, fans out updates, then logs/evaluates on completion
- crates/swactor-dp-mnist/run_training.py: spawn the Aggregator plus two MnistWorkers (identical initial weights, disjoint shards), run 750 rounds, and poll the inbox for `log`/`done`
- crates/swactor-dp-mnist/pyproject.toml: declare torch/torchvision/numpy deps, an editable local `swactor` source, and the PyTorch CPU index
- Cargo.toml: add a `[profile.bench]` retaining debug symbols (`debug = true`, `strip = false`) for profiling
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add Criterion multi-threaded runtime benchmarks and a live dashboard load example.
- benches/mt_benchmarks.rs: add a Criterion suite with `mt/single_actor`, `mt/multi_actor`, `mt/ring`, and `mt/spawn` benchmarks over 2-4 threads, using `iter_custom` and a `wait_for_n_done` helper for deterministic completion
- crates/runtime-dashboard/examples/bench_dashboard.rs: add an example driving sink/ring/spawner load scenarios through `start_dashboard`/`DashboardConfig` for sustained cross-worker visualization
- Cargo.toml: register the `mt_benchmarks` Criterion bench target with `harness = false`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Trim runtime/worker/channel per coverage-fuzz findings; expand runtime_api tests; drop worker_benchmarks.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add a cargo-fuzz harness exercising the runtime under arbitrary actor spawn/send/inbox sequences in both single- and multi-threaded modes.
- fuzz/Cargo.toml: add an isolated `swactor-fuzz` package (own standalone workspace) wiring libfuzzer-sys + arbitrary against the `swactor` crate, with `fuzz_runtime` and `fuzz_mt` bin targets
- fuzz/fuzz_targets/fuzz_runtime.rs: single-threaded model target running randomized scenarios over Echo/Counter/Forwarder/Spawner/Bomber/Noop/WrongType actors with shared-inbox draining and invariant checks
- fuzz/fuzz_targets/fuzz_mt.rs: multi-threaded target driving a live `Runtime`/`RuntimeHandle` across 2-4 threads with spawn/send/burst/inbox actions and `check_invariants` (worker/actor counts) after each run
- fuzz/fuzz_targets: both targets derive a `FuzzInput`/`Action` model from `arbitrary`, cap the action count, and support opt-in run tracing via the `FUZZ_LOG` env var
- .gitignore: ignore fuzz corpus and artifact output
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Split the monolithic crate into a Cargo workspace with the Python and Wasm bindings as separate member crates.
- Cargo.toml: declare a `[workspace]` with members `.`/`crates/swactor-python`/`crates/swactor-wasm`, remove the `python` feature and pyo3 dependency, and change root crate-type from `["cdylib","rlib"]` to `["rlib"]`
- crates/swactor-python: new cdylib crate re-exporting the PyO3 bindings (Runtime/RuntimeConfig/RuntimeHandle/Inbox/Ctx/ActorAddress/RuntimeStats), depending on `swactor` + pyo3; pyproject.toml and uv.lock relocated here from the root
- crates/swactor-wasm: new cdylib crate moved from top-level `wasm/`, depending on `swactor` with `no_random` features
- src/actor.rs: widen `Actor::new`, `AnyActor`, `ContextInner`, and `Ctx::raw_inner` to `pub` so the separate binding crates can drive the runtime
- src/lib.rs: delete the in-tree `python` module and `#[pymodule]`, and gate the `no_random` RNG behind `all(feature = "no_random", not(feature = "getrandom"))`
- tools/: relocate package.json/package-lock.json; drop the now-duplicate `wasm/Cargo.lock`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Flatten the worker module, relocate its tests into a crate-level test module, and delete stale generated artifacts.
- src/worker.rs: flatten the `worker/mod.rs` directory into a single `worker.rs` module file
- src/lib.rs: wire in a new `crate_test` module for crate-level tests
- src/crate_test/mod.rs: move the worker's `tests.rs` into a crate-level `crate_test` module, updating the `Worker` import to `crate::worker`
- deps.dot, deps.html: remove checked-in dependency-graph artifacts
- spectral_report.txt: remove the stale static-analysis report
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Use spectral analysis tool to find spurious edges in the code DAG, refactoring to prune.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Asking the agent to refactor to reduce spectral complexity, it worked. Trivial change, but this did reduce code complexity.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
