Remove the datastore crate, the top-level design/orchestration/ring specs, the
benches, and the ci config. Add the dashboard host telemetry sampler
(cpu/disk/net/gpu/mem). Localize the pipeline-parallel e2e stub/mock paths.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Promote pipeline-parallel-inference to a first-class app and consolidate observability on the datastream wire, decoupling the dashboard crate from `distribution`.
- apps/pipeline-parallel-inference: move the example out of `examples/` into `apps/` as its own workspace, rename binaries to `pp-worker`/`pp-orchestrator`, and strip release binaries
- cluster: add `ClusterNode`, a synchronous facade over the actorized distribution protocol (IrohDriver + per-node Runtime hosting Swim/Registry/Metadata/Directory actors with a `MembershipFanout`), replacing ad-hoc `driver.node()`/`tick()` call sites
- fleet: add per-node fleet telemetry that ships identity/resource records as `DatastreamFrame`s over the cluster transport to the orchestrator's `DatastreamSink`, folded into a `FleetView` on a 3s tick
- provision: add best-effort, opt-in SSH boot-phase telemetry (`PP_DEPLOY_KEY`) that streams rented-node boot logs onto the orchestrator's datastream as `proc.boot.<stage>.*`
- dashboard: rewire the crate dependency from `distribution` to `datastream`, drop the standalone `swactor-datastream-dashboard` binary, and rewrite `datastream_source.rs` to demux per-node frames into Overview/Distribution/Fleet views with live-node TTL filtering
- distribution: refresh dist/netmap plugin copy and README from "Kademlia routing" to gossip-directory terminology
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Strip the collector/aggregator/postproc/snapshot, vastai sampler+shipper,
host/iroh/subprocess/swim introspection, relay observability, sink/spool, and the diag
binaries; drop the t_diag_* tests. Remove DiagEvent emission from iroh_driver. Add
datastream emit/wire (mux + NoopSink/UdpFrameSink/ClusterFrameSink) and rewire the
dashboard onto datastream_source.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Remove the unused TCP transport and dead CLI/simulation scaffolding and collapse scattered single-purpose modules into consolidated files across the dashboard, process, and simulation crates.
- crates/transport: drop the TCP transport (src/tcp.rs and its tcp feature), leaving only ed25519 identity and encoding utilities behind the iroh transport
- crates/dashboard: collapse actor_detail_html/actors_html/dashboard_html/topology_html into a single html.rs, drop command/parse.rs, and inline the trace types into lib.rs
- crates/process: fold driver, pipeline_types, queue, subscriber, waker, and local pipes/signal/wait into local/mod.rs, pipeline.rs, and a unified types.rs, consolidating the public re-exports
- crates/simulation: remove the ci subdirectory (local_sim, sim) and dead node/config/trace modules, and flatten the distribution subdirectory into top-level files
- crates/datastore: remove the unused store_cli, cli, and in-memory storage, and deduplicate crypto.rs across datastore and distribution (about 190 lines of shared code removed)
- crates/distribution: drop dead codec code and trim the messages module
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Introduce a gossip-converged pooled datastore protocol layered on SWIM piggybacking, backed by a reusable gossip-channel abstraction, plus a Docker-free multi-process cluster test runner.
- crates/distribution/src/gossip_channel.rs: add the GossipChannel trait (piggyback on SWIM messages) and a budget-limited DisseminationBuffer<T> that replaces the four duplicated Lambda*ceil(log2(n)) dissemination copies
- crates/datastore/src/pool: add PoolDisseminator (CRDT state for membership/capacity/content-location/ACL with join/leave/announce) and PoolCoordinator (placement-aware CRUD actor delegating to the co-located DatastoreNode)
- crates/shared-types/src/pool.rs: add shared pool protocol types (PoolId plus member/capacity/content-location/ACL entries and PoolConfig) consumed by both distribution and datastore
- crates/dashboard/src/pool_html.rs: add a live pool dashboard page (membership, capacity, content locations) and add pool_tests integration coverage
- xtask/src/sim_cluster.rs: add the sim-cluster runner that spawns N swactor nodes over iroh through a local relay server, reusing the docker cluster scenarios without Docker
- crates/distribution/src/iroh_driver.rs: add relay-URL resolution (cache, then SWIM gossip, then home relay) with a 2s connect timeout to back the relay-based connections
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Make distribution and deployment more stable. Consolidate the logic for a generic swactor node.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Stand up a runnable distribution stack on top of the core node logic.
- distribution: add NodeDriver bridging DistributedNode to real TCP I/O
(TcpTransport/TcpAcceptor), translating NodeActions to/from wire messages;
refine swim probe timing and transport wiring.
- node: new swactor-node binary (crates/node) hosting a real node over TCP.
- tests/docker: multi-host LAN cluster harness (compose, run-lan-cluster.sh,
cluster + lan_cluster integration tests) exercising the full SWIM path.
- simulation: cluster_scenarios integration + distribution property coverage.
- docs: reorganize into distribution/, runtime/, diagrams/, connectome/; add
DOCKER_REALIZATION + SIMULATION_TESTING realization notes.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
