Zachery Aaron Shores-Chmielewski
ae9ca3bcf3
Promote pipeline-parallel-inference to a first-class app and consolidate observability on the datastream wire, decoupling the dashboard crate from `distribution`. - apps/pipeline-parallel-inference: move the example out of `examples/` into `apps/` as its own workspace, rename binaries to `pp-worker`/`pp-orchestrator`, and strip release binaries - cluster: add `ClusterNode`, a synchronous facade over the actorized distribution protocol (IrohDriver + per-node Runtime hosting Swim/Registry/Metadata/Directory actors with a `MembershipFanout`), replacing ad-hoc `driver.node()`/`tick()` call sites - fleet: add per-node fleet telemetry that ships identity/resource records as `DatastreamFrame`s over the cluster transport to the orchestrator's `DatastreamSink`, folded into a `FleetView` on a 3s tick - provision: add best-effort, opt-in SSH boot-phase telemetry (`PP_DEPLOY_KEY`) that streams rented-node boot logs onto the orchestrator's datastream as `proc.boot.<stage>.*` - dashboard: rewire the crate dependency from `distribution` to `datastream`, drop the standalone `swactor-datastream-dashboard` binary, and rewrite `datastream_source.rs` to demux per-node frames into Overview/Distribution/Fleet views with live-node TTL filtering - distribution: refresh dist/netmap plugin copy and README from "Kademlia routing" to gossip-directory terminology Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com> |
||
|---|---|---|
| .. | ||
| src | ||
| tests | ||
| Cargo.toml | ||
| FORMAL_VERIFICATION_REPORT.md | ||
| README.md | ||
swactor-datastore
Distributed content-addressed datastore built on swactor. Objects are split into fixed-size chunks, identified by their blake3 hash, and replicated across a peer-to-peer network via epidemic gossip.
Building
Node binary (HTTP server + actor runtime):
cargo build -p swactor-datastore --features node
CLI client:
cargo build -p swactor-datastore --features cli
Both at once:
cargo build -p swactor-datastore --features node,cli
Node
Start a datastore node:
swactor-store-node
Flags
| Flag | Default | Description |
|---|---|---|
--port |
9091 |
HTTP API port |
--storage-path |
(in-memory) | Directory for persistent storage |
--dashboard-port |
(disabled) | Runtime dashboard port |
--chunk-size |
1048576 |
Chunk size in bytes (1 MB) |
--gc-interval |
1000 |
GC interval in ticks (~100s) |
--disseminate-interval |
50 |
Gossip interval in ticks (~5s) |
--config |
(none) | Path to a TOML config file |
Example with persistent storage and dashboard:
swactor-store-node --storage-path ./data --dashboard-port 9090
Config file
Create a store.toml and pass it with --config:
port = 9091
storage_path = "./my-data"
dashboard_port = 9090
chunk_size = 1048576
gc_interval = 1000
disseminate_interval = 50
CLI flags override config file values. Omitted fields use built-in defaults.
swactor-store-node --config store.toml --port 8080
CLI
The swactor-store command talks to a running node over HTTP.
Status
swactor-store status
Put
swactor-store put photo.jpg --name "vacation"
Get (metadata)
swactor-store get <hash>
Get (download)
swactor-store get <hash> --output photo.jpg
Delete
swactor-store delete <hash>
List (local)
swactor-store list
List (swarm-wide)
swactor-store list --all
Filter by name
swactor-store list --name vacation
Use --url to point at a different node:
swactor-store --url http://192.168.1.50:9091 list
Web UI
Visit http://<host>:<port>/ in a browser. The UI supports uploading, listing, downloading, inspecting, and deleting objects — works on desktop and mobile.
HTTP API
| Method | Path | Description |
|---|---|---|
GET |
/api/status |
Node identity |
POST |
/api/put?name=... |
Upload (body = raw bytes) |
GET |
/api/list |
List objects (?all=true for swarm) |
GET |
/api/get?hash=... |
Object metadata + manifest |
GET |
/api/data?hash=... |
Download reassembled binary |
POST |
/api/delete?hash=... |
Delete object |