Commit graph

12 commits

Author SHA1 Message Date
8363be74cd feat: docker provisioning
Treat Docker as a first-class provisioning provider for the local e2e cluster and add the weight-shard fetch/validate/bind lifecycle behind it.

- docker_cluster_provisioning: model Docker as a provider adapter behind a `DockerCli` boundary (`run_container`/`inspect_ssh_endpoint`/`remove_force`) exposing only per-node ownership primitives
- local_e2e_cluster: wire Docker provisioning into the e2e driver (default `swactor-mvp-local-e2e-cluster` image) across the bootstrap/teardown flow
- weight_shards: add `ModelArtifactRef` (parses `hf://repo@rev/path`), `ShardAssignment`, `ShardManifest`, and `ValidatedShard` with digest-based validation
- shard_fetch: add `ShardLocator` (digest/split/stage -> uri + cache key), `ShardCache`/`ShardFetcher` traits, a `ShardFetchCoordinator`, and typed `FetchError`s
- shard_weight_lifecycle: add the `ShardWeightLifecycle` state machine (Idle->Assigned->Located->Fetching->Fetched->Validating->Binding->Ready/Faulted) with a `WorkerShardBinder` trait
- provisioner/telemetry: route provision logs onto the datastream via per-node/stream channels (`submit_bytes`) and add shard_fetch/shard_weight_lifecycle/weight_shards guarantee tests

Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-28 22:54:02 +04:00
61aa5b972d feat(dashboard): datastream-backed dashboard
Add a universal live datastream explorer view to the HTTP dashboard and point the e2e cluster at it by default.

- live_explorer: add the `LiveDatastreamExplorer` `DashboardView`, registered in `start_dashboard`, served at `/view/datastream/live` over the existing `/events` and `/api/frames` endpoints
- live_explorer_page: ship a 500-line self-contained page that auto-connects, renders streams/channels/frames metrics, and supports level drill-down (streams->channels->frames), payload-kind filtering, and per-channel retention limits
- local_e2e_cluster: switch the dashboard's default opened URL from `/view/mvp/cluster` to `/view/datastream/live`

Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-28 13:42:48 +04:00
789bf1fe19 feat(mvp-system): implement node and docker cluster provisioning
node_provisioning: NodeManager record FSM and transient BootstrapSession SSH core per
the provisioning spec. docker_cluster_provisioning provider; add guarantee tests for
both.


Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-28 11:57:56 +04:00
7935849bcb docs(mvp-system): draft MVP node provisioning spec
Add MVP_NODE_PROVISIONING_SPEC.md covering NodeManager, provider leases, SSH
BootstrapSession, datastream log forwarding, and lease teardown.


Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-26 12:13:28 +04:00
0ffc5fd275 feat: add mvp provisioning subsystem and datastream transport
mvp-system: provisioner actor, provisioning module, node_agent, dashboard_view,
observability_surface; expand gpu_worker ctl/ingress/egress and run_plan.
iroh-driver: replace relay binary with datastream_transport; datastream gains endpoint
abstraction. Archive pipeline-parallel-inference app to old-pipeline-parallel-inference.


Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-25 16:30:18 +04:00
cf996d9697 refactor(dashboard): rebuild around swactor worker view
Replace telemetry/history/plugin/topology/warnings layer with store/view and
swactor worker_page/worker_view fed by datastream frames. Add mvp-system
local_e2e_cluster harness; archive pipeline-parallel-inference to
old-pipeline-parallel-inference.


Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-25 11:29:16 +04:00
9f63b9331d feat(mvp-system): add engine_builder module
Introduce pool/planner/launcher/runtime_stack/model/roles primitives for topology
construction and cluster launch. Drop the core guarantees module entirely; rework
worker bootstrap.


Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-24 19:23:09 +04:00
2a26447184 refactor(core): prune std supervisor and registry surface
Drop children/monitor/resource/service/supervisor/timer/router registries, keeping
groups, naming, watching, and ctx/runtime ext. Decouple Environment keys from
ServiceRegistry; trim g10/g6_g7 guarantee modules. Consolidate tests/std_extension into
core_extension_seams.


Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-24 15:04:39 +04:00
e5df800c83 refactor: consolidate dist tests, add mvp actors and e2e
Fold the fragmented distribution swim/routing/gossip tests into swim_core, routing,
and swim_actor. Prune the dashboard tui and command surfaces. Add mvp-system actors
(node_agent, orchestrator, stage_controller), the local_e2e harness, and gpu worker
e2e.


Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-24 13:30:29 +04:00
0d1b95695d refactor: drop datastore crate, stale specs, and benches
Remove the datastore crate, the top-level design/orchestration/ring specs, the
benches, and the ci config. Add the dashboard host telemetry sampler
(cpu/disk/net/gpu/mem). Localize the pipeline-parallel e2e stub/mock paths.


Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-23 19:42:28 +04:00
46fe9afa1b feat(mvp): implement mvp-system modules and in-crate tests
Implement arena_manager, device_bridge, driver_pumps, edge_establisher, gpu_worker
ctl/egress/ingress/process-adapter, orchestrator run-fsm and token-endpoint, run_plan,
stage_controller, tx_rx_edge_actor, weight_lifecycle, and the remaining modules. Move
guarantee tests from tests/mvp_system into crates/mvp-system/src/tests; add the
tinygrad device-bridge backend helper.


Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-23 17:51:34 +04:00
b5de4c9bd6 docs(mvp): lay out mvp-system spec and contracts
Add MVP_SYSTEM_SPEC plus per-component contract docs (arena manager, device bridge,
gpu worker ctl/ingress/egress/process-adapter, orchestrator run-fsm/token-endpoint,
run plan, stage controller, tx_rx edge, weights). Scaffold the guarantee tests against
the (empty) mvp-system crate.


Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-06-23 11:12:45 +04:00