Replace fleet/actor-overview/actor-dossier/workers views and the root link
list with one ControlPlaneView serving / and /view/fleet: node cards fuse
machine stats with actor rollup, node click opens machine detail + roster,
actor click opens an in-page dossier (identity, message diet, sparkline,
sampled message history via /api/view/fleet/detail).
- Message history folds view-side from messages_processed deltas: 16-receipt
ring, 250ms spacing, sampled-out counters — noisy actors cannot flood the
page and producers stay untouched.
- Stale streams render in a separate collapsed pool; superseded life
generations are evicted immediately; stale pool hard-capped at 50.
- Unified navbar injected server-side from the view registry; pages opt in
with a <!--swactor:nav--> placeholder so app-registered views appear
automatically. Hardcoded per-page navs stripped.
- Rust type names are the actor display name; address is the unique key.
Stale doc comment claiming types are not on the frame fixed.
`cargo xtask provisioning-reconciler-demo [--port n] [--nodes n]` boots a
lightweight orchestrator for visual, human-checked E2E confirmation of the
provisioning reconciler: swactor engine + real ClusterDriver + demo
provider, with node children re-exec'ing the same xtask binary in node
role and joining the supervisor over real iroh connections.
- supervisor actor owns driver/provider/shape on a 250ms wall-clock tick,
mirroring the production ClusterReconciler poll semantics; emits
prov.reconciler.events/snapshot plus per-node lifecycle/status streams
- k8s-styled reconciler view: ready/desired header, node stage cards,
commands-out and events-in feeds
- dashboard `demo-control` feature: POST /control/{kill,provision,remove}
+ Fleet Control view; regular builds compile none of it (symbol-verified)
- fleet cards fold proc.<node>.lifecycle and node.status heartbeats into
per-node pid/state pills that stay live
- hardening: exe resolution survives binary replacement by rebuilds,
spawn failures feed back as BootstrapFailed so the reconciler retries
instead of wedging at SshReady, teardown skips exit waits for
never-started children
Verified in-browser: boot 3/3 converged with real joins; dashboard kill
dips and fully recovers with a replacement; provision +1 → 4/4; remove −2
graceful teardown → 2/2; child process count matches reconciler nodes.
The crate is the per-node metrics/logging pipe with a universal
subscriber endpoint, but "datastream" kept getting misread as a general
messaging plane. Rename crate, module paths, and public API
(`DatastreamEndpoint` → `TelemetryEndpoint`, etc.) so misuse is visible
on sight.
Renamed contracts (all in-repo producers/consumers migrated):
- env vars `MYELIN_DATASTREAM_*` → `MYELIN_TELEMETRY_*`
- artifact `datastream.ndjson` → `telemetry.ndjson`
- actor names `telemetry-publisher` / `telemetry-sink`
- wire ALPN `swactor/telemetry/0`
- `DATASTREAM_SPEC.md` → `TELEMETRY_SPEC.md`
Also fixes two latent test breaks: `process` and `iroh-driver` tests
imported `DatastreamEvent` from the crate root, which was never
re-exported; they now use the observer path `telemetry::frame::`.
The datastream is metrics/logging only; control decisions must never branch
on a frame. This was a recurring cultural problem with no structural
enforcement. This change makes it a compile-time and CI-enforced fact.
datastream crate (lib.rs):
- Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root.
is now a compile error (E0425). These types live
only in datastream::frame::* and are documented as the observer surface.
- Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain
re-exported at root for producer-side callers.
orchestration/app.rs:
- Extracted all frame-touching code (CollectedDatastreamFrame,
drain_datastream_connections, update_load_progress_from_frame,
drain_frames, archive_collected_frame, pump, OrchDatastream,
DashboardSupport) into two new observability modules:
frame_collector.rs and orch_datastream.rs.
- The orchestrator now interacts through a FrameCollector whose
drain/drain_with_progress methods take closures; it never names Frame,
DatastreamEvent, or CollectedDatastreamFrame.
- StageLoadProgress (the one control-relevant signal previously scraped
from frame payloads) is extracted inside FrameCollector and handed to
the control loop as plain data.
xtask:
- New check-telemetry-isolation command scans control-plane modules
(orchestration/, distribution/, data-plane/, provisioning/) for
forbidden frame-type references and fails the build if any are found.
Verified: workspace builds (myelin + dashboard feature), datastream 29
tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes
clean.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add project licensing and a full README, plus a minimal WebAssembly ping-pong example that exercises the actor runtime.
- `LICENSE`: add the full GNU AGPL-3.0 text and set `license = "AGPL-3.0-only"` on every package (`swactor`, each crate, `apps/myelin`, `tools/vastai`, `xtask`)
- `README.md`: rewrite from a stub into a full project overview, covering features (actor_id routing, WASM, iroh QUIC/SWIM, OTP-style std, process manager, zero-copy objects, datastream metrics, dashboard), architecture, examples, developing, status, and license
- `examples/ping-pong`: new standalone workspace (`pingpong` cdylib) where two actors volley on a single-threaded `wasm` runtime driven by a Node host via `tick()`, demonstrating spawning, message passing, and death monitoring (`watching`)
- `examples/ping-pong`: add a host/run harness -- `run.sh` (wasm-pack build + `run.mjs`), `serve.sh` (dashboard + static demo), `index.html`, and a pinned `Cargo.lock`
- `.gitignore`: stop ignoring `.loop/`, `.deployment-notes/`, and `.omp/`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add read-only actor overview and per-actor dossier pages to the dashboard, fed by enriched per-actor runtime snapshots.
- `dashboard/swactor/actor_view`: new `ActorPanelView`, a tolerant frame consumer over `runtime.actors`/`runtime.stats` that folds per-actor snapshots and serves `/view/swactor/actor-overview` (roster) and `/view/swactor/actor-dossier` (per-actor detail), each backed by an embedded HTML template (`actor_overview.html`, `actor_dossier.html`)
- `dashboard`: register both views in `DashboardHandle` and export `actor_overview_view()`/`actor_dossier_view()` from the swactor module
- `swactor` core: enrich `ActorSnapshot` with `actor_type` and `message_type` (populated from `slot.actor.metadata()` in `ActorPool`) and add `ActorAddress::to_full_hex()` for untruncated display
- `myelin/orchestration`: publish actor stats to the dashboard via a `runtime.actors` channel producer (`stats_hook_on`) threaded through the distribution stack, and carry the orchestrator actor address into readiness signaling
- workspace `Cargo.toml`: add `default-members` for native iteration and a centralized `[workspace.dependencies] tokio` so members share one feature set; `dashboard/Cargo.toml` switches to `tokio.workspace = true`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Promote the `mvp-system` workspace library crate to a standalone application at `apps/myelin`, rebranding the MVP system along with its binaries, node image, and spec.
- workspace `Cargo.toml`: swap member `crates/mvp-system` -> `apps/myelin` and drop `apps` from `exclude` so the app joins the workspace
- `apps/myelin/Cargo.toml`: declare package `myelin` with `autobins = false` and explicit `[[bin]]` targets `myelin-worker`, `myelin-orchestrator`, `myelin-chat`
- `apps/myelin/src`: move the whole `mvp-system` source tree and rebrand module surfaces (`chat/mod.rs`, `prompt/mod.rs`); add `bin/chat.rs` (`myelin::run_chat_from_args`) and delete the old `mvp_chat.rs`
- `apps/myelin/node-image`: relocate the worker image assets from `apps/mvp-node/` (Dockerfile, Dockerfile.base, tinygrad_worker.py, entrypoint, e2e script) and rename `MVP_SYSTEM_SPEC.md` -> `MYELIN_SPEC.md`
- `xtask`: rewrite build/reference paths for the rename (~1000-line churn); add `crates/dashboard/ACTOR_PANEL_SPEC.md`
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Convert the remaining thread/mpsc-based provider monitor and stage-shard fetch loops into swactor actors, propagate worker-crash reasons through the actor message chain, and validate cached stage shards before reuse.
- orchestration/provider_adapters/vastai: replace the thread+AtomicBool VastAiProviderMonitor with VastAiProviderMonitorActor driven by Poll/Stop messages on its own RuntimeHandle (schedule_provider_monitor_poll), preserving the status/terminal-failure observation logic
- node/worker_node_runtime: convert the blocking stage-shard-fetch mpsc loop into StageShardFetchActor (Start/PollChild/ProcessLine/ReaderError/ReaderClosed via ExternalSender, on_stop kills and joins the child) reporting Progress/Done/Failed
- staging/gguf_shard: add validate_stage_shard_cache (tensor count/alignment/name/dims/type) and mix STAGE_SHARD_CACHE_FORMAT_VERSION into the shard_cache_key; materialize_stage_shard_with_process now validates a cached shard, emitting StageShardCacheInvalid and refetching when stale
- node/actor + orchestration/actor: NodeAgentMsg::WorkerCrashed now carries Option<reason>, surfaced as ObserveStageFault{reason}/StageFault{reason} up to the orchestrator; NodeAgentActor records last_worker_crash
- orchestration/app: complete_bootstrap now runs after runtime-ready without dropping stop handles (ProvisionedClusterGuard) and folds worker-crash reasons into stage-fault errors; chat/runtime adds cached-model config selection
- xtask: relax the data-path requirement to accept activation-step-executed OR downstream-activation-loaded
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
- Split arena/ring/object-record into a new data-plane crate and node/plugin contracts
into a provisioning crate.
- Reorganize mvp-system into orchestration, staging, node, chat, and worker modules;
extract binaries into chat/runtime and node/worker_node_runtime.
- Add MVP_SYSTEM_MODULE_BOUNDARY_SPEC.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Land end-to-end canonical benchmark observability and a synthetic datastream-connectivity preflight across the orchestrator, chat, worker-node, and Python tinygrad worker, plus an xtask validator, so a 4-stage 7B pipeline run is fully diagnosable.
- benchmark_observability: expand stamp() with canonical producer fields (producer_component/instance_id/process_id/sequence, wall_clock_unix_ms, monotonic_ms, clock_source) and schema_version so every event shares one envelope shape
- orchestrator_app + bin/{mvp_chat,worker_node}: stamp OrchBootstrap/OrchPromptEvent/ChatProgress/NodeEvent/SamplerHealth with the canonical fields plus span_id/parent_span_id, and add a 4-phase synthetic datastream preflight (ProducerConfigured/Connected/SyntheticEventSent/Observed) plus an endpoint_config_snapshot event on each process
- apps/mvp-node/tinygrad_worker: add apply_canonical_envelope()/datastream_endpoint_snapshot() and emit_python_datastream_preflight() mirroring the Rust preflight, and enrich benchmark_stamp() with the same producer fields
- bin/worker_node: pass MVP_DATASTREAM_ENDPOINT_ID/MVP_BENCHMARK_PRODUCER_INSTANCE/MVP_IROH_ENDPOINT_ADDR_MASK/MVP_IROH_RELAY_MODE env to the spawned tinygrad worker so its stamps identify the stage
- bin/mvp_chat: add --pipeline-parallel as an alias for --pipeline-stages (with a duplicate-guard) and bump recursion_limit
- xtask: add a benchmark-observability validator (ValidatorFinding/BenchmarkValidation, validate_benchmark_observability, canonical-stamp and stage/edge checks, evidence + gap-report builders) with tests for missing python datastream connectivity, wrong run_id, and missing span_id
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Distribute only each stage's GGUF layer slice over HTTP, add sampler and weight-load health telemetry, and harden node-image build, orchestrator provisioning, and the VastAI lease/search path.
- gguf_shard (new): StageShardPlan and plan_stage_shard parse the GGUF directory and compute coalesced per-stage tensor byte ranges; materialize_stage_shard_http fetches only those ranges (plus the header) to build a stage-local GGUF, with planned_fetch_bytes accounting.
- orchestrator_app: build a BTreeMap<u32, StageShardPlan> from the run plan for HuggingFace sources, thread stage_shard_plan through StageProvisionWire and weight-load, emit stage_shard_plan summaries, and add liveness phases (prefetching/fetching_stage_shard, cache_ready, stage_shard_ready).
- worker_node: add a stage-shard-fetcher subcommand and materialize_stage_shard_with_process that spawns the fetcher, streams its stdout/stderr as stage_shard_fetch events (StageShardCacheReady/StageShardReady), caches under MVP_MODEL_CACHE_DIR, and feeds the local shard path into load_weights.
- worker_node: add NODE_SAMPLER_CHANNEL and SamplerHealth telemetry (gpu/cpu/net samplers emit started/waiting/ready/failed) plus structured helper stdout/stderr streaming (wait_for_helper_event/drain_worker_stderr).
- node_image: expand node-image build/push handling for the deploy path.
- tools/vastai: extend lease, search, and types and drop unused pricing code.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Complete an 8-stage pipeline-parallel run over VastAI by provisioning stages high-to-low, adding per-stage/per-step metrics, host anti-colocation, and provider state-timeout guardrails.
- orchestrator_app: select the next weight-load stage by max index (provision stages high-to-low for parallel spread), add a throttled "loaded N of M; waiting on stage X" stage_provision_wait headline, and surface min_compute_cap/state_timeout_secs in the config dump.
- orchestrator_app: enrich pipeline_token_in/out and tokenizer_decode events with token_count/token_ids/generated_index.
- worker_node: add timing metrics across the data path (helper_execute_ms, egress_ring_read_ms, send_ms, ingress_ring_write_ms, object_load_ms), refactor take_complete_ingress_record into IngressRecordBytes (object_id/sequence/extent/flags), and emit a new object_loaded event.
- vastai_provisioning: track leased host_ids and blacklist already-leased hosts in later ProvisionRequests so stages don't co-locate, and tag SSH-bootstrap retry logs with the attempt number.
- tools/vastai: add min_compute_cap (PP_MIN_COMPUTE_CAP) filter/search query and a LifecyclePolicy state_timeout (PP_STATE_TIMEOUT_SECS) that fails instances stuck in a non-running status instead of polling forever.
- xtask: raise the check timeout to 1800s/30s grace, drop --skip-rebuild for VastAI, aggregate per-stage StepExecuted metrics, add a vastai summary section, and write failure artifacts on abort.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Lift the VastAI two-stage cap and let mvp-chat-check drive an arbitrary N-stage deployment through a new --pipeline-stages flag.
- orchestrator_app: remove the provider=vastai >2 stage cap and replace the two-stage plan test with an eight-stage plan test (8 specs, no mounts, remote GGUF, max-context).
- xtask: refactor scenario parsing into MvpChatCheckInvocation carrying an optional pipeline_stages, parse a `--pipeline-stages n` flag (rejecting 0/missing values), and default multinode/docker to 2 while VastAI uses the explicit count.
- xtask: thread the invocation through run_mvp_chat_check/run_mvp_chat_check_process and update the usage text and scenario tests (including `--vastai --pipeline-stages 8`).
- gitignore: ignore .deployment-notes/.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Turn mvp-chat-check into a benchmarking harness with per-stage latency capture, a run envelope, summary artifacts, and a mvp-chat-compare command for delta analysis.
- xtask: add mvp-chat-compare <baseline> <candidate> that checks comparability (schema/scenario/workload/model/provider/pipeline_stages) and prints deltas for total, prepare, standup-to-RPC, and per-prompt roundtrip/first-token/decode/text-decode ms.
- xtask: build_benchmark_summary now writes stdout/stderr/prompts/redacted-config/summary artifacts with per-artifact byte counts and a vastai summary section; rename the dump log to datastream.ndjson.
- xtask: add write_failure_artifacts so failed checks still emit a failure summary with the prompt-corpus blake3 and artifact sizes.
- mvp_chat: add the mvp.chat.benchmark channel and emit_benchmark_envelope (BenchmarkRunEnvelope with model/runtime/provider/workload detail), and tag prompt events with prompt_index and a blake3 prompt_hash.
- tinygrad_worker: add per-phase latency metrics (encode/decode/text-decode/first-token elapsed_ms; stage_execution_ms/record_write_ms on execute_step; ring_readable/encode_prompt/decode_tokens elapsed_ms plus payload sizes).
- vastai_provisioning: emit VastAiLeaseReady and VastAiSshEndpointReady provider lines (contract/offer/host/gpu/dph, ssh host/port/user) for observability.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Land the first working two-stage pipeline-parallel run over VastAI, wiring a real inter-stage data path with observability, a max-price offer cap, and remote-image reuse.
- orchestrator_app: raise the VastAI pipeline-stage cap from 1 to 2 and let VastAI pipeline planning resolve the HuggingFace GGUF from the default local cached-model metadata path instead of requiring host mounts; add --vastai-max-dph-total (CLI/env/TOML) config.
- vastai_provisioning: make complete_bootstrap a no-op so the SSH bootstrap log tail stays alive past runtime-ready until node stop, preserving post-ready worker logs; add a test asserting the tail is only stopped on NodeStop.
- worker_node: emit data-path NodeEvents across the pipeline (iroh_edge_stream_arrived/bytes_read/bytes_sent, egress_ring_read, ingress_ring_write) with edge/byte metadata.
- tools/vastai: add max_dph_total (PP_MAX_DPH_TOTAL) to SelectionPolicy, the reachable-offer filter, and the search query, and improve the empty-pool error message.
- xtask: pass --skip-rebuild for the VastAI scenario and gate it on a new require_vastai_data_path_facts plus GPU facts (ring install, activation object load/step, interstage handoff, iroh edge read/sent).
- mvp_chat: add ChatModelConfig (model id/gguf/tokenizer/max-context) forwarded to the orchestrator; for VastAI + skip-rebuild, emit skip events and reuse the remote node image without a local build.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add a `--vastai` acceptance scenario to mvp-chat-check that provisions real VastAI nodes and verifies the remote provider data path, and standardize the VastAI API key on VAST_API_KEY.
- xtask: add the VastAi variant and `--vastai` flag to the mvp-chat-check scenario, passing `--vastai --yes --endpoint-addr-mask relay-only` (no --cached-model) and gating it on a new require_vastai_network_facts check (node_spec workers, ProvisionStart, provider_start, iroh_driver ready).
- xtask: track VastAI dump-log facts (vastai_node_spec_worker_count, vastai_provision_start_nodes, vastai_provider_start_nodes) via record_vastai_provision_dump_log_event and skip the local ChatProgress span assertion for the remote scenario.
- config: rename the VastAI key env var to VAST_API_KEY in ResolvedVastAiConfig validation while keeping MVP_VASTAI_API_KEY/VASTAI_API_KEY fallbacks.
- orchestrator_app: resolve the api key with VAST_API_KEY first, then MVP_VASTAI_API_KEY/VASTAI_API_KEY, and update the missing-key error messages.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Get the multinode pipeline-parallel prompt loop actually running locally by supporting both tinygrad LLM backends and switching the Docker scenario to direct addressing.
- tinygrad_worker.py (load_pipeline_stage_model): try the modern tinygrad.llm gguf/model modules first and fall back to the legacy tinygrad.apps.llm TransformerBlock on ModuleNotFoundError, with PipelineStageTinygradModel constructing blocks positionally when no TransformerConfig exists
- tinygrad_worker.py (load_weights): drop the Transformer.from_gguf whole-model branch and its TinygradAppsLlmPartialStageUnsupported fatal, so partial pipeline stages build through load_pipeline_stage_model on either backend
- xtask (MultinodeDocker): stop forcing --relay-mode default --endpoint-addr-mask relay-only, so the scenario runs over the Docker network with full/direct addresses
- xtask (dump-log facts): rename relay_masked_* facts to docker_*, assert multiple workers join the coordinator via direct addresses (direct_addr_count > 0), and relax the benchmark report to skip the ensure_worker_binary span for the Docker scenario
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add endpoint-address masking and a relay-only advertisement path so the multinode Docker mvp-chat scenario can run with direct addresses stripped.
- endpoint_advertisement: add EndpointAddrMask (Full/RelayOnly) parsed from --endpoint-addr-mask/MVP_IROH_ENDPOINT_ADDR_MASK, and advertised_endpoint that rebuilds an EndpointAddr from relay URLs only, rejecting relay-only without a relay URL
- orchestrator_app: mask the coordinator endpoint before advertising it, thread the masked collector endpoint into datastream subscribe/runtime-ready acks, surface endpoint_addr_mask/has_relay/direct_addr_count in iroh_driver and node_spec events, forward the mask env to workers, and add a 60s RUNTIME_READY_TIMEOUT to the runtime-ready barriers
- worker_node: advertise the masked self endpoint in the iroh_driver ready and coordinator_join events and propagate it through runtime_ready_local and PendingRuntimeReady
- mvp-chat: add --relay-mode/--relay-url/--endpoint-addr-mask plus a [relay] toml section, require (with a Vast.ai fallback) a relay URL when relay-only, and forward all three to the orchestrator CLI
- node_image: resolve the worker binary to a workspace-relative path for the Docker COPY via docker_build_context_path, rejecting paths outside the build context
- xtask/specs: run MultinodeDocker with --relay-mode default --endpoint-addr-mask relay-only, add dump-log fact checks for relay-masked orchestrator/node/coordinator advertisement, and document the mask/relay flags in mvp_chat.md
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
- Pull ~7.4k lines out of the orchestrator bin into a new orchestrator_app library
module.
- Wire a local single-node GPU prompt loop into the mvp_chat bin; touch
gpu_worker_ingress_parser.
- Grow xtask and the mvp-node tinygrad worker.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Add end-to-end timing instrumentation and an xtask benchmark report for mvp-chat runs.
- benchmark_observability: add a shared stamping module — stamp(component) emitting schema/pid/monotonic+wall ms from a process-global start and sequence counter, plus unix_ms_now() — stamped onto every mvp-chat/orchestrator/worker-node event and frame-archive record
- mvp-chat: thread a run_id (new --run-id, defaults to 1) through config and the orchestrator CLI, add per-phase started/ready/failed emits for ensure_orch_binary/ensure_worker_binary/prepare_node_image, and a prompt_complete record carrying tokens_generated/elapsed_ms/final_text bytes
- orchestrator/worker-node: stamp bootstrap and prompt events, add arrival_unix_ms to archived frames, propagate MVP_RUN_ID/MVP_LOGICAL_NODE_ID/MVP_STAGE_INDEX into the tinygrad worker, default the device to CPU for the process provider, emit a prompt_rpc started span, and drop the MVP_TINYGRAD_TEST_MODE passthrough
- tinygrad_worker.py: stamp every control() event and tag it with run/node/stage env, add a CPU:X86 fallback when clang is absent, and remove the test_mode() short-circuits
- xtask: replace the flat dump-log fact assertions with a benchmark report builder (build_benchmark_report) that requires named spans (prepare_runtime, ensure_*_binary, weights_loaded, prompt_rpc) and emits per-prompt first-token/decode/tokens-per-second latency; wrap the cargo run in XtaskBenchmark synthetic frames and pass a unix-ms --run-id
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Move iroh_driver and the relay binary out of distribution into a dedicated
crates/iroh-driver (lib re-exports IrohDriver; relay bin renamed). Remove the node crate
and the single-gpu-inference example; drop the docker/datastream demo. Slim
pipeline-parallel vastai.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Remove the datastore crate, the top-level design/orchestration/ring specs, the
benches, and the ci config. Add the dashboard host telemetry sampler
(cpu/disk/net/gpu/mem). Localize the pipeline-parallel e2e stub/mock paths.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Replaces the generic SimNode/gossip/dashboard framework with a virtual-time
discrete-event engine (priority queue ordered by time/node/fiber/seq), a TOML spec
parser, bundle writer, replay, divergence detector, lint, and postproc, plus the
parity-bar test harness with fixtures and xtask parity-lock tooling. Rewrites
transport identity/crypto and adds the SPEC/TESTING_SPEC/OBSERVABILITY/NORTH_STAR docs.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Remove co-dependencies for different modules found in `crates` and migrate the development history to a new repository. The docs were stale, and largely not getting used, so simply deleted for now. When code stabilizes more, they will become useful again.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Initial experiments in 'formal verification' of authorization tooling. Current state is not to be trusted, but we are not in a security critical situation, so that is fine.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Introduce a gossip-converged pooled datastore protocol layered on SWIM piggybacking, backed by a reusable gossip-channel abstraction, plus a Docker-free multi-process cluster test runner.
- crates/distribution/src/gossip_channel.rs: add the GossipChannel trait (piggyback on SWIM messages) and a budget-limited DisseminationBuffer<T> that replaces the four duplicated Lambda*ceil(log2(n)) dissemination copies
- crates/datastore/src/pool: add PoolDisseminator (CRDT state for membership/capacity/content-location/ACL with join/leave/announce) and PoolCoordinator (placement-aware CRUD actor delegating to the co-located DatastoreNode)
- crates/shared-types/src/pool.rs: add shared pool protocol types (PoolId plus member/capacity/content-location/ACL entries and PoolConfig) consumed by both distribution and datastore
- crates/dashboard/src/pool_html.rs: add a live pool dashboard page (membership, capacity, content locations) and add pool_tests integration coverage
- xtask/src/sim_cluster.rs: add the sim-cluster runner that spawns N swactor nodes over iroh through a local relay server, reusing the docker cluster scenarios without Docker
- crates/distribution/src/iroh_driver.rs: add relay-URL resolution (cache, then SWIM gossip, then home relay) with a 2s connect timeout to back the relay-based connections
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Make distribution and deployment more stable. Consolidate the logic for a generic swactor node.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
Content addressable datastore. Allows you to configure a node to store and stream large blobs of data, and retrieve them from any swactor-connected node.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
