Zachery Aaron Shores-Chmielewski
ae9ca3bcf3
Promote pipeline-parallel-inference to a first-class app and consolidate observability on the datastream wire, decoupling the dashboard crate from `distribution`. - apps/pipeline-parallel-inference: move the example out of `examples/` into `apps/` as its own workspace, rename binaries to `pp-worker`/`pp-orchestrator`, and strip release binaries - cluster: add `ClusterNode`, a synchronous facade over the actorized distribution protocol (IrohDriver + per-node Runtime hosting Swim/Registry/Metadata/Directory actors with a `MembershipFanout`), replacing ad-hoc `driver.node()`/`tick()` call sites - fleet: add per-node fleet telemetry that ships identity/resource records as `DatastreamFrame`s over the cluster transport to the orchestrator's `DatastreamSink`, folded into a `FleetView` on a 3s tick - provision: add best-effort, opt-in SSH boot-phase telemetry (`PP_DEPLOY_KEY`) that streams rented-node boot logs onto the orchestrator's datastream as `proc.boot.<stage>.*` - dashboard: rewire the crate dependency from `distribution` to `datastream`, drop the standalone `swactor-datastream-dashboard` binary, and rewrite `datastream_source.rs` to demux per-node frames into Overview/Distribution/Fleet views with live-node TTL filtering - distribution: refresh dist/netmap plugin copy and README from "Kademlia routing" to gossip-directory terminology Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
42 lines
1.6 KiB
Bash
Executable file
42 lines
1.6 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# run-equivalence-tests.sh — slow-lane runner for the sliced-vs-full
|
|
# equivalence tests (TEST_SPEC §12).
|
|
#
|
|
# These tests are `#[ignore]` because each one spawns a real-tinygrad
|
|
# worker per stage, loads the full llama3.2:1b GGUF on every worker, and
|
|
# runs CPU forward passes for up to `EQUIVALENCE_MAX_TOKENS` decode steps
|
|
# per prompt. A single test typically takes minutes; the full set takes
|
|
# tens of minutes on a workstation. Don't put these in per-commit CI;
|
|
# run them on touch to the worker, the actor's worker-IPC, or the
|
|
# message codec — they are the load-bearing correctness check.
|
|
#
|
|
# Requirements:
|
|
# * Working Python with tinygrad installed and reachable.
|
|
# * Enough disk space for the GGUF (~700MB) under
|
|
# $HOME/.cache/tinygrad (or wherever tinygrad's `fetch` caches).
|
|
#
|
|
# Usage:
|
|
# scripts/run-equivalence-tests.sh # all §12 tests
|
|
# scripts/run-equivalence-tests.sh say_hello # name-filter
|
|
# PP_TEST_THREADS=1 scripts/run-equivalence-tests.sh
|
|
set -euo pipefail
|
|
|
|
cd "$(dirname "$0")/.."
|
|
|
|
THREADS="${PP_TEST_THREADS:-1}"
|
|
|
|
# Default to the union of every §12 test; allow a positional name-filter
|
|
# (matched as a Cargo test-name substring) for spot-checking one case.
|
|
FILTER="${1:-sliced_}"
|
|
|
|
echo "== running TEST_SPEC §12 equivalence tests (filter='$FILTER', threads=$THREADS)"
|
|
echo " each test spawns N+1 real-tinygrad workers; expect minutes per case."
|
|
|
|
exec cargo test \
|
|
-p pipeline-parallel-inference \
|
|
--test t_integration \
|
|
-- \
|
|
--ignored \
|
|
--test-threads "$THREADS" \
|
|
--nocapture \
|
|
"$FILTER"
|