feat(engine): substrate-neutral execution engine abstraction
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
2026-08-10 20:23:03 +00:00
|
|
|
//! Composite engine and cloneable scheduler handle.
|
|
|
|
|
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
use parking_lot::{Condvar, Mutex};
|
|
|
|
|
use std::sync::atomic::{AtomicBool, Ordering};
|
feat(engine): substrate-neutral execution engine abstraction
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
2026-08-10 20:23:03 +00:00
|
|
|
use std::sync::{Arc, Weak};
|
|
|
|
|
use std::time::Duration;
|
|
|
|
|
|
|
|
|
|
use crate::backend::{Capabilities, EngineError, ExecutionBackend};
|
2026-08-16 13:27:38 +00:00
|
|
|
use crate::time::{EngineInstant, Interval, Timeout, Timer};
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
use swactor::actor::{ActorAddress, Message};
|
|
|
|
|
use swactor::runtime::{ExternalSender, Runtime, RuntimeParts};
|
feat(engine): substrate-neutral execution engine abstraction
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
2026-08-10 20:23:03 +00:00
|
|
|
|
2026-08-11 12:08:06 +00:00
|
|
|
/// The composite engine: retains a configured core runtime handle and its
|
|
|
|
|
/// execution backend, and owns one core-driving loop per worker.
|
feat(engine): substrate-neutral execution engine abstraction
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
2026-08-10 20:23:03 +00:00
|
|
|
///
|
|
|
|
|
/// Construct with [`Engine::new`]; obtain a scheduler handle with
|
|
|
|
|
/// [`Engine::handle`].
|
|
|
|
|
pub struct Engine {
|
runtime: checkpoint distributed execution and retained-node deployment
Integrate namespace and source-route lifecycle changes, contextual process cleanup, Python binding updates, and Myelin worker/orchestrator recovery. Keep the shared control contracts, deployment identity fencing, SSH bootstrap adapters, paid admission accounting, and VastAI cleanup implementation together with their consumers.
Migrate Iroh dependencies and telemetry transport/collection with dashboard and demo callsites, workspace build configuration, and actor-control-flow policy updates. This is an intermediate development checkpoint, not paid-provider qualification.
Review verification: contextual_process_guarantees (4 tests), telemetry_transport (4 tests), and shared control contracts (5 tests) passed. Historical five-node redeployment and campaign execution passed individually; complete ordered qualification remains pending.
2026-09-14 09:20:56 +00:00
|
|
|
backend: Arc<dyn ExecutionBackend>,
|
2026-08-11 12:08:06 +00:00
|
|
|
/// Retained so the engine owns the runtime handle it drives for its full
|
|
|
|
|
/// lifetime. Core workers are moved into substrate tasks at construction.
|
2026-08-22 17:18:25 +00:00
|
|
|
_runtime: Runtime,
|
feat(engine): substrate-neutral execution engine abstraction
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
2026-08-10 20:23:03 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl Engine {
|
2026-08-11 12:08:06 +00:00
|
|
|
/// Construct an engine over `parts` driven by `backend`.
|
feat(engine): substrate-neutral execution engine abstraction
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
2026-08-10 20:23:03 +00:00
|
|
|
///
|
2026-08-11 12:08:06 +00:00
|
|
|
/// The runtime parts must be fully configured beforehand; after construction
|
|
|
|
|
/// the engine owns every worker and is their sole driver. Construction fails
|
|
|
|
|
/// if `backend` does not advertise a capability the engine requires (at
|
|
|
|
|
/// minimum, `tasks`).
|
2026-08-16 13:27:38 +00:00
|
|
|
pub fn new(parts: RuntimeParts, backend: impl ExecutionBackend) -> Result<Self, EngineError> {
|
feat(engine): substrate-neutral execution engine abstraction
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
2026-08-10 20:23:03 +00:00
|
|
|
let backend: Arc<dyn ExecutionBackend> = Arc::new(backend);
|
|
|
|
|
if !backend.capabilities().tasks {
|
|
|
|
|
return Err(EngineError::MissingRequiredCapability);
|
|
|
|
|
}
|
2026-08-11 12:08:06 +00:00
|
|
|
let runtime = parts.runtime().clone();
|
|
|
|
|
let workers = parts.into_workers();
|
|
|
|
|
// Install one core-driving loop per worker. This is substrate-neutral —
|
|
|
|
|
// no Tokio feature gate — so core progression does not silently
|
|
|
|
|
// disappear when an alternate backend is used (ENGINE_SPEC.md).
|
|
|
|
|
crate::core_driver::install(workers, &backend);
|
2026-08-22 17:18:25 +00:00
|
|
|
Ok(Engine {
|
|
|
|
|
backend,
|
runtime: checkpoint distributed execution and retained-node deployment
Integrate namespace and source-route lifecycle changes, contextual process cleanup, Python binding updates, and Myelin worker/orchestrator recovery. Keep the shared control contracts, deployment identity fencing, SSH bootstrap adapters, paid admission accounting, and VastAI cleanup implementation together with their consumers.
Migrate Iroh dependencies and telemetry transport/collection with dashboard and demo callsites, workspace build configuration, and actor-control-flow policy updates. This is an intermediate development checkpoint, not paid-provider qualification.
Review verification: contextual_process_guarantees (4 tests), telemetry_transport (4 tests), and shared control contracts (5 tests) passed. Historical five-node redeployment and campaign execution passed individually; complete ordered qualification remains pending.
2026-09-14 09:20:56 +00:00
|
|
|
_runtime: runtime,
|
2026-08-22 17:18:25 +00:00
|
|
|
})
|
feat(engine): substrate-neutral execution engine abstraction
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
2026-08-10 20:23:03 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Return a clonable handle for scheduling engine work.
|
|
|
|
|
///
|
|
|
|
|
/// The handle holds a *weak* backend reference, so handles — and engine
|
|
|
|
|
/// work that captures them — never keep the backend alive. Dropping the
|
|
|
|
|
/// [`Engine`] releases the backend (and its owned runtime / core-driver
|
|
|
|
|
/// task) once no other strong reference remains (ENGINE_SPEC.md).
|
|
|
|
|
pub fn handle(&self) -> EngineHandle {
|
|
|
|
|
EngineHandle {
|
|
|
|
|
backend: Arc::downgrade(&self.backend),
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// A cloneable scheduler handle.
|
|
|
|
|
///
|
|
|
|
|
/// Schedules work and reads engine time without exposing the underlying
|
|
|
|
|
/// backend; in particular it never hands out a raw `tokio::runtime::Handle`.
|
|
|
|
|
/// The handle holds a **weak** backend reference: it does not keep the engine
|
|
|
|
|
/// or its backend alive. Using a handle after its engine has been dropped
|
|
|
|
|
/// degrades gracefully — scheduled work is dropped, timers never fire, and
|
|
|
|
|
/// capability checks report no capabilities — rather than retaining the
|
|
|
|
|
/// backend (ENGINE_SPEC.md).
|
|
|
|
|
#[derive(Clone)]
|
|
|
|
|
pub struct EngineHandle {
|
|
|
|
|
backend: Weak<dyn ExecutionBackend>,
|
|
|
|
|
}
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
/// A clonable substrate route for one-shot blocking I/O work.
|
|
|
|
|
///
|
|
|
|
|
/// Domain actors decide which effect to execute; this handle only moves its
|
|
|
|
|
/// mechanics onto the engine backend's dedicated blocking pool.
|
|
|
|
|
#[derive(Clone)]
|
|
|
|
|
pub struct BlockingWorkSender {
|
|
|
|
|
backend: Weak<dyn ExecutionBackend>,
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl BlockingWorkSender {
|
|
|
|
|
/// Submit one blocking I/O operation without occupying an actor worker.
|
|
|
|
|
///
|
|
|
|
|
/// Returns the operation unchanged if the owning engine has stopped.
|
|
|
|
|
pub fn submit(&self, work: crate::BoxWork) -> Result<(), crate::BoxWork> {
|
|
|
|
|
let Some(backend) = self.backend.upgrade() else {
|
|
|
|
|
return Err(work);
|
|
|
|
|
};
|
|
|
|
|
backend.spawn_blocking(work);
|
|
|
|
|
Ok(())
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Cancellation handle for an engine-owned actor message timer.
|
|
|
|
|
///
|
|
|
|
|
/// Cancellation is idempotent. A timer may already have fired when cancellation
|
|
|
|
|
/// races its deadline, so actor messages should still carry an operation or
|
|
|
|
|
/// generation identity that lets the receiver reject stale work.
|
|
|
|
|
#[derive(Clone, Debug)]
|
|
|
|
|
pub struct ActorTimer {
|
|
|
|
|
cancelled: Arc<AtomicBool>,
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl ActorTimer {
|
|
|
|
|
/// Prevent this timer from delivering future messages.
|
|
|
|
|
pub fn cancel(&self) {
|
|
|
|
|
self.cancelled.store(true, Ordering::Release);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Report whether cancellation has been requested.
|
|
|
|
|
pub fn is_cancelled(&self) -> bool {
|
|
|
|
|
self.cancelled.load(Ordering::Acquire)
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
enum CompletionState<T> {
|
|
|
|
|
Pending,
|
|
|
|
|
Ready(T),
|
|
|
|
|
Consumed,
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// One actor-owned terminal observation for a synchronous process entrypoint.
|
|
|
|
|
///
|
2026-08-27 22:09:26 +00:00
|
|
|
/// The waiting thread cannot poll or advance domain state. An actor decides
|
|
|
|
|
/// when the operation is complete and publishes the value; [`Self::wait`]
|
|
|
|
|
/// blocks until then, while [`Self::wait_deadline`] bounds the wait for
|
|
|
|
|
/// entrypoints whose failure must be observable even if the actor stalls.
|
|
|
|
|
///
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
pub struct ActorCompletion<T> {
|
|
|
|
|
inner: Arc<(Mutex<CompletionState<T>>, Condvar)>,
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl<T> Clone for ActorCompletion<T> {
|
|
|
|
|
fn clone(&self) -> Self {
|
|
|
|
|
Self {
|
|
|
|
|
inner: Arc::clone(&self.inner),
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl<T> Default for ActorCompletion<T> {
|
|
|
|
|
fn default() -> Self {
|
|
|
|
|
Self::new()
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl<T> ActorCompletion<T> {
|
|
|
|
|
pub fn new() -> Self {
|
|
|
|
|
Self {
|
|
|
|
|
inner: Arc::new((Mutex::new(CompletionState::Pending), Condvar::new())),
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Publish the terminal observation once.
|
|
|
|
|
pub fn complete(&self, value: T) -> Result<(), T> {
|
|
|
|
|
let (state, ready) = &*self.inner;
|
|
|
|
|
let mut state = state.lock();
|
|
|
|
|
if !matches!(*state, CompletionState::Pending) {
|
|
|
|
|
return Err(value);
|
|
|
|
|
}
|
|
|
|
|
*state = CompletionState::Ready(value);
|
|
|
|
|
ready.notify_all();
|
|
|
|
|
Ok(())
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Block the process entrypoint until the owning actor completes.
|
|
|
|
|
pub fn wait(&self) -> T {
|
|
|
|
|
let (state, ready) = &*self.inner;
|
|
|
|
|
let mut state = state.lock();
|
|
|
|
|
loop {
|
|
|
|
|
if matches!(*state, CompletionState::Pending) {
|
|
|
|
|
ready.wait(&mut state);
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
match std::mem::replace(&mut *state, CompletionState::Consumed) {
|
|
|
|
|
CompletionState::Ready(value) => return value,
|
|
|
|
|
CompletionState::Consumed => {
|
|
|
|
|
panic!("ActorCompletion::wait called after value was consumed")
|
|
|
|
|
}
|
|
|
|
|
CompletionState::Pending => unreachable!("pending state handled above"),
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
2026-08-27 22:09:26 +00:00
|
|
|
|
|
|
|
|
/// Block until the owning actor completes or `timeout` elapses.
|
|
|
|
|
///
|
|
|
|
|
/// Returns `None` on timeout with the completion still pending, so a
|
|
|
|
|
/// bootstrap entrypoint can fail boundedly instead of hanging forever.
|
|
|
|
|
pub fn wait_deadline(&self, timeout: Duration) -> Option<T> {
|
|
|
|
|
let (state, ready) = &*self.inner;
|
|
|
|
|
let mut state = state.lock();
|
|
|
|
|
let deadline = std::time::Instant::now() + timeout;
|
|
|
|
|
loop {
|
|
|
|
|
if !matches!(*state, CompletionState::Pending) {
|
|
|
|
|
return match std::mem::replace(&mut *state, CompletionState::Consumed) {
|
|
|
|
|
CompletionState::Ready(value) => Some(value),
|
|
|
|
|
CompletionState::Consumed => {
|
|
|
|
|
panic!("ActorCompletion::wait called after value was consumed")
|
|
|
|
|
}
|
|
|
|
|
CompletionState::Pending => unreachable!("pending state handled above"),
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
let now = std::time::Instant::now();
|
|
|
|
|
if now >= deadline {
|
|
|
|
|
return None;
|
|
|
|
|
}
|
|
|
|
|
ready.wait_until(&mut state, deadline);
|
|
|
|
|
}
|
|
|
|
|
}
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
}
|
feat(engine): substrate-neutral execution engine abstraction
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
2026-08-10 20:23:03 +00:00
|
|
|
|
|
|
|
|
impl EngineHandle {
|
|
|
|
|
/// Upgrade to the live backend, or `None` if the owning engine is gone.
|
|
|
|
|
fn backend(&self) -> Option<Arc<dyn ExecutionBackend>> {
|
|
|
|
|
self.backend.upgrade()
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Schedule `task` as cooperative engine work.
|
|
|
|
|
///
|
|
|
|
|
/// A no-op once the owning engine has been dropped: the work is discarded
|
|
|
|
|
/// rather than keeping the backend alive.
|
|
|
|
|
pub fn spawn<F>(&self, task: F)
|
|
|
|
|
where
|
|
|
|
|
F: Future<Output = ()> + Send + 'static,
|
|
|
|
|
{
|
|
|
|
|
if let Some(backend) = self.backend() {
|
|
|
|
|
backend.spawn(Box::pin(task));
|
|
|
|
|
}
|
|
|
|
|
}
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
/// Create a route to the backend's dedicated blocking-I/O pool.
|
|
|
|
|
pub fn blocking_work_sender(&self) -> BlockingWorkSender {
|
|
|
|
|
BlockingWorkSender {
|
|
|
|
|
backend: self.backend.clone(),
|
feat(engine): substrate-neutral execution engine abstraction
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
2026-08-10 20:23:03 +00:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Produce a future that completes after `delay`.
|
|
|
|
|
///
|
|
|
|
|
/// Once the owning engine has been dropped this returns a timer that never
|
|
|
|
|
/// fires.
|
|
|
|
|
pub fn timer(&self, delay: Duration) -> Timer {
|
|
|
|
|
match self.backend() {
|
2026-08-16 13:27:38 +00:00
|
|
|
Some(backend) => Timer {
|
|
|
|
|
inner: backend.timer(delay),
|
|
|
|
|
},
|
feat(engine): substrate-neutral execution engine abstraction
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
2026-08-10 20:23:03 +00:00
|
|
|
None => Timer::closed(),
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Produce a future that recurs every `period`.
|
|
|
|
|
pub fn interval(&self, period: Duration) -> Interval {
|
|
|
|
|
Interval {
|
|
|
|
|
period,
|
|
|
|
|
backend: self.backend.clone(),
|
|
|
|
|
current: None,
|
|
|
|
|
}
|
|
|
|
|
}
|
feat(myelin): enforce actor-owned control flow
Architecture enforcement:
- Install a repository-owned rustc wrapper for ordinary cargo check,
build, and test commands. Resolve compiler item identities so renamed
imports and helper wrappers cannot hide spawning, timing, blocking,
polling, thread, or runtime-driving capabilities.
- Define the execution-owner crates and reject dependencies from those
substrates back into Myelin policy. Add compile-pass and compile-fail
contracts for actor helpers, execution owners, test waits, forbidden
capabilities, suppression attempts, and owner dependency inversions.
Execution ownership:
- Add engine-owned actor timers with cancellation and generation identity,
then migrate lifecycle deadlines and protocol ticks off application
tasks. Keep networking, process output, telemetry, and blocking provider
calls in their approved I/O substrates.
- Move process spawn, wait, signal, Unix listener, and output-following
mechanics into swactor-process. Isolate Vast.ai blocking HTTP mechanics
behind its adapter while actors retain retry, recovery, and provisioning
decisions.
Myelin control flow:
- Rework manual control, worker lifecycle, provisioning, provider recovery,
job deployment, distribution, edge orchestration, and shutdown as actor
state transitions and typed effects. Preserve durable provider adoption
and command outcomes across graceful and abrupt restarts.
- Replace controller loops and timer-forwarding tasks with actor messages;
leave substrate tasks as cancellable observation streams with no durable
policy state.
Properties and resource ownership:
- Add deterministic engine and component properties, a stateful mock-VastAI
lifecycle model, persisted regression cases, controlled fault injection,
and a bounded nightly workflow covering restart and teardown behavior.
- Terminate reply observers, cancel telemetry collectors, bound dashboard
projections, and release child observers, file descriptors, process
records, and inode-verified Unix sockets on every terminal path.
Verified with the compiler-policy contracts, 105 Myelin library tests, 32
swactor-process tests, telemetry cancellation contracts, randomized
stateful restart cases, cargo check, and formatting checks.
2026-08-19 21:38:14 +00:00
|
|
|
|
|
|
|
|
/// Schedule one typed message for delivery after `delay`.
|
|
|
|
|
///
|
|
|
|
|
/// The engine owns the timer task; domain code receives no future or
|
|
|
|
|
/// scheduling callback. The actor receiving `message` owns the deadline
|
|
|
|
|
/// decision and should reject stale operation identities.
|
|
|
|
|
pub fn send_after<M>(
|
|
|
|
|
&self,
|
|
|
|
|
delay: Duration,
|
|
|
|
|
sender: ExternalSender,
|
|
|
|
|
actor: ActorAddress,
|
|
|
|
|
message: M,
|
|
|
|
|
) -> ActorTimer
|
|
|
|
|
where
|
|
|
|
|
M: Message,
|
|
|
|
|
{
|
|
|
|
|
let actor_timer = ActorTimer {
|
|
|
|
|
cancelled: Arc::new(AtomicBool::new(false)),
|
|
|
|
|
};
|
|
|
|
|
let cancelled = Arc::clone(&actor_timer.cancelled);
|
|
|
|
|
let timer = self.timer(delay);
|
|
|
|
|
self.spawn(async move {
|
|
|
|
|
timer.await;
|
|
|
|
|
if !cancelled.load(Ordering::Acquire) {
|
|
|
|
|
let _ = sender.send_to(actor, message);
|
|
|
|
|
}
|
|
|
|
|
});
|
|
|
|
|
actor_timer
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Schedule a cloned typed message after every `period`.
|
|
|
|
|
///
|
|
|
|
|
/// Delivery stops after cancellation or when the actor address no longer
|
|
|
|
|
/// accepts messages.
|
|
|
|
|
pub fn send_every<M>(
|
|
|
|
|
&self,
|
|
|
|
|
period: Duration,
|
|
|
|
|
sender: ExternalSender,
|
|
|
|
|
actor: ActorAddress,
|
|
|
|
|
message: M,
|
|
|
|
|
) -> ActorTimer
|
|
|
|
|
where
|
|
|
|
|
M: Message,
|
|
|
|
|
{
|
|
|
|
|
let actor_timer = ActorTimer {
|
|
|
|
|
cancelled: Arc::new(AtomicBool::new(false)),
|
|
|
|
|
};
|
|
|
|
|
let cancelled = Arc::clone(&actor_timer.cancelled);
|
|
|
|
|
let handle = self.clone();
|
|
|
|
|
let mut interval = Box::pin(handle.interval(period));
|
|
|
|
|
self.spawn(std::future::poll_fn(move |cx| {
|
|
|
|
|
if cancelled.load(Ordering::Acquire) {
|
|
|
|
|
return std::task::Poll::Ready(());
|
|
|
|
|
}
|
|
|
|
|
match std::future::Future::poll(interval.as_mut(), cx) {
|
|
|
|
|
std::task::Poll::Ready(()) => {
|
|
|
|
|
if cancelled.load(Ordering::Acquire)
|
|
|
|
|
|| sender.send_to(actor, message.clone()).is_err()
|
|
|
|
|
{
|
|
|
|
|
std::task::Poll::Ready(())
|
|
|
|
|
} else {
|
|
|
|
|
cx.waker().wake_by_ref();
|
|
|
|
|
std::task::Poll::Pending
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
std::task::Poll::Pending => std::task::Poll::Pending,
|
|
|
|
|
}
|
|
|
|
|
}));
|
|
|
|
|
actor_timer
|
|
|
|
|
}
|
feat(engine): substrate-neutral execution engine abstraction
Introduce the swactor engine: a swactor-owned composite that retains a
selected execution substrate, drives the core runtime, and hosts the
async/blocking/timer work that backs actors. Integrations receive one
cloneable EngineHandle and never construct or borrow a raw Tokio
runtime/handle.
Engine crate (crates/engine):
- The contract: spawn / spawn_blocking / timer / interval / now, a
per-implementation capability model with construction-time binding
(require()), and engine-owned time. The engine owns all progression;
actor handlers stay synchronous and never .await.
- TokioBackend owns the Tokio runtime and schedules core ticks and
supporting futures on it; SteppingBackend is a single-threaded
deterministic scheduler with virtual time (the non-Tokio portability
proof). Core is driven through its existing tick() surface; a
self-rescheduling CoreDriver is installed at construction and is the
sole place permitted to call try_tick.
iroh-driver:
- Receives an EngineHandle instead of a raw Tokio Handle. Accepts,
reads, dials, writes, endpoint construction, and teardown schedule
through it; required capabilities (tasks/timers/io) are validated
before the endpoint binds. Engine-hosted interval pumps drive
actor-bridge, datastream, and edge ingress.
myelin:
- One node/orchestrator engine owns core, protocol tick injection, and
transport progression; the application loop only drains
integration-owned queues. Stage-shard process readers, delayed actor
messages, helper stdout/stderr, prompt RPC, and CPU sampling all
schedule through the engine (spawn_blocking / engine tasks / timers).
- Removed the split-engine APIs: install_actor_bridge_pump(period) and
spawn_protocol_ticker(period) use each component's stored engine;
deleted the no-op pump_network callback and its plumbing; deleted the
dashboard raw-Tokio/standalone-runtime conveniences.
Enforcement:
- A clippy disallowed-methods boundary forbids direct runtime/scheduling/
time/core-driving bypasses, denied in swactor-engine, iroh-driver, and
myelin. Retained excluded uses (VastAI provider, provider process
supervision/log capture, OS-signal/stdin/process-control sequencing)
carry narrow allowances with reasons.
Verification:
- Engine contract + unit tests (incl. the SteppingBackend portability
proof), iroh integration tests (capability rejection before binding,
multi-node actor behavior), and a production execution-composition
smoke test that observes engine-driven actor progress with no ambient
Tokio runtime and no manual tick/pump. Workspace all-target/all-feature
clippy and tests are green.
Specs co-located with their crates: ENGINE_SPEC.md in crates/engine,
IROH_DRIVER_SPEC.md in crates/iroh-driver. VastAI remains explicitly out
of scope pending its separate redesign.
2026-08-10 20:23:03 +00:00
|
|
|
/// Race `future` against an engine timer.
|
|
|
|
|
///
|
|
|
|
|
/// Resolves to `Ok` with the future's output if it completes within
|
|
|
|
|
/// `duration`, or [`Err(Elapsed)`](crate::Elapsed) when the timer fires
|
|
|
|
|
pub fn timeout<F: std::future::Future>(&self, duration: Duration, future: F) -> Timeout<F> {
|
|
|
|
|
Timeout::new(self.timer(duration), future)
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Read the engine's monotonic clock.
|
|
|
|
|
///
|
|
|
|
|
/// Falls back to the real wall clock once the owning engine has been
|
|
|
|
|
/// dropped, since the substrate clock is no longer available.
|
|
|
|
|
pub fn now(&self) -> EngineInstant {
|
|
|
|
|
match self.backend() {
|
|
|
|
|
Some(backend) => backend.now(),
|
|
|
|
|
None => EngineInstant::now(),
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Report the backend's advertised capabilities.
|
|
|
|
|
///
|
|
|
|
|
/// Reports no capabilities once the owning engine has been dropped.
|
|
|
|
|
pub fn capabilities(&self) -> Capabilities {
|
|
|
|
|
match self.backend() {
|
|
|
|
|
Some(backend) => backend.capabilities(),
|
|
|
|
|
None => Capabilities::NONE,
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Validate that this engine satisfies `required` before starting work.
|
|
|
|
|
///
|
|
|
|
|
/// Returns `Err` if the backend cannot provide a requested capability, or
|
|
|
|
|
/// if the owning engine has been dropped. Call this before allocating
|
|
|
|
|
/// resources, starting background work, or becoming externally visible so
|
|
|
|
|
/// that an incompatible engine is rejected early (ENGINE_SPEC.md).
|
|
|
|
|
pub fn require(&self, required: Capabilities) -> Result<(), EngineError> {
|
|
|
|
|
match self.backend() {
|
|
|
|
|
Some(backend) if backend.capabilities().satisfies(required) => Ok(()),
|
|
|
|
|
_ => Err(EngineError::MissingRequiredCapability),
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|