fix: these docs are stale
This commit is contained in:
Developer
parent
669599a9e7
commit
1c1d3fad02
6 changed files with 0 additions and 1776 deletions
|
|
@ -1,335 +0,0 @@
|
|||
# Actor Watching — Distributed Death Notifications
|
||||
|
||||
## Problem
|
||||
|
||||
When an actor dies (panic, explicit stop, or its host node leaves the cluster), other actors that depend on it have no way to know. This is the distributed equivalent of `waitpid()` / `SIGCHLD` — the foundational primitive for building supervision, reconnection logic, and self-healing.
|
||||
|
||||
## Design
|
||||
|
||||
### Types
|
||||
|
||||
```rust
|
||||
// src/actor.rs
|
||||
|
||||
/// Why an actor exited.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
|
||||
pub enum ExitReason {
|
||||
/// Actor was explicitly stopped or removed from the pool.
|
||||
Stopped,
|
||||
/// Actor panicked during message handling.
|
||||
Panicked,
|
||||
/// The node hosting the actor left the cluster (SWIM Dead).
|
||||
NodeDown,
|
||||
}
|
||||
|
||||
/// Delivered to watchers when a watched actor exits.
|
||||
///
|
||||
/// Implements `Message` (Clone + Send + Sync + 'static) so it can be
|
||||
/// delivered through normal mailbox channels.
|
||||
#[derive(Debug, Clone)]
|
||||
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
|
||||
pub struct ActorExited {
|
||||
/// The address of the actor that died.
|
||||
pub addr: ActorAddress,
|
||||
/// Why it exited.
|
||||
pub reason: ExitReason,
|
||||
}
|
||||
```
|
||||
|
||||
### API
|
||||
|
||||
```rust
|
||||
// src/actor.rs — extend ContextInner
|
||||
|
||||
pub trait ContextInner {
|
||||
// existing:
|
||||
fn send_any(&self, addr: ActorAddress, msg: Box<dyn Any + Send>) -> Result<(), Error>;
|
||||
fn spawn_any(&self, addr: ActorAddress, actor: Box<dyn AnyActor>);
|
||||
|
||||
// new:
|
||||
fn watch(&self, watcher: ActorAddress, target: ActorAddress);
|
||||
fn unwatch(&self, watcher: ActorAddress, target: ActorAddress);
|
||||
}
|
||||
|
||||
// src/actor.rs — extend Ctx
|
||||
|
||||
impl Ctx<'_> {
|
||||
/// Watch another actor's liveness. If the target dies, this actor
|
||||
/// receives an `ActorExited` message in its mailbox.
|
||||
///
|
||||
/// Watching an already-dead or non-existent actor delivers
|
||||
/// `ActorExited { reason: Stopped }` immediately (on next tick).
|
||||
///
|
||||
/// Calling watch() multiple times on the same target is idempotent —
|
||||
/// only one notification is delivered.
|
||||
pub fn watch(&self, target: ActorAddress) {
|
||||
self.inner.watch(self.self_addr, target);
|
||||
}
|
||||
|
||||
/// Stop watching an actor. No notification will be delivered if the
|
||||
/// target subsequently dies.
|
||||
pub fn unwatch(&self, target: ActorAddress) {
|
||||
self.inner.unwatch(self.self_addr, target);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### WatchRegistry
|
||||
|
||||
A per-worker structure that tracks watch relationships:
|
||||
|
||||
```rust
|
||||
// src/worker.rs
|
||||
|
||||
struct WatchRegistry {
|
||||
/// target -> set of watchers awaiting death notification
|
||||
watchers: HashMap<ActorAddress, HashSet<ActorAddress>>,
|
||||
/// watcher -> set of targets it's watching (reverse index for cleanup)
|
||||
watching: HashMap<ActorAddress, HashSet<ActorAddress>>,
|
||||
}
|
||||
|
||||
impl WatchRegistry {
|
||||
fn watch(&mut self, watcher: ActorAddress, target: ActorAddress) {
|
||||
self.watchers.entry(target).or_default().insert(watcher);
|
||||
self.watching.entry(watcher).or_default().insert(target);
|
||||
}
|
||||
|
||||
fn unwatch(&mut self, watcher: ActorAddress, target: ActorAddress) {
|
||||
if let Some(set) = self.watchers.get_mut(&target) {
|
||||
set.remove(&watcher);
|
||||
if set.is_empty() { self.watchers.remove(&target); }
|
||||
}
|
||||
if let Some(set) = self.watching.get_mut(&watcher) {
|
||||
set.remove(&target);
|
||||
if set.is_empty() { self.watching.remove(&watcher); }
|
||||
}
|
||||
}
|
||||
|
||||
/// Called when an actor dies. Returns (watcher_addr, ActorExited) pairs.
|
||||
fn notify_death(&mut self, target: ActorAddress, reason: ExitReason)
|
||||
-> Vec<(ActorAddress, ActorExited)>
|
||||
{
|
||||
let notification = ActorExited { addr: target, reason };
|
||||
let mut result = Vec::new();
|
||||
|
||||
if let Some(watcher_set) = self.watchers.remove(&target) {
|
||||
for watcher in &watcher_set {
|
||||
result.push((*watcher, notification.clone()));
|
||||
// clean up reverse index
|
||||
if let Some(set) = self.watching.get_mut(watcher) {
|
||||
set.remove(&target);
|
||||
if set.is_empty() { self.watching.remove(watcher); }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
result
|
||||
}
|
||||
|
||||
/// Called when a watcher itself dies. Cleans up all its watching entries.
|
||||
fn cleanup_watcher(&mut self, watcher: &ActorAddress) {
|
||||
if let Some(targets) = self.watching.remove(watcher) {
|
||||
for target in targets {
|
||||
if let Some(set) = self.watchers.get_mut(&target) {
|
||||
set.remove(watcher);
|
||||
if set.is_empty() { self.watchers.remove(&target); }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Integration with tick_once
|
||||
|
||||
The worker's `tick_once` gains **two new mechanisms**:
|
||||
|
||||
**A. Watch/unwatch request processing** — `WorkerContext::watch()` needs to route the request to the correct worker (the one that owns the target). This mirrors how sends work:
|
||||
|
||||
```
|
||||
watch(watcher_A, target_B) called on worker 0:
|
||||
- target_B is on worker 0? → register locally in WatchRegistry
|
||||
- target_B is on worker 1? → send WatchCommand through transfer queue
|
||||
- target_B not in address_map? → may be remote (see Remote Watches below)
|
||||
```
|
||||
|
||||
New envelope variant for internal watch commands:
|
||||
|
||||
```rust
|
||||
// src/delivery.rs
|
||||
|
||||
enum InternalCommand {
|
||||
Watch { watcher: ActorAddress, target: ActorAddress },
|
||||
Unwatch { watcher: ActorAddress, target: ActorAddress },
|
||||
}
|
||||
```
|
||||
|
||||
These are delivered through the existing transfer queue alongside `Envelope`s. The transfer queue type becomes `enum TransferItem { Message(Envelope), Command(InternalCommand) }`, or — simpler — the WatchRegistry is shared (behind Arc<Mutex>) and watch/unwatch are applied directly. The shared approach is better since watches are rare relative to messages.
|
||||
|
||||
**Recommended**: `Arc<Mutex<WatchRegistry>>` shared across workers, owned by Runtime. Workers hold a reference. Contention is negligible because watch/unwatch operations are rare.
|
||||
|
||||
**B. Death notification dispatch** — added to tick_all's panic detection:
|
||||
|
||||
```rust
|
||||
// In ActorPool::tick_all, after catching a panic:
|
||||
Err(_) => {
|
||||
slot.poisoned = true;
|
||||
slot.mailbox.clear();
|
||||
// NEW: collect death notification
|
||||
deaths.push((addr, ExitReason::Panicked));
|
||||
}
|
||||
```
|
||||
|
||||
After tick_all completes, the worker processes `deaths`:
|
||||
|
||||
```rust
|
||||
// In tick_once, after tick_all:
|
||||
for (addr, reason) in deaths {
|
||||
let notifications = watch_registry.lock().notify_death(addr, reason);
|
||||
for (watcher_addr, msg) in notifications {
|
||||
// Deliver ActorExited as a normal message
|
||||
self.deliver_to(watcher_addr, Box::new(msg), tc);
|
||||
}
|
||||
// Also clean up the dead actor's own watches
|
||||
watch_registry.lock().cleanup_watcher(&addr);
|
||||
}
|
||||
```
|
||||
|
||||
### Watching Non-Existent Actors
|
||||
|
||||
If `watch(watcher, target)` is called and `target` doesn't exist in the address map:
|
||||
|
||||
- **Local runtime**: deliver `ActorExited { reason: Stopped }` immediately (on next tick). The actor is already gone.
|
||||
- **Distributed**: the watch request is forwarded to the node that should own the target (via Kademlia resolution). If the target doesn't exist there either, a `ActorExitedNotify` is sent back.
|
||||
|
||||
### Remote Watches (cross-node)
|
||||
|
||||
Wire protocol additions in `crates/distribution/src/messages.rs`:
|
||||
|
||||
```rust
|
||||
/// Request from node A to node B: "notify me if this actor dies"
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct WatchRequest {
|
||||
pub watcher_node: NodeId,
|
||||
pub watcher_addr: ActorAddress,
|
||||
pub target_addr: ActorAddress,
|
||||
}
|
||||
impl NetworkMessage for WatchRequest {
|
||||
fn type_tag() -> &'static str { "swactor_dist::WatchRequest" }
|
||||
}
|
||||
|
||||
/// Request to cancel a remote watch
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct UnwatchRequest {
|
||||
pub watcher_node: NodeId,
|
||||
pub watcher_addr: ActorAddress,
|
||||
pub target_addr: ActorAddress,
|
||||
}
|
||||
impl NetworkMessage for UnwatchRequest {
|
||||
fn type_tag() -> &'static str { "swactor_dist::UnwatchRequest" }
|
||||
}
|
||||
|
||||
/// Notification from target's node to watcher's node
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct ActorExitedNotify {
|
||||
pub target_addr: ActorAddress,
|
||||
pub reason: ExitReason,
|
||||
/// Which watchers on the receiving node should be notified
|
||||
pub watchers: Vec<ActorAddress>,
|
||||
}
|
||||
impl NetworkMessage for ActorExitedNotify {
|
||||
fn type_tag() -> &'static str { "swactor_dist::ActorExitedNotify" }
|
||||
}
|
||||
```
|
||||
|
||||
**Flow**:
|
||||
|
||||
```
|
||||
Node A Node B
|
||||
│ │
|
||||
│ ctx.watch(watcher_A, target_B) │
|
||||
│ │
|
||||
│ ── WatchRequest ──────────────► │
|
||||
│ │ registers remote watch:
|
||||
│ │ target_B → (NodeA, watcher_A)
|
||||
│ │
|
||||
│ ... time passes ... │
|
||||
│ │
|
||||
│ │ target_B panics
|
||||
│ │
|
||||
│ ◄── ActorExitedNotify ──────── │
|
||||
│ │
|
||||
│ delivers ActorExited to │
|
||||
│ watcher_A's mailbox │
|
||||
```
|
||||
|
||||
The `WatchRegistry` on node B stores remote watches with the additional `NodeId` of the watcher's node. On death, it partitions notifications into local (deliver directly) and remote (send `ActorExitedNotify` to the watcher's node).
|
||||
|
||||
### SWIM Integration
|
||||
|
||||
When `handle_membership_change` detects `MemberState::Dead` (`crates/distribution/src/node.rs:269`):
|
||||
|
||||
1. The node maintains a **node actor index**: `NodeId -> Set<ActorAddress>` — all actors known to be on each node. This is populated from:
|
||||
- Directory entries stored locally
|
||||
- Cache entries
|
||||
- Remote watch registrations
|
||||
|
||||
2. On node death:
|
||||
```rust
|
||||
fn handle_membership_change(&mut self, node_id: NodeId, state: MemberState) {
|
||||
match state {
|
||||
MemberState::Dead => {
|
||||
// existing cleanup...
|
||||
self.routing_table.remove(&node_id);
|
||||
self.cache.invalidate_node(&node_id);
|
||||
self.repair_queue.on_node_death(&node_id, &mut self.directory);
|
||||
|
||||
// NEW: notify local watchers about all actors on the dead node
|
||||
if let Some(actor_addrs) = self.node_actor_index.remove(&node_id) {
|
||||
for actor_addr in actor_addrs {
|
||||
let notifications = self.watch_registry.notify_death(
|
||||
actor_addr,
|
||||
ExitReason::NodeDown,
|
||||
);
|
||||
// Queue for delivery to local actors
|
||||
self.pending_exit_notifications.extend(notifications);
|
||||
}
|
||||
}
|
||||
}
|
||||
// ...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
3. The `pending_exit_notifications` are drained by the NodeDriver on the next tick and delivered into the local runtime.
|
||||
|
||||
### Edge Cases
|
||||
|
||||
| Scenario | Behavior |
|
||||
|---|---|
|
||||
| Watch self | Allowed. On death, ActorExited delivered to own mailbox (no-op since dead). |
|
||||
| Watch already-dead actor | `ActorExited { reason: Stopped }` delivered on next tick. |
|
||||
| Watcher dies before target | Cleanup removes all watching entries. No notification delivered. |
|
||||
| Target node suspected (not yet dead) | No notification — wait for SWIM to confirm Dead or Alive. |
|
||||
| Network partition heals | If target was falsely declared Dead, a stale `NodeDown` was sent. The watcher may re-watch. No automatic "un-death" notification. |
|
||||
| Double watch | Idempotent — only one notification per death event. |
|
||||
|
||||
### Files Modified
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `src/actor.rs` | `ExitReason`, `ActorExited`, `watch()`/`unwatch()` on `ContextInner`, `Ctx` |
|
||||
| `src/worker.rs` | `WatchRegistry`, death collection in `tick_all`, notification dispatch in `tick_once` |
|
||||
| `src/runtime.rs` | `Arc<Mutex<WatchRegistry>>` owned by Runtime, passed to workers |
|
||||
| `crates/distribution/src/messages.rs` | `WatchRequest`, `UnwatchRequest`, `ActorExitedNotify` |
|
||||
| `crates/distribution/src/node.rs` | Node actor index, SWIM Dead fan-out, `pending_exit_notifications` |
|
||||
|
||||
### Tests
|
||||
|
||||
- **watch_local_death**: spawn watcher + target, kill target (panic), verify watcher receives `ActorExited { reason: Panicked }`
|
||||
- **unwatch_prevents_notification**: watch then unwatch, kill target, verify no notification
|
||||
- **watch_nonexistent**: watch an address that was never spawned, verify `ActorExited { reason: Stopped }`
|
||||
- **watcher_dies_first**: watch target, kill watcher, kill target — no panic/leak
|
||||
- **cross_worker_watch**: target on worker 0, watcher on worker 1, kill target, verify notification arrives
|
||||
- **idempotent_watch**: watch same target twice, kill target, verify exactly one notification
|
||||
|
|
@ -1,321 +0,0 @@
|
|||
# Cluster-Wide Registry — Distributed Naming
|
||||
|
||||
## Problem
|
||||
|
||||
Actors can only be found by their `ActorAddress` (a random 32-byte ID). The local `AddressMap` maps addresses to workers on a single node. The Kademlia directory maps addresses to `NodeId`. But neither provides **human-readable naming** or **re-discovery after churn**.
|
||||
|
||||
When a node dies and an actor is re-spawned elsewhere, it gets a new `ActorAddress`. Without a name-based registry, every actor that communicated with it needs manual reconfiguration. This doesn't work for churning infrastructure.
|
||||
|
||||
## Design
|
||||
|
||||
### Approach: Gossip-Propagated LWW-Register CRDT
|
||||
|
||||
Each name binding is a **Last-Writer-Wins Register** — the most recent write (by timestamp) wins. This matches SWIM's eventual-consistency model and reuses the existing gossip piggyback mechanism.
|
||||
|
||||
**Why not Raft/consensus?**
|
||||
- Overkill for name resolution. Names don't need linearizability — eventual consistency is fine.
|
||||
- SWIM already solves dissemination. We piggyback registry updates on existing protocol messages for free.
|
||||
- Consensus requires a stable quorum, which conflicts with the "nodes pop in and out" use case.
|
||||
|
||||
**Why not extend Kademlia?**
|
||||
- Kademlia maps `ActorAddress -> NodeId`. Names are a different key space (`String -> ActorAddress`).
|
||||
- Kademlia lookups are multi-hop (iterative). Registry lookups should be local (every node has a full replica).
|
||||
- The registry is small (hundreds to low-thousands of names). Full replication is cheap.
|
||||
|
||||
### Types
|
||||
|
||||
```rust
|
||||
// crates/distribution/src/registry.rs
|
||||
|
||||
/// A single name binding in the cluster registry.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct RegistryEntry {
|
||||
/// Human-readable name (e.g. "worker-pool", "metrics-collector").
|
||||
pub name: String,
|
||||
/// The actor address this name resolves to.
|
||||
pub actor_addr: ActorAddress,
|
||||
/// The node that owns this binding.
|
||||
pub node_id: NodeId,
|
||||
/// Logical timestamp for LWW conflict resolution.
|
||||
pub timestamp: u64,
|
||||
/// Generation — incremented on re-registration of the same name.
|
||||
pub generation: u64,
|
||||
/// Tombstone — true means the name has been unregistered.
|
||||
pub tombstone: bool,
|
||||
}
|
||||
|
||||
/// Events emitted by the registry for subscribers.
|
||||
#[derive(Debug, Clone)]
|
||||
pub enum RegistryEvent {
|
||||
/// A name was registered or updated.
|
||||
Registered {
|
||||
name: String,
|
||||
actor_addr: ActorAddress,
|
||||
node_id: NodeId,
|
||||
},
|
||||
/// A name was unregistered (tombstoned).
|
||||
Unregistered {
|
||||
name: String,
|
||||
previous_addr: ActorAddress,
|
||||
},
|
||||
}
|
||||
|
||||
/// The local replica of the cluster-wide registry.
|
||||
pub struct ClusterRegistry {
|
||||
/// Current state: name -> latest entry.
|
||||
entries: HashMap<String, RegistryEntry>,
|
||||
/// Pending entries to propagate via gossip (not yet disseminated to all).
|
||||
pending: VecDeque<RegistryEntry>,
|
||||
/// Logical clock for this node.
|
||||
clock: u64,
|
||||
/// Recent events for subscribers.
|
||||
events: VecDeque<RegistryEvent>,
|
||||
/// Max events to buffer.
|
||||
max_events: usize,
|
||||
}
|
||||
```
|
||||
|
||||
### CRDT Merge Rule
|
||||
|
||||
```rust
|
||||
impl ClusterRegistry {
|
||||
/// Merge a remote entry. Returns true if the local state changed.
|
||||
pub fn merge(&mut self, remote: RegistryEntry) -> bool {
|
||||
match self.entries.get(&remote.name) {
|
||||
Some(local) => {
|
||||
// LWW: higher timestamp wins.
|
||||
// Tie-break: higher generation, then higher node_id (deterministic).
|
||||
let dominated = remote.timestamp > local.timestamp
|
||||
|| (remote.timestamp == local.timestamp
|
||||
&& remote.generation > local.generation)
|
||||
|| (remote.timestamp == local.timestamp
|
||||
&& remote.generation == local.generation
|
||||
&& remote.node_id.0 > local.node_id.0);
|
||||
|
||||
if dominated {
|
||||
self.apply(remote);
|
||||
true
|
||||
} else {
|
||||
false
|
||||
}
|
||||
}
|
||||
None => {
|
||||
self.apply(remote);
|
||||
true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn apply(&mut self, entry: RegistryEntry) {
|
||||
let event = if entry.tombstone {
|
||||
let prev = self.entries.get(&entry.name)
|
||||
.map(|e| e.actor_addr);
|
||||
RegistryEvent::Unregistered {
|
||||
name: entry.name.clone(),
|
||||
previous_addr: prev.unwrap_or_default(),
|
||||
}
|
||||
} else {
|
||||
RegistryEvent::Registered {
|
||||
name: entry.name.clone(),
|
||||
actor_addr: entry.actor_addr,
|
||||
node_id: entry.node_id,
|
||||
}
|
||||
};
|
||||
self.events.push_back(event);
|
||||
if self.events.len() > self.max_events {
|
||||
self.events.pop_front();
|
||||
}
|
||||
self.entries.insert(entry.name.clone(), entry);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### API
|
||||
|
||||
On `DistributedNode`:
|
||||
|
||||
```rust
|
||||
// crates/distribution/src/node.rs
|
||||
|
||||
impl DistributedNode {
|
||||
/// Register a name -> actor binding on this node.
|
||||
/// The binding is propagated to all cluster members via gossip.
|
||||
pub fn register_name(&mut self, name: &str, actor_addr: ActorAddress) {
|
||||
self.registry.clock += 1;
|
||||
let entry = RegistryEntry {
|
||||
name: name.to_string(),
|
||||
actor_addr,
|
||||
node_id: self.node_id(),
|
||||
timestamp: self.registry.clock,
|
||||
generation: self.registry.next_generation(name),
|
||||
tombstone: false,
|
||||
};
|
||||
self.registry.merge(entry.clone());
|
||||
self.registry.pending.push_back(entry);
|
||||
}
|
||||
|
||||
/// Remove a name binding. Propagated as a tombstone.
|
||||
pub fn unregister_name(&mut self, name: &str) {
|
||||
self.registry.clock += 1;
|
||||
let actor_addr = self.registry.entries.get(name)
|
||||
.map(|e| e.actor_addr)
|
||||
.unwrap_or_default();
|
||||
let entry = RegistryEntry {
|
||||
name: name.to_string(),
|
||||
actor_addr,
|
||||
node_id: self.node_id(),
|
||||
timestamp: self.registry.clock,
|
||||
generation: 0,
|
||||
tombstone: true,
|
||||
};
|
||||
self.registry.merge(entry.clone());
|
||||
self.registry.pending.push_back(entry);
|
||||
}
|
||||
|
||||
/// Resolve a name to an actor address (local replica, eventually consistent).
|
||||
pub fn resolve_name(&self, name: &str) -> Option<(ActorAddress, NodeId)> {
|
||||
self.registry.entries.get(name)
|
||||
.filter(|e| !e.tombstone)
|
||||
.map(|e| (e.actor_addr, e.node_id))
|
||||
}
|
||||
|
||||
/// Drain buffered registry events (for subscribers).
|
||||
pub fn registry_events(&mut self) -> Vec<RegistryEvent> {
|
||||
self.registry.events.drain(..).collect()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
On `Ctx` (actor-level, requires distribution feature):
|
||||
|
||||
```rust
|
||||
// src/actor.rs — requires ContextInner extensions
|
||||
|
||||
impl Ctx<'_> {
|
||||
/// Register this actor under a name in the cluster registry.
|
||||
pub fn register_as(&self, name: &str) {
|
||||
self.inner.register_name(self.self_addr, name);
|
||||
}
|
||||
|
||||
/// Resolve a name to an actor address.
|
||||
pub fn resolve_name(&self, name: &str) -> Option<ActorAddress> {
|
||||
self.inner.resolve_name(name)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `ContextInner` trait gains two new methods:
|
||||
|
||||
```rust
|
||||
pub trait ContextInner {
|
||||
// ... existing methods ...
|
||||
fn register_name(&self, addr: ActorAddress, name: &str) { /* default no-op */ }
|
||||
fn resolve_name(&self, name: &str) -> Option<ActorAddress> { None }
|
||||
}
|
||||
```
|
||||
|
||||
Default implementations return `None` / no-op so that non-distributed runtimes don't break.
|
||||
|
||||
### Gossip Propagation
|
||||
|
||||
Registry entries are piggybacked on SWIM protocol messages, reusing the existing dissemination mechanism.
|
||||
|
||||
Currently, `crates/distribution/src/swim/dissemination.rs` encodes membership updates into the piggyback payload:
|
||||
|
||||
```
|
||||
piggyback bytes = bincode(Vec<MembershipUpdate>)
|
||||
```
|
||||
|
||||
Extended format:
|
||||
|
||||
```
|
||||
piggyback bytes = bincode(PiggybackPayload {
|
||||
membership: Vec<MembershipUpdate>,
|
||||
registry: Vec<RegistryEntry>, // NEW
|
||||
})
|
||||
```
|
||||
|
||||
```rust
|
||||
// crates/distribution/src/swim/dissemination.rs
|
||||
|
||||
#[derive(Serialize, Deserialize)]
|
||||
struct PiggybackPayload {
|
||||
membership: Vec<MembershipUpdate>,
|
||||
registry: Vec<RegistryEntry>,
|
||||
}
|
||||
```
|
||||
|
||||
The dissemination buffer manages registry entries the same way as membership updates:
|
||||
- Each entry has a dissemination count (how many times it's been piggybacked).
|
||||
- After `log2(N) + 1` disseminations (where N = cluster size), the entry is retired.
|
||||
- Piggyback space is shared: membership updates take priority, registry entries fill remaining space.
|
||||
|
||||
### Node Death Handling
|
||||
|
||||
When SWIM marks a node as `Dead`:
|
||||
|
||||
```rust
|
||||
fn handle_membership_change(&mut self, node_id: NodeId, state: MemberState) {
|
||||
if state == MemberState::Dead {
|
||||
// ... existing cleanup ...
|
||||
|
||||
// NEW: tombstone all registry entries owned by the dead node
|
||||
let to_tombstone: Vec<String> = self.registry.entries.iter()
|
||||
.filter(|(_, e)| e.node_id == node_id && !e.tombstone)
|
||||
.map(|(name, _)| name.clone())
|
||||
.collect();
|
||||
|
||||
for name in to_tombstone {
|
||||
self.registry.clock += 1;
|
||||
let entry = RegistryEntry {
|
||||
name: name.clone(),
|
||||
tombstone: true,
|
||||
timestamp: self.registry.clock,
|
||||
// ... fill from existing entry ...
|
||||
};
|
||||
self.registry.merge(entry.clone());
|
||||
self.registry.pending.push_back(entry);
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Interaction with Actor Watching
|
||||
|
||||
The registry and watching system compose naturally:
|
||||
|
||||
1. Actor A resolves name "service-X" → gets address B on Node 2.
|
||||
2. Actor A calls `ctx.watch(B)`.
|
||||
3. Node 2 dies. Actor A receives `ActorExited { addr: B, reason: NodeDown }`.
|
||||
4. A supervisor re-spawns "service-X" on Node 3 → new address C.
|
||||
5. The supervisor calls `register_name("service-X", C)`.
|
||||
6. Gossip propagates the update.
|
||||
7. Actor A (or anyone) calls `resolve_name("service-X")` → gets address C.
|
||||
8. Actor A calls `ctx.watch(C)` to resume monitoring.
|
||||
|
||||
### Tombstone Garbage Collection
|
||||
|
||||
Tombstones accumulate over time. GC strategy:
|
||||
|
||||
- Tombstones older than `tombstone_ttl` (default: 1 hour of logical clock ticks) are eligible for removal.
|
||||
- GC runs periodically (e.g., every 1000 ticks).
|
||||
- A tombstone is only removed if it has been fully disseminated (dissemination count >= threshold).
|
||||
|
||||
### Files Modified
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `crates/distribution/src/registry.rs` | **New file**: `ClusterRegistry`, `RegistryEntry`, `RegistryEvent`, CRDT merge |
|
||||
| `crates/distribution/src/lib.rs` | `pub mod registry;` |
|
||||
| `crates/distribution/src/node.rs` | `register_name`, `unregister_name`, `resolve_name`, node death tombstoning |
|
||||
| `crates/distribution/src/swim/dissemination.rs` | `PiggybackPayload` extended with registry entries |
|
||||
| `src/actor.rs` | `register_name`/`resolve_name` on `ContextInner` (default no-op), `Ctx` wrappers |
|
||||
|
||||
### Tests
|
||||
|
||||
- **register_and_resolve**: register a name, resolve it, verify correct address
|
||||
- **lww_conflict**: two nodes register same name concurrently, verify latest timestamp wins
|
||||
- **tombstone_propagation**: register name, unregister, verify tombstone propagates and resolve returns None
|
||||
- **node_death_tombstones**: 3-node cluster, register name on node B, kill node B, verify name is tombstoned on surviving nodes
|
||||
- **re_registration**: register name, unregister, re-register with new address, verify resolution
|
||||
- **gossip_convergence**: register name on node A, verify all nodes resolve it after gossip settles
|
||||
|
|
@ -1,306 +0,0 @@
|
|||
# Node Capabilities — Hardware Detection & Placement Constraints
|
||||
|
||||
## Problem
|
||||
|
||||
Swactor targets heterogeneous clusters: some nodes have GPUs, others have large RAM, others are lightweight ARM devices. When spawning an actor (e.g., a model inference worker), the system needs to place it on a node with the right hardware. Today, placement is round-robin — no awareness of what each node can do.
|
||||
|
||||
## Design
|
||||
|
||||
### Separate Crate
|
||||
|
||||
`crates/capabilities/` is a **standalone crate** with no dependency on the swactor core runtime. It's a pure detection + constraint-matching library.
|
||||
|
||||
```toml
|
||||
# crates/capabilities/Cargo.toml
|
||||
[package]
|
||||
name = "swactor-capabilities"
|
||||
version = "0.1.0"
|
||||
edition = "2024"
|
||||
|
||||
[features]
|
||||
default = ["detect"]
|
||||
detect = ["dep:sysinfo"]
|
||||
gpu-nvidia = []
|
||||
# gpu-vulkan = [] # future
|
||||
|
||||
[dependencies]
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
sysinfo = { version = "0.33", optional = true }
|
||||
```
|
||||
|
||||
### Types
|
||||
|
||||
```rust
|
||||
// crates/capabilities/src/lib.rs
|
||||
|
||||
/// A capability value. Kept simple — three variants cover all practical needs.
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub enum CapValue {
|
||||
Bool(bool),
|
||||
Int(i64),
|
||||
Str(String),
|
||||
}
|
||||
|
||||
/// All capabilities of a node.
|
||||
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
|
||||
pub struct NodeCapabilities {
|
||||
labels: BTreeMap<String, CapValue>,
|
||||
}
|
||||
|
||||
impl NodeCapabilities {
|
||||
pub fn new() -> Self { Self::default() }
|
||||
|
||||
/// Get a label value.
|
||||
pub fn get(&self, key: &str) -> Option<&CapValue> {
|
||||
self.labels.get(key)
|
||||
}
|
||||
|
||||
/// Set a label.
|
||||
pub fn set(&mut self, key: impl Into<String>, value: CapValue) {
|
||||
self.labels.insert(key.into(), value);
|
||||
}
|
||||
|
||||
/// Merge in additional labels (overwriting on conflict).
|
||||
pub fn with_labels(mut self, extra: BTreeMap<String, CapValue>) -> Self {
|
||||
self.labels.extend(extra);
|
||||
self
|
||||
}
|
||||
|
||||
/// Check if all constraints in a requirement are satisfied.
|
||||
pub fn satisfies(&self, requirement: &PlacementRequirement) -> bool {
|
||||
requirement.constraints.iter().all(|c| self.satisfies_one(c))
|
||||
}
|
||||
|
||||
fn satisfies_one(&self, constraint: &PlacementConstraint) -> bool {
|
||||
match constraint {
|
||||
PlacementConstraint::Equals(key, expected) => {
|
||||
self.labels.get(key.as_str()) == Some(expected)
|
||||
}
|
||||
PlacementConstraint::MinInt(key, min) => {
|
||||
matches!(self.labels.get(key.as_str()), Some(CapValue::Int(v)) if *v >= *min)
|
||||
}
|
||||
PlacementConstraint::HasLabel(key) => {
|
||||
self.labels.contains_key(key.as_str())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// All labels as a reference.
|
||||
pub fn labels(&self) -> &BTreeMap<String, CapValue> {
|
||||
&self.labels
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Auto-Detection
|
||||
|
||||
```rust
|
||||
impl NodeCapabilities {
|
||||
/// Auto-detect system capabilities.
|
||||
/// Always detects arch and os. Feature-gated backends detect more.
|
||||
pub fn detect() -> Self {
|
||||
let mut caps = Self::new();
|
||||
|
||||
// Always available (no feature gate)
|
||||
caps.set("arch", CapValue::Str(std::env::consts::ARCH.to_string()));
|
||||
caps.set("os", CapValue::Str(std::env::consts::OS.to_string()));
|
||||
|
||||
#[cfg(feature = "detect")]
|
||||
{
|
||||
Self::detect_sysinfo(&mut caps);
|
||||
}
|
||||
|
||||
#[cfg(feature = "gpu-nvidia")]
|
||||
{
|
||||
Self::detect_nvidia(&mut caps);
|
||||
}
|
||||
|
||||
caps
|
||||
}
|
||||
|
||||
#[cfg(feature = "detect")]
|
||||
fn detect_sysinfo(caps: &mut Self) {
|
||||
use sysinfo::System;
|
||||
let sys = System::new_all();
|
||||
|
||||
caps.set("cpu_count", CapValue::Int(sys.cpus().len() as i64));
|
||||
caps.set("ram_mb", CapValue::Int((sys.total_memory() / (1024 * 1024)) as i64));
|
||||
|
||||
if let Ok(hostname) = hostname::get() {
|
||||
if let Some(name) = hostname.to_str() {
|
||||
caps.set("hostname", CapValue::Str(name.to_string()));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "gpu-nvidia")]
|
||||
fn detect_nvidia(caps: &mut Self) {
|
||||
// Shell out to nvidia-smi for maximum compatibility.
|
||||
// Parsing XML output is more robust than CSV for varying driver versions.
|
||||
let output = std::process::Command::new("nvidia-smi")
|
||||
.args(["--query-gpu=name,memory.total", "--format=csv,noheader,nounits"])
|
||||
.output();
|
||||
|
||||
match output {
|
||||
Ok(out) if out.status.success() => {
|
||||
let stdout = String::from_utf8_lossy(&out.stdout);
|
||||
let lines: Vec<&str> = stdout.trim().lines().collect();
|
||||
caps.set("gpu_nvidia", CapValue::Bool(true));
|
||||
caps.set("gpu_count", CapValue::Int(lines.len() as i64));
|
||||
// First GPU's VRAM as representative
|
||||
if let Some(line) = lines.first() {
|
||||
let parts: Vec<&str> = line.split(", ").collect();
|
||||
if let Some(name) = parts.first() {
|
||||
caps.set("gpu_name", CapValue::Str(name.trim().to_string()));
|
||||
}
|
||||
if let Some(vram) = parts.get(1).and_then(|s| s.trim().parse::<i64>().ok()) {
|
||||
caps.set("gpu_vram_mb", CapValue::Int(vram));
|
||||
}
|
||||
}
|
||||
}
|
||||
_ => {
|
||||
caps.set("gpu_nvidia", CapValue::Bool(false));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Placement Constraints
|
||||
|
||||
```rust
|
||||
/// A single constraint on node capabilities.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub enum PlacementConstraint {
|
||||
/// Label must exist and equal the given value.
|
||||
Equals(String, CapValue),
|
||||
/// Label must exist and be >= the given integer value.
|
||||
MinInt(String, i64),
|
||||
/// Label must exist (any value).
|
||||
HasLabel(String),
|
||||
}
|
||||
|
||||
/// A full placement requirement. All constraints must be satisfied (AND).
|
||||
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
|
||||
pub struct PlacementRequirement {
|
||||
pub constraints: Vec<PlacementConstraint>,
|
||||
}
|
||||
|
||||
impl PlacementRequirement {
|
||||
pub fn new() -> Self { Self::default() }
|
||||
|
||||
/// Builder: require a label equals a value.
|
||||
pub fn equals(mut self, key: impl Into<String>, value: CapValue) -> Self {
|
||||
self.constraints.push(PlacementConstraint::Equals(key.into(), value));
|
||||
self
|
||||
}
|
||||
|
||||
/// Builder: require an integer label >= min.
|
||||
pub fn min_int(mut self, key: impl Into<String>, min: i64) -> Self {
|
||||
self.constraints.push(PlacementConstraint::MinInt(key.into(), min));
|
||||
self
|
||||
}
|
||||
|
||||
/// Builder: require a label exists.
|
||||
pub fn has(mut self, key: impl Into<String>) -> Self {
|
||||
self.constraints.push(PlacementConstraint::HasLabel(key.into()));
|
||||
self
|
||||
}
|
||||
|
||||
/// Check if empty (no constraints — any node is acceptable).
|
||||
pub fn is_empty(&self) -> bool {
|
||||
self.constraints.is_empty()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Integration with Distribution
|
||||
|
||||
**NodeRecord extension** (`crates/distribution/src/types.rs`):
|
||||
|
||||
```rust
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct NodeRecord {
|
||||
pub node_id: NodeId,
|
||||
pub addr: SocketAddr,
|
||||
pub state: MemberState,
|
||||
pub incarnation: u64,
|
||||
// NEW (optional — backwards compatible):
|
||||
#[serde(default, skip_serializing_if = "Option::is_none")]
|
||||
pub capabilities: Option<swactor_capabilities::NodeCapabilities>,
|
||||
}
|
||||
```
|
||||
|
||||
**Capabilities flow**:
|
||||
|
||||
1. On startup, the node detects capabilities: `NodeCapabilities::detect().with_labels(operator_labels)`.
|
||||
2. Capabilities are included in the node's own `NodeRecord`.
|
||||
3. When a node joins (via `JoinResponse`), it receives other nodes' capabilities.
|
||||
4. Capabilities are piggybacked on SWIM protocol messages (membership updates already carry `NodeRecord`).
|
||||
|
||||
**Cluster-level placement** (new function in distribution):
|
||||
|
||||
```rust
|
||||
// crates/distribution/src/node.rs
|
||||
|
||||
impl DistributedNode {
|
||||
/// Find nodes that satisfy a placement requirement.
|
||||
/// Returns matching nodes sorted by preference (e.g., least loaded first).
|
||||
pub fn find_suitable_nodes(
|
||||
&self,
|
||||
requirement: &PlacementRequirement,
|
||||
) -> Vec<NodeRecord> {
|
||||
self.members()
|
||||
.into_iter()
|
||||
.filter(|node| {
|
||||
node.capabilities.as_ref()
|
||||
.map(|caps| caps.satisfies(requirement))
|
||||
.unwrap_or(requirement.is_empty())
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Example Usage
|
||||
|
||||
```rust
|
||||
// Operator starts a node with custom labels:
|
||||
let caps = NodeCapabilities::detect()
|
||||
.with_labels(btreemap! {
|
||||
"role".into() => CapValue::Str("inference".into()),
|
||||
"region".into() => CapValue::Str("us-east".into()),
|
||||
});
|
||||
|
||||
// An actor specifies placement requirements:
|
||||
let requirement = PlacementRequirement::new()
|
||||
.has("gpu_nvidia")
|
||||
.min_int("gpu_vram_mb", 8000)
|
||||
.equals("region", CapValue::Str("us-east".into()));
|
||||
|
||||
// Supervisor finds suitable nodes:
|
||||
let nodes = dist_node.find_suitable_nodes(&requirement);
|
||||
```
|
||||
|
||||
### Files Modified
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `crates/capabilities/` | **New crate** |
|
||||
| `crates/capabilities/Cargo.toml` | Package definition, feature flags |
|
||||
| `crates/capabilities/src/lib.rs` | `NodeCapabilities`, `CapValue`, `PlacementConstraint`, `PlacementRequirement`, detection |
|
||||
| `crates/distribution/Cargo.toml` | Optional dependency on `swactor-capabilities` |
|
||||
| `crates/distribution/src/types.rs` | Optional `capabilities` field on `NodeRecord` |
|
||||
| `crates/distribution/src/node.rs` | `find_suitable_nodes()`, capabilities in join flow |
|
||||
| `Cargo.toml` | Add `crates/capabilities` to workspace members |
|
||||
|
||||
### Tests
|
||||
|
||||
- **detect_basics**: `NodeCapabilities::detect()` always has `arch` and `os` labels
|
||||
- **satisfies_equals**: constraint matches/doesn't match
|
||||
- **satisfies_min_int**: integer comparison works correctly
|
||||
- **satisfies_has_label**: existence check works
|
||||
- **empty_requirement**: matches any node
|
||||
- **combined_constraints**: multiple constraints all must pass (AND)
|
||||
- **custom_labels**: operator labels merge correctly, override detection
|
||||
- **find_suitable_nodes**: integration test with mock node records and varying capabilities
|
||||
|
|
@ -1,398 +0,0 @@
|
|||
# Command Interface — Frontend-Agnostic Dispatch
|
||||
|
||||
## Problem
|
||||
|
||||
The runtime has an investigate/REPL protocol (`crates/runtime-dashboard/src/investigate.rs`) that accepts text commands and returns JSON. It works, but it's hardcoded to stdin/stdout and tightly coupled to the dashboard crate. We need the same commands accessible from:
|
||||
|
||||
- Terminal CLI (stdin/stdout)
|
||||
- TUI (the existing ratatui dashboard)
|
||||
- REST API (the existing HTTP server)
|
||||
- Future: WebSocket, remote CLI, programmatic SDK
|
||||
|
||||
And we need write commands (spawn, stop, drain) — not just read-only inspection.
|
||||
|
||||
## Design
|
||||
|
||||
### Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────────────────┐
|
||||
│ Frontends │
|
||||
│ │
|
||||
│ ┌──────────┐ ┌─────────┐ ┌──────────┐ ┌──────────┐ │
|
||||
│ │CLI / REPL│ │ TUI │ │REST /api/│ │ Future │ │
|
||||
│ │(stdin/ │ │(ratatui │ │cmd?name= │ │(websocket│ │
|
||||
│ │ stdout) │ │ events) │ │&arg=val │ │ etc.) │ │
|
||||
│ └────┬─────┘ └────┬────┘ └────┬─────┘ └────┬─────┘ │
|
||||
│ │ │ │ │ │
|
||||
│ └─────────────┴─────┬──────┴──────────────┘ │
|
||||
└───────────────────────────┼──────────────────────────────┘
|
||||
│
|
||||
┌────────▼────────┐
|
||||
│ CommandRouter │
|
||||
│ │
|
||||
│ name → handler │
|
||||
│ dispatch() │
|
||||
└────────┬────────┘
|
||||
│
|
||||
┌──────────────┼──────────────┐
|
||||
│ │ │
|
||||
┌──────▼──────┐ ┌────▼────┐ ┌───────▼───────┐
|
||||
│ Built-in │ │Built-in │ │ Custom │
|
||||
│ Read Cmds │ │Write │ │ (actor- │
|
||||
│ (overview, │ │Cmds │ │ registered) │
|
||||
│ workers, │ │(spawn, │ │ │
|
||||
│ actors, │ │ stop, │ │ │
|
||||
│ hot, ...) │ │ drain) │ │ │
|
||||
└─────────────┘ └─────────┘ └────────────────┘
|
||||
```
|
||||
|
||||
### Core Types
|
||||
|
||||
```rust
|
||||
// crates/command/src/lib.rs
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::Arc;
|
||||
|
||||
/// A command request.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct CommandRequest {
|
||||
/// Command name (e.g., "overview", "spawn", "actors").
|
||||
pub command: String,
|
||||
/// Named arguments. Values are JSON for flexibility.
|
||||
pub args: HashMap<String, serde_json::Value>,
|
||||
}
|
||||
|
||||
/// A command response. Always JSON-serializable.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct CommandResponse {
|
||||
pub ok: bool,
|
||||
pub command: String,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub data: Option<serde_json::Value>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub error: Option<String>,
|
||||
}
|
||||
|
||||
impl CommandResponse {
|
||||
pub fn ok(command: &str, data: impl Serialize) -> Self {
|
||||
Self {
|
||||
ok: true,
|
||||
command: command.to_string(),
|
||||
data: Some(serde_json::to_value(data).unwrap_or(serde_json::Value::Null)),
|
||||
error: None,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn err(command: &str, msg: impl Into<String>) -> Self {
|
||||
Self {
|
||||
ok: false,
|
||||
command: command.to_string(),
|
||||
data: None,
|
||||
error: Some(msg.into()),
|
||||
}
|
||||
}
|
||||
|
||||
/// Serialize to a single JSON line (for REPL protocol).
|
||||
pub fn to_json_line(&self) -> String {
|
||||
serde_json::to_string(self).unwrap_or_else(|e| {
|
||||
format!(r#"{{"ok":false,"command":"","error":"serialization: {e}"}}"#)
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### CommandHandler Trait
|
||||
|
||||
```rust
|
||||
/// Metadata about a command, used for help text and validation.
|
||||
pub struct CommandMeta {
|
||||
pub name: &'static str,
|
||||
pub description: &'static str,
|
||||
pub usage: &'static str,
|
||||
/// Whether this command mutates state (spawn, stop, etc.)
|
||||
pub is_write: bool,
|
||||
}
|
||||
|
||||
/// A command handler. Implementations are stateless — all state
|
||||
/// comes through CommandContext.
|
||||
pub trait CommandHandler: Send + Sync {
|
||||
fn meta(&self) -> CommandMeta;
|
||||
fn handle(&self, args: &HashMap<String, serde_json::Value>, ctx: &CommandContext) -> CommandResponse;
|
||||
}
|
||||
```
|
||||
|
||||
### CommandContext
|
||||
|
||||
```rust
|
||||
/// Context available to command handlers.
|
||||
///
|
||||
/// Contains references to runtime subsystems. Optional fields allow
|
||||
/// commands to work in both standalone and distributed configurations.
|
||||
pub struct CommandContext {
|
||||
pub runtime: Arc<swactor::runtime::Runtime>,
|
||||
pub stats_collector: Option<Arc<crate::StatsCollector>>,
|
||||
// Distribution (only present when running distributed)
|
||||
pub dist_node: Option<Arc<std::sync::Mutex<distribution::node::DistributedNode>>>,
|
||||
// Extensibility: arbitrary typed data that custom commands can access
|
||||
extensions: HashMap<std::any::TypeId, Box<dyn std::any::Any + Send + Sync>>,
|
||||
}
|
||||
|
||||
impl CommandContext {
|
||||
/// Retrieve a typed extension.
|
||||
pub fn get_ext<T: 'static + Send + Sync>(&self) -> Option<&T> {
|
||||
self.extensions.get(&std::any::TypeId::of::<T>())
|
||||
.and_then(|b| b.downcast_ref())
|
||||
}
|
||||
|
||||
/// Add a typed extension.
|
||||
pub fn set_ext<T: 'static + Send + Sync>(&mut self, val: T) {
|
||||
self.extensions.insert(std::any::TypeId::of::<T>(), Box::new(val));
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### CommandRouter
|
||||
|
||||
```rust
|
||||
/// Central command dispatch.
|
||||
pub struct CommandRouter {
|
||||
handlers: HashMap<String, Box<dyn CommandHandler>>,
|
||||
}
|
||||
|
||||
impl CommandRouter {
|
||||
pub fn new() -> Self {
|
||||
Self { handlers: HashMap::new() }
|
||||
}
|
||||
|
||||
/// Register all built-in commands.
|
||||
pub fn with_builtins(mut self) -> Self {
|
||||
self.register(Box::new(builtins::HelpCommand));
|
||||
self.register(Box::new(builtins::OverviewCommand));
|
||||
self.register(Box::new(builtins::WorkersCommand));
|
||||
self.register(Box::new(builtins::WorkerCommand));
|
||||
self.register(Box::new(builtins::ActorsCommand));
|
||||
self.register(Box::new(builtins::ActorCommand));
|
||||
self.register(Box::new(builtins::HotCommand));
|
||||
self.register(Box::new(builtins::PhasesCommand));
|
||||
self.register(Box::new(builtins::DiffCommand));
|
||||
// Write commands
|
||||
self.register(Box::new(builtins::SpawnCommand));
|
||||
self.register(Box::new(builtins::StopCommand));
|
||||
self.register(Box::new(builtins::ShutdownCommand));
|
||||
// Distribution-aware commands (no-op if dist_node is None)
|
||||
self.register(Box::new(builtins::NodesCommand));
|
||||
self.register(Box::new(builtins::RegistryCommand));
|
||||
self.register(Box::new(builtins::ResolveCommand));
|
||||
self.register(Box::new(builtins::DrainCommand));
|
||||
self
|
||||
}
|
||||
|
||||
/// Register a custom command handler.
|
||||
pub fn register(&mut self, handler: Box<dyn CommandHandler>) {
|
||||
let name = handler.meta().name.to_string();
|
||||
self.handlers.insert(name, handler);
|
||||
}
|
||||
|
||||
/// Dispatch a command request.
|
||||
pub fn dispatch(&self, req: &CommandRequest, ctx: &CommandContext) -> CommandResponse {
|
||||
match self.handlers.get(&req.command) {
|
||||
Some(handler) => handler.handle(&req.args, ctx),
|
||||
None => CommandResponse::err(
|
||||
&req.command,
|
||||
format!("unknown command `{}` — try `help`", req.command),
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
/// List all registered commands (for help text).
|
||||
pub fn commands(&self) -> Vec<&CommandMeta> {
|
||||
// sorted by name for stable output
|
||||
let mut metas: Vec<_> = self.handlers.values()
|
||||
.map(|h| h.meta())
|
||||
.collect();
|
||||
metas.sort_by_key(|m| m.name);
|
||||
metas
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Built-in Commands (MVP)
|
||||
|
||||
**Read commands** (extracted from existing `investigate.rs`):
|
||||
|
||||
| Command | Args | Description | Source |
|
||||
|---------|------|-------------|--------|
|
||||
| `help` | — | List all commands | `cmd_help()` |
|
||||
| `overview` | — | Runtime summary | `cmd_overview()` |
|
||||
| `workers` | — | Per-worker stats | `cmd_workers()` |
|
||||
| `worker` | `id: int` | Single worker detail | `cmd_worker()` |
|
||||
| `actors` | `sort`, `limit`, `worker` | List actors | `cmd_actors()` |
|
||||
| `actor` | `prefix: str` | Find by address prefix | `cmd_actor()` |
|
||||
| `hot` | `n: int` | Top N by mailbox depth | `cmd_hot()` |
|
||||
| `phases` | `worker: int?` | Tick phase breakdown | `cmd_phases()` |
|
||||
| `diff` | `seconds: float` | Snapshot delta | `cmd_diff()` |
|
||||
| `nodes` | — | Cluster member list | **new** |
|
||||
| `registry` | — | All registered names | **new** |
|
||||
| `resolve` | `name: str` | Look up a name | **new** |
|
||||
|
||||
**Write commands** (new):
|
||||
|
||||
| Command | Args | Description |
|
||||
|---------|------|-------------|
|
||||
| `stop` | `prefix` or `name` | Stop an actor (poison + cleanup) |
|
||||
| `drain` | `node: str?` | Stop accepting new actors on a node, let existing drain |
|
||||
| `shutdown` | `node: str?` | Graceful shutdown (drain + stop all) |
|
||||
| `spawn` | `factory`, `node?`, `constraints?` | Spawn from a registered factory |
|
||||
|
||||
`spawn` requires a **factory registry** — actors register factory functions that can be invoked by name:
|
||||
|
||||
```rust
|
||||
pub trait ActorFactory: Send + Sync {
|
||||
fn name(&self) -> &str;
|
||||
fn spawn(&self, runtime: &Runtime, args: &HashMap<String, serde_json::Value>)
|
||||
-> Result<ActorAddress, String>;
|
||||
}
|
||||
```
|
||||
|
||||
### Migration from investigate.rs
|
||||
|
||||
The existing `crates/runtime-dashboard/src/investigate.rs` has 9 command functions. Migration strategy:
|
||||
|
||||
1. Create `crates/command/src/builtins/` with one file per command (or grouped by category).
|
||||
2. Each `cmd_*` function becomes a `CommandHandler` impl. The logic is identical — just restructured.
|
||||
3. `dispatch_repl` becomes `CommandRouter::dispatch` with a text-to-`CommandRequest` parser.
|
||||
4. `dispatch_command` (HTTP) becomes `CommandRouter::dispatch` with query-param-to-`CommandRequest` parser.
|
||||
5. `run_investigate` remains in the dashboard crate as a thin loop over `CommandRouter`.
|
||||
|
||||
Example extraction:
|
||||
|
||||
```rust
|
||||
// crates/command/src/builtins/overview.rs
|
||||
|
||||
pub struct OverviewCommand;
|
||||
|
||||
impl CommandHandler for OverviewCommand {
|
||||
fn meta(&self) -> CommandMeta {
|
||||
CommandMeta {
|
||||
name: "overview",
|
||||
description: "Summary: worker count, actor count, total messages, panics",
|
||||
usage: "overview",
|
||||
is_write: false,
|
||||
}
|
||||
}
|
||||
|
||||
fn handle(&self, _args: &HashMap<String, serde_json::Value>, ctx: &CommandContext) -> CommandResponse {
|
||||
let stats = ctx.enriched_stats();
|
||||
let total_msgs: u64 = stats.workers.iter().map(|w| w.messages_processed).sum();
|
||||
// ... same logic as existing cmd_overview ...
|
||||
CommandResponse::ok("overview", serde_json::json!({
|
||||
"workers": stats.num_workers,
|
||||
"actors": stats.actor_details.len(),
|
||||
"total_messages_processed": total_msgs,
|
||||
// ...
|
||||
}))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Frontend Adapters
|
||||
|
||||
Each frontend is a thin adapter that converts its input format into `CommandRequest` and `CommandResponse` back to its output format.
|
||||
|
||||
**REPL adapter** (stdin/stdout):
|
||||
|
||||
```rust
|
||||
// crates/command/src/adapters/repl.rs
|
||||
|
||||
pub fn parse_line(line: &str) -> CommandRequest {
|
||||
let parts: Vec<&str> = line.split_whitespace().collect();
|
||||
let command = parts.first().unwrap_or(&"help").to_string();
|
||||
let args = parse_positional_and_flags(&parts[1..]);
|
||||
CommandRequest { command, args }
|
||||
}
|
||||
```
|
||||
|
||||
**REST adapter** (HTTP query params):
|
||||
|
||||
```rust
|
||||
// crates/command/src/adapters/rest.rs
|
||||
|
||||
pub fn from_query_params(params: &HashMap<String, String>) -> CommandRequest {
|
||||
let command = params.get("cmd").cloned().unwrap_or_else(|| "help".into());
|
||||
let args: HashMap<String, serde_json::Value> = params.iter()
|
||||
.filter(|(k, _)| *k != "cmd")
|
||||
.map(|(k, v)| (k.clone(), serde_json::Value::String(v.clone())))
|
||||
.collect();
|
||||
CommandRequest { command, args }
|
||||
}
|
||||
```
|
||||
|
||||
**TUI adapter**: TUI input field text → `parse_line()` → `dispatch()` → render response in panel.
|
||||
|
||||
### Custom Commands (actor-registered)
|
||||
|
||||
Actors can register command handlers at runtime through the `Ctx`:
|
||||
|
||||
```rust
|
||||
impl Ctx<'_> {
|
||||
/// Register a command that routes to this actor.
|
||||
/// When the command is invoked, a CommandInvocation message
|
||||
/// is sent to this actor's mailbox.
|
||||
pub fn register_command(&self, name: &str, description: &str) {
|
||||
self.inner.register_command(self.self_addr, name, description);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
When a custom command is dispatched:
|
||||
|
||||
1. Router finds it's actor-registered.
|
||||
2. Sends `CommandInvocation { command, args, reply_addr }` to the actor's mailbox.
|
||||
3. The actor processes it and sends `CommandResult { data }` back to `reply_addr`.
|
||||
4. Router waits on a one-shot inbox (with timeout, e.g. 5 seconds).
|
||||
5. Returns the response.
|
||||
|
||||
```rust
|
||||
/// Sent to an actor when its registered command is invoked.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct CommandInvocation {
|
||||
pub command: String,
|
||||
pub args: HashMap<String, serde_json::Value>,
|
||||
pub reply_addr: ActorAddress,
|
||||
}
|
||||
|
||||
/// Sent back by the actor with the command result.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct CommandResult {
|
||||
pub data: serde_json::Value,
|
||||
}
|
||||
```
|
||||
|
||||
This mechanism allows any actor to expose operational endpoints without modifying the command crate.
|
||||
|
||||
### Files Modified
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `crates/command/` | **New crate** |
|
||||
| `crates/command/Cargo.toml` | Dependencies: swactor, serde, serde_json |
|
||||
| `crates/command/src/lib.rs` | `CommandRouter`, `CommandHandler`, `CommandRequest`/`Response`, `CommandContext` |
|
||||
| `crates/command/src/builtins/` | Built-in command handlers (mod.rs + per-command files) |
|
||||
| `crates/command/src/adapters/` | REPL and REST input parsers |
|
||||
| `crates/runtime-dashboard/src/investigate.rs` | Refactored: thin REPL loop over CommandRouter |
|
||||
| `crates/runtime-dashboard/src/server.rs` | REST endpoints use CommandRouter |
|
||||
| `crates/runtime-dashboard/Cargo.toml` | Depends on `crates/command` |
|
||||
| `Cargo.toml` | Add `crates/command` to workspace |
|
||||
|
||||
### Tests
|
||||
|
||||
- **dispatch_known_command**: `overview` returns `ok: true` with expected fields
|
||||
- **dispatch_unknown_command**: returns `ok: false` with helpful error
|
||||
- **help_lists_all**: `help` response includes all registered command names
|
||||
- **parse_repl_line**: `"actors --sort mailbox --limit 5"` → correct `CommandRequest`
|
||||
- **parse_query_params**: `{cmd: "actor", prefix: "a1b2"}` → correct `CommandRequest`
|
||||
- **custom_command_dispatch**: register actor command, invoke, verify response
|
||||
- **custom_command_timeout**: registered actor doesn't respond, verify timeout error
|
||||
- **write_command_stop**: stop an actor via command, verify it's poisoned
|
||||
|
|
@ -1,263 +0,0 @@
|
|||
# Supervision — User-Space Self-Healing
|
||||
|
||||
## Problem
|
||||
|
||||
When nodes churn (spot instances preempted, hardware rebooted, network partitions), actors on those nodes are lost. Something needs to detect the loss and re-spawn the actors on surviving nodes. This is the "self-healing" property of a distributed OS.
|
||||
|
||||
## Design Principle: Supervision Is User-Space
|
||||
|
||||
Supervision is **not** a runtime primitive. It is a pattern built from:
|
||||
|
||||
- **Actor Watching** (01) — detect death
|
||||
- **Cluster Registry** (02) — re-register under the same name
|
||||
- **Node Capabilities** (03) — find a suitable replacement node
|
||||
|
||||
The runtime provides the low-level mechanisms. Supervision is a library actor that composes them. This keeps the kernel minimal and lets users customize supervision policy without forking the runtime.
|
||||
|
||||
## Supervisor Actor
|
||||
|
||||
```rust
|
||||
/// A supervised child definition.
|
||||
struct SupervisedChild {
|
||||
/// Human-readable name (registered in cluster registry).
|
||||
name: String,
|
||||
/// Factory function to create the actor.
|
||||
factory: Box<dyn ActorFactory>,
|
||||
/// Placement constraints for the child.
|
||||
constraints: PlacementRequirement,
|
||||
/// Current address (None if not yet spawned or dead).
|
||||
current_addr: Option<ActorAddress>,
|
||||
/// Number of restarts so far.
|
||||
restart_count: u32,
|
||||
/// Maximum restarts before giving up (0 = unlimited).
|
||||
max_restarts: u32,
|
||||
/// Backoff state for restart delays.
|
||||
last_restart: Option<Instant>,
|
||||
}
|
||||
|
||||
/// Restart strategy for a supervisor.
|
||||
#[derive(Debug, Clone)]
|
||||
enum RestartStrategy {
|
||||
/// Restart only the failed child.
|
||||
OneForOne,
|
||||
/// If any child fails, restart all children.
|
||||
AllForOne,
|
||||
/// Don't restart — just notify (for monitoring supervisors).
|
||||
Notify,
|
||||
}
|
||||
|
||||
/// The supervisor actor.
|
||||
struct Supervisor {
|
||||
children: Vec<SupervisedChild>,
|
||||
strategy: RestartStrategy,
|
||||
}
|
||||
```
|
||||
|
||||
## Message Protocol
|
||||
|
||||
```rust
|
||||
/// Messages the supervisor handles.
|
||||
enum SupervisorMsg {
|
||||
/// A watched child died.
|
||||
Exited(ActorExited),
|
||||
/// External request to add a child.
|
||||
AddChild {
|
||||
name: String,
|
||||
factory: Box<dyn ActorFactory>,
|
||||
constraints: PlacementRequirement,
|
||||
},
|
||||
/// External request to remove a child.
|
||||
RemoveChild { name: String },
|
||||
/// Query: what children are running?
|
||||
Status { reply_to: ActorAddress },
|
||||
}
|
||||
```
|
||||
|
||||
## Lifecycle
|
||||
|
||||
### Startup
|
||||
|
||||
```rust
|
||||
impl Supervisor {
|
||||
fn start(&mut self, ctx: &Ctx) {
|
||||
for child in &mut self.children {
|
||||
match self.spawn_child(ctx, child) {
|
||||
Ok(addr) => {
|
||||
child.current_addr = Some(addr);
|
||||
ctx.watch(addr);
|
||||
ctx.register_as(&child.name); // or register the child
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!("supervisor: failed to spawn {}: {e}", child.name);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Death Handling
|
||||
|
||||
```rust
|
||||
impl ActorInterface for Supervisor {
|
||||
type Incoming = SupervisorMsg;
|
||||
type Response = ();
|
||||
|
||||
fn handle(&mut self, ctx: &Ctx, msg: SupervisorMsg) {
|
||||
match msg {
|
||||
SupervisorMsg::Exited(exited) => {
|
||||
match self.strategy {
|
||||
RestartStrategy::OneForOne => {
|
||||
self.restart_one(ctx, &exited);
|
||||
}
|
||||
RestartStrategy::AllForOne => {
|
||||
self.restart_all(ctx);
|
||||
}
|
||||
RestartStrategy::Notify => {
|
||||
// Just log — don't restart
|
||||
}
|
||||
}
|
||||
}
|
||||
// ... other messages ...
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Restart Flow (OneForOne)
|
||||
|
||||
```
|
||||
1. Receive ActorExited { addr: X, reason: NodeDown }
|
||||
2. Find child with current_addr == X → child "worker-3"
|
||||
3. Check restart_count < max_restarts
|
||||
4. Query capabilities: find_suitable_nodes(child.constraints)
|
||||
5. Pick best node (least loaded, or local if possible)
|
||||
6. Spawn child on selected node via factory
|
||||
7. Watch new address
|
||||
8. Register child.name → new address in cluster registry
|
||||
9. Update child.current_addr
|
||||
10. Increment child.restart_count
|
||||
```
|
||||
|
||||
```rust
|
||||
impl Supervisor {
|
||||
fn restart_one(&mut self, ctx: &Ctx, exited: &ActorExited) {
|
||||
let child = match self.children.iter_mut()
|
||||
.find(|c| c.current_addr == Some(exited.addr))
|
||||
{
|
||||
Some(c) => c,
|
||||
None => return, // not our child
|
||||
};
|
||||
|
||||
child.current_addr = None;
|
||||
|
||||
if child.max_restarts > 0 && child.restart_count >= child.max_restarts {
|
||||
eprintln!(
|
||||
"supervisor: child {} exceeded max restarts ({}), giving up",
|
||||
child.name, child.max_restarts,
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
// Spawn replacement
|
||||
match self.spawn_child(ctx, child) {
|
||||
Ok(addr) => {
|
||||
child.current_addr = Some(addr);
|
||||
child.restart_count += 1;
|
||||
ctx.watch(addr);
|
||||
// Re-register name → new address
|
||||
// (done via the registry, which gossips to all nodes)
|
||||
}
|
||||
Err(e) => {
|
||||
eprintln!("supervisor: failed to restart {}: {e}", child.name);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Restart Flow (AllForOne)
|
||||
|
||||
When any child dies:
|
||||
1. Stop all other children (send stop signal).
|
||||
2. Wait for all `ActorExited` notifications.
|
||||
3. Restart all children in order.
|
||||
|
||||
This is useful for interdependent actor groups where partial restart doesn't make sense.
|
||||
|
||||
## Capability-Aware Placement
|
||||
|
||||
The supervisor uses `find_suitable_nodes()` from the distribution crate:
|
||||
|
||||
```rust
|
||||
fn spawn_child(&self, ctx: &Ctx, child: &SupervisedChild)
|
||||
-> Result<ActorAddress, String>
|
||||
{
|
||||
// If constraints are empty, spawn locally
|
||||
if child.constraints.is_empty() {
|
||||
return child.factory.spawn(ctx);
|
||||
}
|
||||
|
||||
// Find suitable remote nodes
|
||||
let nodes = dist_node.find_suitable_nodes(&child.constraints);
|
||||
if nodes.is_empty() {
|
||||
return Err("no nodes satisfy placement constraints".into());
|
||||
}
|
||||
|
||||
// Pick the least-loaded suitable node
|
||||
let target_node = &nodes[0]; // TODO: sort by load
|
||||
|
||||
// Spawn remotely (requires remote spawn protocol — future work)
|
||||
// For now: if current node satisfies, spawn locally
|
||||
// Otherwise: send spawn request to target node
|
||||
todo!("remote spawn")
|
||||
}
|
||||
```
|
||||
|
||||
**Note**: Remote spawn (telling another node to create an actor) is not yet part of the runtime. The supervisor design accounts for it, but the initial implementation will only support local spawn + re-registration.
|
||||
|
||||
## Relationship to Existing `spawn_restartable`
|
||||
|
||||
The core runtime already has `spawn_restartable` with `factory` and `max_restarts`. This is a **local-only** recovery mechanism — when an actor panics, the same worker restarts it.
|
||||
|
||||
The supervisor pattern extends this to **cluster-wide** recovery:
|
||||
|
||||
| Feature | `spawn_restartable` | Supervisor |
|
||||
|---|---|---|
|
||||
| Scope | Single worker | Cluster-wide |
|
||||
| Trigger | Panic | Panic, stop, or node death |
|
||||
| Placement | Same worker | Capability-aware, any node |
|
||||
| Naming | No | Yes (cluster registry) |
|
||||
| Strategy | Always restart | OneForOne, AllForOne, Notify |
|
||||
| Implementation | Runtime internal | User-space actor |
|
||||
|
||||
They complement each other: `spawn_restartable` handles fast local recovery (no network round-trip); the supervisor handles node-level failures.
|
||||
|
||||
## Future Extensions
|
||||
|
||||
- **Restart backoff**: exponential backoff between restarts to avoid thrashing.
|
||||
- **Health checks**: periodic health probes (not just death detection).
|
||||
- **Cascading supervisors**: supervisor trees (supervisor watches sub-supervisor).
|
||||
- **Declarative spec**: TOML/YAML file defining supervision topology, loaded at startup.
|
||||
- **Migration (not restart)**: move a running actor's state to another node (requires persistence, out of scope).
|
||||
|
||||
## Files
|
||||
|
||||
This is a library actor, not a runtime change. Implementation lives in:
|
||||
|
||||
| File | Content |
|
||||
|------|---------|
|
||||
| `crates/supervision/src/lib.rs` | `Supervisor`, `SupervisedChild`, `RestartStrategy` |
|
||||
| `crates/supervision/src/factory.rs` | `ActorFactory` trait, factory registry |
|
||||
| `crates/supervision/Cargo.toml` | Depends on `swactor`, `swactor-capabilities`, `distribution` |
|
||||
|
||||
Or, if the scope doesn't warrant a separate crate, it can live in `src/supervision.rs` behind a feature flag.
|
||||
|
||||
## Tests
|
||||
|
||||
- **one_for_one_restart**: supervisor with 3 children, kill one, verify only that one restarts
|
||||
- **all_for_one_restart**: supervisor with 3 children, kill one, verify all restart
|
||||
- **max_restarts_exceeded**: child dies repeatedly, verify supervisor gives up after max
|
||||
- **name_re_registration**: child dies and restarts, verify name resolves to new address
|
||||
- **capability_placement**: child with GPU constraint, verify spawned on GPU node (or error if none available)
|
||||
- **supervisor_itself_dies**: verify children are stopped (or orphaned — design decision)
|
||||
|
|
@ -1,153 +0,0 @@
|
|||
# Swactor Distributed OS — Design Overview
|
||||
|
||||
## Vision
|
||||
|
||||
Swactor is evolving from a local actor runtime into a **distributed operating system** for running long-lived daemons across heterogeneous, churning machines.
|
||||
|
||||
**Primary use case**: Wire together personal hardware today; add on-demand spot compute (vast.ai, etc.) tomorrow. Machines pop in and out of the network. The system self-heals.
|
||||
|
||||
**Design principles**:
|
||||
|
||||
- **Churn is the norm**, not the exception. Every subsystem assumes nodes can disappear at any time.
|
||||
- **General and flexible**. Minimal assumptions about what a node looks like — feature-gate hardware-specific code.
|
||||
- **Layered**. The core runtime stays minimal. OS features are opt-in crates. Supervision is user-space, not kernel.
|
||||
- **Frontend-agnostic**. Operational interfaces (commands, inspection) work identically from CLI, TUI, REST, or future transports.
|
||||
|
||||
## Current Capabilities
|
||||
|
||||
| OS Concept | What Swactor Has Today |
|
||||
|---|---|
|
||||
| Processes | Actor spawn/stop, lifecycle hooks, restartable with factory + max_restarts |
|
||||
| Scheduling | Worker threads, fairness budget (64 msgs/tick), load-aware placement |
|
||||
| IPC | Typed send, request/reply, per-actor VecDeque mailboxes |
|
||||
| Naming | String-keyed `AddressMap` — node-local only |
|
||||
| Fault tolerance | `catch_unwind` for panics, factory restart, dead actor cleanup |
|
||||
| Backpressure | Per-actor mailbox capacity, DropNewest/DropOldest overflow |
|
||||
| Networking | SWIM membership (Lifeguard extensions), TCP transport, bincode wire protocol |
|
||||
| Directory | Kademlia DHT for `ActorAddress -> NodeId` resolution |
|
||||
| Monitoring | WorkerStats/RuntimeStats, TUI dashboard, REST `/api/*`, investigate REPL |
|
||||
| Distribution | Multi-node cluster, 5-node Docker test suite |
|
||||
|
||||
## What's Missing (This Design)
|
||||
|
||||
| Feature | Document | Priority |
|
||||
|---|---|---|
|
||||
| Actor Watching | [01-actor-watching.md](./01-actor-watching.md) | Foundation for everything |
|
||||
| Cluster Registry | [02-cluster-registry.md](./02-cluster-registry.md) | Actors find each other across nodes |
|
||||
| Node Capabilities | [03-node-capabilities.md](./03-node-capabilities.md) | Heterogeneous placement |
|
||||
| Command Interface | [04-command-interface.md](./04-command-interface.md) | Operational control |
|
||||
| Supervision | [05-supervision.md](./05-supervision.md) | Self-healing (user-space) |
|
||||
|
||||
## Crate Structure (After This Work)
|
||||
|
||||
```
|
||||
swactor/ # core runtime — no network deps
|
||||
src/
|
||||
actor.rs # +watch/unwatch on ContextInner, ExitReason, ActorExited
|
||||
worker.rs # +WatchRegistry, death notification phase
|
||||
delivery.rs # (unchanged)
|
||||
runtime.rs # (unchanged)
|
||||
...
|
||||
crates/
|
||||
capabilities/ # NEW — hardware detection + placement constraints
|
||||
src/lib.rs # NodeCapabilities, CapValue, PlacementConstraint
|
||||
command/ # NEW — frontend-agnostic command dispatch
|
||||
src/lib.rs # CommandRouter, CommandHandler, CommandRequest/Response
|
||||
src/builtins/ # Built-in command handlers
|
||||
distribution/ # SWIM + Kademlia + cluster registry
|
||||
src/
|
||||
registry.rs # NEW — ClusterRegistry, gossip-propagated naming
|
||||
node.rs # +register_name, resolve_name, capabilities
|
||||
messages.rs # +WatchRequest, UnwatchRequest, ActorExitedNotify
|
||||
...
|
||||
runtime-dashboard/ # TUI + REST — refactored to use command crate
|
||||
python/ # PyO3 bindings
|
||||
simulation/ # Network simulation
|
||||
wasm/ # WASM bindings
|
||||
```
|
||||
|
||||
## Dependency Graph
|
||||
|
||||
```
|
||||
┌──────────────┐
|
||||
│ capabilities │ (standalone — only serde + sysinfo)
|
||||
└──────┬───────┘
|
||||
│ optional
|
||||
┌─────────┐ ┌──────▼────────┐
|
||||
│ swactor │◄─────│ distribution │
|
||||
│ (core) │ │ +registry │
|
||||
└────┬────┘ └──────┬────────┘
|
||||
│ │
|
||||
│ ┌──────▼────────┐
|
||||
└──────────►│ command │
|
||||
└──────┬────────┘
|
||||
│
|
||||
┌──────▼────────────┐
|
||||
│ runtime-dashboard │
|
||||
│ (TUI + REST) │
|
||||
└───────────────────┘
|
||||
```
|
||||
|
||||
Key constraints:
|
||||
- `capabilities` has **zero** dependency on `swactor` — it's a standalone detection library.
|
||||
- `command` depends on `swactor` (needs `Runtime`, stats types) but NOT on `distribution`.
|
||||
- `distribution` optionally depends on `capabilities` (for `NodeRecord` labels).
|
||||
- `runtime-dashboard` depends on both `command` and optionally `distribution`.
|
||||
|
||||
## Feature Flags
|
||||
|
||||
### Core `swactor`
|
||||
|
||||
| Flag | Purpose |
|
||||
|---|---|
|
||||
| `getrandom` (default) | Cryptographic RNG for actor addresses |
|
||||
| `serde` | Serialization for types |
|
||||
| `tracing` | Structured logging |
|
||||
| `transport` | Transport-agnostic remote messaging |
|
||||
| `watching` (new) | Watch API on ContextInner/Ctx |
|
||||
|
||||
### `swactor-capabilities`
|
||||
|
||||
| Flag | Purpose |
|
||||
|---|---|
|
||||
| `detect` (default) | Auto-detect CPU, RAM, hostname via sysinfo |
|
||||
| `gpu-nvidia` | Detect NVIDIA GPU via nvidia-smi |
|
||||
| `gpu-vulkan` | Detect GPU via Vulkan API |
|
||||
|
||||
### `distribution`
|
||||
|
||||
| Flag | Purpose |
|
||||
|---|---|
|
||||
| `registry` (new, default) | Cluster-wide gossip-propagated naming |
|
||||
| `capabilities` (new) | NodeCapabilities on NodeRecord |
|
||||
|
||||
## Implementation Order
|
||||
|
||||
Each feature is one PR, in dependency order:
|
||||
|
||||
```
|
||||
PR 1: Actor Watching (local only)
|
||||
└──► PR 2: Command Interface
|
||||
└──► PR 3: Cluster Registry
|
||||
└──► PR 4: Node Capabilities
|
||||
└──► PR 5: Remote Watching (cross-node)
|
||||
└──► PR 6: Supervisor library
|
||||
```
|
||||
|
||||
**PR 1 — Actor Watching (local)**
|
||||
Adds `WatchRegistry`, `ActorExited`, `ExitReason` to core runtime. Testable without any distribution. Foundation for everything else.
|
||||
|
||||
**PR 2 — Command Interface**
|
||||
Extracts investigate.rs into `crates/command/`. Adds write commands (spawn, stop, drain). Immediately useful for operations.
|
||||
|
||||
**PR 3 — Cluster Registry**
|
||||
Gossip-propagated naming in `crates/distribution/src/registry.rs`. Actors can find each other by name across nodes.
|
||||
|
||||
**PR 4 — Node Capabilities**
|
||||
`crates/capabilities/` with auto-detection and placement constraints. Integrates with distribution for capability-aware placement.
|
||||
|
||||
**PR 5 — Remote Watching**
|
||||
Wire protocol for cross-node watches. SWIM Dead triggers `ActorExited { reason: NodeDown }` for all actors on that node.
|
||||
|
||||
**PR 6 — Supervisor Library**
|
||||
User-space supervisor pattern. Composes watching + registry + capabilities to auto-respawn actors after churn.
|
||||
Loading…
Reference in a new issue