Add watch/unwatch API to the actor system so actors can monitor each other's liveness. When a watched actor dies (panic or stop), watchers receive an ActorExited notification via on_actor_exit(). - ExitReason enum (Stopped, Panicked, NodeDown) and ActorExited struct - ContextInner::watch()/unwatch() + Ctx typed wrappers - ActorInterface::on_actor_exit() default method (system message fallback) - WatchRegistry in worker with bidirectional tracking - Death notification dispatch as phase 5b in tick_once - Runtime-level watch for external callers - 10 behavioral tests in tests/watch_api.rs - Design documents for OS features in docs/os-design/ Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
15 KiB
Command Interface — Frontend-Agnostic Dispatch
Problem
The runtime has an investigate/REPL protocol (crates/runtime-dashboard/src/investigate.rs) that accepts text commands and returns JSON. It works, but it's hardcoded to stdin/stdout and tightly coupled to the dashboard crate. We need the same commands accessible from:
- Terminal CLI (stdin/stdout)
- TUI (the existing ratatui dashboard)
- REST API (the existing HTTP server)
- Future: WebSocket, remote CLI, programmatic SDK
And we need write commands (spawn, stop, drain) — not just read-only inspection.
Design
Architecture
┌──────────────────────────────────────────────────────────┐
│ Frontends │
│ │
│ ┌──────────┐ ┌─────────┐ ┌──────────┐ ┌──────────┐ │
│ │CLI / REPL│ │ TUI │ │REST /api/│ │ Future │ │
│ │(stdin/ │ │(ratatui │ │cmd?name= │ │(websocket│ │
│ │ stdout) │ │ events) │ │&arg=val │ │ etc.) │ │
│ └────┬─────┘ └────┬────┘ └────┬─────┘ └────┬─────┘ │
│ │ │ │ │ │
│ └─────────────┴─────┬──────┴──────────────┘ │
└───────────────────────────┼──────────────────────────────┘
│
┌────────▼────────┐
│ CommandRouter │
│ │
│ name → handler │
│ dispatch() │
└────────┬────────┘
│
┌──────────────┼──────────────┐
│ │ │
┌──────▼──────┐ ┌────▼────┐ ┌───────▼───────┐
│ Built-in │ │Built-in │ │ Custom │
│ Read Cmds │ │Write │ │ (actor- │
│ (overview, │ │Cmds │ │ registered) │
│ workers, │ │(spawn, │ │ │
│ actors, │ │ stop, │ │ │
│ hot, ...) │ │ drain) │ │ │
└─────────────┘ └─────────┘ └────────────────┘
Core Types
// crates/command/src/lib.rs
use std::collections::HashMap;
use std::sync::Arc;
/// A command request.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CommandRequest {
/// Command name (e.g., "overview", "spawn", "actors").
pub command: String,
/// Named arguments. Values are JSON for flexibility.
pub args: HashMap<String, serde_json::Value>,
}
/// A command response. Always JSON-serializable.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CommandResponse {
pub ok: bool,
pub command: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub data: Option<serde_json::Value>,
#[serde(skip_serializing_if = "Option::is_none")]
pub error: Option<String>,
}
impl CommandResponse {
pub fn ok(command: &str, data: impl Serialize) -> Self {
Self {
ok: true,
command: command.to_string(),
data: Some(serde_json::to_value(data).unwrap_or(serde_json::Value::Null)),
error: None,
}
}
pub fn err(command: &str, msg: impl Into<String>) -> Self {
Self {
ok: false,
command: command.to_string(),
data: None,
error: Some(msg.into()),
}
}
/// Serialize to a single JSON line (for REPL protocol).
pub fn to_json_line(&self) -> String {
serde_json::to_string(self).unwrap_or_else(|e| {
format!(r#"{{"ok":false,"command":"","error":"serialization: {e}"}}"#)
})
}
}
CommandHandler Trait
/// Metadata about a command, used for help text and validation.
pub struct CommandMeta {
pub name: &'static str,
pub description: &'static str,
pub usage: &'static str,
/// Whether this command mutates state (spawn, stop, etc.)
pub is_write: bool,
}
/// A command handler. Implementations are stateless — all state
/// comes through CommandContext.
pub trait CommandHandler: Send + Sync {
fn meta(&self) -> CommandMeta;
fn handle(&self, args: &HashMap<String, serde_json::Value>, ctx: &CommandContext) -> CommandResponse;
}
CommandContext
/// Context available to command handlers.
///
/// Contains references to runtime subsystems. Optional fields allow
/// commands to work in both standalone and distributed configurations.
pub struct CommandContext {
pub runtime: Arc<swactor::runtime::Runtime>,
pub stats_collector: Option<Arc<crate::StatsCollector>>,
// Distribution (only present when running distributed)
pub dist_node: Option<Arc<std::sync::Mutex<distribution::node::DistributedNode>>>,
// Extensibility: arbitrary typed data that custom commands can access
extensions: HashMap<std::any::TypeId, Box<dyn std::any::Any + Send + Sync>>,
}
impl CommandContext {
/// Retrieve a typed extension.
pub fn get_ext<T: 'static + Send + Sync>(&self) -> Option<&T> {
self.extensions.get(&std::any::TypeId::of::<T>())
.and_then(|b| b.downcast_ref())
}
/// Add a typed extension.
pub fn set_ext<T: 'static + Send + Sync>(&mut self, val: T) {
self.extensions.insert(std::any::TypeId::of::<T>(), Box::new(val));
}
}
CommandRouter
/// Central command dispatch.
pub struct CommandRouter {
handlers: HashMap<String, Box<dyn CommandHandler>>,
}
impl CommandRouter {
pub fn new() -> Self {
Self { handlers: HashMap::new() }
}
/// Register all built-in commands.
pub fn with_builtins(mut self) -> Self {
self.register(Box::new(builtins::HelpCommand));
self.register(Box::new(builtins::OverviewCommand));
self.register(Box::new(builtins::WorkersCommand));
self.register(Box::new(builtins::WorkerCommand));
self.register(Box::new(builtins::ActorsCommand));
self.register(Box::new(builtins::ActorCommand));
self.register(Box::new(builtins::HotCommand));
self.register(Box::new(builtins::PhasesCommand));
self.register(Box::new(builtins::DiffCommand));
// Write commands
self.register(Box::new(builtins::SpawnCommand));
self.register(Box::new(builtins::StopCommand));
self.register(Box::new(builtins::ShutdownCommand));
// Distribution-aware commands (no-op if dist_node is None)
self.register(Box::new(builtins::NodesCommand));
self.register(Box::new(builtins::RegistryCommand));
self.register(Box::new(builtins::ResolveCommand));
self.register(Box::new(builtins::DrainCommand));
self
}
/// Register a custom command handler.
pub fn register(&mut self, handler: Box<dyn CommandHandler>) {
let name = handler.meta().name.to_string();
self.handlers.insert(name, handler);
}
/// Dispatch a command request.
pub fn dispatch(&self, req: &CommandRequest, ctx: &CommandContext) -> CommandResponse {
match self.handlers.get(&req.command) {
Some(handler) => handler.handle(&req.args, ctx),
None => CommandResponse::err(
&req.command,
format!("unknown command `{}` — try `help`", req.command),
),
}
}
/// List all registered commands (for help text).
pub fn commands(&self) -> Vec<&CommandMeta> {
// sorted by name for stable output
let mut metas: Vec<_> = self.handlers.values()
.map(|h| h.meta())
.collect();
metas.sort_by_key(|m| m.name);
metas
}
}
Built-in Commands (MVP)
Read commands (extracted from existing investigate.rs):
| Command | Args | Description | Source |
|---|---|---|---|
help |
— | List all commands | cmd_help() |
overview |
— | Runtime summary | cmd_overview() |
workers |
— | Per-worker stats | cmd_workers() |
worker |
id: int |
Single worker detail | cmd_worker() |
actors |
sort, limit, worker |
List actors | cmd_actors() |
actor |
prefix: str |
Find by address prefix | cmd_actor() |
hot |
n: int |
Top N by mailbox depth | cmd_hot() |
phases |
worker: int? |
Tick phase breakdown | cmd_phases() |
diff |
seconds: float |
Snapshot delta | cmd_diff() |
nodes |
— | Cluster member list | new |
registry |
— | All registered names | new |
resolve |
name: str |
Look up a name | new |
Write commands (new):
| Command | Args | Description |
|---|---|---|
stop |
prefix or name |
Stop an actor (poison + cleanup) |
drain |
node: str? |
Stop accepting new actors on a node, let existing drain |
shutdown |
node: str? |
Graceful shutdown (drain + stop all) |
spawn |
factory, node?, constraints? |
Spawn from a registered factory |
spawn requires a factory registry — actors register factory functions that can be invoked by name:
pub trait ActorFactory: Send + Sync {
fn name(&self) -> &str;
fn spawn(&self, runtime: &Runtime, args: &HashMap<String, serde_json::Value>)
-> Result<ActorAddress, String>;
}
Migration from investigate.rs
The existing crates/runtime-dashboard/src/investigate.rs has 9 command functions. Migration strategy:
- Create
crates/command/src/builtins/with one file per command (or grouped by category). - Each
cmd_*function becomes aCommandHandlerimpl. The logic is identical — just restructured. dispatch_replbecomesCommandRouter::dispatchwith a text-to-CommandRequestparser.dispatch_command(HTTP) becomesCommandRouter::dispatchwith query-param-to-CommandRequestparser.run_investigateremains in the dashboard crate as a thin loop overCommandRouter.
Example extraction:
// crates/command/src/builtins/overview.rs
pub struct OverviewCommand;
impl CommandHandler for OverviewCommand {
fn meta(&self) -> CommandMeta {
CommandMeta {
name: "overview",
description: "Summary: worker count, actor count, total messages, panics",
usage: "overview",
is_write: false,
}
}
fn handle(&self, _args: &HashMap<String, serde_json::Value>, ctx: &CommandContext) -> CommandResponse {
let stats = ctx.enriched_stats();
let total_msgs: u64 = stats.workers.iter().map(|w| w.messages_processed).sum();
// ... same logic as existing cmd_overview ...
CommandResponse::ok("overview", serde_json::json!({
"workers": stats.num_workers,
"actors": stats.actor_details.len(),
"total_messages_processed": total_msgs,
// ...
}))
}
}
Frontend Adapters
Each frontend is a thin adapter that converts its input format into CommandRequest and CommandResponse back to its output format.
REPL adapter (stdin/stdout):
// crates/command/src/adapters/repl.rs
pub fn parse_line(line: &str) -> CommandRequest {
let parts: Vec<&str> = line.split_whitespace().collect();
let command = parts.first().unwrap_or(&"help").to_string();
let args = parse_positional_and_flags(&parts[1..]);
CommandRequest { command, args }
}
REST adapter (HTTP query params):
// crates/command/src/adapters/rest.rs
pub fn from_query_params(params: &HashMap<String, String>) -> CommandRequest {
let command = params.get("cmd").cloned().unwrap_or_else(|| "help".into());
let args: HashMap<String, serde_json::Value> = params.iter()
.filter(|(k, _)| *k != "cmd")
.map(|(k, v)| (k.clone(), serde_json::Value::String(v.clone())))
.collect();
CommandRequest { command, args }
}
TUI adapter: TUI input field text → parse_line() → dispatch() → render response in panel.
Custom Commands (actor-registered)
Actors can register command handlers at runtime through the Ctx:
impl Ctx<'_> {
/// Register a command that routes to this actor.
/// When the command is invoked, a CommandInvocation message
/// is sent to this actor's mailbox.
pub fn register_command(&self, name: &str, description: &str) {
self.inner.register_command(self.self_addr, name, description);
}
}
When a custom command is dispatched:
- Router finds it's actor-registered.
- Sends
CommandInvocation { command, args, reply_addr }to the actor's mailbox. - The actor processes it and sends
CommandResult { data }back toreply_addr. - Router waits on a one-shot inbox (with timeout, e.g. 5 seconds).
- Returns the response.
/// Sent to an actor when its registered command is invoked.
#[derive(Debug, Clone)]
pub struct CommandInvocation {
pub command: String,
pub args: HashMap<String, serde_json::Value>,
pub reply_addr: ActorAddress,
}
/// Sent back by the actor with the command result.
#[derive(Debug, Clone)]
pub struct CommandResult {
pub data: serde_json::Value,
}
This mechanism allows any actor to expose operational endpoints without modifying the command crate.
Files Modified
| File | Change |
|---|---|
crates/command/ |
New crate |
crates/command/Cargo.toml |
Dependencies: swactor, serde, serde_json |
crates/command/src/lib.rs |
CommandRouter, CommandHandler, CommandRequest/Response, CommandContext |
crates/command/src/builtins/ |
Built-in command handlers (mod.rs + per-command files) |
crates/command/src/adapters/ |
REPL and REST input parsers |
crates/runtime-dashboard/src/investigate.rs |
Refactored: thin REPL loop over CommandRouter |
crates/runtime-dashboard/src/server.rs |
REST endpoints use CommandRouter |
crates/runtime-dashboard/Cargo.toml |
Depends on crates/command |
Cargo.toml |
Add crates/command to workspace |
Tests
- dispatch_known_command:
overviewreturnsok: truewith expected fields - dispatch_unknown_command: returns
ok: falsewith helpful error - help_lists_all:
helpresponse includes all registered command names - parse_repl_line:
"actors --sort mailbox --limit 5"→ correctCommandRequest - parse_query_params:
{cmd: "actor", prefix: "a1b2"}→ correctCommandRequest - custom_command_dispatch: register actor command, invoke, verify response
- custom_command_timeout: registered actor doesn't respond, verify timeout error
- write_command_stop: stop an actor via command, verify it's poisoned