swactor/docs/os-design/04-command-interface.md
Developer 473999d1df feat: actor watching — local death notifications
Add watch/unwatch API to the actor system so actors can monitor each
other's liveness. When a watched actor dies (panic or stop), watchers
receive an ActorExited notification via on_actor_exit().

- ExitReason enum (Stopped, Panicked, NodeDown) and ActorExited struct
- ContextInner::watch()/unwatch() + Ctx typed wrappers
- ActorInterface::on_actor_exit() default method (system message fallback)
- WatchRegistry in worker with bidirectional tracking
- Death notification dispatch as phase 5b in tick_once
- Runtime-level watch for external callers
- 10 behavioral tests in tests/watch_api.rs
- Design documents for OS features in docs/os-design/

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-13 07:25:27 +00:00

398 lines
15 KiB
Markdown

# Command Interface — Frontend-Agnostic Dispatch
## Problem
The runtime has an investigate/REPL protocol (`crates/runtime-dashboard/src/investigate.rs`) that accepts text commands and returns JSON. It works, but it's hardcoded to stdin/stdout and tightly coupled to the dashboard crate. We need the same commands accessible from:
- Terminal CLI (stdin/stdout)
- TUI (the existing ratatui dashboard)
- REST API (the existing HTTP server)
- Future: WebSocket, remote CLI, programmatic SDK
And we need write commands (spawn, stop, drain) — not just read-only inspection.
## Design
### Architecture
```
┌──────────────────────────────────────────────────────────┐
│ Frontends │
│ │
│ ┌──────────┐ ┌─────────┐ ┌──────────┐ ┌──────────┐ │
│ │CLI / REPL│ │ TUI │ │REST /api/│ │ Future │ │
│ │(stdin/ │ │(ratatui │ │cmd?name= │ │(websocket│ │
│ │ stdout) │ │ events) │ │&arg=val │ │ etc.) │ │
│ └────┬─────┘ └────┬────┘ └────┬─────┘ └────┬─────┘ │
│ │ │ │ │ │
│ └─────────────┴─────┬──────┴──────────────┘ │
└───────────────────────────┼──────────────────────────────┘
│
┌────────▼────────┐
│ CommandRouter │
│ │
│ name → handler │
│ dispatch() │
└────────┬────────┘
│
┌──────────────┼──────────────┐
│ │ │
┌──────▼──────┐ ┌────▼────┐ ┌───────▼───────┐
│ Built-in │ │Built-in │ │ Custom │
│ Read Cmds │ │Write │ │ (actor- │
│ (overview, │ │Cmds │ │ registered) │
│ workers, │ │(spawn, │ │ │
│ actors, │ │ stop, │ │ │
│ hot, ...) │ │ drain) │ │ │
└─────────────┘ └─────────┘ └────────────────┘
```
### Core Types
```rust
// crates/command/src/lib.rs
use std::collections::HashMap;
use std::sync::Arc;
/// A command request.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CommandRequest {
/// Command name (e.g., "overview", "spawn", "actors").
pub command: String,
/// Named arguments. Values are JSON for flexibility.
pub args: HashMap<String, serde_json::Value>,
}
/// A command response. Always JSON-serializable.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CommandResponse {
pub ok: bool,
pub command: String,
#[serde(skip_serializing_if = "Option::is_none")]
pub data: Option<serde_json::Value>,
#[serde(skip_serializing_if = "Option::is_none")]
pub error: Option<String>,
}
impl CommandResponse {
pub fn ok(command: &str, data: impl Serialize) -> Self {
Self {
ok: true,
command: command.to_string(),
data: Some(serde_json::to_value(data).unwrap_or(serde_json::Value::Null)),
error: None,
}
}
pub fn err(command: &str, msg: impl Into<String>) -> Self {
Self {
ok: false,
command: command.to_string(),
data: None,
error: Some(msg.into()),
}
}
/// Serialize to a single JSON line (for REPL protocol).
pub fn to_json_line(&self) -> String {
serde_json::to_string(self).unwrap_or_else(|e| {
format!(r#"{{"ok":false,"command":"","error":"serialization: {e}"}}"#)
})
}
}
```
### CommandHandler Trait
```rust
/// Metadata about a command, used for help text and validation.
pub struct CommandMeta {
pub name: &'static str,
pub description: &'static str,
pub usage: &'static str,
/// Whether this command mutates state (spawn, stop, etc.)
pub is_write: bool,
}
/// A command handler. Implementations are stateless — all state
/// comes through CommandContext.
pub trait CommandHandler: Send + Sync {
fn meta(&self) -> CommandMeta;
fn handle(&self, args: &HashMap<String, serde_json::Value>, ctx: &CommandContext) -> CommandResponse;
}
```
### CommandContext
```rust
/// Context available to command handlers.
///
/// Contains references to runtime subsystems. Optional fields allow
/// commands to work in both standalone and distributed configurations.
pub struct CommandContext {
pub runtime: Arc<swactor::runtime::Runtime>,
pub stats_collector: Option<Arc<crate::StatsCollector>>,
// Distribution (only present when running distributed)
pub dist_node: Option<Arc<std::sync::Mutex<distribution::node::DistributedNode>>>,
// Extensibility: arbitrary typed data that custom commands can access
extensions: HashMap<std::any::TypeId, Box<dyn std::any::Any + Send + Sync>>,
}
impl CommandContext {
/// Retrieve a typed extension.
pub fn get_ext<T: 'static + Send + Sync>(&self) -> Option<&T> {
self.extensions.get(&std::any::TypeId::of::<T>())
.and_then(|b| b.downcast_ref())
}
/// Add a typed extension.
pub fn set_ext<T: 'static + Send + Sync>(&mut self, val: T) {
self.extensions.insert(std::any::TypeId::of::<T>(), Box::new(val));
}
}
```
### CommandRouter
```rust
/// Central command dispatch.
pub struct CommandRouter {
handlers: HashMap<String, Box<dyn CommandHandler>>,
}
impl CommandRouter {
pub fn new() -> Self {
Self { handlers: HashMap::new() }
}
/// Register all built-in commands.
pub fn with_builtins(mut self) -> Self {
self.register(Box::new(builtins::HelpCommand));
self.register(Box::new(builtins::OverviewCommand));
self.register(Box::new(builtins::WorkersCommand));
self.register(Box::new(builtins::WorkerCommand));
self.register(Box::new(builtins::ActorsCommand));
self.register(Box::new(builtins::ActorCommand));
self.register(Box::new(builtins::HotCommand));
self.register(Box::new(builtins::PhasesCommand));
self.register(Box::new(builtins::DiffCommand));
// Write commands
self.register(Box::new(builtins::SpawnCommand));
self.register(Box::new(builtins::StopCommand));
self.register(Box::new(builtins::ShutdownCommand));
// Distribution-aware commands (no-op if dist_node is None)
self.register(Box::new(builtins::NodesCommand));
self.register(Box::new(builtins::RegistryCommand));
self.register(Box::new(builtins::ResolveCommand));
self.register(Box::new(builtins::DrainCommand));
self
}
/// Register a custom command handler.
pub fn register(&mut self, handler: Box<dyn CommandHandler>) {
let name = handler.meta().name.to_string();
self.handlers.insert(name, handler);
}
/// Dispatch a command request.
pub fn dispatch(&self, req: &CommandRequest, ctx: &CommandContext) -> CommandResponse {
match self.handlers.get(&req.command) {
Some(handler) => handler.handle(&req.args, ctx),
None => CommandResponse::err(
&req.command,
format!("unknown command `{}` — try `help`", req.command),
),
}
}
/// List all registered commands (for help text).
pub fn commands(&self) -> Vec<&CommandMeta> {
// sorted by name for stable output
let mut metas: Vec<_> = self.handlers.values()
.map(|h| h.meta())
.collect();
metas.sort_by_key(|m| m.name);
metas
}
}
```
### Built-in Commands (MVP)
**Read commands** (extracted from existing `investigate.rs`):
| Command | Args | Description | Source |
|---------|------|-------------|--------|
| `help` | — | List all commands | `cmd_help()` |
| `overview` | — | Runtime summary | `cmd_overview()` |
| `workers` | — | Per-worker stats | `cmd_workers()` |
| `worker` | `id: int` | Single worker detail | `cmd_worker()` |
| `actors` | `sort`, `limit`, `worker` | List actors | `cmd_actors()` |
| `actor` | `prefix: str` | Find by address prefix | `cmd_actor()` |
| `hot` | `n: int` | Top N by mailbox depth | `cmd_hot()` |
| `phases` | `worker: int?` | Tick phase breakdown | `cmd_phases()` |
| `diff` | `seconds: float` | Snapshot delta | `cmd_diff()` |
| `nodes` | — | Cluster member list | **new** |
| `registry` | — | All registered names | **new** |
| `resolve` | `name: str` | Look up a name | **new** |
**Write commands** (new):
| Command | Args | Description |
|---------|------|-------------|
| `stop` | `prefix` or `name` | Stop an actor (poison + cleanup) |
| `drain` | `node: str?` | Stop accepting new actors on a node, let existing drain |
| `shutdown` | `node: str?` | Graceful shutdown (drain + stop all) |
| `spawn` | `factory`, `node?`, `constraints?` | Spawn from a registered factory |
`spawn` requires a **factory registry** — actors register factory functions that can be invoked by name:
```rust
pub trait ActorFactory: Send + Sync {
fn name(&self) -> &str;
fn spawn(&self, runtime: &Runtime, args: &HashMap<String, serde_json::Value>)
-> Result<ActorAddress, String>;
}
```
### Migration from investigate.rs
The existing `crates/runtime-dashboard/src/investigate.rs` has 9 command functions. Migration strategy:
1. Create `crates/command/src/builtins/` with one file per command (or grouped by category).
2. Each `cmd_*` function becomes a `CommandHandler` impl. The logic is identical — just restructured.
3. `dispatch_repl` becomes `CommandRouter::dispatch` with a text-to-`CommandRequest` parser.
4. `dispatch_command` (HTTP) becomes `CommandRouter::dispatch` with query-param-to-`CommandRequest` parser.
5. `run_investigate` remains in the dashboard crate as a thin loop over `CommandRouter`.
Example extraction:
```rust
// crates/command/src/builtins/overview.rs
pub struct OverviewCommand;
impl CommandHandler for OverviewCommand {
fn meta(&self) -> CommandMeta {
CommandMeta {
name: "overview",
description: "Summary: worker count, actor count, total messages, panics",
usage: "overview",
is_write: false,
}
}
fn handle(&self, _args: &HashMap<String, serde_json::Value>, ctx: &CommandContext) -> CommandResponse {
let stats = ctx.enriched_stats();
let total_msgs: u64 = stats.workers.iter().map(|w| w.messages_processed).sum();
// ... same logic as existing cmd_overview ...
CommandResponse::ok("overview", serde_json::json!({
"workers": stats.num_workers,
"actors": stats.actor_details.len(),
"total_messages_processed": total_msgs,
// ...
}))
}
}
```
### Frontend Adapters
Each frontend is a thin adapter that converts its input format into `CommandRequest` and `CommandResponse` back to its output format.
**REPL adapter** (stdin/stdout):
```rust
// crates/command/src/adapters/repl.rs
pub fn parse_line(line: &str) -> CommandRequest {
let parts: Vec<&str> = line.split_whitespace().collect();
let command = parts.first().unwrap_or(&"help").to_string();
let args = parse_positional_and_flags(&parts[1..]);
CommandRequest { command, args }
}
```
**REST adapter** (HTTP query params):
```rust
// crates/command/src/adapters/rest.rs
pub fn from_query_params(params: &HashMap<String, String>) -> CommandRequest {
let command = params.get("cmd").cloned().unwrap_or_else(|| "help".into());
let args: HashMap<String, serde_json::Value> = params.iter()
.filter(|(k, _)| *k != "cmd")
.map(|(k, v)| (k.clone(), serde_json::Value::String(v.clone())))
.collect();
CommandRequest { command, args }
}
```
**TUI adapter**: TUI input field text → `parse_line()` → `dispatch()` → render response in panel.
### Custom Commands (actor-registered)
Actors can register command handlers at runtime through the `Ctx`:
```rust
impl Ctx<'_> {
/// Register a command that routes to this actor.
/// When the command is invoked, a CommandInvocation message
/// is sent to this actor's mailbox.
pub fn register_command(&self, name: &str, description: &str) {
self.inner.register_command(self.self_addr, name, description);
}
}
```
When a custom command is dispatched:
1. Router finds it's actor-registered.
2. Sends `CommandInvocation { command, args, reply_addr }` to the actor's mailbox.
3. The actor processes it and sends `CommandResult { data }` back to `reply_addr`.
4. Router waits on a one-shot inbox (with timeout, e.g. 5 seconds).
5. Returns the response.
```rust
/// Sent to an actor when its registered command is invoked.
#[derive(Debug, Clone)]
pub struct CommandInvocation {
pub command: String,
pub args: HashMap<String, serde_json::Value>,
pub reply_addr: ActorAddress,
}
/// Sent back by the actor with the command result.
#[derive(Debug, Clone)]
pub struct CommandResult {
pub data: serde_json::Value,
}
```
This mechanism allows any actor to expose operational endpoints without modifying the command crate.
### Files Modified
| File | Change |
|------|--------|
| `crates/command/` | **New crate** |
| `crates/command/Cargo.toml` | Dependencies: swactor, serde, serde_json |
| `crates/command/src/lib.rs` | `CommandRouter`, `CommandHandler`, `CommandRequest`/`Response`, `CommandContext` |
| `crates/command/src/builtins/` | Built-in command handlers (mod.rs + per-command files) |
| `crates/command/src/adapters/` | REPL and REST input parsers |
| `crates/runtime-dashboard/src/investigate.rs` | Refactored: thin REPL loop over CommandRouter |
| `crates/runtime-dashboard/src/server.rs` | REST endpoints use CommandRouter |
| `crates/runtime-dashboard/Cargo.toml` | Depends on `crates/command` |
| `Cargo.toml` | Add `crates/command` to workspace |
### Tests
- **dispatch_known_command**: `overview` returns `ok: true` with expected fields
- **dispatch_unknown_command**: returns `ok: false` with helpful error
- **help_lists_all**: `help` response includes all registered command names
- **parse_repl_line**: `"actors --sort mailbox --limit 5"` → correct `CommandRequest`
- **parse_query_params**: `{cmd: "actor", prefix: "a1b2"}` → correct `CommandRequest`
- **custom_command_dispatch**: register actor command, invoke, verify response
- **custom_command_timeout**: registered actor doesn't respond, verify timeout error
- **write_command_stop**: stop an actor via command, verify it's poisoned