Add watch/unwatch API to the actor system so actors can monitor each other's liveness. When a watched actor dies (panic or stop), watchers receive an ActorExited notification via on_actor_exit(). - ExitReason enum (Stopped, Panicked, NodeDown) and ActorExited struct - ContextInner::watch()/unwatch() + Ctx typed wrappers - ActorInterface::on_actor_exit() default method (system message fallback) - WatchRegistry in worker with bidirectional tracking - Death notification dispatch as phase 5b in tick_once - Runtime-level watch for external callers - 10 behavioral tests in tests/watch_api.rs - Design documents for OS features in docs/os-design/ Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
398 lines
15 KiB
Markdown
398 lines
15 KiB
Markdown
# Command Interface — Frontend-Agnostic Dispatch
|
|
|
|
## Problem
|
|
|
|
The runtime has an investigate/REPL protocol (`crates/runtime-dashboard/src/investigate.rs`) that accepts text commands and returns JSON. It works, but it's hardcoded to stdin/stdout and tightly coupled to the dashboard crate. We need the same commands accessible from:
|
|
|
|
- Terminal CLI (stdin/stdout)
|
|
- TUI (the existing ratatui dashboard)
|
|
- REST API (the existing HTTP server)
|
|
- Future: WebSocket, remote CLI, programmatic SDK
|
|
|
|
And we need write commands (spawn, stop, drain) — not just read-only inspection.
|
|
|
|
## Design
|
|
|
|
### Architecture
|
|
|
|
```
|
|
┌──────────────────────────────────────────────────────────┐
|
|
│ Frontends │
|
|
│ │
|
|
│ ┌──────────┐ ┌─────────┐ ┌──────────┐ ┌──────────┐ │
|
|
│ │CLI / REPL│ │ TUI │ │REST /api/│ │ Future │ │
|
|
│ │(stdin/ │ │(ratatui │ │cmd?name= │ │(websocket│ │
|
|
│ │ stdout) │ │ events) │ │&arg=val │ │ etc.) │ │
|
|
│ └────┬─────┘ └────┬────┘ └────┬─────┘ └────┬─────┘ │
|
|
│ │ │ │ │ │
|
|
│ └─────────────┴─────┬──────┴──────────────┘ │
|
|
└───────────────────────────┼──────────────────────────────┘
|
|
│
|
|
┌────────▼────────┐
|
|
│ CommandRouter │
|
|
│ │
|
|
│ name → handler │
|
|
│ dispatch() │
|
|
└────────┬────────┘
|
|
│
|
|
┌──────────────┼──────────────┐
|
|
│ │ │
|
|
┌──────▼──────┐ ┌────▼────┐ ┌───────▼───────┐
|
|
│ Built-in │ │Built-in │ │ Custom │
|
|
│ Read Cmds │ │Write │ │ (actor- │
|
|
│ (overview, │ │Cmds │ │ registered) │
|
|
│ workers, │ │(spawn, │ │ │
|
|
│ actors, │ │ stop, │ │ │
|
|
│ hot, ...) │ │ drain) │ │ │
|
|
└─────────────┘ └─────────┘ └────────────────┘
|
|
```
|
|
|
|
### Core Types
|
|
|
|
```rust
|
|
// crates/command/src/lib.rs
|
|
|
|
use std::collections::HashMap;
|
|
use std::sync::Arc;
|
|
|
|
/// A command request.
|
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
|
pub struct CommandRequest {
|
|
/// Command name (e.g., "overview", "spawn", "actors").
|
|
pub command: String,
|
|
/// Named arguments. Values are JSON for flexibility.
|
|
pub args: HashMap<String, serde_json::Value>,
|
|
}
|
|
|
|
/// A command response. Always JSON-serializable.
|
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
|
pub struct CommandResponse {
|
|
pub ok: bool,
|
|
pub command: String,
|
|
#[serde(skip_serializing_if = "Option::is_none")]
|
|
pub data: Option<serde_json::Value>,
|
|
#[serde(skip_serializing_if = "Option::is_none")]
|
|
pub error: Option<String>,
|
|
}
|
|
|
|
impl CommandResponse {
|
|
pub fn ok(command: &str, data: impl Serialize) -> Self {
|
|
Self {
|
|
ok: true,
|
|
command: command.to_string(),
|
|
data: Some(serde_json::to_value(data).unwrap_or(serde_json::Value::Null)),
|
|
error: None,
|
|
}
|
|
}
|
|
|
|
pub fn err(command: &str, msg: impl Into<String>) -> Self {
|
|
Self {
|
|
ok: false,
|
|
command: command.to_string(),
|
|
data: None,
|
|
error: Some(msg.into()),
|
|
}
|
|
}
|
|
|
|
/// Serialize to a single JSON line (for REPL protocol).
|
|
pub fn to_json_line(&self) -> String {
|
|
serde_json::to_string(self).unwrap_or_else(|e| {
|
|
format!(r#"{{"ok":false,"command":"","error":"serialization: {e}"}}"#)
|
|
})
|
|
}
|
|
}
|
|
```
|
|
|
|
### CommandHandler Trait
|
|
|
|
```rust
|
|
/// Metadata about a command, used for help text and validation.
|
|
pub struct CommandMeta {
|
|
pub name: &'static str,
|
|
pub description: &'static str,
|
|
pub usage: &'static str,
|
|
/// Whether this command mutates state (spawn, stop, etc.)
|
|
pub is_write: bool,
|
|
}
|
|
|
|
/// A command handler. Implementations are stateless — all state
|
|
/// comes through CommandContext.
|
|
pub trait CommandHandler: Send + Sync {
|
|
fn meta(&self) -> CommandMeta;
|
|
fn handle(&self, args: &HashMap<String, serde_json::Value>, ctx: &CommandContext) -> CommandResponse;
|
|
}
|
|
```
|
|
|
|
### CommandContext
|
|
|
|
```rust
|
|
/// Context available to command handlers.
|
|
///
|
|
/// Contains references to runtime subsystems. Optional fields allow
|
|
/// commands to work in both standalone and distributed configurations.
|
|
pub struct CommandContext {
|
|
pub runtime: Arc<swactor::runtime::Runtime>,
|
|
pub stats_collector: Option<Arc<crate::StatsCollector>>,
|
|
// Distribution (only present when running distributed)
|
|
pub dist_node: Option<Arc<std::sync::Mutex<distribution::node::DistributedNode>>>,
|
|
// Extensibility: arbitrary typed data that custom commands can access
|
|
extensions: HashMap<std::any::TypeId, Box<dyn std::any::Any + Send + Sync>>,
|
|
}
|
|
|
|
impl CommandContext {
|
|
/// Retrieve a typed extension.
|
|
pub fn get_ext<T: 'static + Send + Sync>(&self) -> Option<&T> {
|
|
self.extensions.get(&std::any::TypeId::of::<T>())
|
|
.and_then(|b| b.downcast_ref())
|
|
}
|
|
|
|
/// Add a typed extension.
|
|
pub fn set_ext<T: 'static + Send + Sync>(&mut self, val: T) {
|
|
self.extensions.insert(std::any::TypeId::of::<T>(), Box::new(val));
|
|
}
|
|
}
|
|
```
|
|
|
|
### CommandRouter
|
|
|
|
```rust
|
|
/// Central command dispatch.
|
|
pub struct CommandRouter {
|
|
handlers: HashMap<String, Box<dyn CommandHandler>>,
|
|
}
|
|
|
|
impl CommandRouter {
|
|
pub fn new() -> Self {
|
|
Self { handlers: HashMap::new() }
|
|
}
|
|
|
|
/// Register all built-in commands.
|
|
pub fn with_builtins(mut self) -> Self {
|
|
self.register(Box::new(builtins::HelpCommand));
|
|
self.register(Box::new(builtins::OverviewCommand));
|
|
self.register(Box::new(builtins::WorkersCommand));
|
|
self.register(Box::new(builtins::WorkerCommand));
|
|
self.register(Box::new(builtins::ActorsCommand));
|
|
self.register(Box::new(builtins::ActorCommand));
|
|
self.register(Box::new(builtins::HotCommand));
|
|
self.register(Box::new(builtins::PhasesCommand));
|
|
self.register(Box::new(builtins::DiffCommand));
|
|
// Write commands
|
|
self.register(Box::new(builtins::SpawnCommand));
|
|
self.register(Box::new(builtins::StopCommand));
|
|
self.register(Box::new(builtins::ShutdownCommand));
|
|
// Distribution-aware commands (no-op if dist_node is None)
|
|
self.register(Box::new(builtins::NodesCommand));
|
|
self.register(Box::new(builtins::RegistryCommand));
|
|
self.register(Box::new(builtins::ResolveCommand));
|
|
self.register(Box::new(builtins::DrainCommand));
|
|
self
|
|
}
|
|
|
|
/// Register a custom command handler.
|
|
pub fn register(&mut self, handler: Box<dyn CommandHandler>) {
|
|
let name = handler.meta().name.to_string();
|
|
self.handlers.insert(name, handler);
|
|
}
|
|
|
|
/// Dispatch a command request.
|
|
pub fn dispatch(&self, req: &CommandRequest, ctx: &CommandContext) -> CommandResponse {
|
|
match self.handlers.get(&req.command) {
|
|
Some(handler) => handler.handle(&req.args, ctx),
|
|
None => CommandResponse::err(
|
|
&req.command,
|
|
format!("unknown command `{}` — try `help`", req.command),
|
|
),
|
|
}
|
|
}
|
|
|
|
/// List all registered commands (for help text).
|
|
pub fn commands(&self) -> Vec<&CommandMeta> {
|
|
// sorted by name for stable output
|
|
let mut metas: Vec<_> = self.handlers.values()
|
|
.map(|h| h.meta())
|
|
.collect();
|
|
metas.sort_by_key(|m| m.name);
|
|
metas
|
|
}
|
|
}
|
|
```
|
|
|
|
### Built-in Commands (MVP)
|
|
|
|
**Read commands** (extracted from existing `investigate.rs`):
|
|
|
|
| Command | Args | Description | Source |
|
|
|---------|------|-------------|--------|
|
|
| `help` | — | List all commands | `cmd_help()` |
|
|
| `overview` | — | Runtime summary | `cmd_overview()` |
|
|
| `workers` | — | Per-worker stats | `cmd_workers()` |
|
|
| `worker` | `id: int` | Single worker detail | `cmd_worker()` |
|
|
| `actors` | `sort`, `limit`, `worker` | List actors | `cmd_actors()` |
|
|
| `actor` | `prefix: str` | Find by address prefix | `cmd_actor()` |
|
|
| `hot` | `n: int` | Top N by mailbox depth | `cmd_hot()` |
|
|
| `phases` | `worker: int?` | Tick phase breakdown | `cmd_phases()` |
|
|
| `diff` | `seconds: float` | Snapshot delta | `cmd_diff()` |
|
|
| `nodes` | — | Cluster member list | **new** |
|
|
| `registry` | — | All registered names | **new** |
|
|
| `resolve` | `name: str` | Look up a name | **new** |
|
|
|
|
**Write commands** (new):
|
|
|
|
| Command | Args | Description |
|
|
|---------|------|-------------|
|
|
| `stop` | `prefix` or `name` | Stop an actor (poison + cleanup) |
|
|
| `drain` | `node: str?` | Stop accepting new actors on a node, let existing drain |
|
|
| `shutdown` | `node: str?` | Graceful shutdown (drain + stop all) |
|
|
| `spawn` | `factory`, `node?`, `constraints?` | Spawn from a registered factory |
|
|
|
|
`spawn` requires a **factory registry** — actors register factory functions that can be invoked by name:
|
|
|
|
```rust
|
|
pub trait ActorFactory: Send + Sync {
|
|
fn name(&self) -> &str;
|
|
fn spawn(&self, runtime: &Runtime, args: &HashMap<String, serde_json::Value>)
|
|
-> Result<ActorAddress, String>;
|
|
}
|
|
```
|
|
|
|
### Migration from investigate.rs
|
|
|
|
The existing `crates/runtime-dashboard/src/investigate.rs` has 9 command functions. Migration strategy:
|
|
|
|
1. Create `crates/command/src/builtins/` with one file per command (or grouped by category).
|
|
2. Each `cmd_*` function becomes a `CommandHandler` impl. The logic is identical — just restructured.
|
|
3. `dispatch_repl` becomes `CommandRouter::dispatch` with a text-to-`CommandRequest` parser.
|
|
4. `dispatch_command` (HTTP) becomes `CommandRouter::dispatch` with query-param-to-`CommandRequest` parser.
|
|
5. `run_investigate` remains in the dashboard crate as a thin loop over `CommandRouter`.
|
|
|
|
Example extraction:
|
|
|
|
```rust
|
|
// crates/command/src/builtins/overview.rs
|
|
|
|
pub struct OverviewCommand;
|
|
|
|
impl CommandHandler for OverviewCommand {
|
|
fn meta(&self) -> CommandMeta {
|
|
CommandMeta {
|
|
name: "overview",
|
|
description: "Summary: worker count, actor count, total messages, panics",
|
|
usage: "overview",
|
|
is_write: false,
|
|
}
|
|
}
|
|
|
|
fn handle(&self, _args: &HashMap<String, serde_json::Value>, ctx: &CommandContext) -> CommandResponse {
|
|
let stats = ctx.enriched_stats();
|
|
let total_msgs: u64 = stats.workers.iter().map(|w| w.messages_processed).sum();
|
|
// ... same logic as existing cmd_overview ...
|
|
CommandResponse::ok("overview", serde_json::json!({
|
|
"workers": stats.num_workers,
|
|
"actors": stats.actor_details.len(),
|
|
"total_messages_processed": total_msgs,
|
|
// ...
|
|
}))
|
|
}
|
|
}
|
|
```
|
|
|
|
### Frontend Adapters
|
|
|
|
Each frontend is a thin adapter that converts its input format into `CommandRequest` and `CommandResponse` back to its output format.
|
|
|
|
**REPL adapter** (stdin/stdout):
|
|
|
|
```rust
|
|
// crates/command/src/adapters/repl.rs
|
|
|
|
pub fn parse_line(line: &str) -> CommandRequest {
|
|
let parts: Vec<&str> = line.split_whitespace().collect();
|
|
let command = parts.first().unwrap_or(&"help").to_string();
|
|
let args = parse_positional_and_flags(&parts[1..]);
|
|
CommandRequest { command, args }
|
|
}
|
|
```
|
|
|
|
**REST adapter** (HTTP query params):
|
|
|
|
```rust
|
|
// crates/command/src/adapters/rest.rs
|
|
|
|
pub fn from_query_params(params: &HashMap<String, String>) -> CommandRequest {
|
|
let command = params.get("cmd").cloned().unwrap_or_else(|| "help".into());
|
|
let args: HashMap<String, serde_json::Value> = params.iter()
|
|
.filter(|(k, _)| *k != "cmd")
|
|
.map(|(k, v)| (k.clone(), serde_json::Value::String(v.clone())))
|
|
.collect();
|
|
CommandRequest { command, args }
|
|
}
|
|
```
|
|
|
|
**TUI adapter**: TUI input field text → `parse_line()` → `dispatch()` → render response in panel.
|
|
|
|
### Custom Commands (actor-registered)
|
|
|
|
Actors can register command handlers at runtime through the `Ctx`:
|
|
|
|
```rust
|
|
impl Ctx<'_> {
|
|
/// Register a command that routes to this actor.
|
|
/// When the command is invoked, a CommandInvocation message
|
|
/// is sent to this actor's mailbox.
|
|
pub fn register_command(&self, name: &str, description: &str) {
|
|
self.inner.register_command(self.self_addr, name, description);
|
|
}
|
|
}
|
|
```
|
|
|
|
When a custom command is dispatched:
|
|
|
|
1. Router finds it's actor-registered.
|
|
2. Sends `CommandInvocation { command, args, reply_addr }` to the actor's mailbox.
|
|
3. The actor processes it and sends `CommandResult { data }` back to `reply_addr`.
|
|
4. Router waits on a one-shot inbox (with timeout, e.g. 5 seconds).
|
|
5. Returns the response.
|
|
|
|
```rust
|
|
/// Sent to an actor when its registered command is invoked.
|
|
#[derive(Debug, Clone)]
|
|
pub struct CommandInvocation {
|
|
pub command: String,
|
|
pub args: HashMap<String, serde_json::Value>,
|
|
pub reply_addr: ActorAddress,
|
|
}
|
|
|
|
/// Sent back by the actor with the command result.
|
|
#[derive(Debug, Clone)]
|
|
pub struct CommandResult {
|
|
pub data: serde_json::Value,
|
|
}
|
|
```
|
|
|
|
This mechanism allows any actor to expose operational endpoints without modifying the command crate.
|
|
|
|
### Files Modified
|
|
|
|
| File | Change |
|
|
|------|--------|
|
|
| `crates/command/` | **New crate** |
|
|
| `crates/command/Cargo.toml` | Dependencies: swactor, serde, serde_json |
|
|
| `crates/command/src/lib.rs` | `CommandRouter`, `CommandHandler`, `CommandRequest`/`Response`, `CommandContext` |
|
|
| `crates/command/src/builtins/` | Built-in command handlers (mod.rs + per-command files) |
|
|
| `crates/command/src/adapters/` | REPL and REST input parsers |
|
|
| `crates/runtime-dashboard/src/investigate.rs` | Refactored: thin REPL loop over CommandRouter |
|
|
| `crates/runtime-dashboard/src/server.rs` | REST endpoints use CommandRouter |
|
|
| `crates/runtime-dashboard/Cargo.toml` | Depends on `crates/command` |
|
|
| `Cargo.toml` | Add `crates/command` to workspace |
|
|
|
|
### Tests
|
|
|
|
- **dispatch_known_command**: `overview` returns `ok: true` with expected fields
|
|
- **dispatch_unknown_command**: returns `ok: false` with helpful error
|
|
- **help_lists_all**: `help` response includes all registered command names
|
|
- **parse_repl_line**: `"actors --sort mailbox --limit 5"` → correct `CommandRequest`
|
|
- **parse_query_params**: `{cmd: "actor", prefix: "a1b2"}` → correct `CommandRequest`
|
|
- **custom_command_dispatch**: register actor command, invoke, verify response
|
|
- **custom_command_timeout**: registered actor doesn't respond, verify timeout error
|
|
- **write_command_stop**: stop an actor via command, verify it's poisoned
|