swactor/docs/os-design/03-node-capabilities.md

# Node Capabilities — Hardware Detection & Placement Constraints

## Problem

Swactor targets heterogeneous clusters: some nodes have GPUs, others have large RAM, others are lightweight ARM devices. When spawning an actor (e.g., a model inference worker), the system needs to place it on a node with the right hardware. Today, placement is round-robin — no awareness of what each node can do.

## Design

### Separate Crate

`crates/capabilities/` is a **standalone crate** with no dependency on the swactor core runtime. It's a pure detection + constraint-matching library.

```toml
# crates/capabilities/Cargo.toml
[package]
name = "swactor-capabilities"
version = "0.1.0"
edition = "2024"

[features]
default = ["detect"]
detect = ["dep:sysinfo"]
gpu-nvidia = []
# gpu-vulkan = []   # future

[dependencies]
serde = { version = "1", features = ["derive"] }
sysinfo = { version = "0.33", optional = true }
```

### Types

```rust
// crates/capabilities/src/lib.rs

/// A capability value. Kept simple — three variants cover all practical needs.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub enum CapValue {
    Bool(bool),
    Int(i64),
    Str(String),
}

/// All capabilities of a node.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct NodeCapabilities {
    labels: BTreeMap<String, CapValue>,
}

impl NodeCapabilities {
    pub fn new() -> Self { Self::default() }

    /// Get a label value.
    pub fn get(&self, key: &str) -> Option<&CapValue> {
        self.labels.get(key)
    }

    /// Set a label.
    pub fn set(&mut self, key: impl Into<String>, value: CapValue) {
        self.labels.insert(key.into(), value);
    }

    /// Merge in additional labels (overwriting on conflict).
    pub fn with_labels(mut self, extra: BTreeMap<String, CapValue>) -> Self {
        self.labels.extend(extra);
        self
    }

    /// Check if all constraints in a requirement are satisfied.
    pub fn satisfies(&self, requirement: &PlacementRequirement) -> bool {
        requirement.constraints.iter().all(|c| self.satisfies_one(c))
    }

    fn satisfies_one(&self, constraint: &PlacementConstraint) -> bool {
        match constraint {
            PlacementConstraint::Equals(key, expected) => {
                self.labels.get(key.as_str()) == Some(expected)
            }
            PlacementConstraint::MinInt(key, min) => {
                matches!(self.labels.get(key.as_str()), Some(CapValue::Int(v)) if *v >= *min)
            }
            PlacementConstraint::HasLabel(key) => {
                self.labels.contains_key(key.as_str())
            }
        }
    }

    /// All labels as a reference.
    pub fn labels(&self) -> &BTreeMap<String, CapValue> {
        &self.labels
    }
}
```

### Auto-Detection

```rust
impl NodeCapabilities {
    /// Auto-detect system capabilities.
    /// Always detects arch and os. Feature-gated backends detect more.
    pub fn detect() -> Self {
        let mut caps = Self::new();

        // Always available (no feature gate)
        caps.set("arch", CapValue::Str(std::env::consts::ARCH.to_string()));
        caps.set("os", CapValue::Str(std::env::consts::OS.to_string()));

        #[cfg(feature = "detect")]
        {
            Self::detect_sysinfo(&mut caps);
        }

        #[cfg(feature = "gpu-nvidia")]
        {
            Self::detect_nvidia(&mut caps);
        }

        caps
    }

    #[cfg(feature = "detect")]
    fn detect_sysinfo(caps: &mut Self) {
        use sysinfo::System;
        let sys = System::new_all();

        caps.set("cpu_count", CapValue::Int(sys.cpus().len() as i64));
        caps.set("ram_mb", CapValue::Int((sys.total_memory() / (1024 * 1024)) as i64));

        if let Ok(hostname) = hostname::get() {
            if let Some(name) = hostname.to_str() {
                caps.set("hostname", CapValue::Str(name.to_string()));
            }
        }
    }

    #[cfg(feature = "gpu-nvidia")]
    fn detect_nvidia(caps: &mut Self) {
        // Shell out to nvidia-smi for maximum compatibility.
        // Parsing XML output is more robust than CSV for varying driver versions.
        let output = std::process::Command::new("nvidia-smi")
            .args(["--query-gpu=name,memory.total", "--format=csv,noheader,nounits"])
            .output();

        match output {
            Ok(out) if out.status.success() => {
                let stdout = String::from_utf8_lossy(&out.stdout);
                let lines: Vec<&str> = stdout.trim().lines().collect();
                caps.set("gpu_nvidia", CapValue::Bool(true));
                caps.set("gpu_count", CapValue::Int(lines.len() as i64));
                // First GPU's VRAM as representative
                if let Some(line) = lines.first() {
                    let parts: Vec<&str> = line.split(", ").collect();
                    if let Some(name) = parts.first() {
                        caps.set("gpu_name", CapValue::Str(name.trim().to_string()));
                    }
                    if let Some(vram) = parts.get(1).and_then(|s| s.trim().parse::<i64>().ok()) {
                        caps.set("gpu_vram_mb", CapValue::Int(vram));
                    }
                }
            }
            _ => {
                caps.set("gpu_nvidia", CapValue::Bool(false));
            }
        }
    }
}
```

### Placement Constraints

```rust
/// A single constraint on node capabilities.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub enum PlacementConstraint {
    /// Label must exist and equal the given value.
    Equals(String, CapValue),
    /// Label must exist and be >= the given integer value.
    MinInt(String, i64),
    /// Label must exist (any value).
    HasLabel(String),
}

/// A full placement requirement. All constraints must be satisfied (AND).
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct PlacementRequirement {
    pub constraints: Vec<PlacementConstraint>,
}

impl PlacementRequirement {
    pub fn new() -> Self { Self::default() }

    /// Builder: require a label equals a value.
    pub fn equals(mut self, key: impl Into<String>, value: CapValue) -> Self {
        self.constraints.push(PlacementConstraint::Equals(key.into(), value));
        self
    }

    /// Builder: require an integer label >= min.
    pub fn min_int(mut self, key: impl Into<String>, min: i64) -> Self {
        self.constraints.push(PlacementConstraint::MinInt(key.into(), min));
        self
    }

    /// Builder: require a label exists.
    pub fn has(mut self, key: impl Into<String>) -> Self {
        self.constraints.push(PlacementConstraint::HasLabel(key.into()));
        self
    }

    /// Check if empty (no constraints — any node is acceptable).
    pub fn is_empty(&self) -> bool {
        self.constraints.is_empty()
    }
}
```

### Integration with Distribution

**NodeRecord extension** (`crates/distribution/src/types.rs`):

```rust
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct NodeRecord {
    pub node_id: NodeId,
    pub addr: SocketAddr,
    pub state: MemberState,
    pub incarnation: u64,
    // NEW (optional — backwards compatible):
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub capabilities: Option<swactor_capabilities::NodeCapabilities>,
}
```

**Capabilities flow**:

1. On startup, the node detects capabilities: `NodeCapabilities::detect().with_labels(operator_labels)`.
2. Capabilities are included in the node's own `NodeRecord`.
3. When a node joins (via `JoinResponse`), it receives other nodes' capabilities.
4. Capabilities are piggybacked on SWIM protocol messages (membership updates already carry `NodeRecord`).

**Cluster-level placement** (new function in distribution):

```rust
// crates/distribution/src/node.rs

impl DistributedNode {
    /// Find nodes that satisfy a placement requirement.
    /// Returns matching nodes sorted by preference (e.g., least loaded first).
    pub fn find_suitable_nodes(
        &self,
        requirement: &PlacementRequirement,
    ) -> Vec<NodeRecord> {
        self.members()
            .into_iter()
            .filter(|node| {
                node.capabilities.as_ref()
                    .map(|caps| caps.satisfies(requirement))
                    .unwrap_or(requirement.is_empty())
            })
            .collect()
    }
}
```

### Example Usage

```rust
// Operator starts a node with custom labels:
let caps = NodeCapabilities::detect()
    .with_labels(btreemap! {
        "role".into() => CapValue::Str("inference".into()),
        "region".into() => CapValue::Str("us-east".into()),
    });

// An actor specifies placement requirements:
let requirement = PlacementRequirement::new()
    .has("gpu_nvidia")
    .min_int("gpu_vram_mb", 8000)
    .equals("region", CapValue::Str("us-east".into()));

// Supervisor finds suitable nodes:
let nodes = dist_node.find_suitable_nodes(&requirement);
```

### Files Modified

| File | Change |
|------|--------|
| `crates/capabilities/` | **New crate** |
| `crates/capabilities/Cargo.toml` | Package definition, feature flags |
| `crates/capabilities/src/lib.rs` | `NodeCapabilities`, `CapValue`, `PlacementConstraint`, `PlacementRequirement`, detection |
| `crates/distribution/Cargo.toml` | Optional dependency on `swactor-capabilities` |
| `crates/distribution/src/types.rs` | Optional `capabilities` field on `NodeRecord` |
| `crates/distribution/src/node.rs` | `find_suitable_nodes()`, capabilities in join flow |
| `Cargo.toml` | Add `crates/capabilities` to workspace members |

### Tests

- **detect_basics**: `NodeCapabilities::detect()` always has `arch` and `os` labels
- **satisfies_equals**: constraint matches/doesn't match
- **satisfies_min_int**: integer comparison works correctly
- **satisfies_has_label**: existence check works
- **empty_requirement**: matches any node
- **combined_constraints**: multiple constraints all must pass (AND)
- **custom_labels**: operator labels merge correctly, override detection
- **find_suitable_nodes**: integration test with mock node records and varying capabilities