swactor/docs/os-design/03-node-capabilities.md

Node Capabilities — Hardware Detection & Placement Constraints

Problem

Swactor targets heterogeneous clusters: some nodes have GPUs, others have large RAM, others are lightweight ARM devices. When spawning an actor (e.g., a model inference worker), the system needs to place it on a node with the right hardware. Today, placement is round-robin — no awareness of what each node can do.

Design

Separate Crate

crates/capabilities/ is a standalone crate with no dependency on the swactor core runtime. It's a pure detection + constraint-matching library.

# crates/capabilities/Cargo.toml
[package]
name = "swactor-capabilities"
version = "0.1.0"
edition = "2024"

[features]
default = ["detect"]
detect = ["dep:sysinfo"]
gpu-nvidia = []
# gpu-vulkan = []   # future

[dependencies]
serde = { version = "1", features = ["derive"] }
sysinfo = { version = "0.33", optional = true }

Types

// crates/capabilities/src/lib.rs

/// A capability value. Kept simple — three variants cover all practical needs.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub enum CapValue {
    Bool(bool),
    Int(i64),
    Str(String),
}

/// All capabilities of a node.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct NodeCapabilities {
    labels: BTreeMap<String, CapValue>,
}

impl NodeCapabilities {
    pub fn new() -> Self { Self::default() }

    /// Get a label value.
    pub fn get(&self, key: &str) -> Option<&CapValue> {
        self.labels.get(key)
    }

    /// Set a label.
    pub fn set(&mut self, key: impl Into<String>, value: CapValue) {
        self.labels.insert(key.into(), value);
    }

    /// Merge in additional labels (overwriting on conflict).
    pub fn with_labels(mut self, extra: BTreeMap<String, CapValue>) -> Self {
        self.labels.extend(extra);
        self
    }

    /// Check if all constraints in a requirement are satisfied.
    pub fn satisfies(&self, requirement: &PlacementRequirement) -> bool {
        requirement.constraints.iter().all(|c| self.satisfies_one(c))
    }

    fn satisfies_one(&self, constraint: &PlacementConstraint) -> bool {
        match constraint {
            PlacementConstraint::Equals(key, expected) => {
                self.labels.get(key.as_str()) == Some(expected)
            }
            PlacementConstraint::MinInt(key, min) => {
                matches!(self.labels.get(key.as_str()), Some(CapValue::Int(v)) if *v >= *min)
            }
            PlacementConstraint::HasLabel(key) => {
                self.labels.contains_key(key.as_str())
            }
        }
    }

    /// All labels as a reference.
    pub fn labels(&self) -> &BTreeMap<String, CapValue> {
        &self.labels
    }
}

Auto-Detection

impl NodeCapabilities {
    /// Auto-detect system capabilities.
    /// Always detects arch and os. Feature-gated backends detect more.
    pub fn detect() -> Self {
        let mut caps = Self::new();

        // Always available (no feature gate)
        caps.set("arch", CapValue::Str(std::env::consts::ARCH.to_string()));
        caps.set("os", CapValue::Str(std::env::consts::OS.to_string()));

        #[cfg(feature = "detect")]
        {
            Self::detect_sysinfo(&mut caps);
        }

        #[cfg(feature = "gpu-nvidia")]
        {
            Self::detect_nvidia(&mut caps);
        }

        caps
    }

    #[cfg(feature = "detect")]
    fn detect_sysinfo(caps: &mut Self) {
        use sysinfo::System;
        let sys = System::new_all();

        caps.set("cpu_count", CapValue::Int(sys.cpus().len() as i64));
        caps.set("ram_mb", CapValue::Int((sys.total_memory() / (1024 * 1024)) as i64));

        if let Ok(hostname) = hostname::get() {
            if let Some(name) = hostname.to_str() {
                caps.set("hostname", CapValue::Str(name.to_string()));
            }
        }
    }

    #[cfg(feature = "gpu-nvidia")]
    fn detect_nvidia(caps: &mut Self) {
        // Shell out to nvidia-smi for maximum compatibility.
        // Parsing XML output is more robust than CSV for varying driver versions.
        let output = std::process::Command::new("nvidia-smi")
            .args(["--query-gpu=name,memory.total", "--format=csv,noheader,nounits"])
            .output();

        match output {
            Ok(out) if out.status.success() => {
                let stdout = String::from_utf8_lossy(&out.stdout);
                let lines: Vec<&str> = stdout.trim().lines().collect();
                caps.set("gpu_nvidia", CapValue::Bool(true));
                caps.set("gpu_count", CapValue::Int(lines.len() as i64));
                // First GPU's VRAM as representative
                if let Some(line) = lines.first() {
                    let parts: Vec<&str> = line.split(", ").collect();
                    if let Some(name) = parts.first() {
                        caps.set("gpu_name", CapValue::Str(name.trim().to_string()));
                    }
                    if let Some(vram) = parts.get(1).and_then(|s| s.trim().parse::<i64>().ok()) {
                        caps.set("gpu_vram_mb", CapValue::Int(vram));
                    }
                }
            }
            _ => {
                caps.set("gpu_nvidia", CapValue::Bool(false));
            }
        }
    }
}

Placement Constraints

/// A single constraint on node capabilities.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub enum PlacementConstraint {
    /// Label must exist and equal the given value.
    Equals(String, CapValue),
    /// Label must exist and be >= the given integer value.
    MinInt(String, i64),
    /// Label must exist (any value).
    HasLabel(String),
}

/// A full placement requirement. All constraints must be satisfied (AND).
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct PlacementRequirement {
    pub constraints: Vec<PlacementConstraint>,
}

impl PlacementRequirement {
    pub fn new() -> Self { Self::default() }

    /// Builder: require a label equals a value.
    pub fn equals(mut self, key: impl Into<String>, value: CapValue) -> Self {
        self.constraints.push(PlacementConstraint::Equals(key.into(), value));
        self
    }

    /// Builder: require an integer label >= min.
    pub fn min_int(mut self, key: impl Into<String>, min: i64) -> Self {
        self.constraints.push(PlacementConstraint::MinInt(key.into(), min));
        self
    }

    /// Builder: require a label exists.
    pub fn has(mut self, key: impl Into<String>) -> Self {
        self.constraints.push(PlacementConstraint::HasLabel(key.into()));
        self
    }

    /// Check if empty (no constraints — any node is acceptable).
    pub fn is_empty(&self) -> bool {
        self.constraints.is_empty()
    }
}

Integration with Distribution

NodeRecord extension (crates/distribution/src/types.rs):

#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct NodeRecord {
    pub node_id: NodeId,
    pub addr: SocketAddr,
    pub state: MemberState,
    pub incarnation: u64,
    // NEW (optional — backwards compatible):
    #[serde(default, skip_serializing_if = "Option::is_none")]
    pub capabilities: Option<swactor_capabilities::NodeCapabilities>,
}

Capabilities flow:

1. On startup, the node detects capabilities: NodeCapabilities::detect().with_labels(operator_labels).
2. Capabilities are included in the node's own NodeRecord.
3. When a node joins (via JoinResponse), it receives other nodes' capabilities.
4. Capabilities are piggybacked on SWIM protocol messages (membership updates already carry NodeRecord).

Cluster-level placement (new function in distribution):

// crates/distribution/src/node.rs

impl DistributedNode {
    /// Find nodes that satisfy a placement requirement.
    /// Returns matching nodes sorted by preference (e.g., least loaded first).
    pub fn find_suitable_nodes(
        &self,
        requirement: &PlacementRequirement,
    ) -> Vec<NodeRecord> {
        self.members()
            .into_iter()
            .filter(|node| {
                node.capabilities.as_ref()
                    .map(|caps| caps.satisfies(requirement))
                    .unwrap_or(requirement.is_empty())
            })
            .collect()
    }
}

Example Usage

// Operator starts a node with custom labels:
let caps = NodeCapabilities::detect()
    .with_labels(btreemap! {
        "role".into() => CapValue::Str("inference".into()),
        "region".into() => CapValue::Str("us-east".into()),
    });

// An actor specifies placement requirements:
let requirement = PlacementRequirement::new()
    .has("gpu_nvidia")
    .min_int("gpu_vram_mb", 8000)
    .equals("region", CapValue::Str("us-east".into()));

// Supervisor finds suitable nodes:
let nodes = dist_node.find_suitable_nodes(&requirement);

Files Modified

File	Change
crates/capabilities/	New crate
crates/capabilities/Cargo.toml	Package definition, feature flags
crates/capabilities/src/lib.rs	NodeCapabilities, CapValue, PlacementConstraint, PlacementRequirement, detection
crates/distribution/Cargo.toml	Optional dependency on swactor-capabilities
crates/distribution/src/types.rs	Optional capabilities field on NodeRecord
crates/distribution/src/node.rs	find_suitable_nodes(), capabilities in join flow
Cargo.toml	Add crates/capabilities to workspace members

Tests

• detect_basics: NodeCapabilities::detect() always has arch and os labels
• satisfies_equals: constraint matches/doesn't match
• satisfies_min_int: integer comparison works correctly
• satisfies_has_label: existence check works
• empty_requirement: matches any node
• combined_constraints: multiple constraints all must pass (AND)
• custom_labels: operator labels merge correctly, override detection
• find_suitable_nodes: integration test with mock node records and varying capabilities