swactor/docs/os-design/02-cluster-registry.md

Cluster-Wide Registry — Distributed Naming

Problem

Actors can only be found by their ActorAddress (a random 32-byte ID). The local AddressMap maps addresses to workers on a single node. The Kademlia directory maps addresses to NodeId. But neither provides human-readable naming or re-discovery after churn.

When a node dies and an actor is re-spawned elsewhere, it gets a new ActorAddress. Without a name-based registry, every actor that communicated with it needs manual reconfiguration. This doesn't work for churning infrastructure.

Design

Approach: Gossip-Propagated LWW-Register CRDT

Each name binding is a Last-Writer-Wins Register — the most recent write (by timestamp) wins. This matches SWIM's eventual-consistency model and reuses the existing gossip piggyback mechanism.

Why not Raft/consensus?

• Overkill for name resolution. Names don't need linearizability — eventual consistency is fine.
• SWIM already solves dissemination. We piggyback registry updates on existing protocol messages for free.
• Consensus requires a stable quorum, which conflicts with the "nodes pop in and out" use case.

Why not extend Kademlia?

• Kademlia maps ActorAddress -> NodeId. Names are a different key space (String -> ActorAddress).
• Kademlia lookups are multi-hop (iterative). Registry lookups should be local (every node has a full replica).
• The registry is small (hundreds to low-thousands of names). Full replication is cheap.

Types

// crates/distribution/src/registry.rs

/// A single name binding in the cluster registry.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct RegistryEntry {
    /// Human-readable name (e.g. "worker-pool", "metrics-collector").
    pub name: String,
    /// The actor address this name resolves to.
    pub actor_addr: ActorAddress,
    /// The node that owns this binding.
    pub node_id: NodeId,
    /// Logical timestamp for LWW conflict resolution.
    pub timestamp: u64,
    /// Generation — incremented on re-registration of the same name.
    pub generation: u64,
    /// Tombstone — true means the name has been unregistered.
    pub tombstone: bool,
}

/// Events emitted by the registry for subscribers.
#[derive(Debug, Clone)]
pub enum RegistryEvent {
    /// A name was registered or updated.
    Registered {
        name: String,
        actor_addr: ActorAddress,
        node_id: NodeId,
    },
    /// A name was unregistered (tombstoned).
    Unregistered {
        name: String,
        previous_addr: ActorAddress,
    },
}

/// The local replica of the cluster-wide registry.
pub struct ClusterRegistry {
    /// Current state: name -> latest entry.
    entries: HashMap<String, RegistryEntry>,
    /// Pending entries to propagate via gossip (not yet disseminated to all).
    pending: VecDeque<RegistryEntry>,
    /// Logical clock for this node.
    clock: u64,
    /// Recent events for subscribers.
    events: VecDeque<RegistryEvent>,
    /// Max events to buffer.
    max_events: usize,
}

CRDT Merge Rule

impl ClusterRegistry {
    /// Merge a remote entry. Returns true if the local state changed.
    pub fn merge(&mut self, remote: RegistryEntry) -> bool {
        match self.entries.get(&remote.name) {
            Some(local) => {
                // LWW: higher timestamp wins.
                // Tie-break: higher generation, then higher node_id (deterministic).
                let dominated = remote.timestamp > local.timestamp
                    || (remote.timestamp == local.timestamp
                        && remote.generation > local.generation)
                    || (remote.timestamp == local.timestamp
                        && remote.generation == local.generation
                        && remote.node_id.0 > local.node_id.0);

                if dominated {
                    self.apply(remote);
                    true
                } else {
                    false
                }
            }
            None => {
                self.apply(remote);
                true
            }
        }
    }

    fn apply(&mut self, entry: RegistryEntry) {
        let event = if entry.tombstone {
            let prev = self.entries.get(&entry.name)
                .map(|e| e.actor_addr);
            RegistryEvent::Unregistered {
                name: entry.name.clone(),
                previous_addr: prev.unwrap_or_default(),
            }
        } else {
            RegistryEvent::Registered {
                name: entry.name.clone(),
                actor_addr: entry.actor_addr,
                node_id: entry.node_id,
            }
        };
        self.events.push_back(event);
        if self.events.len() > self.max_events {
            self.events.pop_front();
        }
        self.entries.insert(entry.name.clone(), entry);
    }
}

API

On DistributedNode:

// crates/distribution/src/node.rs

impl DistributedNode {
    /// Register a name -> actor binding on this node.
    /// The binding is propagated to all cluster members via gossip.
    pub fn register_name(&mut self, name: &str, actor_addr: ActorAddress) {
        self.registry.clock += 1;
        let entry = RegistryEntry {
            name: name.to_string(),
            actor_addr,
            node_id: self.node_id(),
            timestamp: self.registry.clock,
            generation: self.registry.next_generation(name),
            tombstone: false,
        };
        self.registry.merge(entry.clone());
        self.registry.pending.push_back(entry);
    }

    /// Remove a name binding. Propagated as a tombstone.
    pub fn unregister_name(&mut self, name: &str) {
        self.registry.clock += 1;
        let actor_addr = self.registry.entries.get(name)
            .map(|e| e.actor_addr)
            .unwrap_or_default();
        let entry = RegistryEntry {
            name: name.to_string(),
            actor_addr,
            node_id: self.node_id(),
            timestamp: self.registry.clock,
            generation: 0,
            tombstone: true,
        };
        self.registry.merge(entry.clone());
        self.registry.pending.push_back(entry);
    }

    /// Resolve a name to an actor address (local replica, eventually consistent).
    pub fn resolve_name(&self, name: &str) -> Option<(ActorAddress, NodeId)> {
        self.registry.entries.get(name)
            .filter(|e| !e.tombstone)
            .map(|e| (e.actor_addr, e.node_id))
    }

    /// Drain buffered registry events (for subscribers).
    pub fn registry_events(&mut self) -> Vec<RegistryEvent> {
        self.registry.events.drain(..).collect()
    }
}

On Ctx (actor-level, requires distribution feature):

// src/actor.rs — requires ContextInner extensions

impl Ctx<'_> {
    /// Register this actor under a name in the cluster registry.
    pub fn register_as(&self, name: &str) {
        self.inner.register_name(self.self_addr, name);
    }

    /// Resolve a name to an actor address.
    pub fn resolve_name(&self, name: &str) -> Option<ActorAddress> {
        self.inner.resolve_name(name)
    }
}

The ContextInner trait gains two new methods:

pub trait ContextInner {
    // ... existing methods ...
    fn register_name(&self, addr: ActorAddress, name: &str) { /* default no-op */ }
    fn resolve_name(&self, name: &str) -> Option<ActorAddress> { None }
}

Default implementations return None / no-op so that non-distributed runtimes don't break.

Gossip Propagation

Registry entries are piggybacked on SWIM protocol messages, reusing the existing dissemination mechanism.

Currently, crates/distribution/src/swim/dissemination.rs encodes membership updates into the piggyback payload:

piggyback bytes = bincode(Vec<MembershipUpdate>)

Extended format:

piggyback bytes = bincode(PiggybackPayload {
    membership: Vec<MembershipUpdate>,
    registry: Vec<RegistryEntry>,       // NEW
})

// crates/distribution/src/swim/dissemination.rs

#[derive(Serialize, Deserialize)]
struct PiggybackPayload {
    membership: Vec<MembershipUpdate>,
    registry: Vec<RegistryEntry>,
}

The dissemination buffer manages registry entries the same way as membership updates:

• Each entry has a dissemination count (how many times it's been piggybacked).
• After log2(N) + 1 disseminations (where N = cluster size), the entry is retired.
• Piggyback space is shared: membership updates take priority, registry entries fill remaining space.

Node Death Handling

When SWIM marks a node as Dead:

fn handle_membership_change(&mut self, node_id: NodeId, state: MemberState) {
    if state == MemberState::Dead {
        // ... existing cleanup ...

        // NEW: tombstone all registry entries owned by the dead node
        let to_tombstone: Vec<String> = self.registry.entries.iter()
            .filter(|(_, e)| e.node_id == node_id && !e.tombstone)
            .map(|(name, _)| name.clone())
            .collect();

        for name in to_tombstone {
            self.registry.clock += 1;
            let entry = RegistryEntry {
                name: name.clone(),
                tombstone: true,
                timestamp: self.registry.clock,
                // ... fill from existing entry ...
            };
            self.registry.merge(entry.clone());
            self.registry.pending.push_back(entry);
        }
    }
}

Interaction with Actor Watching

The registry and watching system compose naturally:

1. Actor A resolves name "service-X" → gets address B on Node 2.
2. Actor A calls ctx.watch(B).
3. Node 2 dies. Actor A receives ActorExited { addr: B, reason: NodeDown }.
4. A supervisor re-spawns "service-X" on Node 3 → new address C.
5. The supervisor calls register_name("service-X", C).
6. Gossip propagates the update.
7. Actor A (or anyone) calls resolve_name("service-X") → gets address C.
8. Actor A calls ctx.watch(C) to resume monitoring.

Tombstone Garbage Collection

Tombstones accumulate over time. GC strategy:

• Tombstones older than tombstone_ttl (default: 1 hour of logical clock ticks) are eligible for removal.
• GC runs periodically (e.g., every 1000 ticks).
• A tombstone is only removed if it has been fully disseminated (dissemination count >= threshold).

Files Modified

File	Change
crates/distribution/src/registry.rs	New file: ClusterRegistry, RegistryEntry, RegistryEvent, CRDT merge
crates/distribution/src/lib.rs	pub mod registry;
crates/distribution/src/node.rs	register_name, unregister_name, resolve_name, node death tombstoning
crates/distribution/src/swim/dissemination.rs	PiggybackPayload extended with registry entries
src/actor.rs	register_name/resolve_name on ContextInner (default no-op), Ctx wrappers

Tests

• register_and_resolve: register a name, resolve it, verify correct address
• lww_conflict: two nodes register same name concurrently, verify latest timestamp wins
• tombstone_propagation: register name, unregister, verify tombstone propagates and resolve returns None
• node_death_tombstones: 3-node cluster, register name on node B, kill node B, verify name is tombstoned on surviving nodes
• re_registration: register name, unregister, re-register with new address, verify resolution
• gossip_convergence: register name on node A, verify all nodes resolve it after gossip settles