swactor/docs/distribution/kademlia.md

Kademlia DHT

The Kademlia layer provides a distributed actor directory. Actors are mapped to nodes using XOR-distance routing over a 256-bit keyspace, with iterative lookups, quorum reads, and automatic repair on node failure.

Routing Table

256 k-buckets indexed by XOR(self_id, target).leading_zeros(). Each bucket holds up to K nodes (default 20) in LRU order — most-recently-seen at tail.

┌─ RoutingTable ──────────────────────────────────────────────────────────┐
│                                                                         │
│  self_id: NodeId                                                        │
│  buckets: [KBucket; 256]                                                │
│                                                                         │
│  Bucket[i] holds nodes where XOR distance has exactly i leading zeros.  │
│  Bucket 0 = farthest half of keyspace.                                  │
│  Bucket 255 = nearest neighbor.                                         │
│                                                                         │
│  Each bucket:                                                           │
│    nodes:        VecDeque<NodeEntry>   (LRU, capacity K)                │
│    replacements: VecDeque<NodeEntry>   (overflow cache)                 │
│                                                                         │
│  On insert into a full bucket:                                          │
│    → new node goes to replacement cache                                 │
│    → only promoted when an existing node is evicted                     │
│    → prefers long-lived nodes (Kademlia stability heuristic)            │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

The closest(target, count) method scans all buckets, sorts by XOR distance, and returns the count nearest entries. This is used both for iterative lookups and for selecting STORE targets.

Iterative Lookup

See actor_resolution.svg for the registration and resolution datapaths.

The NodeLookup state machine drives iterative FIND_NODE:

1. Seed with the K closest nodes from the local routing table.
2. Query alpha (default 3) closest unqueried nodes in parallel.
3. Responses bring new closer nodes — incorporate and repeat.
4. Terminate when all K closest nodes have been queried, or max rounds (20).

  Start(target)
      │
      v
  seed = routing_table.closest(target, K)
      │
      v
  ┌───────────────────┐
  │  next_round()     │◄──────────────────┐
  │  pick α closest   │                   │
  │  unqueried nodes  │                   │
  └────────┬──────────┘                   │
           │                              │
           v                              │
  ┌───────────────────┐      ┌────────────┴───────────┐
  │ Query(node, addr) │─────►│ handle_response(from,  │
  │ × α in parallel   │      │   closer_nodes)        │
  └───────────────────┘      │ incorporate new nodes   │
                             └────────────────────────┘
           │
      all K closest queried
      or max_rounds exceeded
           │
           v
  Done { closest: Vec<(NodeId, SocketAddr)> }

The lookup is pure — it produces LookupActions and the caller translates them into network I/O.

Directory Shards & Quorum

Each node holds a DirectoryShard — a local map from ActorAddress to Vec<DirectoryEntry>. Entries are signed by the spawning node's keypair:

pub struct DirectoryEntry {
    pub actor_addr: ActorAddress,
    pub node_id: NodeId,          // who spawned the actor
    pub generation: u64,          // bumped on re-registration
    pub signature: Signature,     // ed25519 over (actor_addr, node_id, generation)
}

STORE replicates entries to the r closest nodes in XOR space. FIND_VALUE does quorum reads with signature verification:

• Entries are grouped by (node_id, generation).
• Groups meeting quorum (f + 1 agreement) are candidates.
• Highest generation among quorum groups wins.
• Invalid signatures are silently rejected.

Resolution returns QuorumResult::Resolved(entry), QuorumResult::NoQuorum(all), or QuorumResult::NotFound.

Repair & Republish

Two mechanisms maintain directory integrity under churn:

RepairQueue — On node death, on_node_death(dead_id, shard) extracts all entries authored by the dead node and queues them for re-STORE to the new r-closest nodes.

RepublishTracker — Each node periodically re-STOREs its locally-spawned actor entries at a configurable interval. This counteracts topology drift: as nodes join and leave, the "r-closest" set shifts, and republishing ensures entries migrate to the current closest nodes.

See crates/distribution/DESIGN_NOTES.md for the full rationale behind re-replication and republishing.

Where Things Live

Type	File	Role
RoutingTable	kademlia/routing_table.rs	256 k-buckets, XOR routing
KBucket	kademlia/routing_table.rs	LRU bucket + replacement cache
NodeEntry	kademlia/routing_table.rs	Node in routing table
NodeLookup	kademlia/lookup.rs	Iterative FIND_NODE state machine
LookupAction	kademlia/lookup.rs	Lookup I/O actions
DirectoryShard	kademlia/directory.rs	Local directory storage
QuorumResult	kademlia/directory.rs	Quorum read outcomes
RepairQueue	kademlia/repair.rs	Death-triggered re-replication
RepublishTracker	kademlia/repair.rs	Periodic re-STORE
NodeId	types.rs	256-bit identity / XOR key
DirectoryEntry	types.rs	Signed actor→node binding
Signature	types.rs	Ed25519 signature (64 bytes)