137 lines
6.4 KiB
Markdown
137 lines
6.4 KiB
Markdown
|
# Kademlia DHT
|
|||
|
|
|
|||
|
|
The Kademlia layer provides a distributed actor directory. Actors are mapped
|
|||
|
|
to nodes using XOR-distance routing over a 256-bit keyspace, with iterative
|
|||
|
|
lookups, quorum reads, and automatic repair on node failure.
|
|||
|
|
|
|||
|
|
## Routing Table
|
|||
|
|
|
|||
|
|
256 k-buckets indexed by `XOR(self_id, target).leading_zeros()`. Each bucket
|
|||
|
|
holds up to `K` nodes (default 20) in LRU order — most-recently-seen at tail.
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
┌─ RoutingTable ──────────────────────────────────────────────────────────┐
|
|||
|
|
│ │
|
|||
|
|
│ self_id: NodeId │
|
|||
|
|
│ buckets: [KBucket; 256] │
|
|||
|
|
│ │
|
|||
|
|
│ Bucket[i] holds nodes where XOR distance has exactly i leading zeros. │
|
|||
|
|
│ Bucket 0 = farthest half of keyspace. │
|
|||
|
|
│ Bucket 255 = nearest neighbor. │
|
|||
|
|
│ │
|
|||
|
|
│ Each bucket: │
|
|||
|
|
│ nodes: VecDeque<NodeEntry> (LRU, capacity K) │
|
|||
|
|
│ replacements: VecDeque<NodeEntry> (overflow cache) │
|
|||
|
|
│ │
|
|||
|
|
│ On insert into a full bucket: │
|
|||
|
|
│ → new node goes to replacement cache │
|
|||
|
|
│ → only promoted when an existing node is evicted │
|
|||
|
|
│ → prefers long-lived nodes (Kademlia stability heuristic) │
|
|||
|
|
│ │
|
|||
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The `closest(target, count)` method scans all buckets, sorts by XOR distance,
|
|||
|
|
and returns the `count` nearest entries. This is used both for iterative
|
|||
|
|
lookups and for selecting STORE targets.
|
|||
|
|
|
|||
|
|
## Iterative Lookup
|
|||
|
|
|
|||
|
|
See [actor_resolution.svg](actor_resolution.svg) for the registration and
|
|||
|
|
resolution datapaths.
|
|||
|
|
|
|||
|
|
The `NodeLookup` state machine drives iterative `FIND_NODE`:
|
|||
|
|
|
|||
|
|
1. Seed with the `K` closest nodes from the local routing table.
|
|||
|
|
2. Query `alpha` (default 3) closest unqueried nodes in parallel.
|
|||
|
|
3. Responses bring new closer nodes — incorporate and repeat.
|
|||
|
|
4. Terminate when all `K` closest nodes have been queried, or max rounds (20).
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
Start(target)
|
|||
|
|
│
|
|||
|
|
v
|
|||
|
|
seed = routing_table.closest(target, K)
|
|||
|
|
│
|
|||
|
|
v
|
|||
|
|
┌───────────────────┐
|
|||
|
|
│ next_round() │◄──────────────────┐
|
|||
|
|
│ pick α closest │ │
|
|||
|
|
│ unqueried nodes │ │
|
|||
|
|
└────────┬──────────┘ │
|
|||
|
|
│ │
|
|||
|
|
v │
|
|||
|
|
┌───────────────────┐ ┌────────────┴───────────┐
|
|||
|
|
│ Query(node, addr) │─────►│ handle_response(from, │
|
|||
|
|
│ × α in parallel │ │ closer_nodes) │
|
|||
|
|
└───────────────────┘ │ incorporate new nodes │
|
|||
|
|
└────────────────────────┘
|
|||
|
|
│
|
|||
|
|
all K closest queried
|
|||
|
|
or max_rounds exceeded
|
|||
|
|
│
|
|||
|
|
v
|
|||
|
|
Done { closest: Vec<(NodeId, SocketAddr)> }
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
The lookup is pure — it produces `LookupAction`s and the caller translates
|
|||
|
|
them into network I/O.
|
|||
|
|
|
|||
|
|
## Directory Shards & Quorum
|
|||
|
|
|
|||
|
|
Each node holds a `DirectoryShard` — a local map from `ActorAddress` to
|
|||
|
|
`Vec<DirectoryEntry>`. Entries are signed by the spawning node's keypair:
|
|||
|
|
|
|||
|
|
```rust
|
|||
|
|
pub struct DirectoryEntry {
|
|||
|
|
pub actor_addr: ActorAddress,
|
|||
|
|
pub node_id: NodeId, // who spawned the actor
|
|||
|
|
pub generation: u64, // bumped on re-registration
|
|||
|
|
pub signature: Signature, // ed25519 over (actor_addr, node_id, generation)
|
|||
|
|
}
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
STORE replicates entries to the `r` closest nodes in XOR space.
|
|||
|
|
FIND_VALUE does quorum reads with signature verification:
|
|||
|
|
|
|||
|
|
- Entries are grouped by `(node_id, generation)`.
|
|||
|
|
- Groups meeting quorum (`f + 1` agreement) are candidates.
|
|||
|
|
- Highest generation among quorum groups wins.
|
|||
|
|
- Invalid signatures are silently rejected.
|
|||
|
|
|
|||
|
|
Resolution returns `QuorumResult::Resolved(entry)`,
|
|||
|
|
`QuorumResult::NoQuorum(all)`, or `QuorumResult::NotFound`.
|
|||
|
|
|
|||
|
|
## Repair & Republish
|
|||
|
|
|
|||
|
|
Two mechanisms maintain directory integrity under churn:
|
|||
|
|
|
|||
|
|
**RepairQueue** — On node death, `on_node_death(dead_id, shard)` extracts
|
|||
|
|
all entries authored by the dead node and queues them for re-STORE to the
|
|||
|
|
new `r`-closest nodes.
|
|||
|
|
|
|||
|
|
**RepublishTracker** — Each node periodically re-STOREs its locally-spawned
|
|||
|
|
actor entries at a configurable interval. This counteracts topology drift:
|
|||
|
|
as nodes join and leave, the "r-closest" set shifts, and republishing ensures
|
|||
|
|
entries migrate to the current closest nodes.
|
|||
|
|
|
|||
|
|
See `crates/distribution/DESIGN_NOTES.md` for the full rationale behind
|
|||
|
|
re-replication and republishing.
|
|||
|
|
|
|||
|
|
## Where Things Live
|
|||
|
|
|
|||
|
|
| Type | File | Role |
|
|||
|
|
|------|------|------|
|
|||
|
|
| `RoutingTable` | `kademlia/routing_table.rs` | 256 k-buckets, XOR routing |
|
|||
|
|
| `KBucket` | `kademlia/routing_table.rs` | LRU bucket + replacement cache |
|
|||
|
|
| `NodeEntry` | `kademlia/routing_table.rs` | Node in routing table |
|
|||
|
|
| `NodeLookup` | `kademlia/lookup.rs` | Iterative FIND_NODE state machine |
|
|||
|
|
| `LookupAction` | `kademlia/lookup.rs` | Lookup I/O actions |
|
|||
|
|
| `DirectoryShard` | `kademlia/directory.rs` | Local directory storage |
|
|||
|
|
| `QuorumResult` | `kademlia/directory.rs` | Quorum read outcomes |
|
|||
|
|
| `RepairQueue` | `kademlia/repair.rs` | Death-triggered re-replication |
|
|||
|
|
| `RepublishTracker` | `kademlia/repair.rs` | Periodic re-STORE |
|
|||
|
|
| `NodeId` | `types.rs` | 256-bit identity / XOR key |
|
|||
|
|
| `DirectoryEntry` | `types.rs` | Signed actor→node binding |
|
|||
|
|
| `Signature` | `types.rs` | Ed25519 signature (64 bytes) |
|