zacheryasc
402a106beb
Major feature addition. For full details read `./docs/development_history/DISTRIBUTION.md` Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
136 lines
6.4 KiB
Markdown
136 lines
6.4 KiB
Markdown
# Kademlia DHT
|
||
|
||
The Kademlia layer provides a distributed actor directory. Actors are mapped
|
||
to nodes using XOR-distance routing over a 256-bit keyspace, with iterative
|
||
lookups, quorum reads, and automatic repair on node failure.
|
||
|
||
## Routing Table
|
||
|
||
256 k-buckets indexed by `XOR(self_id, target).leading_zeros()`. Each bucket
|
||
holds up to `K` nodes (default 20) in LRU order — most-recently-seen at tail.
|
||
|
||
```
|
||
┌─ RoutingTable ──────────────────────────────────────────────────────────┐
|
||
│ │
|
||
│ self_id: NodeId │
|
||
│ buckets: [KBucket; 256] │
|
||
│ │
|
||
│ Bucket[i] holds nodes where XOR distance has exactly i leading zeros. │
|
||
│ Bucket 0 = farthest half of keyspace. │
|
||
│ Bucket 255 = nearest neighbor. │
|
||
│ │
|
||
│ Each bucket: │
|
||
│ nodes: VecDeque<NodeEntry> (LRU, capacity K) │
|
||
│ replacements: VecDeque<NodeEntry> (overflow cache) │
|
||
│ │
|
||
│ On insert into a full bucket: │
|
||
│ → new node goes to replacement cache │
|
||
│ → only promoted when an existing node is evicted │
|
||
│ → prefers long-lived nodes (Kademlia stability heuristic) │
|
||
│ │
|
||
└─────────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
The `closest(target, count)` method scans all buckets, sorts by XOR distance,
|
||
and returns the `count` nearest entries. This is used both for iterative
|
||
lookups and for selecting STORE targets.
|
||
|
||
## Iterative Lookup
|
||
|
||
See [actor_resolution.svg](actor_resolution.svg) for the registration and
|
||
resolution datapaths.
|
||
|
||
The `NodeLookup` state machine drives iterative `FIND_NODE`:
|
||
|
||
1. Seed with the `K` closest nodes from the local routing table.
|
||
2. Query `alpha` (default 3) closest unqueried nodes in parallel.
|
||
3. Responses bring new closer nodes — incorporate and repeat.
|
||
4. Terminate when all `K` closest nodes have been queried, or max rounds (20).
|
||
|
||
```
|
||
Start(target)
|
||
│
|
||
v
|
||
seed = routing_table.closest(target, K)
|
||
│
|
||
v
|
||
┌───────────────────┐
|
||
│ next_round() │◄──────────────────┐
|
||
│ pick α closest │ │
|
||
│ unqueried nodes │ │
|
||
└────────┬──────────┘ │
|
||
│ │
|
||
v │
|
||
┌───────────────────┐ ┌────────────┴───────────┐
|
||
│ Query(node, addr) │─────►│ handle_response(from, │
|
||
│ × α in parallel │ │ closer_nodes) │
|
||
└───────────────────┘ │ incorporate new nodes │
|
||
└────────────────────────┘
|
||
│
|
||
all K closest queried
|
||
or max_rounds exceeded
|
||
│
|
||
v
|
||
Done { closest: Vec<(NodeId, SocketAddr)> }
|
||
```
|
||
|
||
The lookup is pure — it produces `LookupAction`s and the caller translates
|
||
them into network I/O.
|
||
|
||
## Directory Shards & Quorum
|
||
|
||
Each node holds a `DirectoryShard` — a local map from `ActorAddress` to
|
||
`Vec<DirectoryEntry>`. Entries are signed by the spawning node's keypair:
|
||
|
||
```rust
|
||
pub struct DirectoryEntry {
|
||
pub actor_addr: ActorAddress,
|
||
pub node_id: NodeId, // who spawned the actor
|
||
pub generation: u64, // bumped on re-registration
|
||
pub signature: Signature, // ed25519 over (actor_addr, node_id, generation)
|
||
}
|
||
```
|
||
|
||
STORE replicates entries to the `r` closest nodes in XOR space.
|
||
FIND_VALUE does quorum reads with signature verification:
|
||
|
||
- Entries are grouped by `(node_id, generation)`.
|
||
- Groups meeting quorum (`f + 1` agreement) are candidates.
|
||
- Highest generation among quorum groups wins.
|
||
- Invalid signatures are silently rejected.
|
||
|
||
Resolution returns `QuorumResult::Resolved(entry)`,
|
||
`QuorumResult::NoQuorum(all)`, or `QuorumResult::NotFound`.
|
||
|
||
## Repair & Republish
|
||
|
||
Two mechanisms maintain directory integrity under churn:
|
||
|
||
**RepairQueue** — On node death, `on_node_death(dead_id, shard)` extracts
|
||
all entries authored by the dead node and queues them for re-STORE to the
|
||
new `r`-closest nodes.
|
||
|
||
**RepublishTracker** — Each node periodically re-STOREs its locally-spawned
|
||
actor entries at a configurable interval. This counteracts topology drift:
|
||
as nodes join and leave, the "r-closest" set shifts, and republishing ensures
|
||
entries migrate to the current closest nodes.
|
||
|
||
See `crates/distribution/DESIGN_NOTES.md` for the full rationale behind
|
||
re-replication and republishing.
|
||
|
||
## Where Things Live
|
||
|
||
| Type | File | Role |
|
||
|------|------|------|
|
||
| `RoutingTable` | `kademlia/routing_table.rs` | 256 k-buckets, XOR routing |
|
||
| `KBucket` | `kademlia/routing_table.rs` | LRU bucket + replacement cache |
|
||
| `NodeEntry` | `kademlia/routing_table.rs` | Node in routing table |
|
||
| `NodeLookup` | `kademlia/lookup.rs` | Iterative FIND_NODE state machine |
|
||
| `LookupAction` | `kademlia/lookup.rs` | Lookup I/O actions |
|
||
| `DirectoryShard` | `kademlia/directory.rs` | Local directory storage |
|
||
| `QuorumResult` | `kademlia/directory.rs` | Quorum read outcomes |
|
||
| `RepairQueue` | `kademlia/repair.rs` | Death-triggered re-replication |
|
||
| `RepublishTracker` | `kademlia/repair.rs` | Periodic re-STORE |
|
||
| `NodeId` | `types.rs` | 256-bit identity / XOR key |
|
||
| `DirectoryEntry` | `types.rs` | Signed actor→node binding |
|
||
| `Signature` | `types.rs` | Ed25519 signature (64 bytes) |
|