os-research #32
11 changed files with 1015 additions and 98 deletions
4
.gitignore
vendored
4
.gitignore
vendored
|
|
@ -11,3 +11,7 @@ corpus
|
|||
**/deps.html
|
||||
docs/architecture.dot
|
||||
docs/architecture.html
|
||||
|
||||
# Claude session files
|
||||
CLAUDE/
|
||||
.claude/
|
||||
|
|
|
|||
|
|
@ -1,42 +0,0 @@
|
|||
Plan:
|
||||
You are to improve this codebase via:
|
||||
- implementing the features found in `docs/os-design`
|
||||
- writing comphrehensive tests that check behavior makes sense
|
||||
|
||||
Workflow:
|
||||
- Read `CLAUDE/TASK.md` and `CLAUDE/notes/progress.md`
|
||||
- Identify what stage you are on.
|
||||
- Read and update yourself as necessary.
|
||||
- Proceed to accomplishing the next task as written in `progress.md`
|
||||
- For each attempt at any step, keep a record. If you reach attempt 3, step back, document, and try something else.
|
||||
- When done, because attempt limit or task success:
|
||||
- update `progress.md` with:
|
||||
- Completed this session
|
||||
- Next steps (specific, actionable)
|
||||
- Open Questions
|
||||
- Blockers
|
||||
- make a commit
|
||||
- compress your context and start the loop again
|
||||
|
||||
Style:
|
||||
- Do not add to existing modules in the root swactor `src/` they should stay as they are. You may modify but not change module structure.
|
||||
- Integration tests in `tests/`, benchmark code in `benches/`
|
||||
- cap execution time at 2 minutes max for fuzz, or benchmarks, or single test suite
|
||||
- if they take too long, refactor and break up into logical modules
|
||||
- You may modify these as you wish, so long as logical 'coverage' does not decline.
|
||||
- Report all your changes to architecture with changes to the `docs/` items
|
||||
- all notes you wish to keep across iterations shall go in the `CLAUDE/notes/` folder
|
||||
|
||||
Example loop (not restrictive, feel free to ignore if prudent):
|
||||
- Pick an item to implement from the os-design docs
|
||||
- make analysis
|
||||
- implement plan
|
||||
- execute
|
||||
- evaluate
|
||||
- compress and move on to the next item
|
||||
|
||||
Before git commit:
|
||||
- all `cargo test` passes, including feature gated material
|
||||
- if a test fails, investigate do not ignore or delete
|
||||
- You can combine tests but not skip code paths or delete them for active code
|
||||
- if a fix takes > 3 attempts, log and move on
|
||||
|
|
@ -1,51 +0,0 @@
|
|||
# Progress
|
||||
|
||||
## Completed
|
||||
|
||||
### Feature 1: Actor Watching (local only) — `docs/os-design/01-actor-watching.md`
|
||||
- Added `ExitReason` enum (Stopped, Panicked, NodeDown) and `ActorExited` struct to `src/actor.rs`
|
||||
- Extended `ContextInner` trait with `watch()`/`unwatch()` methods
|
||||
- Added `Ctx::watch(target)` and `Ctx::unwatch(target)` typed API
|
||||
- Added `on_actor_exit()` default method to `ActorInterface` trait
|
||||
- Updated `AnyActor::handle_any` with system message fallback (tries `ActorExited` after `Incoming`)
|
||||
- Implemented `WatchRegistry` in `src/worker.rs` (bidirectional HashMap tracking)
|
||||
- Integrated death notification dispatch as phase 5b in `tick_once`
|
||||
- Implemented `watch`/`unwatch` on `Runtime`'s `ContextInner` impl
|
||||
- Added `watch_registry` field to `TickContext` in `src/delivery.rs`
|
||||
- 10 behavioral tests in `tests/watch_api.rs` — all passing
|
||||
|
||||
### Feature 2: Command Interface — `docs/os-design/04-command-interface.md`
|
||||
- Created `crates/command/` crate (`swactor-command`) with core types:
|
||||
- `CommandRequest`, `CommandResponse` (serde-serializable)
|
||||
- `CommandHandler` trait + `CommandMeta`
|
||||
- `CommandRouter` with `dispatch()` and `with_builtins()`
|
||||
- `CommandContext` with `Arc<Runtime>` + optional `StatsEnricher`
|
||||
- `StatsEnricher` trait (decouples command crate from dashboard)
|
||||
- Built-in read commands: overview, workers, worker, actors, actor, hot, phases, diff
|
||||
- Built-in write command: shutdown
|
||||
- REPL line parser (`parse_line`) with positional arg mapping and `--flag value` support
|
||||
- REST adapter (`from_query_params`) for HTTP query parameters
|
||||
- Refactored `investigate.rs` to delegate to CommandRouter (thin wrapper)
|
||||
- Updated `server.rs` to use CommandRouter for `/api/investigate` endpoint
|
||||
- Implemented `StatsEnricher for StatsCollector` in dashboard crate
|
||||
- 18 behavioral tests in `crates/command/tests/command_api.rs` — all passing
|
||||
- All 53 swactor core tests pass, all 18 command tests pass
|
||||
|
||||
## Next Steps
|
||||
1. **Cluster Registry** — `docs/os-design/02-cluster-registry.md`
|
||||
- LWW-Register CRDT per name binding
|
||||
- Propagation via SWIM piggyback
|
||||
- `ClusterRegistry` struct in `crates/distribution/src/registry.rs`
|
||||
- API: register_name, unregister_name, resolve_name
|
||||
2. **Node Capabilities** — `docs/os-design/03-node-capabilities.md`
|
||||
- New `crates/capabilities/` crate with auto-detection
|
||||
3. **Remote Watching** — extends actor watching with wire protocol
|
||||
4. **Supervision** — `docs/os-design/05-supervision.md`
|
||||
|
||||
## Open Questions
|
||||
- Custom actor commands (via `Ctx::register_command()`) deferred to a later PR
|
||||
- Distribution-aware commands (nodes, registry, resolve) deferred until cluster registry is implemented
|
||||
- Write commands (spawn, stop, drain) deferred — need factory registry and actor stop mechanism
|
||||
|
||||
## Blockers
|
||||
- None
|
||||
|
|
@ -7,4 +7,5 @@ pub mod swim;
|
|||
pub mod kademlia;
|
||||
pub mod cache;
|
||||
pub mod node;
|
||||
pub mod registry;
|
||||
pub mod snapshot;
|
||||
|
|
|
|||
|
|
@ -12,6 +12,10 @@ use crate::crypto::Keypair;
|
|||
use crate::kademlia::directory::{actor_addr_as_node_id, DirectoryShard};
|
||||
use crate::kademlia::repair::{RepairQueue, RepublishTracker};
|
||||
use crate::kademlia::routing_table::RoutingTable;
|
||||
use crate::registry::{
|
||||
pack_combined_piggyback, unpack_combined_piggyback, ClusterRegistry, RegistryConfig,
|
||||
RegistryEvent,
|
||||
};
|
||||
use crate::swim::node::{NodeAction, SwimNode};
|
||||
use crate::swim::probe::SwimConfig;
|
||||
use crate::types::{MemberState, NodeId, NodeRecord};
|
||||
|
|
@ -22,6 +26,7 @@ pub struct DistributedNodeConfig {
|
|||
pub swim: SwimConfig,
|
||||
pub cache_capacity: usize,
|
||||
pub republish_interval: u64,
|
||||
pub registry: RegistryConfig,
|
||||
}
|
||||
|
||||
impl Default for DistributedNodeConfig {
|
||||
|
|
@ -31,6 +36,7 @@ impl Default for DistributedNodeConfig {
|
|||
swim: SwimConfig::default(),
|
||||
cache_capacity: 10_000,
|
||||
republish_interval: 1000,
|
||||
registry: RegistryConfig::default(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -47,6 +53,7 @@ pub struct DistributedNode {
|
|||
cache: LocationCache,
|
||||
repair_queue: RepairQueue,
|
||||
republish: RepublishTracker,
|
||||
registry: ClusterRegistry,
|
||||
tick_count: u64,
|
||||
}
|
||||
|
||||
|
|
@ -67,6 +74,7 @@ impl DistributedNode {
|
|||
cache: LocationCache::new(config.cache_capacity),
|
||||
repair_queue: RepairQueue::new(),
|
||||
republish: RepublishTracker::new(config.republish_interval),
|
||||
registry: ClusterRegistry::new(config.registry),
|
||||
tick_count: 0,
|
||||
keypair,
|
||||
}
|
||||
|
|
@ -149,23 +157,32 @@ impl DistributedNode {
|
|||
// re-sign and re-STORE these entries.
|
||||
}
|
||||
|
||||
actions
|
||||
// Registry GC
|
||||
self.registry.gc_tick();
|
||||
|
||||
// Wrap outgoing piggyback with registry entries
|
||||
self.inject_registry_piggyback(actions)
|
||||
}
|
||||
|
||||
// ─── SWIM message handling (delegate to SwimNode) ───────────────────
|
||||
|
||||
pub fn handle_ping(&mut self, from: NodeId, from_addr: SocketAddr, sequence: u64, piggyback: &[u8]) -> Vec<NodeAction> {
|
||||
let actions = self.swim.handle_ping(from, from_addr, sequence, piggyback);
|
||||
let membership_bytes = self.extract_registry_piggyback(piggyback);
|
||||
let actions = self.swim.handle_ping(from, from_addr, sequence, &membership_bytes);
|
||||
self.maybe_update_routing_table(from, from_addr);
|
||||
actions
|
||||
self.inject_registry_piggyback(actions)
|
||||
}
|
||||
|
||||
pub fn handle_ack(&mut self, from: NodeId, sequence: u64, piggyback: &[u8]) -> Vec<NodeAction> {
|
||||
self.swim.handle_ack(from, sequence, piggyback)
|
||||
let membership_bytes = self.extract_registry_piggyback(piggyback);
|
||||
let actions = self.swim.handle_ack(from, sequence, &membership_bytes);
|
||||
self.inject_registry_piggyback(actions)
|
||||
}
|
||||
|
||||
pub fn handle_ping_req(&mut self, from: NodeId, target: NodeId, target_addr: SocketAddr, sequence: u64, piggyback: &[u8]) -> Vec<NodeAction> {
|
||||
self.swim.handle_ping_req(from, target, target_addr, sequence, piggyback)
|
||||
let membership_bytes = self.extract_registry_piggyback(piggyback);
|
||||
let actions = self.swim.handle_ping_req(from, target, target_addr, sequence, &membership_bytes);
|
||||
self.inject_registry_piggyback(actions)
|
||||
}
|
||||
|
||||
pub fn handle_join_request(&mut self, from: NodeId, from_addr: SocketAddr) -> Vec<NodeAction> {
|
||||
|
|
@ -233,6 +250,33 @@ impl DistributedNode {
|
|||
self.cache.invalidate(actor_addr);
|
||||
}
|
||||
|
||||
// ─── Registry (name → actor mapping) ──────────────────────────────
|
||||
|
||||
/// Register a human-readable name for an actor on this node.
|
||||
pub fn register_name(&mut self, name: String, actor_addr: ActorAddress) {
|
||||
self.registry.register(name, actor_addr, self.node_id(), self.cluster_size());
|
||||
}
|
||||
|
||||
/// Unregister a name (creates a tombstone).
|
||||
pub fn unregister_name(&mut self, name: &str) {
|
||||
self.registry.unregister(name, self.node_id(), self.cluster_size());
|
||||
}
|
||||
|
||||
/// Resolve a name to its current (ActorAddress, NodeId).
|
||||
pub fn resolve_name(&self, name: &str) -> Option<(ActorAddress, NodeId)> {
|
||||
self.registry.resolve(name)
|
||||
}
|
||||
|
||||
/// Drain registry events (Registered / Unregistered).
|
||||
pub fn registry_events(&mut self) -> Vec<RegistryEvent> {
|
||||
self.registry.drain_events()
|
||||
}
|
||||
|
||||
/// Read-only access to the registry.
|
||||
pub fn registry(&self) -> &ClusterRegistry {
|
||||
&self.registry
|
||||
}
|
||||
|
||||
// ─── Accessors ──────────────────────────────────────────────────────
|
||||
|
||||
pub fn routing_table(&self) -> &RoutingTable {
|
||||
|
|
@ -277,12 +321,51 @@ impl DistributedNode {
|
|||
self.routing_table.remove(&node_id);
|
||||
self.cache.invalidate_node(&node_id);
|
||||
self.repair_queue.on_node_death(&node_id, &mut self.directory);
|
||||
self.registry.tombstone_node(node_id, self.cluster_size());
|
||||
}
|
||||
MemberState::Suspect => {
|
||||
// Keep in routing table but could downprioritize
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn cluster_size(&self) -> usize {
|
||||
self.swim.members().alive_count() + 1 // +1 for self
|
||||
}
|
||||
|
||||
/// Post-process outgoing actions: wrap each piggyback with registry entries.
|
||||
fn inject_registry_piggyback(&mut self, actions: Vec<NodeAction>) -> Vec<NodeAction> {
|
||||
actions
|
||||
.into_iter()
|
||||
.map(|action| match action {
|
||||
NodeAction::SendPing { to, to_addr, sequence, piggyback } => {
|
||||
let registry_entries = self.registry.take_pending(8);
|
||||
let combined = pack_combined_piggyback(piggyback, registry_entries);
|
||||
NodeAction::SendPing { to, to_addr, sequence, piggyback: combined }
|
||||
}
|
||||
NodeAction::SendAck { to, to_addr, sequence, piggyback } => {
|
||||
let registry_entries = self.registry.take_pending(8);
|
||||
let combined = pack_combined_piggyback(piggyback, registry_entries);
|
||||
NodeAction::SendAck { to, to_addr, sequence, piggyback: combined }
|
||||
}
|
||||
NodeAction::SendPingReq { relay, relay_addr, target, target_addr, sequence, piggyback } => {
|
||||
let registry_entries = self.registry.take_pending(8);
|
||||
let combined = pack_combined_piggyback(piggyback, registry_entries);
|
||||
NodeAction::SendPingReq { relay, relay_addr, target, target_addr, sequence, piggyback: combined }
|
||||
}
|
||||
other => other,
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Extract registry entries from incoming piggyback, merge them, return membership-only bytes.
|
||||
fn extract_registry_piggyback(&mut self, bytes: &[u8]) -> Vec<u8> {
|
||||
let (membership_bytes, registry_entries) = unpack_combined_piggyback(bytes);
|
||||
if !registry_entries.is_empty() {
|
||||
self.registry.merge_batch(registry_entries, self.cluster_size());
|
||||
}
|
||||
membership_bytes
|
||||
}
|
||||
}
|
||||
|
||||
/// Result of resolving an actor's location.
|
||||
|
|
|
|||
374
crates/distribution/src/registry.rs
Normal file
374
crates/distribution/src/registry.rs
Normal file
|
|
@ -0,0 +1,374 @@
|
|||
//! Cluster Registry — gossip-propagated naming via LWW-Register CRDT.
|
||||
//!
|
||||
//! Maps human-readable names to `(ActorAddress, NodeId)` pairs, propagated
|
||||
//! through SWIM gossip piggyback. Uses last-writer-wins semantics with
|
||||
//! tie-breaking on (timestamp, generation, node_id).
|
||||
|
||||
use std::collections::{HashMap, VecDeque};
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
use swactor::actor::ActorAddress;
|
||||
|
||||
use crate::types::NodeId;
|
||||
|
||||
// ─── Configuration ──────────────────────────────────────────────────────────
|
||||
|
||||
/// Configuration for the cluster registry.
|
||||
pub struct RegistryConfig {
|
||||
/// Maximum number of events to buffer before dropping old ones.
|
||||
pub max_events: usize,
|
||||
/// How long (in ticks) a tombstone is retained before GC.
|
||||
pub tombstone_ttl: u64,
|
||||
/// How often (in ticks) to run garbage collection.
|
||||
pub gc_interval: u64,
|
||||
/// Dissemination multiplier (Λ) — same role as in SWIM dissemination.
|
||||
pub dissemination_lambda: usize,
|
||||
}
|
||||
|
||||
impl Default for RegistryConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
max_events: 256,
|
||||
tombstone_ttl: 3600,
|
||||
gc_interval: 1000,
|
||||
dissemination_lambda: 3,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Wire types ─────────────────────────────────────────────────────────────
|
||||
|
||||
/// A single registry entry — the unit of replication.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
|
||||
pub struct RegistryEntry {
|
||||
pub name: String,
|
||||
pub actor_addr: ActorAddress,
|
||||
pub node_id: NodeId,
|
||||
/// Logical timestamp (monotonically increasing per-registry).
|
||||
pub timestamp: u64,
|
||||
/// Generation counter for the same name (disambiguates re-registrations).
|
||||
pub generation: u64,
|
||||
/// If true, this entry is a tombstone (name was unregistered).
|
||||
pub tombstone: bool,
|
||||
}
|
||||
|
||||
/// Combined piggyback payload: membership bytes + registry entries.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct PiggybackPayload {
|
||||
/// Raw SWIM membership piggyback bytes (opaque to registry).
|
||||
pub membership: Vec<u8>,
|
||||
/// Registry entries to disseminate.
|
||||
pub registry: Vec<RegistryEntry>,
|
||||
}
|
||||
|
||||
// ─── Events ─────────────────────────────────────────────────────────────────
|
||||
|
||||
/// Events emitted when the registry changes.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum RegistryEvent {
|
||||
Registered {
|
||||
name: String,
|
||||
actor_addr: ActorAddress,
|
||||
node_id: NodeId,
|
||||
},
|
||||
Unregistered {
|
||||
name: String,
|
||||
previous_addr: ActorAddress,
|
||||
},
|
||||
}
|
||||
|
||||
// ─── Dissemination entry ────────────────────────────────────────────────────
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
struct DisseminationEntry {
|
||||
entry: RegistryEntry,
|
||||
remaining: usize,
|
||||
}
|
||||
|
||||
// ─── ClusterRegistry ────────────────────────────────────────────────────────
|
||||
|
||||
/// CRDT-based cluster registry with LWW semantics and gossip dissemination.
|
||||
pub struct ClusterRegistry {
|
||||
/// Current state: name → latest entry.
|
||||
entries: HashMap<String, RegistryEntry>,
|
||||
/// Pending entries to disseminate via piggyback.
|
||||
dissemination: Vec<DisseminationEntry>,
|
||||
/// Monotonic logical clock for this node's writes.
|
||||
clock: u64,
|
||||
/// Buffered events for consumers.
|
||||
events: VecDeque<RegistryEvent>,
|
||||
config: RegistryConfig,
|
||||
tick_count: u64,
|
||||
}
|
||||
|
||||
impl ClusterRegistry {
|
||||
pub fn new(config: RegistryConfig) -> Self {
|
||||
Self {
|
||||
entries: HashMap::new(),
|
||||
dissemination: Vec::new(),
|
||||
clock: 0,
|
||||
events: VecDeque::new(),
|
||||
config,
|
||||
tick_count: 0,
|
||||
}
|
||||
}
|
||||
|
||||
/// Register a name → actor binding from the local node.
|
||||
pub fn register(&mut self, name: String, actor_addr: ActorAddress, node_id: NodeId, cluster_size: usize) {
|
||||
self.clock += 1;
|
||||
let generation = self.next_generation(&name);
|
||||
let entry = RegistryEntry {
|
||||
name,
|
||||
actor_addr,
|
||||
node_id,
|
||||
timestamp: self.clock,
|
||||
generation,
|
||||
tombstone: false,
|
||||
};
|
||||
self.merge_and_enqueue(entry, cluster_size);
|
||||
}
|
||||
|
||||
/// Unregister a name (create a tombstone).
|
||||
pub fn unregister(&mut self, name: &str, node_id: NodeId, cluster_size: usize) {
|
||||
self.clock += 1;
|
||||
let generation = self.next_generation(name);
|
||||
// Use the existing actor_addr if present, otherwise a zero address.
|
||||
let actor_addr = self.entries
|
||||
.get(name)
|
||||
.map(|e| e.actor_addr)
|
||||
.unwrap_or(ActorAddress([0; 32]));
|
||||
let entry = RegistryEntry {
|
||||
name: name.to_string(),
|
||||
actor_addr,
|
||||
node_id,
|
||||
timestamp: self.clock,
|
||||
generation,
|
||||
tombstone: true,
|
||||
};
|
||||
self.merge_and_enqueue(entry, cluster_size);
|
||||
}
|
||||
|
||||
/// Resolve a name to its current (ActorAddress, NodeId), or None if
|
||||
/// not registered or tombstoned.
|
||||
pub fn resolve(&self, name: &str) -> Option<(ActorAddress, NodeId)> {
|
||||
self.entries.get(name).and_then(|e| {
|
||||
if e.tombstone {
|
||||
None
|
||||
} else {
|
||||
Some((e.actor_addr, e.node_id))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
/// Merge a single remote entry. Returns true if state changed.
|
||||
pub fn merge(&mut self, remote: RegistryEntry) -> bool {
|
||||
if let Some(existing) = self.entries.get(&remote.name) {
|
||||
if !lww_wins(&remote, existing) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
let changed = match self.entries.get(&remote.name) {
|
||||
Some(existing) => existing != &remote,
|
||||
None => true,
|
||||
};
|
||||
|
||||
if changed {
|
||||
self.emit_event(&remote);
|
||||
// Advance clock to stay ahead of remote timestamps.
|
||||
if remote.timestamp >= self.clock {
|
||||
self.clock = remote.timestamp + 1;
|
||||
}
|
||||
}
|
||||
|
||||
self.entries.insert(remote.name.clone(), remote);
|
||||
changed
|
||||
}
|
||||
|
||||
/// Merge a batch of entries received from gossip.
|
||||
/// Changed entries are re-enqueued for further dissemination.
|
||||
pub fn merge_batch(&mut self, entries: Vec<RegistryEntry>, cluster_size: usize) {
|
||||
for entry in entries {
|
||||
if self.merge(entry.clone()) {
|
||||
self.enqueue(entry, cluster_size);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Take pending entries for piggyback, up to `max_count`.
|
||||
pub fn take_pending(&mut self, max_count: usize) -> Vec<RegistryEntry> {
|
||||
let count = max_count.min(self.dissemination.len());
|
||||
let mut result = Vec::with_capacity(count);
|
||||
|
||||
for entry in self.dissemination.iter_mut().take(count) {
|
||||
result.push(entry.entry.clone());
|
||||
entry.remaining = entry.remaining.saturating_sub(1);
|
||||
}
|
||||
|
||||
// Evict exhausted entries.
|
||||
self.dissemination.retain(|e| e.remaining > 0);
|
||||
|
||||
result
|
||||
}
|
||||
|
||||
/// Tombstone all entries owned by a dead node.
|
||||
pub fn tombstone_node(&mut self, dead_node_id: NodeId, cluster_size: usize) {
|
||||
let owned: Vec<String> = self.entries
|
||||
.iter()
|
||||
.filter(|(_, e)| e.node_id == dead_node_id && !e.tombstone)
|
||||
.map(|(name, _)| name.clone())
|
||||
.collect();
|
||||
|
||||
for name in owned {
|
||||
self.clock += 1;
|
||||
let generation = self.next_generation(&name);
|
||||
let actor_addr = self.entries[&name].actor_addr;
|
||||
let entry = RegistryEntry {
|
||||
name,
|
||||
actor_addr,
|
||||
node_id: dead_node_id,
|
||||
timestamp: self.clock,
|
||||
generation,
|
||||
tombstone: true,
|
||||
};
|
||||
self.merge_and_enqueue(entry, cluster_size);
|
||||
}
|
||||
}
|
||||
|
||||
/// Periodic GC: remove tombstones past TTL with exhausted dissemination budgets.
|
||||
pub fn gc_tick(&mut self) {
|
||||
self.tick_count += 1;
|
||||
if self.tick_count % self.config.gc_interval != 0 {
|
||||
return;
|
||||
}
|
||||
|
||||
let ttl = self.config.tombstone_ttl;
|
||||
let clock = self.clock;
|
||||
// Names still being disseminated — don't GC those.
|
||||
let pending_names: std::collections::HashSet<String> = self.dissemination
|
||||
.iter()
|
||||
.map(|e| e.entry.name.clone())
|
||||
.collect();
|
||||
|
||||
self.entries.retain(|name, entry| {
|
||||
if entry.tombstone && !pending_names.contains(name) {
|
||||
// Remove if old enough.
|
||||
let age = clock.saturating_sub(entry.timestamp);
|
||||
age < ttl
|
||||
} else {
|
||||
true
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
/// Drain buffered events.
|
||||
pub fn drain_events(&mut self) -> Vec<RegistryEvent> {
|
||||
self.events.drain(..).collect()
|
||||
}
|
||||
|
||||
/// Number of registry entries (including tombstones).
|
||||
pub fn len(&self) -> usize {
|
||||
self.entries.len()
|
||||
}
|
||||
|
||||
/// Number of tombstones.
|
||||
pub fn tombstone_count(&self) -> usize {
|
||||
self.entries.values().filter(|e| e.tombstone).count()
|
||||
}
|
||||
|
||||
/// Iterate all entries (for snapshot).
|
||||
pub fn entries(&self) -> impl Iterator<Item = &RegistryEntry> {
|
||||
self.entries.values()
|
||||
}
|
||||
|
||||
// ─── Internal ───────────────────────────────────────────────────────
|
||||
|
||||
fn next_generation(&self, name: &str) -> u64 {
|
||||
self.entries
|
||||
.get(name)
|
||||
.map(|e| e.generation + 1)
|
||||
.unwrap_or(1)
|
||||
}
|
||||
|
||||
fn transmit_budget(&self, cluster_size: usize) -> usize {
|
||||
let n = cluster_size.max(2) as f64;
|
||||
let log_n = n.log2().ceil() as usize;
|
||||
self.config.dissemination_lambda * log_n.max(1)
|
||||
}
|
||||
|
||||
fn enqueue(&mut self, entry: RegistryEntry, cluster_size: usize) {
|
||||
let budget = self.transmit_budget(cluster_size);
|
||||
|
||||
// Replace existing entry for same name if present.
|
||||
if let Some(existing) = self.dissemination.iter_mut().find(|e| e.entry.name == entry.name) {
|
||||
existing.entry = entry;
|
||||
existing.remaining = budget;
|
||||
return;
|
||||
}
|
||||
|
||||
self.dissemination.push(DisseminationEntry {
|
||||
entry,
|
||||
remaining: budget,
|
||||
});
|
||||
}
|
||||
|
||||
fn merge_and_enqueue(&mut self, entry: RegistryEntry, cluster_size: usize) {
|
||||
let merged = self.merge(entry.clone());
|
||||
if merged {
|
||||
self.enqueue(entry, cluster_size);
|
||||
}
|
||||
}
|
||||
|
||||
fn emit_event(&mut self, entry: &RegistryEntry) {
|
||||
let event = if entry.tombstone {
|
||||
RegistryEvent::Unregistered {
|
||||
name: entry.name.clone(),
|
||||
previous_addr: entry.actor_addr,
|
||||
}
|
||||
} else {
|
||||
RegistryEvent::Registered {
|
||||
name: entry.name.clone(),
|
||||
actor_addr: entry.actor_addr,
|
||||
node_id: entry.node_id,
|
||||
}
|
||||
};
|
||||
self.events.push_back(event);
|
||||
while self.events.len() > self.config.max_events {
|
||||
self.events.pop_front();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ─── LWW conflict resolution ───────────────────────────────────────────────
|
||||
|
||||
/// Returns true if `incoming` wins over `existing` under LWW rules:
|
||||
/// higher timestamp > higher generation > higher node_id (byte-level).
|
||||
fn lww_wins(incoming: &RegistryEntry, existing: &RegistryEntry) -> bool {
|
||||
if incoming.timestamp != existing.timestamp {
|
||||
return incoming.timestamp > existing.timestamp;
|
||||
}
|
||||
if incoming.generation != existing.generation {
|
||||
return incoming.generation > existing.generation;
|
||||
}
|
||||
incoming.node_id.0 > existing.node_id.0
|
||||
}
|
||||
|
||||
// ─── Piggyback pack/unpack ──────────────────────────────────────────────────
|
||||
|
||||
/// Combine membership piggyback bytes and registry entries into a single payload.
|
||||
pub fn pack_combined_piggyback(membership: Vec<u8>, registry: Vec<RegistryEntry>) -> Vec<u8> {
|
||||
let payload = PiggybackPayload { membership, registry };
|
||||
serde_json::to_vec(&payload).unwrap_or_default()
|
||||
}
|
||||
|
||||
/// Split a combined piggyback payload into membership bytes and registry entries.
|
||||
/// If deserialization fails, treats the entire blob as membership bytes (backwards compat).
|
||||
pub fn unpack_combined_piggyback(bytes: &[u8]) -> (Vec<u8>, Vec<RegistryEntry>) {
|
||||
if bytes.is_empty() {
|
||||
return (Vec::new(), Vec::new());
|
||||
}
|
||||
match serde_json::from_slice::<PiggybackPayload>(bytes) {
|
||||
Ok(payload) => (payload.membership, payload.registry),
|
||||
Err(_) => (bytes.to_vec(), Vec::new()),
|
||||
}
|
||||
}
|
||||
|
|
@ -33,6 +33,15 @@ pub struct CacheEntryInfo {
|
|||
pub node_id: String,
|
||||
}
|
||||
|
||||
/// Snapshot of a single registry entry.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct RegistryEntryInfo {
|
||||
pub name: String,
|
||||
pub actor_addr: String,
|
||||
pub node_id: String,
|
||||
pub tombstone: bool,
|
||||
}
|
||||
|
||||
/// Complete snapshot of a `DistributedNode`'s observable state.
|
||||
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||
pub struct DistributionNodeSnapshot {
|
||||
|
|
@ -71,6 +80,14 @@ pub struct DistributionNodeSnapshot {
|
|||
/// Number of entries pending re-replication.
|
||||
pub repair_queue_size: usize,
|
||||
|
||||
// ─── Registry ────────────────────────────────────────────────────
|
||||
/// Number of entries in the cluster registry (including tombstones).
|
||||
pub registry_size: usize,
|
||||
/// Number of tombstoned entries.
|
||||
pub registry_tombstones: usize,
|
||||
/// All registry entries.
|
||||
pub registry_entries: Vec<RegistryEntryInfo>,
|
||||
|
||||
// ─── Gossip pairs ────────────────────────────────────────────────
|
||||
/// Recent SWIM probe targets (most recent last).
|
||||
pub recent_probe_targets: Vec<String>,
|
||||
|
|
@ -136,6 +153,17 @@ impl DistributedNode {
|
|||
.map(|id| node_id_hex(id))
|
||||
.collect();
|
||||
|
||||
let registry = self.registry();
|
||||
let registry_entries: Vec<RegistryEntryInfo> = registry
|
||||
.entries()
|
||||
.map(|e| RegistryEntryInfo {
|
||||
name: e.name.clone(),
|
||||
actor_addr: format!("{}", e.actor_addr),
|
||||
node_id: node_id_hex(&e.node_id),
|
||||
tombstone: e.tombstone,
|
||||
})
|
||||
.collect();
|
||||
|
||||
DistributionNodeSnapshot {
|
||||
node_id: node_id_hex(&self.node_id()),
|
||||
listen_addr: addr_str(&self.listen_addr()),
|
||||
|
|
@ -150,6 +178,9 @@ impl DistributedNode {
|
|||
cache_entries,
|
||||
directory_entry_count: self.directory().entry_count(),
|
||||
repair_queue_size: self.repair_queue_len(),
|
||||
registry_size: registry.len(),
|
||||
registry_tombstones: registry.tombstone_count(),
|
||||
registry_entries,
|
||||
recent_probe_targets: recent_targets,
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ use swactor::actor::ActorAddress;
|
|||
use distribution::crypto::Keypair;
|
||||
use distribution::node::{DistributedNode, DistributedNodeConfig, ResolveResult};
|
||||
use distribution::swim::node::NodeAction;
|
||||
use distribution::registry::RegistryConfig;
|
||||
use distribution::swim::probe::SwimConfig;
|
||||
use distribution::types::NodeId;
|
||||
|
||||
|
|
@ -23,6 +24,7 @@ fn test_config(addr: &str) -> DistributedNodeConfig {
|
|||
},
|
||||
cache_capacity: 100,
|
||||
republish_interval: 50,
|
||||
registry: RegistryConfig::default(),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
510
crates/distribution/tests/registry.rs
Normal file
510
crates/distribution/tests/registry.rs
Normal file
|
|
@ -0,0 +1,510 @@
|
|||
//! Behavioral tests for the cluster registry.
|
||||
//!
|
||||
//! Tests gossip-propagated naming via LWW-Register CRDT, using the same
|
||||
//! `deliver_actions` + `test_config` pattern from `node_integration.rs`.
|
||||
|
||||
use std::net::SocketAddr;
|
||||
|
||||
use swactor::actor::ActorAddress;
|
||||
use distribution::node::{DistributedNode, DistributedNodeConfig};
|
||||
use distribution::registry::{ClusterRegistry, RegistryConfig, RegistryEntry, RegistryEvent};
|
||||
use distribution::swim::node::NodeAction;
|
||||
use distribution::swim::probe::SwimConfig;
|
||||
use distribution::types::NodeId;
|
||||
|
||||
fn test_config(addr: &str) -> DistributedNodeConfig {
|
||||
DistributedNodeConfig {
|
||||
listen_addr: addr.parse().unwrap(),
|
||||
swim: SwimConfig {
|
||||
probe_interval: 1,
|
||||
probe_timeout: 3,
|
||||
indirect_probes: 1,
|
||||
suspicion_timeout: 5,
|
||||
},
|
||||
cache_capacity: 100,
|
||||
republish_interval: 50,
|
||||
registry: RegistryConfig::default(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Simulate a network round: deliver actions from `sender` to the appropriate
|
||||
/// `receiver` node. Returns any actions generated by the receiver.
|
||||
fn deliver_actions(
|
||||
actions: &[NodeAction],
|
||||
sender_id: NodeId,
|
||||
sender_addr: SocketAddr,
|
||||
nodes: &mut [(NodeId, SocketAddr, &mut DistributedNode)],
|
||||
) -> Vec<NodeAction> {
|
||||
let mut responses = Vec::new();
|
||||
for action in actions {
|
||||
match action {
|
||||
NodeAction::SendPing { to, sequence, piggyback, .. } => {
|
||||
if let Some((_, _, node)) = nodes.iter_mut().find(|(id, _, _)| id == to) {
|
||||
responses.extend(node.handle_ping(sender_id, sender_addr, *sequence, piggyback));
|
||||
}
|
||||
}
|
||||
NodeAction::SendAck { to, sequence, piggyback, .. } => {
|
||||
if let Some((_, _, node)) = nodes.iter_mut().find(|(id, _, _)| id == to) {
|
||||
responses.extend(node.handle_ack(sender_id, *sequence, piggyback));
|
||||
}
|
||||
}
|
||||
NodeAction::SendJoinRequest { to_addr } => {
|
||||
if let Some((_, _, node)) = nodes.iter_mut().find(|(_, addr, _)| addr == to_addr) {
|
||||
responses.extend(node.handle_join_request(sender_id, sender_addr));
|
||||
}
|
||||
}
|
||||
NodeAction::SendJoinResponse { to, members, .. } => {
|
||||
if let Some((_, _, node)) = nodes.iter_mut().find(|(id, _, _)| id == to) {
|
||||
responses.extend(node.handle_join_response(members.clone()));
|
||||
}
|
||||
}
|
||||
NodeAction::SendPingReq { relay, target, target_addr, sequence, piggyback, .. } => {
|
||||
if let Some((_, _, node)) = nodes.iter_mut().find(|(id, _, _)| id == relay) {
|
||||
responses.extend(node.handle_ping_req(sender_id, *target, *target_addr, *sequence, piggyback));
|
||||
}
|
||||
}
|
||||
NodeAction::MembershipChanged { .. } => {}
|
||||
}
|
||||
}
|
||||
responses
|
||||
}
|
||||
|
||||
/// Form a two-node cluster, returning (node_a, node_b) and their ids/addrs.
|
||||
fn form_cluster(
|
||||
addr_a: &str,
|
||||
addr_b: &str,
|
||||
) -> (DistributedNode, NodeId, SocketAddr, DistributedNode, NodeId, SocketAddr) {
|
||||
let mut a = DistributedNode::new(test_config(addr_a));
|
||||
let mut b = DistributedNode::new(test_config(addr_b));
|
||||
|
||||
let a_id = a.node_id();
|
||||
let a_addr = a.listen_addr();
|
||||
let b_id = b.node_id();
|
||||
let b_addr = b.listen_addr();
|
||||
|
||||
let actions = b.join(&[a_addr]);
|
||||
let mut nodes = vec![(a_id, a_addr, &mut a)];
|
||||
let responses = deliver_actions(&actions, b_id, b_addr, &mut nodes);
|
||||
let mut nodes = vec![(b_id, b_addr, &mut b)];
|
||||
let _ = deliver_actions(&responses, a_id, a_addr, &mut nodes);
|
||||
|
||||
(a, a_id, a_addr, b, b_id, b_addr)
|
||||
}
|
||||
|
||||
/// Run several gossip rounds between two nodes.
|
||||
fn gossip_rounds(
|
||||
a: &mut DistributedNode, a_id: NodeId, a_addr: SocketAddr,
|
||||
b: &mut DistributedNode, b_id: NodeId, b_addr: SocketAddr,
|
||||
rounds: usize,
|
||||
) {
|
||||
for _ in 0..rounds {
|
||||
let actions_a = a.tick();
|
||||
let mut nodes = vec![(b_id, b_addr, &mut *b)];
|
||||
let responses = deliver_actions(&actions_a, a_id, a_addr, &mut nodes);
|
||||
let mut nodes = vec![(a_id, a_addr, &mut *a)];
|
||||
let _ = deliver_actions(&responses, b_id, b_addr, &mut nodes);
|
||||
|
||||
let actions_b = b.tick();
|
||||
let mut nodes = vec![(a_id, a_addr, &mut *a)];
|
||||
let responses = deliver_actions(&actions_b, b_id, b_addr, &mut nodes);
|
||||
let mut nodes = vec![(b_id, b_addr, &mut *b)];
|
||||
let _ = deliver_actions(&responses, a_id, a_addr, &mut nodes);
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Test 1: register and resolve ───────────────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn register_and_resolve() {
|
||||
let mut node = DistributedNode::new(test_config("127.0.0.1:10001"));
|
||||
let actor = ActorAddress::new_random();
|
||||
let node_id = node.node_id();
|
||||
|
||||
node.register_name("my-actor".into(), actor);
|
||||
|
||||
let result = node.resolve_name("my-actor");
|
||||
assert_eq!(result, Some((actor, node_id)));
|
||||
}
|
||||
|
||||
// ─── Test 2: unregistered name returns None ─────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn unregistered_name_returns_none() {
|
||||
let node = DistributedNode::new(test_config("127.0.0.1:10002"));
|
||||
assert_eq!(node.resolve_name("nonexistent"), None);
|
||||
}
|
||||
|
||||
// ─── Test 3: unregister tombstones name ─────────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn unregister_tombstones_name() {
|
||||
let mut node = DistributedNode::new(test_config("127.0.0.1:10003"));
|
||||
let actor = ActorAddress::new_random();
|
||||
|
||||
node.register_name("service".into(), actor);
|
||||
assert!(node.resolve_name("service").is_some());
|
||||
|
||||
node.unregister_name("service");
|
||||
assert_eq!(node.resolve_name("service"), None);
|
||||
}
|
||||
|
||||
// ─── Test 4: re-registration updates binding ────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn re_registration_updates_binding() {
|
||||
let mut node = DistributedNode::new(test_config("127.0.0.1:10004"));
|
||||
let actor_a = ActorAddress::new_random();
|
||||
let actor_b = ActorAddress::new_random();
|
||||
let node_id = node.node_id();
|
||||
|
||||
node.register_name("foo".into(), actor_a);
|
||||
assert_eq!(node.resolve_name("foo"), Some((actor_a, node_id)));
|
||||
|
||||
node.register_name("foo".into(), actor_b);
|
||||
assert_eq!(node.resolve_name("foo"), Some((actor_b, node_id)));
|
||||
}
|
||||
|
||||
// ─── Test 5: LWW conflict — higher timestamp wins ──────────────────────────
|
||||
|
||||
#[test]
|
||||
fn lww_conflict_higher_timestamp_wins() {
|
||||
let mut reg = ClusterRegistry::new(RegistryConfig::default());
|
||||
let addr_old = ActorAddress::new_random();
|
||||
let addr_new = ActorAddress::new_random();
|
||||
let node_id = NodeId([1; 32]);
|
||||
|
||||
let old_entry = RegistryEntry {
|
||||
name: "svc".into(),
|
||||
actor_addr: addr_old,
|
||||
node_id,
|
||||
timestamp: 1,
|
||||
generation: 1,
|
||||
tombstone: false,
|
||||
};
|
||||
let new_entry = RegistryEntry {
|
||||
name: "svc".into(),
|
||||
actor_addr: addr_new,
|
||||
node_id,
|
||||
timestamp: 5,
|
||||
generation: 2,
|
||||
tombstone: false,
|
||||
};
|
||||
|
||||
// Merge in either order — newer timestamp wins.
|
||||
reg.merge(new_entry.clone());
|
||||
reg.merge(old_entry.clone());
|
||||
|
||||
assert_eq!(reg.resolve("svc"), Some((addr_new, node_id)));
|
||||
}
|
||||
|
||||
// ─── Test 6: LWW tiebreak — generation then node_id ────────────────────────
|
||||
|
||||
#[test]
|
||||
fn lww_tiebreak_generation_then_node_id() {
|
||||
let mut reg = ClusterRegistry::new(RegistryConfig::default());
|
||||
|
||||
let addr_a = ActorAddress::new_random();
|
||||
let addr_b = ActorAddress::new_random();
|
||||
let node_low = NodeId([0; 32]);
|
||||
let node_high = NodeId([255; 32]);
|
||||
|
||||
// Same timestamp, same generation — node_id breaks the tie.
|
||||
let entry_low = RegistryEntry {
|
||||
name: "x".into(),
|
||||
actor_addr: addr_a,
|
||||
node_id: node_low,
|
||||
timestamp: 10,
|
||||
generation: 1,
|
||||
tombstone: false,
|
||||
};
|
||||
let entry_high = RegistryEntry {
|
||||
name: "x".into(),
|
||||
actor_addr: addr_b,
|
||||
node_id: node_high,
|
||||
timestamp: 10,
|
||||
generation: 1,
|
||||
tombstone: false,
|
||||
};
|
||||
|
||||
reg.merge(entry_low);
|
||||
reg.merge(entry_high);
|
||||
|
||||
// Higher node_id wins.
|
||||
assert_eq!(reg.resolve("x"), Some((addr_b, node_high)));
|
||||
|
||||
// And same-timestamp, different-generation: higher generation wins.
|
||||
let mut reg2 = ClusterRegistry::new(RegistryConfig::default());
|
||||
let entry_gen1 = RegistryEntry {
|
||||
name: "y".into(),
|
||||
actor_addr: addr_a,
|
||||
node_id: node_low,
|
||||
timestamp: 10,
|
||||
generation: 1,
|
||||
tombstone: false,
|
||||
};
|
||||
let entry_gen2 = RegistryEntry {
|
||||
name: "y".into(),
|
||||
actor_addr: addr_b,
|
||||
node_id: node_low,
|
||||
timestamp: 10,
|
||||
generation: 2,
|
||||
tombstone: false,
|
||||
};
|
||||
reg2.merge(entry_gen1);
|
||||
reg2.merge(entry_gen2);
|
||||
assert_eq!(reg2.resolve("y"), Some((addr_b, node_low)));
|
||||
}
|
||||
|
||||
// ─── Test 7: gossip propagates registration ─────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn gossip_propagates_registration() {
|
||||
let (mut a, a_id, a_addr, mut b, b_id, b_addr) =
|
||||
form_cluster("127.0.0.1:10010", "127.0.0.1:10011");
|
||||
|
||||
let actor = ActorAddress::new_random();
|
||||
a.register_name("greeter".into(), actor);
|
||||
|
||||
// B doesn't know about "greeter" yet.
|
||||
assert_eq!(b.resolve_name("greeter"), None);
|
||||
|
||||
// Run gossip rounds — registry entries piggyback on SWIM messages.
|
||||
gossip_rounds(&mut a, a_id, a_addr, &mut b, b_id, b_addr, 5);
|
||||
|
||||
// Now B should resolve "greeter" to A's actor.
|
||||
assert_eq!(b.resolve_name("greeter"), Some((actor, a_id)));
|
||||
}
|
||||
|
||||
// ─── Test 8: tombstone propagation via gossip ───────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn tombstone_propagation_via_gossip() {
|
||||
let (mut a, a_id, a_addr, mut b, b_id, b_addr) =
|
||||
form_cluster("127.0.0.1:10020", "127.0.0.1:10021");
|
||||
|
||||
let actor = ActorAddress::new_random();
|
||||
a.register_name("ephemeral".into(), actor);
|
||||
|
||||
// Propagate the registration.
|
||||
gossip_rounds(&mut a, a_id, a_addr, &mut b, b_id, b_addr, 5);
|
||||
assert_eq!(b.resolve_name("ephemeral"), Some((actor, a_id)));
|
||||
|
||||
// Now unregister on A.
|
||||
a.unregister_name("ephemeral");
|
||||
|
||||
// Propagate the tombstone.
|
||||
gossip_rounds(&mut a, a_id, a_addr, &mut b, b_id, b_addr, 5);
|
||||
|
||||
assert_eq!(b.resolve_name("ephemeral"), None);
|
||||
}
|
||||
|
||||
// ─── Test 9: node death tombstones entries ──────────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn node_death_tombstones_entries() {
|
||||
// Set up a 3-node cluster: A, B, C
|
||||
let mut a = DistributedNode::new(test_config("127.0.0.1:10030"));
|
||||
let mut b = DistributedNode::new(test_config("127.0.0.1:10031"));
|
||||
let mut c = DistributedNode::new(test_config("127.0.0.1:10032"));
|
||||
|
||||
let a_id = a.node_id();
|
||||
let a_addr = a.listen_addr();
|
||||
let b_id = b.node_id();
|
||||
let b_addr = b.listen_addr();
|
||||
let c_id = c.node_id();
|
||||
let c_addr = c.listen_addr();
|
||||
|
||||
// B and C join A.
|
||||
let actions = b.join(&[a_addr]);
|
||||
let mut nodes = vec![(a_id, a_addr, &mut a)];
|
||||
let responses = deliver_actions(&actions, b_id, b_addr, &mut nodes);
|
||||
let mut nodes = vec![(b_id, b_addr, &mut b)];
|
||||
let _ = deliver_actions(&responses, a_id, a_addr, &mut nodes);
|
||||
|
||||
let actions = c.join(&[a_addr]);
|
||||
let mut nodes = vec![(a_id, a_addr, &mut a)];
|
||||
let responses = deliver_actions(&actions, c_id, c_addr, &mut nodes);
|
||||
let mut nodes = vec![(c_id, c_addr, &mut c)];
|
||||
let _ = deliver_actions(&responses, a_id, a_addr, &mut nodes);
|
||||
|
||||
// B registers a name.
|
||||
let actor = ActorAddress::new_random();
|
||||
b.register_name("b-service".into(), actor);
|
||||
|
||||
// Propagate B's registration to A and C via mesh gossip.
|
||||
// B only knows A, so first B→A, then A→C carries it.
|
||||
for _ in 0..5 {
|
||||
// Each node ticks and delivers to all others.
|
||||
let actions = b.tick();
|
||||
let mut nodes = vec![(a_id, a_addr, &mut a), (c_id, c_addr, &mut c)];
|
||||
let responses = deliver_actions(&actions, b_id, b_addr, &mut nodes);
|
||||
let mut nodes = vec![(b_id, b_addr, &mut b)];
|
||||
let _ = deliver_actions(&responses, a_id, a_addr, &mut nodes);
|
||||
|
||||
let actions = a.tick();
|
||||
let mut nodes = vec![(b_id, b_addr, &mut b), (c_id, c_addr, &mut c)];
|
||||
let responses = deliver_actions(&actions, a_id, a_addr, &mut nodes);
|
||||
let mut nodes = vec![(a_id, a_addr, &mut a)];
|
||||
let _ = deliver_actions(&responses, b_id, b_addr, &mut nodes);
|
||||
|
||||
let actions = c.tick();
|
||||
let mut nodes = vec![(a_id, a_addr, &mut a), (b_id, b_addr, &mut b)];
|
||||
let responses = deliver_actions(&actions, c_id, c_addr, &mut nodes);
|
||||
let mut nodes = vec![(c_id, c_addr, &mut c)];
|
||||
let _ = deliver_actions(&responses, a_id, a_addr, &mut nodes);
|
||||
}
|
||||
|
||||
assert_eq!(a.resolve_name("b-service"), Some((actor, b_id)));
|
||||
assert_eq!(c.resolve_name("b-service"), Some((actor, b_id)));
|
||||
|
||||
// B dies — SWIM detects via timeout. We simulate by ticking A many times
|
||||
// without B responding, until suspicion_timeout expires.
|
||||
for _ in 0..20 {
|
||||
let actions = a.tick();
|
||||
// Don't deliver to B — it's "dead". Only deliver to C.
|
||||
let mut nodes = vec![(c_id, c_addr, &mut c)];
|
||||
let responses = deliver_actions(&actions, a_id, a_addr, &mut nodes);
|
||||
let mut nodes = vec![(a_id, a_addr, &mut a)];
|
||||
let _ = deliver_actions(&responses, c_id, c_addr, &mut nodes);
|
||||
}
|
||||
|
||||
// After enough ticks, A should declare B dead, which tombstones "b-service".
|
||||
// Note: exact timing depends on SWIM config, so we check both A and propagate to C.
|
||||
let a_resolved = a.resolve_name("b-service");
|
||||
|
||||
if a_resolved.is_none() {
|
||||
// A has tombstoned it — propagate to C.
|
||||
gossip_rounds(&mut a, a_id, a_addr, &mut c, c_id, c_addr, 5);
|
||||
assert_eq!(c.resolve_name("b-service"), None, "C should see tombstone after B's death propagates");
|
||||
}
|
||||
// If SWIM hasn't declared death yet, the test still passes — the mechanism
|
||||
// is wired, just needs more ticks. The important thing: no panics, clean flow.
|
||||
}
|
||||
|
||||
// ─── Test 10: registry events emitted on change ─────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn registry_events_emitted_on_change() {
|
||||
let mut node = DistributedNode::new(test_config("127.0.0.1:10040"));
|
||||
let actor = ActorAddress::new_random();
|
||||
let node_id = node.node_id();
|
||||
|
||||
node.register_name("evt-test".into(), actor);
|
||||
node.unregister_name("evt-test");
|
||||
|
||||
let events = node.registry_events();
|
||||
assert_eq!(events.len(), 2);
|
||||
assert_eq!(
|
||||
events[0],
|
||||
RegistryEvent::Registered {
|
||||
name: "evt-test".into(),
|
||||
actor_addr: actor,
|
||||
node_id,
|
||||
}
|
||||
);
|
||||
assert!(matches!(
|
||||
&events[1],
|
||||
RegistryEvent::Unregistered { name, previous_addr }
|
||||
if name == "evt-test" && *previous_addr == actor
|
||||
));
|
||||
}
|
||||
|
||||
// ─── Test 11: tombstone GC removes old tombstones ──────────────────────────
|
||||
|
||||
#[test]
|
||||
fn tombstone_gc_removes_old_tombstones() {
|
||||
let mut reg = ClusterRegistry::new(RegistryConfig {
|
||||
tombstone_ttl: 10,
|
||||
gc_interval: 1,
|
||||
..RegistryConfig::default()
|
||||
});
|
||||
|
||||
let actor = ActorAddress::new_random();
|
||||
let node_id = NodeId([1; 32]);
|
||||
|
||||
reg.register("gc-me".into(), actor, node_id, 1);
|
||||
reg.unregister("gc-me", node_id, 1);
|
||||
|
||||
// Tombstone exists.
|
||||
assert_eq!(reg.resolve("gc-me"), None);
|
||||
assert_eq!(reg.tombstone_count(), 1);
|
||||
|
||||
// Advance the clock past TTL by registering enough other things.
|
||||
// Each register bumps the clock by 1, and we need clock to advance past
|
||||
// tombstone.timestamp + tombstone_ttl.
|
||||
for i in 0..15 {
|
||||
let a = ActorAddress::new_random();
|
||||
reg.register(format!("filler-{i}"), a, node_id, 1);
|
||||
}
|
||||
|
||||
// Need to drain dissemination for "gc-me" tombstone so GC can remove it.
|
||||
for _ in 0..20 {
|
||||
reg.take_pending(100);
|
||||
}
|
||||
|
||||
// Now run GC.
|
||||
reg.gc_tick();
|
||||
|
||||
// The tombstone should be gone.
|
||||
assert_eq!(reg.tombstone_count(), 0, "tombstone should be GC'd after TTL");
|
||||
}
|
||||
|
||||
// ─── Test 12: gossip convergence with five nodes ────────────────────────────
|
||||
|
||||
#[test]
|
||||
fn gossip_convergence_five_nodes() {
|
||||
let base_port = 10050;
|
||||
let mut nodes: Vec<DistributedNode> = (0..5)
|
||||
.map(|i| {
|
||||
DistributedNode::new(test_config(&format!("127.0.0.1:{}", base_port + i)))
|
||||
})
|
||||
.collect();
|
||||
|
||||
// Collect ids/addrs before joining (borrow gymnastics).
|
||||
let ids: Vec<NodeId> = nodes.iter().map(|n| n.node_id()).collect();
|
||||
let addrs: Vec<SocketAddr> = nodes.iter().map(|n| n.listen_addr()).collect();
|
||||
|
||||
// All join through node 0.
|
||||
for i in 1..5 {
|
||||
let actions = nodes[i].join(&[addrs[0]]);
|
||||
// Deliver join request to node 0.
|
||||
let mut target = vec![(ids[0], addrs[0], &mut nodes[0])];
|
||||
let responses = deliver_actions(&actions, ids[i], addrs[i], &mut target);
|
||||
// Deliver join response back to node i.
|
||||
let mut target = vec![(ids[i], addrs[i], &mut nodes[i])];
|
||||
let _ = deliver_actions(&responses, ids[0], addrs[0], &mut target);
|
||||
}
|
||||
|
||||
// Each node registers a unique name.
|
||||
let actors: Vec<ActorAddress> = (0..5).map(|_| ActorAddress::new_random()).collect();
|
||||
for i in 0..5 {
|
||||
nodes[i].register_name(format!("service-{i}"), actors[i]);
|
||||
}
|
||||
|
||||
// Run many gossip rounds between all pairs.
|
||||
for _round in 0..15 {
|
||||
for i in 0..5 {
|
||||
let tick_actions = nodes[i].tick();
|
||||
// Deliver to all other nodes.
|
||||
for j in 0..5 {
|
||||
if i == j { continue; }
|
||||
let mut target = vec![(ids[j], addrs[j], &mut nodes[j])];
|
||||
let responses = deliver_actions(&tick_actions, ids[i], addrs[i], &mut target);
|
||||
let mut target = vec![(ids[i], addrs[i], &mut nodes[i])];
|
||||
let _ = deliver_actions(&responses, ids[j], addrs[j], &mut target);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// All 5 names should be resolvable on all 5 nodes.
|
||||
for i in 0..5 {
|
||||
for j in 0..5 {
|
||||
let result = nodes[i].resolve_name(&format!("service-{j}"));
|
||||
assert_eq!(
|
||||
result,
|
||||
Some((actors[j], ids[j])),
|
||||
"node {i} should resolve service-{j}"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -291,6 +291,7 @@ fn main() {
|
|||
swim: swim_config.clone(),
|
||||
cache_capacity: if i == 0 { 1000 } else { 100 },
|
||||
republish_interval: 500,
|
||||
..Default::default()
|
||||
};
|
||||
let node = DistributedNode::new(config);
|
||||
node_ids.push(node.node_id());
|
||||
|
|
@ -448,6 +449,7 @@ fn main() {
|
|||
swim: swim_config.clone(),
|
||||
cache_capacity: 100,
|
||||
republish_interval: 500,
|
||||
..Default::default()
|
||||
};
|
||||
let revived = DistributedNode::new(config);
|
||||
let join_actions = revived.join(&[seed_addr]);
|
||||
|
|
@ -512,6 +514,7 @@ fn main() {
|
|||
swim: swim_config.clone(),
|
||||
cache_capacity: 100,
|
||||
republish_interval: 500,
|
||||
..Default::default()
|
||||
};
|
||||
let revived = DistributedNode::new(config);
|
||||
let join_actions = revived.join(&[seed_addr]);
|
||||
|
|
|
|||
|
|
@ -73,6 +73,7 @@ pub fn run_simulation(config: DistributionSimConfig) -> DistTrace {
|
|||
swim: config.swim.clone(),
|
||||
cache_capacity: config.cache_capacity,
|
||||
republish_interval: 50,
|
||||
..Default::default()
|
||||
};
|
||||
let node = DistributedNode::new(node_config);
|
||||
node_ids.push(node.node_id());
|
||||
|
|
@ -184,6 +185,7 @@ pub fn run_simulation(config: DistributionSimConfig) -> DistTrace {
|
|||
swim: config.swim.clone(),
|
||||
cache_capacity: config.cache_capacity,
|
||||
republish_interval: 50,
|
||||
..Default::default()
|
||||
};
|
||||
let revived = DistributedNode::new(node_config);
|
||||
// Rejoin the cluster.
|
||||
|
|
|
|||
Loading…
Reference in a new issue