feat: distribution simulation tests #34
Loading…
Reference in a new issue
No description provided.
Delete branch "distribution-sim-tests"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Test cluster behavior in simulation.
Add sim infrastructure for registry testing: - Extend DistributionSnapshot with registry_size, registry_tombstone_count - Add SimAction enum (RegisterName, RegisterNameWithActor, UnregisterName) - Add action_schedule to DistributionSimConfig for mid-sim registry ops - Add run_simulation_with_nodes() returning node references for assertions - Add property checks: registry_propagation, registry_tombstones, etc. 6 registry sim tests: - registry_name_converges_across_cluster (PASS) - split_brain_naming_converges_after_partition_heals (IGNORED — bug) - tombstone_propagates_when_name_owner_dies (IGNORED — bug) - rapid_re_registration_converges_to_latest (PASS) - simultaneous_registration_converges_deterministically (PASS) - multiple_names_from_different_nodes_all_propagate (PASS) BUG FOUND: SwimProbe::check_suspicion_timeouts() calls members.declare_dead() internally before SwimNode::translate_probe_actions() processes the DeclareDead action, so the second declare_dead() returns false and MembershipChanged{Dead} is never emitted. This silently breaks all death-related side effects: RT cleanup, cache invalidation, repair queue population, and registry tombstoning. Authored by Claude, lovingly guided by Zachery Aaron Shores-ChmielewskiBug 1 (swim/node.rs): SwimProbe::check_suspicion_timeouts() calls members.declare_dead() before translate_probe_actions() processes the DeclareDead action. The second declare_dead() returned false (already dead), so MembershipChanged{Dead} was never emitted and the death was never enqueued for dissemination. Fix: remove the redundant declare_dead() call in translate_probe_actions since the probe already performed the mutation. Bug 2 (swim/node.rs): apply_membership_update() — which processes piggyback on every ping/ack/ping_req — updated the internal member list but never emitted NodeAction::MembershipChanged. This meant DistributedNode was blind to all state transitions learned via gossip piggyback (e.g., a dead node refuting via incarnation bump). Fix: return MembershipChanged actions from apply_piggyback and propagate through handle_ping/handle_ack/handle_ping_req. Bug 3 (node.rs): DistributedNode::handle_ping/handle_ack/handle_ping_req never processed MembershipChanged actions from SwimNode — only tick() did. Fix: extract process_membership_changes() helper and call it from all four message paths (tick, handle_ping, handle_ack, handle_ping_req). Additional fixes: - registry.rs: add re_disseminate_all() for anti-entropy on partition heal - node.rs: call re_disseminate_all on MemberState::Alive transitions so registry state accumulated during partition reaches recovering nodes - cluster_scenarios: enable dead_reprobe in 10% message loss test, since correct death dissemination (now working) causes cascading false deaths without a recovery mechanism Authored by Claude, lovingly guided by Zachery Aaron Shores-Chmielewski