swactor/docs/distribution/swim.md

# SWIM Membership Protocol

SWIM (Scalable Weakly-consistent Infection-style Membership) handles failure
detection and membership dissemination. Each node periodically probes a random
peer; if the peer doesn't respond, indirect probes through relays confirm or
deny reachability before the node is suspected and eventually declared dead.

## Probe Cycle

See [swim_probe_cycle.svg](../diagrams/swim_probe_cycle.svg) for the full state machine.

The probe cycle is a pure state machine driven by ticks:

```
    ┌──────────────────────────────────────────────────────────────────────┐
    │                        ProbePhase                                    │
    │                                                                      │
    │   Idle ──probe_interval──► WaitingDirectAck ──timeout──►             │
    │    ▲                            │                                    │
    │    │                        ack received                             │
    │    │                            │                                    │
    │    └────────────────────────────┘                                    │
    │                                                                      │
    │   WaitingDirectAck ──timeout──► WaitingIndirectAck ──timeout──►      │
    │                                        │                             │
    │                                    indirect ack                      │
    │                                        │                             │
    │                                   back to Idle                       │
    │                                                                      │
    │   WaitingIndirectAck ──timeout──► Suspect(target) + start timer      │
    │                                                                      │
    │   Suspicion timer ──timeout──► DeclareDead(target)                   │
    └──────────────────────────────────────────────────────────────────────┘
```

Probe targets are selected via round-robin over a shuffled member list. Each
probe cycle picks one target and sends a direct `Ping`. If no `Ack` arrives
within `probe_timeout` ticks, indirect `PingReq` messages are sent through
`k` relay nodes. If no indirect ack arrives either, the target enters the
Suspect state.

A ring buffer of the last 16 probe targets is maintained for dashboard
visualization.

## Member States & Incarnation

```
┌─ MemberState ────────────────────────────────────────────────────────────┐
│                                                                          │
│  Alive ─────► Suspect ─────► Dead                                        │
│                                                                          │
│  Priority ordering: Dead (2) > Suspect (1) > Alive (0)                  │
│  Within the same incarnation, higher-priority state always wins.         │
│                                                                          │
│  Incarnation number: bumped only by the node itself to refute            │
│  suspicion. Higher incarnation unconditionally overrides any state.      │
│                                                                          │
└──────────────────────────────────────────────────────────────────────────┘
```

The membership list is a CRDT with merge semantics:
1. Higher incarnation wins unconditionally.
2. Same incarnation: higher-priority state wins (`Dead > Suspect > Alive`).
3. Lower incarnation is silently ignored.

When a node receives a `Suspect` about itself, it bumps its incarnation and
disseminates an `Alive` update — this is self-refutation.

## Dissemination Queue

Membership updates are not sent in dedicated messages. Instead, they are
**piggybacked** onto existing protocol messages (Pings, Acks, PingReqs).

Each update has a **transmit budget** of `Lambda * ceil(log2(n))` sends, where
`Lambda` (default 3) is the dissemination multiplier and `n` is the cluster
size. The budget ensures logarithmic scaling: a 10-node cluster sends each
update ~12 times; a 1000-node cluster sends it ~30 times.

Updates are sorted by priority when piggybacking, so `Dead` notifications
propagate faster than routine `Alive` heartbeats.

See `crates/distribution/DESIGN_NOTES.md` for the full rationale.

## Lifeguard Health Multiplier

Inspired by the Hashicorp Lifeguard paper, the `HealthMultiplier` tracks
the local node's network health:

- Each successful ack decreases the health score.
- Each nack/timeout increases it (capped at `max_health_score`).
- The score produces a multiplier `1 + score` that stretches probe intervals
  and timeouts.

```rust
// Dynamic suspicion timeout:
// clamp(base * ceil(log2(n+1)) * multiplier, min, max)
pub fn dynamic_suspicion_timeout(&self, cluster_size: usize) -> u64
```

This prevents unhealthy nodes from generating false accusations: a node
that can't reach peers slows its own probing rather than suspecting everyone.

## Join Protocol

A new node contacts seed addresses with `SendJoinRequest`. The receiving
node adds the newcomer to its member list, enqueues the event for
dissemination, and responds with a `SendJoinResponse` containing the full
membership snapshot. The joiner applies the snapshot, learning about all
existing cluster members in one round-trip.

## Where Things Live

| Type | File | Role |
|------|------|------|
| `SwimProbe` | `swim/probe.rs` | Probe cycle state machine |
| `SwimConfig` | `swim/probe.rs` | Protocol tuning knobs |
| `SwimEvent` / `SwimAction` | `swim/probe.rs` | State machine I/O |
| `MemberList` | `swim/member_list.rs` | Membership CRDT |
| `MemberEntry` | `swim/member_list.rs` | Single member record |
| `SwimNode` | `swim/node.rs` | Integrated SWIM facade |
| `NodeAction` | `swim/node.rs` | High-level network actions |
| `DisseminationQueue` | `swim/dissemination.rs` | Piggybacked update queue |
| `HealthMultiplier` | `swim/lifeguard.rs` | Local health tracking |
| `LifeguardConfig` | `swim/lifeguard.rs` | Lifeguard tuning knobs |
| `MemberState` | `types.rs` | `Alive` / `Suspect` / `Dead` enum |
| `NodeRecord` | `types.rs` | Wire-format membership record |