# Actor Watching — Distributed Death Notifications

## Problem

When an actor dies (panic, explicit stop, or its host node leaves the cluster), other actors that depend on it have no way to know. This is the distributed equivalent of `waitpid()` / `SIGCHLD` — the foundational primitive for building supervision, reconnection logic, and self-healing.

## Design

### Types

```rust
// src/actor.rs

/// Why an actor exited.
#[derive(Debug, Clone, PartialEq, Eq)]
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
pub enum ExitReason {
    /// Actor was explicitly stopped or removed from the pool.
    Stopped,
    /// Actor panicked during message handling.
    Panicked,
    /// The node hosting the actor left the cluster (SWIM Dead).
    NodeDown,
}

/// Delivered to watchers when a watched actor exits.
///
/// Implements `Message` (Clone + Send + Sync + 'static) so it can be
/// delivered through normal mailbox channels.
#[derive(Debug, Clone)]
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
pub struct ActorExited {
    /// The address of the actor that died.
    pub addr: ActorAddress,
    /// Why it exited.
    pub reason: ExitReason,
}
```

### API

```rust
// src/actor.rs — extend ContextInner

pub trait ContextInner {
    // existing:
    fn send_any(&self, addr: ActorAddress, msg: Box<dyn Any + Send>) -> Result<(), Error>;
    fn spawn_any(&self, addr: ActorAddress, actor: Box<dyn AnyActor>);

    // new:
    fn watch(&self, watcher: ActorAddress, target: ActorAddress);
    fn unwatch(&self, watcher: ActorAddress, target: ActorAddress);
}

// src/actor.rs — extend Ctx

impl Ctx<'_> {
    /// Watch another actor's liveness. If the target dies, this actor
    /// receives an `ActorExited` message in its mailbox.
    ///
    /// Watching an already-dead or non-existent actor delivers
    /// `ActorExited { reason: Stopped }` immediately (on next tick).
    ///
    /// Calling watch() multiple times on the same target is idempotent —
    /// only one notification is delivered.
    pub fn watch(&self, target: ActorAddress) {
        self.inner.watch(self.self_addr, target);
    }

    /// Stop watching an actor. No notification will be delivered if the
    /// target subsequently dies.
    pub fn unwatch(&self, target: ActorAddress) {
        self.inner.unwatch(self.self_addr, target);
    }
}
```

### WatchRegistry

A per-worker structure that tracks watch relationships:

```rust
// src/worker.rs

struct WatchRegistry {
    /// target -> set of watchers awaiting death notification
    watchers: HashMap<ActorAddress, HashSet<ActorAddress>>,
    /// watcher -> set of targets it's watching (reverse index for cleanup)
    watching: HashMap<ActorAddress, HashSet<ActorAddress>>,
}

impl WatchRegistry {
    fn watch(&mut self, watcher: ActorAddress, target: ActorAddress) {
        self.watchers.entry(target).or_default().insert(watcher);
        self.watching.entry(watcher).or_default().insert(target);
    }

    fn unwatch(&mut self, watcher: ActorAddress, target: ActorAddress) {
        if let Some(set) = self.watchers.get_mut(&target) {
            set.remove(&watcher);
            if set.is_empty() { self.watchers.remove(&target); }
        }
        if let Some(set) = self.watching.get_mut(&watcher) {
            set.remove(&target);
            if set.is_empty() { self.watching.remove(&watcher); }
        }
    }

    /// Called when an actor dies. Returns (watcher_addr, ActorExited) pairs.
    fn notify_death(&mut self, target: ActorAddress, reason: ExitReason)
        -> Vec<(ActorAddress, ActorExited)>
    {
        let notification = ActorExited { addr: target, reason };
        let mut result = Vec::new();

        if let Some(watcher_set) = self.watchers.remove(&target) {
            for watcher in &watcher_set {
                result.push((*watcher, notification.clone()));
                // clean up reverse index
                if let Some(set) = self.watching.get_mut(watcher) {
                    set.remove(&target);
                    if set.is_empty() { self.watching.remove(watcher); }
                }
            }
        }

        result
    }

    /// Called when a watcher itself dies. Cleans up all its watching entries.
    fn cleanup_watcher(&mut self, watcher: &ActorAddress) {
        if let Some(targets) = self.watching.remove(watcher) {
            for target in targets {
                if let Some(set) = self.watchers.get_mut(&target) {
                    set.remove(watcher);
                    if set.is_empty() { self.watchers.remove(&target); }
                }
            }
        }
    }
}
```

### Integration with tick_once

The worker's `tick_once` gains **two new mechanisms**:

**A. Watch/unwatch request processing** — `WorkerContext::watch()` needs to route the request to the correct worker (the one that owns the target). This mirrors how sends work:

```
watch(watcher_A, target_B) called on worker 0:
  - target_B is on worker 0? → register locally in WatchRegistry
  - target_B is on worker 1? → send WatchCommand through transfer queue
  - target_B not in address_map? → may be remote (see Remote Watches below)
```

New envelope variant for internal watch commands:

```rust
// src/delivery.rs

enum InternalCommand {
    Watch { watcher: ActorAddress, target: ActorAddress },
    Unwatch { watcher: ActorAddress, target: ActorAddress },
}
```

These are delivered through the existing transfer queue alongside `Envelope`s. The transfer queue type becomes `enum TransferItem { Message(Envelope), Command(InternalCommand) }`, or — simpler — the WatchRegistry is shared (behind Arc<Mutex>) and watch/unwatch are applied directly. The shared approach is better since watches are rare relative to messages.

**Recommended**: `Arc<Mutex<WatchRegistry>>` shared across workers, owned by Runtime. Workers hold a reference. Contention is negligible because watch/unwatch operations are rare.

**B. Death notification dispatch** — added to tick_all's panic detection:

```rust
// In ActorPool::tick_all, after catching a panic:
Err(_) => {
    slot.poisoned = true;
    slot.mailbox.clear();
    // NEW: collect death notification
    deaths.push((addr, ExitReason::Panicked));
}
```

After tick_all completes, the worker processes `deaths`:

```rust
// In tick_once, after tick_all:
for (addr, reason) in deaths {
    let notifications = watch_registry.lock().notify_death(addr, reason);
    for (watcher_addr, msg) in notifications {
        // Deliver ActorExited as a normal message
        self.deliver_to(watcher_addr, Box::new(msg), tc);
    }
    // Also clean up the dead actor's own watches
    watch_registry.lock().cleanup_watcher(&addr);
}
```

### Watching Non-Existent Actors

If `watch(watcher, target)` is called and `target` doesn't exist in the address map:

- **Local runtime**: deliver `ActorExited { reason: Stopped }` immediately (on next tick). The actor is already gone.
- **Distributed**: the watch request is forwarded to the node that should own the target (via Kademlia resolution). If the target doesn't exist there either, a `ActorExitedNotify` is sent back.

### Remote Watches (cross-node)

Wire protocol additions in `crates/distribution/src/messages.rs`:

```rust
/// Request from node A to node B: "notify me if this actor dies"
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct WatchRequest {
    pub watcher_node: NodeId,
    pub watcher_addr: ActorAddress,
    pub target_addr: ActorAddress,
}
impl NetworkMessage for WatchRequest {
    fn type_tag() -> &'static str { "swactor_dist::WatchRequest" }
}

/// Request to cancel a remote watch
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct UnwatchRequest {
    pub watcher_node: NodeId,
    pub watcher_addr: ActorAddress,
    pub target_addr: ActorAddress,
}
impl NetworkMessage for UnwatchRequest {
    fn type_tag() -> &'static str { "swactor_dist::UnwatchRequest" }
}

/// Notification from target's node to watcher's node
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ActorExitedNotify {
    pub target_addr: ActorAddress,
    pub reason: ExitReason,
    /// Which watchers on the receiving node should be notified
    pub watchers: Vec<ActorAddress>,
}
impl NetworkMessage for ActorExitedNotify {
    fn type_tag() -> &'static str { "swactor_dist::ActorExitedNotify" }
}
```

**Flow**:

```
Node A                              Node B
  │                                   │
  │  ctx.watch(watcher_A, target_B)   │
  │                                   │
  │  ── WatchRequest ──────────────►  │
  │                                   │  registers remote watch:
  │                                   │  target_B → (NodeA, watcher_A)
  │                                   │
  │         ... time passes ...       │
  │                                   │
  │                                   │  target_B panics
  │                                   │
  │  ◄── ActorExitedNotify ────────   │
  │                                   │
  │  delivers ActorExited to          │
  │  watcher_A's mailbox              │
```

The `WatchRegistry` on node B stores remote watches with the additional `NodeId` of the watcher's node. On death, it partitions notifications into local (deliver directly) and remote (send `ActorExitedNotify` to the watcher's node).

### SWIM Integration

When `handle_membership_change` detects `MemberState::Dead` (`crates/distribution/src/node.rs:269`):

1. The node maintains a **node actor index**: `NodeId -> Set<ActorAddress>` — all actors known to be on each node. This is populated from:
   - Directory entries stored locally
   - Cache entries
   - Remote watch registrations

2. On node death:
   ```rust
   fn handle_membership_change(&mut self, node_id: NodeId, state: MemberState) {
       match state {
           MemberState::Dead => {
               // existing cleanup...
               self.routing_table.remove(&node_id);
               self.cache.invalidate_node(&node_id);
               self.repair_queue.on_node_death(&node_id, &mut self.directory);

               // NEW: notify local watchers about all actors on the dead node
               if let Some(actor_addrs) = self.node_actor_index.remove(&node_id) {
                   for actor_addr in actor_addrs {
                       let notifications = self.watch_registry.notify_death(
                           actor_addr,
                           ExitReason::NodeDown,
                       );
                       // Queue for delivery to local actors
                       self.pending_exit_notifications.extend(notifications);
                   }
               }
           }
           // ...
       }
   }
   ```

3. The `pending_exit_notifications` are drained by the NodeDriver on the next tick and delivered into the local runtime.

### Edge Cases

| Scenario | Behavior |
|---|---|
| Watch self | Allowed. On death, ActorExited delivered to own mailbox (no-op since dead). |
| Watch already-dead actor | `ActorExited { reason: Stopped }` delivered on next tick. |
| Watcher dies before target | Cleanup removes all watching entries. No notification delivered. |
| Target node suspected (not yet dead) | No notification — wait for SWIM to confirm Dead or Alive. |
| Network partition heals | If target was falsely declared Dead, a stale `NodeDown` was sent. The watcher may re-watch. No automatic "un-death" notification. |
| Double watch | Idempotent — only one notification per death event. |

### Files Modified

| File | Change |
|------|--------|
| `src/actor.rs` | `ExitReason`, `ActorExited`, `watch()`/`unwatch()` on `ContextInner`, `Ctx` |
| `src/worker.rs` | `WatchRegistry`, death collection in `tick_all`, notification dispatch in `tick_once` |
| `src/runtime.rs` | `Arc<Mutex<WatchRegistry>>` owned by Runtime, passed to workers |
| `crates/distribution/src/messages.rs` | `WatchRequest`, `UnwatchRequest`, `ActorExitedNotify` |
| `crates/distribution/src/node.rs` | Node actor index, SWIM Dead fan-out, `pending_exit_notifications` |

### Tests

- **watch_local_death**: spawn watcher + target, kill target (panic), verify watcher receives `ActorExited { reason: Panicked }`
- **unwatch_prevents_notification**: watch then unwatch, kill target, verify no notification
- **watch_nonexistent**: watch an address that was never spawned, verify `ActorExited { reason: Stopped }`
- **watcher_dies_first**: watch target, kill watcher, kill target — no panic/leak
- **cross_worker_watch**: target on worker 0, watcher on worker 1, kill target, verify notification arrives
- **idempotent_watch**: watch same target twice, kill target, verify exactly one notification