zacheryasc/swactor-development-history
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
swactor-development-history/pooled-datastore/POOLED_DATASTORE.md

18 KiB
Raw Permalink Blame History

Pooled Datastore & Sim-Cluster — Development History

Adds a gossip-converged pooled storage protocol, a generic gossip channel abstraction, a pool dashboard page, shared pool types, iroh connection hardening, and a Docker-free multi-process sim-cluster test harness.

~18 new/modified files · ~2,400 insertions

Branch: pooled-datastore

Table of Contents

  1. Overview & Motivation
  2. What Was Built
  3. Pooled Storage Protocol
  4. Generic Gossip Channel Abstraction
  5. Pool Disseminator
  6. Pool Coordinator Actor
  7. Shared Pool Types
  8. Dashboard Pool Page
  9. Iroh Connection Hardening
  10. Sim-Cluster Test Harness
  11. Design Decisions & Tradeoffs
  12. Test Coverage
  13. Known Gaps & Future Work

1. Overview & Motivation

The existing datastore provides content-addressed storage on individual nodes, but there is no mechanism for multiple nodes to form a shared storage pool — knowing who has what content, how much capacity each node offers, or where to place new data.

This branch introduces a pooled datastore protocol that layers on top of the existing content-addressed datastore. Multiple nodes join a named pool, gossip their membership/capacity/content-locations via SWIM piggyback, and converge on a shared view of the pool's state. This enables:

  • Content location: find which node(s) hold a given content hash without fan-out queries.
  • Capacity-aware placement: route new writes to the node with the most free space.
  • Pool ACL: optional allow-list to restrict which nodes can join a pool.
  • Live observability: a new dashboard page shows pool membership, capacity bars, content location map, and ACL state in real time via SSE.

Separately, the branch also introduces:

  • A generic gossip channel abstraction (GossipChannel trait + DisseminationBuffer<T>) that replaces the 4 duplicated dissemination patterns in the distribution crate.
  • A sim-cluster test harness (cargo xtask test sim-cluster / cargo xtask sim-cluster) that spawns real multi-process clusters with a local iroh relay — no Docker required.
  • Iroh connection hardening: relay URL resolution cascade and connect timeouts to prevent indefinite hangs during peer connection.

2. What Was Built

Component Crate / Location Lines
Pool types (PoolId, entries, config) crates/shared-types/src/pool.rs ~170
Gossip channel trait + DisseminationBuffer crates/distribution/src/gossip_channel.rs ~270
Pool disseminator (CRDT state + gossip) crates/datastore/src/pool/disseminator.rs ~720
Pool coordinator actor crates/datastore/src/pool/coordinator.rs ~275
Pool messages crates/datastore/src/pool/messages.rs ~80
Pool dashboard HTML/JS crates/dashboard/src/pool_html.rs ~390
Sim-cluster harness xtask/src/sim_cluster.rs ~700
Pool integration tests crates/datastore/tests/pool_tests.rs ~340
Docker compose (dev cluster) tests/docker/docker-compose.dev-cluster.yml ~75

Modified files:

File Change
crates/distribution/src/iroh_driver.rs Relay URL cascade + connect timeout
crates/swactor-node/src/main.rs Seed node relay URL hints for iroh
crates/datastore/tests/dashboard_integration_test.rs Start HTTP standalone
xtask/src/main.rs sim-cluster subcommand + test group
xtask/Cargo.toml reqwest, tokio, iroh-relay deps

3. Pooled Storage Protocol

The pool protocol is a set of four CRDT entry types that converge via gossip:

┌────────────────────────────────────────────────────┐
│                   Pool State (per node)             │
├──────────────┬─────────────┬───────────┬───────────┤
│  Membership  │  Capacity   │  Content  │    ACL    │
│              │             │  Location │           │
│ node→state   │ node→bytes  │ (hash,    │ node→     │
│ (Active/Left)│ (total/used)│  node)→   │  grant/   │
│              │             │  tombstone│  revoke   │
├──────────────┴─────────────┴───────────┴───────────┤
│             Higher generation always wins           │
│          (last-writer-wins register per key)        │
└────────────────────────────────────────────────────┘

Convergence rule: For each entry type, the key is derived from the entry (e.g. node_id for membership, (content_hash, node_id) for content locations). When two entries share a key, the one with the higher generation wins. This makes all merges commutative, associative, and idempotent — a CRDT.

Deletion: Content locations and ACL entries use tombstones (tombstone: true / revoked: true) with a generation bump. Tombstones are garbage collected after a configurable TTL.

Dissemination: All entries go through a shared DisseminationBuffer<PoolEntry> which transmits each entry Λ * ceil(log₂(n)) times before eviction, matching the standard SWIM protocol budget.

4. Generic Gossip Channel Abstraction

File: crates/distribution/src/gossip_channel.rs

Before this branch, SWIM piggyback dissemination was hardcoded for membership updates, directory entries, and dead-letter notifications — each with its own copy of the Λ * ceil(log₂(n)) budget logic.

The new abstraction provides:

  • GossipChannel trait: A topic-tagged channel that produces/consumes Vec<u8> entries for piggyback. Methods: topic_tag(), take_pending_bytes(), apply_incoming_bytes(), re_disseminate_all(), on_node_death(), gc_tick().

  • DisseminationBuffer<T>: A generic budget-limited queue. Entries are enqueued with a transmit budget of Λ * ceil(log₂(n)) and evicted after exhaustion. Supports enqueue, enqueue_or_replace (idempotent upsert), take, re_enqueue_all, and retain.

  • Serialization helpers: serialize_each() and deserialize_each() for converting between typed entries and Vec<u8>.

The pool disseminator is the first consumer, plugging into the distribution layer via SharedPoolChannel which wraps Arc<Mutex<PoolDisseminator>>.

5. Pool Disseminator

File: crates/datastore/src/pool/disseminator.rs

The core state machine. Manages four HashMap tables (membership, capacity, content locations, ACL) and a single DisseminationBuffer<PoolEntry>.

Key methods:

  • Lifecycle: join(), leave() — announce membership state changes.
  • Storage: announce_content(), remove_content(), announce_capacity().
  • ACL: grant_access(), revoke_access(), is_node_authorized().
  • Queries: active_members(), member_count(), content_count(), locate_content(), node_with_most_free_space(), pool_capacity_summary().
  • Dashboard: snapshot_json() — full JSON snapshot for SSE.
  • Internal: merge_entry() applies the higher-generation-wins rule. take_pending_inner() / apply_incoming_inner() drive gossip exchange.

SharedPoolChannel wraps this in an Arc<Mutex<>> and implements GossipChannel, bridging ownership between the PoolCoordinator actor (lifecycle/queries) and DistributedNode (gossip transport).

6. Pool Coordinator Actor

File: crates/datastore/src/pool/coordinator.rs

An actor implementing ActorInterface for PoolCoordinatorMsg. It acts as a placement-aware CRUD facade:

  • PoolPut/Get/Delete/List: Delegates to the co-located DatastoreNode actor. Future: redirect to best node based on capacity.
  • PoolStatus: Queries the disseminator and returns a JSON status snapshot.
  • JoinPool/LeavePool: Checks ACL authorization, then calls the disseminator.
  • GrantPoolAccess/RevokePoolAccess: Manages the allow-list.
  • PoolTick: Periodic capacity re-announcement (every 100 ticks).

7. Shared Pool Types

File: crates/shared-types/src/pool.rs

Types live in shared-types to avoid circular dependencies between distribution and datastore:

  • PoolId: blake3(name_bytes) — 32-byte deterministic pool identifier. Supports hex encoding/decoding and truncated display.
  • PoolMemberEntry: Node membership with Active/Left state.
  • PoolCapacityEntry: Storage capacity announcement (total/used bytes).
  • ContentLocationEntry: Where a content hash is stored, with tombstone support.
  • PoolACLEntry: Authorization grant/revoke with granted_by provenance.
  • PoolEntry: Tagged enum wrapping all four entry types for gossip serialization.
  • PoolConfig: Pool configuration (name, capacity, TTL, GC interval, Λ).

8. Dashboard Pool Page

File: crates/dashboard/src/pool_html.rs

A new /pool page in the dashboard with:

  • Summary cards: pool name, member count, content count, total/used capacity.
  • Capacity bars: per-node usage with color thresholds (green < 70%, orange < 90%, red >= 90%).
  • Members table: node ID (truncated with tooltip), state, total/used/free.
  • Content location map: content hash → replica count → node list.
  • ACL panel: open mode indicator or allow-list table.
  • Join/Leave buttons: POST to /api/pool/join and /api/pool/leave.
  • Live updates: SSE pool events drive real-time state refresh.

9. Iroh Connection Hardening

File: crates/distribution/src/iroh_driver.rs

Two problems fixed:

  1. Relay URL resolution cascade: When connecting to a peer, the driver now tries three sources in order: (a) explicit relay URL cache from prior connections, (b) SWIM metadata gossip (via node.relay_url()), (c) the local node's own home relay. Previously only the explicit cache was checked, causing connections to fail when the cache was empty.

  2. Connect timeout: All endpoint.connect() calls now have a 2-second tokio::time::timeout wrapper. Previously, connections could hang indefinitely if a peer was unreachable.

File: crates/swactor-node/src/main.rs

Seed node addresses now include relay URLs so iroh can locate the seed through the relay server, rather than relying solely on direct addressing.

10. Sim-Cluster Test Harness

File: xtask/src/sim_cluster.rs

A new test stage and development tool that spawns real multi-process swactor clusters without Docker:

Test mode: cargo xtask test sim-cluster

Runs 4 scenarios sequentially, each with a fresh 5-node cluster:

# Scenario Validates
1 Cluster convergence All 5 nodes see >= 4 alive peers, routing table >= 4
2 Node death detection Kill node 2, survivors detect alive drop, dead count >= 1
3 Killed node rejoins Kill node 2, restart it, rejoined node sees alive >= 1
4 Actors resolvable Each node has >= 2 directory entries, total >= 10

Interactive mode: cargo xtask sim-cluster --nodes N

Spawns a persistent cluster for development. Prints dashboard URLs and blocks until Ctrl-C.

Infrastructure

  • Local relay server: Embedded iroh-relay server on an ephemeral port. Nodes connect through the relay rather than requiring direct connectivity.
  • RAII lifecycle: SimCluster owns child processes and SIGTERM's them on drop. RelayServer owns its tokio runtime.
  • Config generation: Each node gets a node.toml with dashboard port, actor count, relay host/port, and optional seed node ID.
  • Seed key discovery: Polls the seed node's key file on disk to extract the public key before spawning joiner nodes.
  • HTTP observation: Polls /api/distribution on each node's dashboard. Uses serde_json::Value to avoid compile-time coupling to protocol types.
  • Node lifecycle: kill_node() sends SIGTERM, restart_node() re-spawns with the same config (non-seed nodes get the seed's public key).

Dev cluster compose

File: tests/docker/docker-compose.dev-cluster.yml

A 3-node Docker Compose file for development with pool configuration (--pool-name dev-pool --pool-capacity 104857600). Uses a bridge network with static IPs.

11. Design Decisions & Tradeoffs

Higher-generation-wins CRDT over vector clocks: Pool entries use a simple monotonic generation counter per entry key. This is sufficient because each entry has a single writer (the node that owns it). Vector clocks would add complexity without benefit since there are no concurrent writers for the same key.

Tombstones with TTL over immediate deletion: Content locations and ACL revocations use tombstones that propagate via gossip before being GC'd. Without tombstones, a deleted entry could be re-introduced by a node that hasn't yet received the deletion.

Shared Arc<Mutex<>> over message-passing for disseminator: The pool disseminator needs to be accessed by both the coordinator actor (for lifecycle/queries) and the distribution layer (for gossip). Rather than adding an actor-to-actor message protocol, the disseminator is wrapped in Arc<Mutex<PoolDisseminator>>. The lock is held only briefly for individual operations.

Sim-cluster over Docker for testing: Docker adds build time, image management, and network configuration complexity. The sim-cluster spawns bare processes on localhost, uses an embedded iroh relay, and tears down in milliseconds. Scenarios that previously required Docker Compose now run with cargo xtask test sim-cluster.

HTTP polling over direct protocol observation: The sim-cluster observes node state via HTTP (/api/distribution) rather than linking against protocol types. This makes the test harness resilient to protocol changes and mirrors how an operator would observe a real cluster.

Pool types in shared-types: Pool entry types live in shared-types rather than datastore to avoid a circular dependency — distribution needs to know about pool entries for gossip serialization, and datastore depends on distribution.

12. Test Coverage

Unit tests (disseminator internals)

In crates/datastore/src/pool/disseminator.rs:

  • join_and_query_members — join lifecycle
  • leave_removes_from_active — leave lifecycle
  • announce_and_locate_content — content announcement + query
  • remove_content_tombstones — tombstone semantics
  • capacity_summary — capacity aggregation
  • acl_grant_and_check / acl_revoke / empty_acl_means_open — ACL logic
  • higher_generation_wins_merge — CRDT merge rule
  • two_disseminators_converge_via_gossip_exchange — two-node gossip
  • three_node_convergence_loop — multi-round gossip convergence

Unit tests (gossip channel)

In crates/distribution/src/gossip_channel.rs:

  • budget_math_* (4 tests) — transmit budget calculation
  • enqueue_take_evicts_after_budget — budget exhaustion
  • enqueue_or_replace_* (2 tests) — idempotent upsert
  • re_enqueue_all_refreshes_budgets — anti-entropy
  • retain_removes_non_matching — predicate-based eviction
  • serialize_deserialize_roundtrip — wire format

Unit tests (shared types)

In crates/shared-types/src/pool.rs:

  • pool_id_from_name_is_deterministic / pool_id_different_names_differ
  • pool_id_hex_roundtrip
  • pool_entry_serde_roundtrip
  • higher_generation_wins_for_membership
  • content_location_tombstone_semantics

Integration tests (pool protocol)

In crates/datastore/tests/pool_tests.rs:

  • two_nodes_converge_on_membership — two-node gossip convergence
  • content_location_propagates_via_gossip — cross-node content discovery
  • leave_propagates_via_gossip — membership leave propagation
  • content_deletion_propagates — tombstone propagation
  • acl_grant_propagates — ACL gossip
  • capacity_propagates_and_summarizes — capacity gossip + aggregation
  • placement_query_picks_node_with_most_space — capacity-aware placement
  • five_node_pool_converges — 5-node full convergence
  • shared_pool_channel_topic_tag — GossipChannel interface
  • gossip_channel_bytes_roundtrip — wire format through GossipChannel

Sim-cluster scenarios (multi-process)

In xtask/src/sim_cluster.rs:

  • Cluster convergence (5 nodes)
  • Node death detection (kill + observe)
  • Killed node rejoins (kill + restart + observe)
  • Actors resolvable (directory entry propagation)

13. Known Gaps & Future Work

  • Remote content fetch: PoolGet currently only checks the local datastore. It should use locate_content() to fetch from the node that actually has the content.
  • Capacity-aware placement: PoolPut delegates to the local datastore. It should use node_with_most_free_space() to route writes to the best node.
  • Actual usage tracking: PoolTick re-announces capacity with used: 0. It should query the BlobStore for actual disk usage.
  • GossipChannel integration: The GossipChannel trait and SharedPoolChannel are built but not yet wired into DistributedNode's piggyback system. The existing hardcoded dissemination channels need to be migrated to the new trait.
  • Dashboard SSE integration: The pool dashboard HTML is built, but the server-side SSE event source for pool events needs to be wired to the PoolDisseminator::snapshot_json() method.
  • Sim-cluster namespace isolation: The harness uses high ephemeral ports for isolation. Full Linux network namespace isolation (as designed in TEST_ISOLATION.md) is a future enhancement.
  • Sim-cluster in CI: The sim-cluster test group is opt-in and excluded from essential/all. Once proven stable, it should be added to CI.