swactor-development-history/datastore/SUMMARY.md

# swactor-datastore: Development History & Status

## Overview

`swactor-datastore` is a distributed personal file/blob storage protocol for small trusted clusters (laptop, phone, browser). It provides content-hash-first addressing with immutable content-addressed objects, replicated via epidemic metadata dissemination across peers.

The implementation is organized as a single Rust crate (`crates/datastore/`) built on the `swactor` actor runtime. It was developed in 7 ordered modules (local single-node operations) followed by a multi-node simulation phase.

**Current state: 93 tests across 9 test files, all passing. Zero warnings.**

---

## Architecture

```
┌─────────────────────────────────────────────────────┐
│                   DatastoreNode                      │  Coordinator/facade
│            (single entry point for callers)           │
├────────────────────┬────────────────────────────────┤
│   MetadataActor    │         BlobStoreActor          │  Long-lived, one per node
│  (object index,    │     (chunk & manifest I/O       │
│   dissemination,   │      via StorageBackend)        │
│   GC orchestration)│                                 │
├────────────────────┴────────────────────────────────┤
│              TransferActor (ephemeral)                │  One per download
│         (chunk tracking, retry, self-termination)     │
├─────────────────────────────────────────────────────┤
│               StorageBackend trait                    │  Pluggable I/O
│    FilesystemBackend  │  InMemoryBackend             │
├─────────────────────────────────────────────────────┤
│            Chunking Engine (pure functions)           │  No I/O, deterministic
│     chunk_blob · reassemble_blob · verify_integrity  │
└─────────────────────────────────────────────────────┘
```

### Core Design Principles

- **Content-hash-first addressing** -- every object identified by `blake3(blob_bytes)`.
- **Immutable content-addressed objects** -- no write conflicts by construction.
- **Names are metadata** -- optional flat strings, not keys.
- **Separation of data and metadata** -- chunks are large opaque blobs; metadata is small and gossiped.
- **Actor-based** -- three actor types coordinate via message passing.
- **Transport-agnostic** -- protocol messages defined as `NetworkMessage` types.
- **Pluggable storage** -- `StorageBackend` trait abstracts I/O.

---

## Module-by-Module Development History

### Module 1: Chunking Engine

**What:** Pure functions for content-addressed blob chunking and reassembly. `chunk_blob()`, `reassemble_blob()`, `verify_integrity()`, plus `ContentHash`, `ObjectManifest`, `ChunkRef` types.

**Key decisions:**
- Fixed-size chunking over content-defined chunking (simpler, deterministic; CDC dedup unnecessary for small clusters)
- BLAKE3 for all hashing (3-7 GB/s, 32-byte output matching `NodeId`)
- Whole-blob hash as content hash rather than Merkle root of chunk hashes
- Empty blob produces a valid zero-chunk manifest

**Tests:** 13 (10 scenario + 3 proptest). Round-trips, edge cases, integrity verification, determinism.

**Files:** `src/chunking.rs`, `src/types.rs`

---

### Module 2: Storage Backend

**What:** `StorageBackend` trait with two implementations: `FilesystemBackend` (2-level hex-sharded dirs, fsync-on-write) and `InMemoryBackend` (HashMap-based for tests/WASM).

**Key decisions:**
- 2-level hex sharding (65,536 possible directories) to avoid hot directories
- In-memory chunk index for O(1) `has_chunk` lookups, populated via scan-on-init
- JSON manifest serialization for debuggability
- `Send` but not `Sync` on the trait (single-actor ownership)
- Idempotent writes and deletes

**Tests:** 12 (9 parameterized across both backends + 2 FS-only + 1 proptest).

**Files:** `src/storage/mod.rs`, `src/storage/in_memory.rs`

---

### Module 3: BlobStoreActor

**What:** Message-driven actor wrapping `Box<dyn StorageBackend>` for chunk/manifest CRUD plus garbage collection. Also introduced the shared test harness (`tests/common/mod.rs`).

**Key decisions:**
- Thin delegation -- actor adds no logic beyond message dispatch
- Explicit `reply_to` pattern (tell, not ask) for response routing
- Fire-and-forget deletes and GC (no reply needed)
- `Box<dyn StorageBackend>` for dynamic dispatch (one backend per instance)
- Single-threaded tick-based testing for determinism

**Tests:** 7 scenario tests through the swactor runtime.

**Established patterns:** `reply_to` pattern, shared test harness with `test_runtime()`, `tick_n()`, `tick_until_recv()`, `DatastoreHarness`.

**Files:** `src/actors/blob_store.rs`, `tests/blob_store_tests.rs`, `tests/common/mod.rs`

---

### Module 4: MetadataActor

**What:** Object metadata index (`HashMap<ContentHash, ObjectEntry>`), manifest cache, SWIM-inspired gossip dissemination queue. Handles local CRUD, DHT protocol messages (`HandleFindObject`, `HandleStoreObject`), and GC tick orchestration.

**Key decisions:**
- Node ID stamping on `PutObject` (prevents spoofing; remote entries retain original owner)
- Idempotent DHT store (insert-if-absent semantics)
- Synthetic empty manifest for `HandleFindObject` when manifest is missing
- Separate `GetObject` (local, error on missing manifest) vs `HandleFindObject` (DHT, synthesizes empty manifest)
- SWIM-style dissemination with budget `Lambda * ceil(log2(n))`, Lambda=3

**Tests:** 10 scenario tests covering CRUD, DHT operations, idempotency, lifecycle.

**Files:** `src/actors/metadata.rs`, `tests/metadata_tests.rs`

---

### Module 5: TransferActor

**What:** Ephemeral per-download actor. Tracks pending/received chunks, forwards received chunks to BlobStoreActor, implements per-chunk retry, self-terminates on completion/failure/cancel.

**Key decisions:**
- Ephemeral actor pattern (one per download, isolates transfer state)
- Passive design -- chunks driven externally via `ChunkReceived`/`ChunkFailed` (decoupled from networking)
- Whole-transfer failure on any chunk exhausting retries
- `max_retries` defaults to 1 (first failure allows retry, second aborts)
- Fire-and-forget chunk persistence (ChunkStored reply silently dropped)

**Tests:** 10 scenario tests covering state machine transitions, retry logic, cancellation, data recovery.

**Files:** `src/actors/transfer.rs`, `tests/transfer_tests.rs`

---

### Module 6: DatastoreNode Coordinator

**What:** Facade actor encapsulating BlobStoreActor + MetadataActor behind a single address. Routes 6 user-facing commands and 5 network protocol variants.

**Key decisions:**
- Pass-through `reply_to` pattern (responses go directly to caller, coordinator never intercepts)
- Inline chunking in `handle_put` (synchronous, no async coordination)
- Fire-and-forget chunk writes (same pattern as TransferActor)
- Immutable state after construction
- `Put` uses `data: Vec<u8>` not `PathBuf` (testable, WASM-compatible)

**Tests:** 12 scenario tests through `NodeHarness`.

**Files:** `src/actors/datastore_node.rs`, `tests/datastore_node_tests.rs`

---

### Module 7: Garbage Collection

**What:** Completed `MetadataActor::gc_tick()` to build a referenced chunk set from all manifests and send `GcUnreferenced` to BlobStoreActor for orphan cleanup.

**Key decisions:**
- `blob_store_addr = None` guard for backward compatibility (GC no-ops when not wired)
- Fire-and-forget `GcUnreferenced` (no reply needed)
- `spawn_metadata_with_config()` helper to wire blob_store_addr before spawning
- Mark-and-sweep: union of all chunk hashes from all manifests = referenced set

**Tests:** 6 scenario tests covering cleanup, preservation, deduplication safety, interval gating, empty-store edge case.

**Files:** `src/actors/metadata.rs` (delta), `tests/gc_tests.rs`, `tests/common/mod.rs` (GcHarness)

---

### Multi-Node Simulation (Phases 3-4)

**What:** Wired up MetadataActor for peer-to-peer metadata dissemination. Added `SetPeers` and `DisseminateTick` messages. Created `MultiNodeHarness` for simulating clusters on a single Runtime. Extended `HandleStoreObject` to carry manifests alongside entries for full metadata replication.

**Key decisions:**
- Single Runtime for simulation -- all nodes' actors share one Runtime; actor addresses are globally unique so cross-node messaging "just works" via `ctx.send()`
- MetadataActor owns peer relationships (simpler than routing through DatastoreNode)
- Manifest dissemination alongside entry dissemination (peers receive both)
- TransferActor stays passive -- tests feed chunks from remote BlobStoreActor (test harness plays the "network adapter" role)
- No automatic remote GET orchestration yet -- DatastoreNode remains a stateless router

**New messages:**
- `MetadataMsg::SetPeers { peers: Vec<ActorAddress> }`
- `MetadataMsg::DisseminateTick`
- `MetadataMsg::HandleStoreObject` extended with `manifest: Option<ObjectManifest>`

**Tests:** 10 new scenario tests in `tests/multi_node_tests.rs`:

| # | Test | Verifies |
|---|------|----------|
| 1 | `metadata_replicates_to_peer_after_dissemination` | Put on 0, disseminate, node 1 finds it |
| 2 | `metadata_replicates_to_all_peers_in_3_node_cluster` | Full cluster replication |
| 3 | `dissemination_budget_expires_after_enough_rounds` | Budget exhaustion, fresh entries still work |
| 4 | `delete_on_origin_does_not_propagate_to_peers` | Delete is local only |
| 5 | `duplicate_put_via_dissemination_is_idempotent` | No duplicate entries on peer |
| 6 | `find_object_on_peer_after_dissemination` | HandleFindObject succeeds on peer |
| 7 | `chunk_transfer_from_remote_blob_store` | TransferActor pulls chunks cross-node |
| 8 | `full_remote_get_scenario` | End-to-end: put on 0, disseminate, transfer 0->1, reassemble matches |
| 9 | `list_across_all_nodes_finds_objects_from_any_node` | Simulated ListSwarm fan-out |
| 10 | `gc_on_one_node_does_not_affect_other_nodes` | GC isolation between nodes |

**Files:** `src/messages.rs`, `src/actors/metadata.rs`, `tests/common/mod.rs` (MultiNodeHarness), `tests/multi_node_tests.rs`

---

## Code Quality Pass

Alongside the multi-node work, a cleanup pass was performed:

- **PROTOCOL.md** -- Added honest `> Status:` annotations to sections 5 (Metadata DHT), 6.3 (Remote GET), 6.6 (ListSwarm), 7 (Actor Architecture), 8 (Wire Protocol), 10 (GC), 12 (CLI), and 13 (Browser API)
- **metadata.rs** -- Updated stale `ListSwarm` "MVP" comment
- **transfer.rs** -- Updated architecture comments describing simulation-ready passive design
- **tests/common/mod.rs** -- Removed 3 unused imports (`ActorInterface`, `Ctx`, `DatastoreNodeMsg`)

---

## Test Summary

| Test File | Count | What |
|-----------|-------|------|
| `chunking_tests.rs` | 13 | Pure function round-trips, edge cases, proptests |
| `storage_tests.rs` | 12 | Backend CRUD, parameterized across FS + InMemory, proptests |
| `blob_store_tests.rs` | 7 | Actor-level chunk/manifest CRUD, GC |
| `metadata_tests.rs` | 10 | Object index, DHT protocol, lifecycle |
| `transfer_tests.rs` | 10 | Download state machine, retry, cancel |
| `datastore_node_tests.rs` | 12 | Coordinator routing, network protocol |
| `datastore_tests.rs` | 13 | Content-addressing properties, proptests |
| `gc_tests.rs` | 6 | Mark-and-sweep GC, dedup safety |
| `multi_node_tests.rs` | 10 | Dissemination, cross-node transfer, GC isolation |
| **Total** | **93** | |

**Testing philosophy:** Scenario/story tests first, property-based tests for invariants, contract tests for serialization. No white-box/structural tests. Low coupling to internals -- tests should survive a refactor.

---

## Future Work

### Near-Term (Next Sessions)

**CLI Implementation (Phase 2)**
- Parser and dispatcher using `clap`
- `[[bin]]` target in Cargo.toml
- `ContentHash::from_hex()` for CLI input
- Commands: `put <path>`, `fetch <hash>` (metadata only), `get <hash> --output <path>` (full download), `delete <hash>`, `list`, `status`
- Single-node only (no networking); spawns its own actor set
- Follow `crates/node/src/main.rs` pattern

**ListSwarm Fan-Out**
- Currently delegates to `ListLocal`. Wire MetadataActor to query all peers and merge/deduplicate results by content hash.

**Automatic Remote GET Orchestration**
- Currently, remote GET requires manual orchestration (test harness or external driver reads chunks from remote BlobStore and feeds them to TransferActor).
- DatastoreNode needs to become stateful: detect local miss, query peers via `HandleFindObject`, spawn TransferActor, coordinate chunk pulling from the remote BlobStoreActor.
- This is the largest remaining architectural change for local functionality.

### Medium-Term

**Transport Integration (iroh/QUIC)**
- Wire `NetworkMessage` types to actual network transport.
- DatastoreNode gains peer management (`AddPeer`/`RemovePeer`) at the node level.
- Replace simulation-only direct actor addressing with network-routed messages.
- Framing: `[4-byte length (big-endian)][JSON payload]` over QUIC streams.

**Anti-Entropy / Repair**
- When a node comes back online, re-disseminate its metadata to peers.
- Periodic full-index comparison between peers to detect and repair drift.

**Active Chunk Pulling in TransferActor**
- `StartDownload` sends `GetChunkRequest` to the source node for each chunk.
- Currently passive (chunks fed externally); make it drive its own downloads.

**Parallel Chunk Fetching**
- TransferActor currently fetches sequentially. Add configurable concurrency (`max_concurrent_transfers` in config already exists).

### Longer-Term

**Browser API (WASM)**
- Expose `list_objects`, `get_object`, `put_object`, `delete_object`, `get_node_status` via WASM bindings.
- Use `InMemoryBackend` (or IndexedDB backend) in the browser.
- Coordinate with `crates/wasm/` for the in-browser swactor runtime.

**Binary Wire Format**
- Replace JSON serialization with bincode or msgpack for `GetChunkResponse` and other payload-heavy messages.

**Content-Defined Chunking (CDC)**
- Replace fixed-size chunking with FastCDC or similar for better cross-object deduplication.
- Transparent to the rest of the system -- only `chunk_blob()` changes; manifest format is the same.

**Streaming / Large File Support**
- Current `Put` takes `data: Vec<u8>` (entire blob in memory). For large files, add streaming chunking that reads from a `Read` source.

**Delete Propagation**
- Currently, delete is local only (by design). Add optional "tombstone dissemination" to remove entries from peers.

**Replication Factor Control**
- Currently, dissemination is epidemic (all peers get everything). Add configurable k-closest replication for the metadata DHT.

**IndexedDB Backend**
- Implement `StorageBackend` for browser IndexedDB for persistent storage in web contexts.

---

## Key Files

| File | Purpose |
|------|---------|
| `src/types.rs` | Core types: `ContentHash`, `ObjectEntry`, `ObjectManifest`, `ChunkRef`, `DatastoreConfig` |
| `src/messages.rs` | All inter-node and intra-node message types |
| `src/chunking.rs` | Pure chunking/reassembly functions |
| `src/storage/mod.rs` | `StorageBackend` trait + `FilesystemBackend` |
| `src/storage/in_memory.rs` | `InMemoryBackend` |
| `src/actors/blob_store.rs` | Chunk/manifest I/O actor |
| `src/actors/metadata.rs` | Object index, dissemination, GC orchestration |
| `src/actors/transfer.rs` | Ephemeral download actor |
| `src/actors/datastore_node.rs` | Coordinator/facade |
| `src/cli.rs` | CLI command type definitions (types only, no implementation) |
| `PROTOCOL.md` | Protocol specification with status annotations |
| `PROTOCOL_IMPLEMENTATION_PLAN.md` | Original 7-module implementation plan |
| `tests/common/mod.rs` | Shared test harness: DatastoreHarness, GcHarness, MultiNodeHarness, NodeHarness |