From f3bd9685bab5095e716c20a486d6a141cd05a99a Mon Sep 17 00:00:00 2001 From: Zachery Aaron Shores-Chmielewski Date: Sun, 15 Feb 2026 20:43:10 +0700 Subject: [PATCH] fix: cli for datastore works --- crates/datastore/PROTOCOL.md | 471 ----------------------------------- 1 file changed, 471 deletions(-) delete mode 100644 crates/datastore/PROTOCOL.md diff --git a/crates/datastore/PROTOCOL.md b/crates/datastore/PROTOCOL.md deleted file mode 100644 index 85dc919..0000000 --- a/crates/datastore/PROTOCOL.md +++ /dev/null @@ -1,471 +0,0 @@ -# Swactor Datastore Protocol Specification - -**Version:** 0.2.0 (MVP) -**Status:** Draft - -## 1. Overview - -The Swactor Datastore is a distributed personal file/blob storage protocol for small trusted clusters (laptop, phone, browser). It provides content-hash-first addressing with immutable content-addressed objects, replicated via a Kademlia-based metadata DHT. - -### Design Principles - -- **Content-hash-first addressing** — every object is identified by `blake3(blob_bytes)`. This is the primary key for all operations. -- **Immutable content-addressed objects** — content hashes are unique identifiers. There are no write conflicts by construction. -- **Names are metadata** — optional flat strings attached to objects, not keys. Multiple objects can share a name; distinguished by content hash. -- **Separation of data and metadata** — chunks are large opaque blobs; metadata is small, gossiped, and queryable. -- **Crash-safe** — fsync before acknowledge on all writes. -- **Actor-based** — three actor types coordinate via message passing within the swactor runtime. -- **Transport-agnostic** — protocol messages defined as `NetworkMessage` types; MVP uses iroh (QUIC + NAT hole-punch + encryption). -- **Pluggable storage** — `StorageBackend` trait abstracts I/O for filesystem (MVP), IndexedDB (browser), etc. - -## 2. Terminology - -| Term | Definition | -|------|-----------| -| **Object** | Content-addressed blob identified by `blake3(blob_bytes)`. May carry an optional human-readable name as metadata. | -| **Blob** | The raw byte content of an object. | -| **Chunk** | A fixed-size (1 MB default) slice of a blob, identified by its blake3 content hash. | -| **Manifest** | An ordered list of `ChunkRef`s describing how to reassemble an object from chunks. Stored under the object's content hash. | -| **ContentHash** | 32-byte blake3 digest. Primary identifier for blobs and DHT key. | -| **ObjectEntry** | Metadata record: content hash, optional name, owner node, tags. | -| **DHT overlay** | A Kademlia distributed hash table for object metadata, separate from the actor directory DHT. Keys are `blake3(blob_bytes)`. | -| **StorageBackend** | Trait abstracting chunk and manifest I/O for pluggable backends (filesystem, IndexedDB, etc.). | -| **Node** | A device running the swactor runtime with a datastore actor set (BlobStoreActor + MetadataActor). | - -## 3. Data Model - -### 3.1 ContentHash - -``` -ContentHash = blake3(data)[0..32] // 32 bytes -``` - -- **Hashing algorithm:** blake3 — 2-3x faster than sha256, tree-hashable (parallel hashing of large chunks), same 32-byte output. Supports streaming hashing for large blobs via `blake3::Hasher`. -- **Display:** first 8 bytes as hex + ellipsis (e.g. `a1b2c3d4e5f6a7b8…`). -- **XOR distance:** bitwise XOR of the 32-byte arrays, used for Kademlia routing in the metadata DHT. - -### 3.2 ObjectEntry - -``` -ObjectEntry { - content_hash: ContentHash, // blake3(entire_blob) — primary identifier - name: Option, // Optional flat string, not a path - node_id: NodeId, // Node that stores the object - tags: BTreeMap, // User-defined key-value tags - size_bytes: u64, // Total object size - created_at: u64, // Wall-clock creation time (informational) -} -``` - -No conflict resolution is needed — content hashes are unique identifiers. Storing the same blob twice is a no-op (same content hash). Different blobs always have different content hashes. - -### 3.3 ObjectManifest - -``` -ObjectManifest { - content_hash: ContentHash, // blake3(entire_blob) — NOT the hash of this manifest - chunks: Vec, // Ordered list of chunks - total_size: u64, // Total object size in bytes - chunk_size: u32, // Fixed chunk size used (e.g. 1MB) - content_type: Option, // MIME type -} - -ChunkRef { - hash: ContentHash, // Content hash of chunk data - offset: u64, // Byte offset in original object - size: u32, // Actual size (last chunk may be smaller) -} -``` - -The `content_hash` field is `blake3(entire_blob)`, computed via a streaming hasher alongside chunking. The manifest is stored and looked up using this content hash as the key. - -### 3.4 Storage Backend - -The `StorageBackend` trait abstracts all chunk and manifest I/O: - -```rust -pub trait StorageBackend: Send { - fn write_chunk(&mut self, hash: &ContentHash, data: &[u8]) -> Result<(), io::Error>; - fn read_chunk(&self, hash: &ContentHash) -> Result>, io::Error>; - fn delete_chunk(&mut self, hash: &ContentHash) -> Result<(), io::Error>; - fn has_chunk(&self, hash: &ContentHash) -> bool; - fn list_chunks(&self) -> Vec; - fn write_manifest(&mut self, manifest: &ObjectManifest) -> Result<(), io::Error>; - fn read_manifest(&self, content_hash: &ContentHash) -> Result, io::Error>; - fn delete_manifest(&mut self, content_hash: &ContentHash) -> Result<(), io::Error>; -} -``` - -#### MVP: FilesystemBackend - -Two-level directory sharding to avoid huge directories: - -``` -{storage_path}/ -├── chunks/ -│ └── {hex[0..2]}/ -│ └── {hex[2..4]}/ -│ └── {full_hex_hash} # Raw chunk bytes -└── manifests/ - └── {hex[0..2]}/ - └── {hex[2..4]}/ - └── {full_hex_hash} # JSON-serialized ObjectManifest -``` - -Example: chunk with hash `abcdef12...` is stored at `chunks/ab/cd/abcdef12...`. - -All writes are fsynced before acknowledging. - -## 4. Content Addressing - -### 4.1 Chunking Algorithm - -Fixed-size chunking (MVP): - -1. Read the input file in `chunk_size` byte blocks (default: 1,048,576 = 1 MB). -2. For each block, compute `ContentHash::of(block)`. -3. Store each chunk via the `StorageBackend`. -4. Build a `Vec` with sequential offsets. -5. Compute `content_hash = blake3(entire_blob)` using a streaming hasher fed alongside chunking. -6. Create the `ObjectManifest` with this `content_hash` and store it via the `StorageBackend` keyed by `content_hash`. - -The last chunk may be smaller than `chunk_size`. - -### 4.2 Reassembly - -1. Read the `ObjectManifest` (by its `content_hash`). -2. For each `ChunkRef` in order, read the chunk by `hash`. -3. Concatenate all chunk data to reconstruct the original blob. -4. Verify: `blake3(reassembled) == content_hash` (optional integrity check). - -## 5. Metadata DHT - -> **Status:** Types and routing table logic exist in the `distribution` crate. `MetadataActor` has a dissemination queue (`enqueue`/`take_pending`) and peer-to-peer replication via `SetPeers` + `DisseminateTick`. Verified in local multi-node simulation. Full Kademlia iterative lookup (FIND_VALUE with α-parallel queries) is not yet implemented — dissemination is epidemic/gossip-style. - -### 5.1 Overlay Design - -The metadata DHT is a **separate Kademlia overlay** from the actor directory. It stores `ObjectEntry` records keyed by `blake3(blob_bytes)` — the content hash of the entire blob. - -This separation ensures: -- Object metadata routing doesn't interfere with actor discovery. -- Different replication factors can be used (objects may be stored on fewer nodes). -- The DHT can be independently tuned for the metadata workload. - -### 5.2 Key Mapping - -``` -DHT key = blake3(blob_bytes) = entry.content_hash -``` - -### 5.3 Store Flow - -When storing object metadata: -1. Use `key = entry.content_hash`. -2. Find the `k` closest nodes to `key` in the metadata DHT routing table. -3. Send `StoreObjectRequest { entry }` to each of the `k` closest nodes. - -### 5.4 Lookup Flow - -When looking up object metadata: -1. Send `FindObjectRequest { content_hash }` to the `α` closest known nodes. -2. Each node responds with either `Found(ObjectEntry)` or `Closer(Vec<(NodeId, SocketAddr)>)`. -3. Continue querying closer nodes until convergence. - -No merge step is needed — content hashes are unique identifiers. - -## 6. Protocol Flows - -### 6.1 PUT — Store an Object - -``` -User MetadataActor BlobStoreActor - │ │ │ - │─── PutObject ────────────>│ │ - │ │ │ - │ │ (chunk the file, │ - │ │ stream blake3 hash) │ - │ │ │ - │ │─── WriteChunk ────────>│ - │ │<── ChunkStored ────────│ (repeat for each chunk) - │ │ │ - │ │─── WriteManifest ─────>│ - │ │<── ManifestStored ─────│ - │ │ │ - │ │ (create ObjectEntry, │ - │ │ store in local index,│ - │ │ enqueue for DHT │ - │ │ dissemination) │ - │ │ │ - │<── PutOk {content_hash} ─│ │ -``` - -### 6.2 GET — Retrieve an Object (Local) - -``` -User MetadataActor BlobStoreActor - │ │ │ - │─── GetObject ────────────>│ │ - │ {content_hash} │ │ - │ │ (lookup content_hash │ - │ │ in local index) │ - │ │ │ - │<── GetOk { entry, │ │ - │ manifest } ──────│ │ - │ │ - │ (for each chunk in manifest) │ - │───────────── ReadChunk ───────────────────────────>│ - │<────────────── ChunkOk ───────────────────────────│ - │ │ - │ (reassemble chunks into original file) │ -``` - -### 6.3 GET — Retrieve an Object (Remote) - -> **Status:** `TransferActor` state machine is functional and stores received chunks to the local `BlobStoreActor`. Chunks must be fed externally (via `ChunkReceived` messages). Automatic chunk pulling from remote nodes is not yet implemented — the test harness or a future network adapter plays the "pull" role. Verified in multi-node simulation. - -``` -User MetadataActor TransferActor Remote BlobStore - │ │ │ │ - │─ GetObject ──>│ │ │ - │ {content_hash}│ │ │ - │ │ (not in local │ │ - │ │ index; DHT │ │ - │ │ lookup) │ │ - │ │ │ │ - │ │─ StartDownload ───>│ │ - │ │ │ │ - │ │ │── GetChunkRequest ─>│ - │ │ │<─ GetChunkResponse ─│ - │ │ │ │ - │ │ │ (repeat for each │ - │ │ │ chunk) │ - │ │ │ │ - │ │<─ TransferComplete │ │ - │ │ │ │ - │<── GetOk ────│ │ (stops self) │ -``` - -### 6.4 DELETE — Remove an Object - -``` -User MetadataActor - │ │ - │─── DeleteObject ─────────>│ - │ {content_hash} │ - │ │ - │ │ (remove from local ) - │ │ (index, best-effort ) - │ │ (notify DHT peers ) - │ │ - │<── DeleteOk │ - │ {content_hash} ────────│ -``` - -Chunk data is **not** immediately deleted. Unreferenced chunks are cleaned up during GC sweeps (see Section 10). - -### 6.5 LIST — List Objects (Local) - -``` -User MetadataActor - │ │ - │─── ListLocal ────────────>│ - │ {name_filter} │ - │ │ - │ │ (filter local index ) - │ │ (by name substring ) - │ │ - │<── ListOk { entries } ───│ -``` - -### 6.6 LIST — List Objects (Swarm-Wide) - -> **Status:** `ListSwarm` currently delegates to `ListLocal` (returns local entries only). Fan-out to peer MetadataActors is not yet wired. Swarm-wide listing is verified in simulation by querying each node and merging results in the test harness. - -``` -User MetadataActor Remote MetadataActors - │ │ │ - │─ ListSwarm ──>│ │ - │ {name_filter} │ │ - │ │── ListObjectsRequest ─>│ (fan-out to all known - │ │<─ ListObjectsResponse ─│ alive nodes) - │ │ │ - │ │ (merge all results, │ - │ │ deduplicate by │ - │ │ content hash) │ - │ │ │ - │<── ListOk ───│ │ -``` - -## 7. Actor Architecture - -> **Status:** All three actor types are fully implemented and tested. `DatastoreNode` coordinator routes commands to internal actors. 83+ tests across 8 test files verify single-node operations. Multi-node dissemination and cross-node transfers verified in simulation. - -### 7.1 BlobStoreActor - -**Responsibility:** Chunk and manifest I/O via `StorageBackend` trait. - -- **State:** `Box` -- **Lifecycle:** Long-lived, one per node. -- **Guarantees:** Delegates to backend; filesystem backend fsyncs before acknowledging. - -**Message types:** `BlobStoreMsg` (see `messages.rs`) - -### 7.2 MetadataActor - -**Responsibility:** Object metadata index, DHT routing. - -- **State:** Local object index (`HashMap`), manifest cache, dissemination queue. -- **Lifecycle:** Long-lived, one per node. -- **Coordinates with:** BlobStoreActor (for manifest storage), remote MetadataActors (DHT operations). - -**Message types:** `MetadataMsg` (see `messages.rs`) - -### 7.3 TransferActor - -**Responsibility:** Downloading an object (all its chunks) from a remote node. - -- **State:** Manifest, pending/received chunk sets, retry counts. -- **Lifecycle:** Ephemeral — spawned per download, self-terminates on completion/failure/cancel. -- **Coordinates with:** Remote BlobStoreActor (chunk requests), local BlobStoreActor (chunk storage). - -**Message types:** `TransferMsg` (see `messages.rs`) - -## 8. Wire Protocol - -> **Status:** All message types are defined with `NetworkMessage` implementations and stable type tags. Serialization is JSON (serde). No transport integration yet — messages are passed directly via actor addresses in simulation. - -### 8.1 Message Types - -All inter-node messages implement `NetworkMessage` with a stable `type_tag()`: - -| Message | type_tag | Direction | -|---------|----------|-----------| -| `GetChunkRequest` | `swactor_datastore::GetChunkRequest` | requester → holder | -| `GetChunkResponse` | `swactor_datastore::GetChunkResponse` | holder → requester | -| `StoreObjectRequest` | `swactor_datastore::StoreObjectRequest` | writer → DHT nodes | -| `FindObjectRequest` | `swactor_datastore::FindObjectRequest` | reader → DHT nodes | -| `FindObjectResponse` | `swactor_datastore::FindObjectResponse` | DHT node → reader | -| `GetManifestRequest` | `swactor_datastore::GetManifestRequest` | requester → holder | -| `GetManifestResponse` | `swactor_datastore::GetManifestResponse` | holder → requester | -| `ListObjectsRequest` | `swactor_datastore::ListObjectsRequest` | requester → remote node | -| `ListObjectsResponse` | `swactor_datastore::ListObjectsResponse` | remote node → requester | - -`FindObjectRequest` contains a `content_hash` field (the `blake3(blob_bytes)` key). - -### 8.2 Serialization - -MVP: serde JSON for all messages. Binary format (bincode or msgpack) planned for later to reduce overhead, especially for `GetChunkResponse` which carries large payloads. - -### 8.3 Framing - -Messages are framed over iroh QUIC streams: -- Each request/response pair uses a single bidirectional stream. -- Message format: `[4-byte length (big-endian)][JSON payload]`. - -## 9. Naming - -Names are **optional flat strings** — human-readable labels attached to objects as metadata. - -- Names are not keys. The content hash is the only primary identifier. -- Multiple objects can share the same name. They are distinguished by content hash. -- Names are simple strings (e.g. `"vacation.jpg"`, `"backup-2024-01"`). No path hierarchy, no separators enforced. -- No conflict resolution is needed — different content always produces different content hashes. - -## 10. Garbage Collection - -> **Status:** Fully implemented. `MetadataActor::gc_tick()` builds a referenced chunk set from all local manifests and sends `GcUnreferenced` to `BlobStoreActor`. Verified with 6 GC-specific tests including deduplication safety, interval gating, and empty-store edge case. - -### 10.1 Entry Removal - -Deleting an object: -1. Remove the `ObjectEntry` from the local index. -2. Best-effort notify DHT peers to remove their replicas. -3. Remove the local manifest. - -### 10.2 Chunk Reference Counting - -Unreferenced chunk cleanup: - -1. Build a referenced set: union of all chunk hashes from all local manifest entries. -2. Send `BlobStoreMsg::GcUnreferenced { referenced }` to the BlobStoreActor. -3. BlobStoreActor diffs its chunk list against the referenced set and deletes unreferenced chunks. - -**Safety:** A chunk may be referenced by multiple objects (deduplication). Only delete when zero references remain. - -### 10.3 GC Schedule - -- `MetadataActor` runs `gc_tick()` every tick. Actual GC sweep happens every `gc_interval` ticks (default: 1000). -- Chunk GC is triggered less frequently (order of minutes) to avoid overhead. - -## 11. Failure Modes - -### 11.1 Node Offline - -- **Metadata persists** in the DHT (replicated to k-closest nodes). Lookups succeed as long as any replica is alive. -- **Chunk fetches fail** if the only copy is on the offline node. The TransferActor retries once, then reports failure. -- **Recovery:** When the node comes back, its metadata is re-disseminated (anti-entropy). - -### 11.2 Transfer Interrupted - -- **Partial state:** Some chunks may be written to the local BlobStoreActor before the transfer fails. -- **Cleanup:** Partially downloaded chunks are not harmful — they're content-addressed and may be useful for future downloads. Unreferenced chunks are cleaned up by GC. -- **Retry:** The user can retry the GET, and only missing chunks need to be fetched (future optimization). - -### 11.3 DHT Inconsistency - -- **Stale metadata:** A node may serve an outdated ObjectEntry. Anti-entropy dissemination ensures replicas converge. -- Content addressing eliminates write conflicts — storing the same content hash twice is idempotent. - -### 11.4 Disk Full - -- `StorageBackend::write_chunk` fails with an I/O error, which is propagated back to the requester as `DatastoreResponse::Error`. -- No partial writes — fsync ensures atomicity (filesystem backend). - -## 12. CLI Interface - -> **Status:** Command types defined in `src/cli.rs`. Parser, dispatcher, and `[[bin]]` target not yet implemented. Planned for a follow-up session. - -``` -swactor-store put [--name