Compare commits
3 commits
52bbc661a2
...
1c8d6ab912
| Author | SHA1 | Date | |
|---|---|---|---|
| 1c8d6ab912 | |||
| 2027e61d84 | |||
| 564c762be9 |
2 changed files with 0 additions and 1151 deletions
|
|
@ -1,403 +0,0 @@
|
|||
# Data-plane SPSC stream MVP
|
||||
|
||||
## Status
|
||||
|
||||
Pending implementation.
|
||||
|
||||
This document is the authoritative plan for the remaining data-plane stream slice. Blob namespace and resolution work is defined separately in `DATA_PLANE_VIRTUAL_BLOB_NAMESPACE_MVP.md`.
|
||||
|
||||
## Goal
|
||||
|
||||
Complete the existing public stream API for one producer and one consumer:
|
||||
|
||||
```python
|
||||
reader = await ctx.data.read_stream(path)
|
||||
|
||||
async with ctx.data.write_stream(path) as writer:
|
||||
await writer.write(data)
|
||||
```
|
||||
|
||||
Swactor must resolve the logical path, match exactly one source with exactly one sink, provision the local arena rings and remote transport, preserve byte order, apply lossless backpressure, propagate terminal state, and clean up every binding.
|
||||
|
||||
The final Myelin acceptance path replaces the temporary Unix-socket result bridge with this implementation.
|
||||
|
||||
## Scope
|
||||
|
||||
The MVP supports only:
|
||||
|
||||
- one producer;
|
||||
- one consumer;
|
||||
- ordered opaque bytes;
|
||||
- lossless delivery while both endpoints remain live;
|
||||
- explicit EOF and fault termination;
|
||||
- bounded buffering and backpressure;
|
||||
- actor-owned protocol state;
|
||||
- arena-backed SPSC byte rings;
|
||||
- local or Iroh-backed transfer chosen internally.
|
||||
|
||||
The MVP does not support:
|
||||
|
||||
- fanout or SPMC streams;
|
||||
- multiple writers;
|
||||
- topics or best-effort delivery;
|
||||
- replay or resume;
|
||||
- reconnecting an existing stream incarnation;
|
||||
- application records, tensor schemas, JSON, dtype, or shape semantics;
|
||||
- application-visible actors, nodes, edges, rings, sockets, or arena offsets.
|
||||
|
||||
## Public semantics
|
||||
|
||||
A stream path names one SPSC rendezvous.
|
||||
|
||||
```text
|
||||
DataPath
|
||||
-> waiting source or sink
|
||||
-> matched source/sink incarnation
|
||||
-> active transfer
|
||||
-> EOF or fault
|
||||
```
|
||||
|
||||
The stream is opaque ordered bytes. Application framing remains application code.
|
||||
|
||||
Each match creates one incarnation. A failed or closed incarnation is never silently resumed or joined to a replacement endpoint.
|
||||
|
||||
### Writer
|
||||
|
||||
```python
|
||||
async with ctx.data.write_stream(path) as writer:
|
||||
await writer.write(data)
|
||||
```
|
||||
|
||||
- Entering the context resolves and authorizes the path, matches the sink, provisions the transfer, and waits for readiness.
|
||||
- `write()` completes only after Swactor has accepted the complete input into its bounded ordered pipeline.
|
||||
- Backpressure suspends `write()` rather than dropping bytes.
|
||||
- A clean context exit publishes EOF exactly once.
|
||||
- Exceptional exit publishes a fault or abort terminal state.
|
||||
- Writes after terminal state fail typed.
|
||||
|
||||
### Reader
|
||||
|
||||
```python
|
||||
reader = await ctx.data.read_stream(path)
|
||||
chunk = await reader.read()
|
||||
```
|
||||
|
||||
- Opening resolves and authorizes the path, matches the source, provisions the transfer, and waits for readiness.
|
||||
- Reads observe source byte order.
|
||||
- EOF is distinguishable from a zero-length temporary read.
|
||||
- Source, transport, ring, and session faults become typed stream failures.
|
||||
- Cancellation releases any held ring view and permits cleanup.
|
||||
|
||||
The precise Python read method shape may follow existing binding conventions, but it must not return arena, ring, edge, actor, or transport identities.
|
||||
|
||||
## Namespace rendezvous
|
||||
|
||||
The centralized namespace service from `DATA_PLANE_VIRTUAL_BLOB_NAMESPACE_MVP.md` gains stream entries without changing blob semantics.
|
||||
|
||||
Conceptually:
|
||||
|
||||
```rust
|
||||
enum NamespaceEntry {
|
||||
Blob(BlobBinding),
|
||||
Stream(StreamBinding),
|
||||
}
|
||||
|
||||
enum StreamBinding {
|
||||
WaitingSink {
|
||||
sink: ActorAddress,
|
||||
},
|
||||
WaitingSource {
|
||||
source: ActorAddress,
|
||||
},
|
||||
Matched {
|
||||
incarnation: u64,
|
||||
source: ActorAddress,
|
||||
sink: ActorAddress,
|
||||
coordinator: ActorAddress,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
The directory actor serializes registration and matching. A path cannot simultaneously name a blob and a stream.
|
||||
|
||||
For the first Myelin flow, orchestration may register the result sink before the job opens its writer. The actor protocol should not require application-managed edge IDs or socket paths.
|
||||
|
||||
## Existing foundation to retain
|
||||
|
||||
`crates/data-plane/src/byte_ring.rs` already provides the correct stream storage foundation:
|
||||
|
||||
- arena-resident SPSC header and data region;
|
||||
- release/acquire `commit` and `consume` cursors;
|
||||
- generation and producer/consumer role enforcement;
|
||||
- `Data`, `Eof`, and `Fault` record framing;
|
||||
- lossless bounded backpressure;
|
||||
- ordering and wraparound handling;
|
||||
- corruption containment.
|
||||
|
||||
Retain its valid guarantee tests.
|
||||
|
||||
Also retain:
|
||||
|
||||
- one inherited arena descriptor;
|
||||
- automatically driven child runtime;
|
||||
- persistent host and child data-plane session actors;
|
||||
- one operation actor per public open;
|
||||
- one host binding/coordinator actor per match;
|
||||
- existing actor routing and Iroh transport;
|
||||
- the rule that protocol state lives in actor FSM fields rather than mutex-protected pending maps.
|
||||
|
||||
## Required byte-ring correction
|
||||
|
||||
The current consuming path returns owned bytes through `recv_record() -> Vec<u8>`. That is not zero-copy.
|
||||
|
||||
Add a pinned borrowed record view:
|
||||
|
||||
```rust
|
||||
struct PinnedRecordView {
|
||||
endpoint_guard: Arc<...>,
|
||||
generation: u64,
|
||||
kind: RecordKind,
|
||||
first_span: ...,
|
||||
second_span: Option<...>,
|
||||
}
|
||||
```
|
||||
|
||||
A record may wrap around the ring and therefore expose two ordered spans. The view pins the committed range. The consumer cursor advances only after the view is released or explicitly consumed.
|
||||
|
||||
Required behavior:
|
||||
|
||||
```text
|
||||
producer commits record
|
||||
-> consumer receives pinned view
|
||||
-> transport/Python consumes one or two spans
|
||||
-> view release publishes consume cursor
|
||||
-> producer capacity becomes available
|
||||
```
|
||||
|
||||
Holding a view intentionally applies backpressure.
|
||||
|
||||
No hot-path stream operation should assemble a record into `Vec<u8>` merely to cross the host/child boundary or feed the transport.
|
||||
|
||||
## Actor notification model
|
||||
|
||||
Actor messages replace eventfd wakeups and polling.
|
||||
|
||||
Messages indicate that protocol state may have advanced; ring cursors remain authoritative.
|
||||
|
||||
Conceptually:
|
||||
|
||||
```text
|
||||
producer commits data
|
||||
-> ProgressAvailable notification
|
||||
-> consumer checks committed cursor
|
||||
|
||||
consumer advances consume cursor
|
||||
-> CapacityAvailable notification
|
||||
-> producer retries pending reservation
|
||||
```
|
||||
|
||||
Notifications may be coalesced or redundant. Correctness must not depend on wake counts.
|
||||
|
||||
Each operation actor owns its pending action in FSM fields. Do not add a shared request-ID map or protocol mutex.
|
||||
|
||||
## Standard happy path
|
||||
|
||||
```text
|
||||
Sink registration
|
||||
-> namespace records waiting sink
|
||||
|
||||
Python write_stream(path)
|
||||
-> PyDataPlane delegates to general DataPlane
|
||||
-> ChildDataPlaneSessionActor spawns WriteStreamOperationActor
|
||||
-> host session authorizes canonical path
|
||||
-> namespace matches source operation with waiting sink
|
||||
-> StreamCoordinatorActor is created
|
||||
|
||||
Coordinator
|
||||
-> resolves persistent endpoint actors to nodes
|
||||
-> allocates and installs arena SPSC rings
|
||||
-> establishes local or Iroh transfer
|
||||
-> waits for source and sink readiness
|
||||
-> sends Opened to both operation paths
|
||||
|
||||
Python writer.write(bytes)
|
||||
-> reserves ring capacity
|
||||
-> writes bytes into producer spans
|
||||
-> release-commits Data record
|
||||
-> actor notification drives downstream progress
|
||||
-> transport consumes pinned source record
|
||||
-> destination producer commits to consumer ring
|
||||
-> reader consumes bytes in order
|
||||
-> consume progress releases backpressure
|
||||
|
||||
Clean writer exit
|
||||
-> commits EOF once
|
||||
-> reader observes all preceding bytes, then EOF
|
||||
-> coordinator quiesces pumps
|
||||
-> rings are uninstalled and leases released
|
||||
-> namespace incarnation terminates
|
||||
```
|
||||
|
||||
## Backpressure
|
||||
|
||||
Backpressure is end-to-end and lossless:
|
||||
|
||||
```text
|
||||
slow reader
|
||||
-> destination ring fills
|
||||
-> network receive stops advancing
|
||||
-> source transport stops draining
|
||||
-> source ring fills
|
||||
-> writer.write awaits capacity
|
||||
```
|
||||
|
||||
No layer may drop bytes, overwrite unread bytes, busy-spin, or create an unbounded overflow queue.
|
||||
|
||||
A held reader or transport view is part of the backpressure mechanism and must keep the relevant range pinned.
|
||||
|
||||
## Ordering
|
||||
|
||||
For one stream incarnation:
|
||||
|
||||
- producer writes have one total order;
|
||||
- the consumer observes exactly that order;
|
||||
- EOF follows every successfully accepted data byte;
|
||||
- a fault terminates the incarnation and no later data is delivered;
|
||||
- transport chunk boundaries are not application-visible stream boundaries.
|
||||
|
||||
The existing Iroh QUIC byte stream already provides remote ordered delivery. Swactor must not implement another TCP-like acknowledgement, retransmission, congestion-control, or reorder protocol.
|
||||
|
||||
## Terminal behavior
|
||||
|
||||
Each incarnation has exactly one terminal result:
|
||||
|
||||
```text
|
||||
EOF
|
||||
Fault
|
||||
Cancelled
|
||||
SessionClosed
|
||||
```
|
||||
|
||||
Terminal state is sticky. Duplicate or late terminal messages are ignored after the first accepted terminal transition.
|
||||
|
||||
A clean writer close produces EOF. Writer failure, transport failure, ring corruption, endpoint death, or exceptional writer exit produces a fault. Cancellation and session shutdown reclaim resources without presenting successful EOF.
|
||||
|
||||
## Python binding boundary
|
||||
|
||||
General functionality belongs in `crates/data-plane`:
|
||||
|
||||
- stream reader and writer types;
|
||||
- operation actors;
|
||||
- pinned ring record views;
|
||||
- ring lifetime guards;
|
||||
- backpressure and terminal FSMs;
|
||||
- namespace and transfer protocols.
|
||||
|
||||
Python owns only:
|
||||
|
||||
- `PyStreamReader` and `PyStreamWriter` wrappers;
|
||||
- Rust-future to awaitable conversion;
|
||||
- buffer exposure where used;
|
||||
- async context-manager behavior;
|
||||
- typed exception conversion.
|
||||
|
||||
Python must not manually tick the runtime or poll shared cursors.
|
||||
|
||||
## Myelin migration
|
||||
|
||||
The current result path uses a temporary Unix-domain socket exposed through `legacy_output` and `SWACTOR_DATA_PLANE_OUTPUT`.
|
||||
|
||||
After actor-driven streams satisfy the acceptance path:
|
||||
|
||||
1. Register the Myelin result sink at `/runs/<run-id>/results/inference`.
|
||||
2. Let `/runs/self/results/inference` resolve through `JobContext`.
|
||||
3. Route `ctx.data.write_stream(...)` through the SPSC stream implementation.
|
||||
4. Remove the Unix listener, socket path, legacy environment variable, connector, and compatibility code.
|
||||
5. Preserve application-owned JSON/result framing.
|
||||
6. Rerun the complete Tinygrad CUDA scenario.
|
||||
|
||||
Expected result:
|
||||
|
||||
```text
|
||||
CUDA -> [2.75, -8.75]
|
||||
```
|
||||
|
||||
## Failure behavior
|
||||
|
||||
| Failure | Required behavior |
|
||||
|---|---|
|
||||
| Path absent or unmatched under chosen open policy | Typed path/rendezvous failure or pending operation, never fallback socket |
|
||||
| Unauthorized path | Fail before ring or edge provisioning |
|
||||
| Duplicate source or sink | Typed namespace conflict |
|
||||
| Arena exhaustion | Fail both sides and release partial provisioning |
|
||||
| Ring generation mismatch or corruption | Fault incarnation and stop using the ring |
|
||||
| Iroh stream fault | Fault both endpoint operations |
|
||||
| Writer exception | Publish fault/abort, never clean EOF |
|
||||
| Reader or writer cancellation | Release held views and terminate binding |
|
||||
| Child or host session death | Stop bindings, quiesce pumps, reclaim leases |
|
||||
| EOF | Deliver all prior bytes, then complete reader exactly once |
|
||||
|
||||
## Behavioral invariants
|
||||
|
||||
1. One stream incarnation has exactly one producer and one consumer.
|
||||
2. A live path cannot simultaneously name a blob and a stream.
|
||||
3. Neither endpoint receives `Opened` before required rings and transport are ready.
|
||||
4. Successfully accepted bytes are delivered exactly once and in order while both endpoints remain live.
|
||||
5. The producer never overwrites unread bytes.
|
||||
6. The consumer never observes uncommitted bytes.
|
||||
7. A pinned record prevents its range from being consumed or reused.
|
||||
8. Backpressure is bounded and lossless.
|
||||
9. EOF follows all accepted data and appears exactly once.
|
||||
10. A fault prevents subsequent data or EOF success.
|
||||
11. Every incarnation reaches exactly one terminal outcome.
|
||||
12. Every arena lease is released only after pumps and views are quiescent.
|
||||
13. Wake notifications are hints; ring state is authoritative.
|
||||
14. Python never observes actor, node, edge, ring, socket, or arena identities.
|
||||
15. Swactor never interprets application byte schemas.
|
||||
|
||||
## Implementation sequence
|
||||
|
||||
1. Extend the centralized namespace with SPSC stream source/sink registration and atomic matching.
|
||||
2. Define shared stream session, operation, coordinator, progress, and terminal messages.
|
||||
3. Implement pinned borrowed byte-ring record views with wraparound spans.
|
||||
4. Add actor-driven capacity and data-progress notifications.
|
||||
5. Implement general `StreamReader` and `StreamWriter` operation actors in `crates/data-plane`.
|
||||
6. Provision local arena rings and Iroh edges through one coordinator FSM.
|
||||
7. Add thin Python reader/writer wrappers and async context behavior.
|
||||
8. Add focused tests for ordering, backpressure, wraparound, held views, EOF, faults, cancellation, and lease cleanup.
|
||||
9. Substitute the SPSC path for Myelin's temporary Unix result bridge.
|
||||
10. Remove all legacy stream socket and environment plumbing.
|
||||
11. Run affected Rust and Python suites.
|
||||
12. Exercise the full Tinygrad CUDA scenario and verify the deterministic result.
|
||||
|
||||
## Verification
|
||||
|
||||
Focused behavioral proof must cover:
|
||||
|
||||
- source-first or sink-first rendezvous behavior selected for the MVP;
|
||||
- concurrent independent SPSC paths;
|
||||
- exact byte ordering through wraparound;
|
||||
- writer suspension and resumption under backpressure;
|
||||
- no consume advancement while a view is pinned;
|
||||
- EOF after all prior data;
|
||||
- fault dominance over later progress;
|
||||
- cancellation and session-failure reclamation;
|
||||
- no stream payload assembly into `Vec<u8>` on zero-copy host/child paths;
|
||||
- absence of the legacy Unix stream bridge after migration;
|
||||
- the complete Myelin CUDA result path.
|
||||
|
||||
Affected suites include:
|
||||
|
||||
```bash
|
||||
cargo test -p data-plane
|
||||
cargo test -p myelin --lib
|
||||
PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 cargo check -p python
|
||||
```
|
||||
|
||||
Python binding tests run after rebuilding the extension from `crates/bindings/python`.
|
||||
|
||||
## Final boundary
|
||||
|
||||
The remaining stream MVP is intentionally narrow:
|
||||
|
||||
> One logical path rendezvous-matches one writer with one reader. Swactor provisions ordered bounded SPSC byte movement, uses actor messages for progress, applies lossless backpressure, propagates EOF or faults, and removes the temporary Unix bridge without exposing transport mechanics to applications.
|
||||
|
|
@ -1,748 +0,0 @@
|
|||
# Data-plane virtual blob namespace MVP
|
||||
|
||||
## Purpose
|
||||
|
||||
This document defines the MVP for registering, resolving, transferring, and exposing fixed-size blobs through Swactor.
|
||||
|
||||
The immediate acceptance path is Myelin loading `apps/myelin/jobs/tiny_linear.weights` from disk, registering it at `/models/tiny-linear/weights`, transferring it into a job's arena, and exposing it to Python as a read-only blob.
|
||||
|
||||
The design deliberately targets the centralized happy path:
|
||||
|
||||
- one authoritative namespace actor;
|
||||
- one current blob target per path;
|
||||
- one disk-backed source implementation;
|
||||
- complete eager transfer into the current arena-backed Python view;
|
||||
- existing actor routing and Iroh byte transport.
|
||||
|
||||
Content identities, verification, replication, leaderless consistency, lazy paging, and additional source types are deferred.
|
||||
|
||||
## Core abstraction
|
||||
|
||||
A blob is a fixed-length virtual byte space.
|
||||
|
||||
A `DataPath` behaves like a named pointer:
|
||||
|
||||
```text
|
||||
DataPath -> current blob target
|
||||
```
|
||||
|
||||
The path is the stable identity. It is not tied to a content hash or immutable object ID.
|
||||
|
||||
```text
|
||||
/models/tiny-linear/weights
|
||||
-> current FileBlobSourceActor
|
||||
```
|
||||
|
||||
A later registration may atomically replace the target:
|
||||
|
||||
```text
|
||||
/models/tiny-linear/weights
|
||||
-> replacement source
|
||||
```
|
||||
|
||||
### Dereference semantics
|
||||
|
||||
`read_blob(path)` dereferences the path once.
|
||||
|
||||
```text
|
||||
read begins while path -> target A
|
||||
path is later changed -> target B
|
||||
existing read continues against A
|
||||
new reads resolve B
|
||||
```
|
||||
|
||||
An in-flight operation is never redirected midway through transfer.
|
||||
|
||||
This is pointer replacement, not live shared mutation of bytes underneath an existing view. Supporting visible in-place mutation requires a separate coherence model and is outside the MVP.
|
||||
|
||||
## Developer experience
|
||||
|
||||
A developer registers a disk-backed blob with one control-plane operation:
|
||||
|
||||
```rust
|
||||
data.register(
|
||||
"/models/tiny-linear/weights",
|
||||
blob::file(weights_path),
|
||||
)
|
||||
.await?;
|
||||
```
|
||||
|
||||
A job reads it by path:
|
||||
|
||||
```python
|
||||
weights = await ctx.data.read_blob(
|
||||
"/models/tiny-linear/weights",
|
||||
)
|
||||
|
||||
with weights.map() as mapped:
|
||||
values = struct.unpack_from("<6f", mapped)
|
||||
```
|
||||
|
||||
Swactor owns everything between these calls:
|
||||
|
||||
- namespace consistency;
|
||||
- file opening and source lifetime;
|
||||
- actor and node resolution;
|
||||
- transfer negotiation;
|
||||
- destination arena allocation;
|
||||
- completion and failure;
|
||||
- lease and source cleanup.
|
||||
|
||||
The developer does not supply:
|
||||
|
||||
- byte length;
|
||||
- digest;
|
||||
- provider actor;
|
||||
- directory actor;
|
||||
- node identity;
|
||||
- edge identity;
|
||||
- source chunks;
|
||||
- provenance metadata;
|
||||
- cleanup behavior.
|
||||
|
||||
## Central namespace
|
||||
|
||||
One authoritative `DataDirectoryActor` owns the cluster namespace:
|
||||
|
||||
```rust
|
||||
struct DataDirectoryActor {
|
||||
blobs: BTreeMap<DataPath, BlobBinding>,
|
||||
}
|
||||
|
||||
struct BlobBinding {
|
||||
source: ActorAddress,
|
||||
length: u64,
|
||||
revision: u64,
|
||||
}
|
||||
```
|
||||
|
||||
The actor serializes all registration, replacement, lookup, and removal operations.
|
||||
|
||||
Conceptual protocol:
|
||||
|
||||
```rust
|
||||
enum DataDirectoryIn {
|
||||
Register {
|
||||
path: DataPath,
|
||||
source: ActorAddress,
|
||||
operation_id: OperationId,
|
||||
length: u64,
|
||||
reply_to: ActorAddress,
|
||||
},
|
||||
Resolve {
|
||||
path: DataPath,
|
||||
reply_to: ActorAddress,
|
||||
},
|
||||
Unregister {
|
||||
path: DataPath,
|
||||
operation_id: OperationId,
|
||||
reply_to: ActorAddress,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
`Register` is an atomic durable store:
|
||||
|
||||
- an absent path receives its first binding;
|
||||
- an existing path has its binding replaced;
|
||||
- the namespace revision advances once;
|
||||
- success is returned only after the new binding and operation result are durable and authoritative.
|
||||
|
||||
`Unregister` follows the same durable commit rule. A client-generated `OperationId` makes mutation retry idempotent across a crash after commit but before reply:
|
||||
|
||||
```text
|
||||
same operation ID + same request -> return the committed result
|
||||
same operation ID + different request -> reject
|
||||
```
|
||||
|
||||
The revision is actor-message fencing and recovery state. It is not a content identity and is not exposed to applications.
|
||||
|
||||
### Availability and restart model
|
||||
|
||||
The orchestrator hosts the one authoritative directory for the MVP:
|
||||
|
||||
- no consensus;
|
||||
- no replication;
|
||||
- no leader election;
|
||||
- exactly one orchestrator instance may own the namespace;
|
||||
- namespace mutations and current bindings are persisted locally.
|
||||
|
||||
If the orchestrator or directory is unavailable, operations that still require namespace authority wait for service rediscovery instead of failing merely because the authority restarted. Callers may cancel or impose their own deadline. Reads that already selected a source no longer depend on the directory and may complete while it is unavailable.
|
||||
|
||||
On restart, the orchestrator restores the durable namespace, reconstructs recoverable sources, starts a new `DataDirectoryActor`, increments its authority epoch, and publishes the actor through stable service discovery. Waiting clients discover the new actor and retry unresolved or idempotent operations.
|
||||
|
||||
Already-returned `Blob`s, mapped views, job arenas, and independently hosted source actors do not depend on the orchestrator process. A transfer whose source actor died with the orchestrator may fail; it is never silently redirected to a replacement target.
|
||||
|
||||
Concurrent orchestrators and recovery from network partitions are outside the MVP. Deployment must provide singleton process ownership; split-brain safety requires external fencing or a future consensus design.
|
||||
|
||||
## Registration lifecycle
|
||||
|
||||
The control-plane registration call performs:
|
||||
|
||||
```text
|
||||
open file
|
||||
-> determine fixed length
|
||||
-> create disk source behavior
|
||||
-> make its actor routable
|
||||
-> durably commit its recovery descriptor and namespace binding
|
||||
-> atomically install it in DataDirectoryActor
|
||||
```
|
||||
|
||||
Registration persists until:
|
||||
|
||||
- explicit `unregister(path)`;
|
||||
- replacement by a later registration.
|
||||
|
||||
It survives orchestrator restart and does not depend on retaining an application-visible registration guard. A registration acknowledged before a crash is present after recovery.
|
||||
|
||||
### Replacement
|
||||
|
||||
A later registration atomically replaces the target:
|
||||
|
||||
```rust
|
||||
data.register(
|
||||
"/models/tiny-linear/weights",
|
||||
blob::file(new_weights_path),
|
||||
)
|
||||
.await?;
|
||||
```
|
||||
|
||||
```text
|
||||
before swap:
|
||||
new reads -> old source
|
||||
|
||||
after swap:
|
||||
new reads -> new source
|
||||
```
|
||||
|
||||
Existing reads retain the old source binding until completion or failure.
|
||||
|
||||
The retired source remains alive while active operations use it. Once it is no longer the namespace target and has no active transfers, Swactor closes it.
|
||||
|
||||
`unregister(path)` follows the same lifetime discipline:
|
||||
|
||||
```text
|
||||
remove namespace binding
|
||||
-> reject new reads with PathNotFound
|
||||
-> permit already-pinned reads to finish
|
||||
-> retire source when unreferenced
|
||||
```
|
||||
|
||||
## Disk source behavior
|
||||
|
||||
The MVP implements one concrete source:
|
||||
|
||||
```text
|
||||
FileBlobSourceActor
|
||||
```
|
||||
|
||||
It owns:
|
||||
|
||||
- an opened read-only file;
|
||||
- the fixed length observed at registration;
|
||||
- private source information needed to read it;
|
||||
- active source transfer bindings.
|
||||
|
||||
The public namespace record contains no file path. The durable namespace store retains a private recovery descriptor sufficient to reopen the source after orchestrator restart; that descriptor is never sent to Python or returned by path resolution.
|
||||
|
||||
The trusted-cluster MVP contract is sufficient:
|
||||
|
||||
- the registrant supplies a stable file at a restart-stable private location;
|
||||
- Swactor retains the opened file while the source actor is live;
|
||||
- recovery reopens the file and checks the fixed registration length before republishing the source;
|
||||
- source length is fixed;
|
||||
- digest and mutation verification are deferred.
|
||||
|
||||
If recovery cannot reopen or stat the file at the recorded length, the binding remains known but unavailable and reads fail with a typed source failure until control replaces or unregisters it. Recovery must not silently bind different bytes.
|
||||
|
||||
HTTP, object storage, caching, and remote-origin plugins are not implemented now. The source boundary and durable descriptor remain opaque so `DataDirectoryActor` does not depend on disk-specific fields.
|
||||
|
||||
## Swactor service installation
|
||||
|
||||
The namespace and source plumbing are Swactor services, not application setup.
|
||||
|
||||
Internally:
|
||||
|
||||
```text
|
||||
Swactor/Myelin orchestrator runtime
|
||||
-> opens durable namespace store
|
||||
-> restores current bindings and idempotency records
|
||||
-> reconstructs recoverable source actors
|
||||
-> starts DataDirectoryActor with a new authority epoch
|
||||
-> publishes the namespace service through stable discovery
|
||||
```
|
||||
|
||||
Host job sessions discover the namespace service internally. They retain the logical service locator rather than treating one runtime actor address as permanent.
|
||||
|
||||
Application code does not:
|
||||
|
||||
- spawn the directory;
|
||||
- publish or retain its actor address;
|
||||
- wire routes;
|
||||
- pass it through job configuration;
|
||||
- implement reconnect or mutation retry.
|
||||
|
||||
The existing distribution `DirectoryActor` remains responsible for locating persistent source actors on nodes. The data directory resolves logical paths; the distribution directory resolves actor locations; stable service discovery locates the current data-directory actor after restart.
|
||||
|
||||
## Python API
|
||||
|
||||
### `read_blob`
|
||||
|
||||
```python
|
||||
blob = await ctx.data.read_blob(path)
|
||||
```
|
||||
|
||||
The public API guarantees:
|
||||
|
||||
- the path was canonicalized and authorized;
|
||||
- one current source target was selected;
|
||||
- the returned `Blob` represents one fixed byte space;
|
||||
- namespace-authority restart before resolution stalls the awaitable rather than changing its meaning;
|
||||
- source, placement, and recovery details remain opaque.
|
||||
|
||||
The MVP implementation additionally:
|
||||
|
||||
- allocates the complete destination lease;
|
||||
- transfers all bytes;
|
||||
- checks exact length;
|
||||
- seals the lease;
|
||||
- returns only after the complete blob is ready.
|
||||
|
||||
That eager behavior is not a permanent public placement promise. A future implementation may provide lazy virtual mappings without changing the `Blob` abstraction.
|
||||
|
||||
### `Blob`
|
||||
|
||||
```python
|
||||
blob.length
|
||||
blob.digest # normally None in this MVP
|
||||
blob.map()
|
||||
```
|
||||
|
||||
### `BlobView`
|
||||
|
||||
```python
|
||||
with blob.map() as mapped:
|
||||
consume(mapped)
|
||||
```
|
||||
|
||||
`BlobView` is the public name. `ArenaView` exposes the current backing strategy.
|
||||
|
||||
The MVP `BlobView`:
|
||||
|
||||
- wraps the existing arena-backed view;
|
||||
- exports Python's read-only buffer protocol;
|
||||
- holds the blob lease alive;
|
||||
- prevents close while buffer exports remain active;
|
||||
- never constructs Python `bytes` or `bytearray`.
|
||||
|
||||
## Standard read path
|
||||
|
||||
```text
|
||||
Python:
|
||||
ctx.data.read_blob(path)
|
||||
|
||||
PyDataPlane:
|
||||
converts Rust future to Python awaitable
|
||||
|
||||
DataPlane:
|
||||
asks ChildDataPlaneSessionActor
|
||||
|
||||
ChildDataPlaneSessionActor:
|
||||
spawns ReadBlobOperationActor
|
||||
|
||||
ReadBlobOperationActor:
|
||||
sends OpenReadBlob to HostDataPlaneSessionActor
|
||||
|
||||
HostDataPlaneSessionActor:
|
||||
expands /runs/self
|
||||
checks JobContext read prefixes
|
||||
spawns HostReadBlobBindingActor
|
||||
|
||||
HostReadBlobBindingActor:
|
||||
discovers the current DataDirectoryActor
|
||||
waits and rediscovers while namespace authority is unavailable
|
||||
asks DataDirectoryActor to resolve path
|
||||
|
||||
DataDirectoryActor:
|
||||
returns one current source actor, fixed length, and binding revision
|
||||
|
||||
HostReadBlobBindingActor:
|
||||
pins that source for this operation
|
||||
allocates FillingRead lease in child arena
|
||||
asks source actor to begin transfer
|
||||
|
||||
FileBlobSourceActor:
|
||||
spawns one source transfer binding
|
||||
reads the opened file
|
||||
sends bytes through existing Iroh transfer
|
||||
|
||||
Destination binding:
|
||||
writes incoming chunks into final arena payload
|
||||
tracks exact received length
|
||||
seals the lease after source completion
|
||||
|
||||
HostReadBlobBindingActor:
|
||||
sends BlobOpened through child session
|
||||
|
||||
ReadBlobOperationActor:
|
||||
validates sealed lease
|
||||
constructs Blob and lease guard
|
||||
completes original local ask
|
||||
|
||||
PyDataPlane:
|
||||
returns PyBlob
|
||||
```
|
||||
|
||||
If authority disappears before resolution completes, the binding returns to discovery and keeps the original awaitable pending. It does not allocate a destination lease while waiting. Cancellation stops discovery. After a source binding has been returned, that logical read is pinned and is never re-resolved to a newer path target.
|
||||
|
||||
The current Iroh API produces `WireEvent::BytesRead(Vec<u8>)`, so the MVP performs one copy from each network buffer into the final arena payload. It must not add another blob-assembly buffer or pass blob bytes through the reusable byte ring.
|
||||
|
||||
## Python `write_blob`
|
||||
|
||||
`write_blob` updates the same namespace:
|
||||
|
||||
```python
|
||||
async with ctx.data.write_blob(path, length=n) as blob:
|
||||
with blob.map() as mapped:
|
||||
mapped[:] = result
|
||||
```
|
||||
|
||||
Flow:
|
||||
|
||||
```text
|
||||
allocate writable candidate
|
||||
-> Python fills candidate
|
||||
-> clean context exit seals candidate
|
||||
-> host binding becomes a readable source
|
||||
-> discover or wait for namespace authority
|
||||
-> durably and atomically replace path binding
|
||||
```
|
||||
|
||||
Exceptional exit:
|
||||
|
||||
```text
|
||||
abort candidate
|
||||
-> publish nothing
|
||||
-> retain previous path target
|
||||
```
|
||||
|
||||
No separate host-session `published` namespace remains. Disk registrations and Python-produced blobs are both current targets in `DataDirectoryActor`.
|
||||
|
||||
A sealed candidate waiting for directory recovery retains its arena lease. Cancellation or child-session close aborts it and reclaims the lease. A committed publication is retried by operation ID if the directory crashes before its reply.
|
||||
|
||||
## Reuse from the current implementation
|
||||
|
||||
Retain:
|
||||
|
||||
- `DataPath` parsing and segment-safe prefix checks;
|
||||
- `/runs/self` expansion in `JobContext`;
|
||||
- host-side read/write authorization;
|
||||
- child session and per-operation actors;
|
||||
- host binding actors;
|
||||
- arena allocator actor;
|
||||
- `AllocationKind::FillingRead`;
|
||||
- incoming chunk writes into the final lease;
|
||||
- exact-length sealing;
|
||||
- `Blob`, lease guard, and Python buffer lifetime handling;
|
||||
- Iroh data transfer;
|
||||
- one inherited arena descriptor.
|
||||
|
||||
Refactor:
|
||||
|
||||
- `IncomingBlobSourceActor` becomes destination transfer machinery rather than a session-local namespace owner;
|
||||
- current `BeginBlobSource`, `BlobSourceChunk`, `FinishBlobSource`, and `FailBlobSource` calls become internal transfer events;
|
||||
- `HostDataPlaneSessionActor::blobs` and `published` no longer define application paths.
|
||||
|
||||
Remove from application setup:
|
||||
|
||||
- `InputBlobAssignment` edge/path registration;
|
||||
- manual source begin/push/finish calls;
|
||||
- application-managed model edge IDs;
|
||||
- `include_bytes!` as the production model source.
|
||||
|
||||
## Myelin acceptance path
|
||||
|
||||
Control side:
|
||||
|
||||
```rust
|
||||
data.register(
|
||||
"/models/tiny-linear/weights",
|
||||
blob::file("apps/myelin/jobs/tiny_linear.weights"),
|
||||
)
|
||||
.await?;
|
||||
```
|
||||
|
||||
Job side:
|
||||
|
||||
```python
|
||||
weights = await ctx.data.read_blob(
|
||||
"/models/tiny-linear/weights",
|
||||
)
|
||||
|
||||
with weights.map() as mapped:
|
||||
values = struct.unpack_from("<6f", mapped)
|
||||
```
|
||||
|
||||
Swactor resolves the source actor, negotiates the transfer, fills the job arena, and returns the blob.
|
||||
|
||||
The final scenario remains:
|
||||
|
||||
```text
|
||||
CUDA -> [2.75, -8.75]
|
||||
```
|
||||
|
||||
The temporary result-stream bridge remains until the later SPSC stream slice replaces it.
|
||||
|
||||
## State machines
|
||||
|
||||
### Namespace binding
|
||||
|
||||
```text
|
||||
Absent
|
||||
-> Bound(revision)
|
||||
-> Replaced(new revision)
|
||||
-> Unbound(new revision)
|
||||
```
|
||||
|
||||
Every mutation is durably committed before acknowledgement. Replacement installs the new source atomically. The previous source transitions independently toward retirement.
|
||||
|
||||
### Destination read binding
|
||||
|
||||
```text
|
||||
DiscoveringDirectory
|
||||
-> Resolving
|
||||
-> Allocating
|
||||
-> Negotiating
|
||||
-> Filling
|
||||
-> Sealing
|
||||
-> Granted
|
||||
-> Released
|
||||
|
||||
DiscoveringDirectory or Resolving
|
||||
-> WaitingForDirectory
|
||||
-> DiscoveringDirectory
|
||||
|
||||
any pre-grant state
|
||||
-> Faulted
|
||||
-> Releasing
|
||||
```
|
||||
|
||||
### File source
|
||||
|
||||
```text
|
||||
Opening
|
||||
-> Bound
|
||||
-> Serving
|
||||
-> Retiring
|
||||
-> Closed
|
||||
```
|
||||
|
||||
Each read uses an operation-specific source binding. The persistent source actor does not require a request-ID hashmap.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
| Failure | Behavior |
|
||||
|---|---|
|
||||
| Path absent | `PathNotFound` |
|
||||
| Unauthorized read | `Unauthorized` before discovery or transfer |
|
||||
| File open/stat failure during registration | Registration fails; existing binding remains |
|
||||
| File recovery open/stat/length failure | Binding remains known but unavailable; reads return a typed source failure |
|
||||
| Source route unavailable after resolution | Read fails with source or session failure; it is not redirected |
|
||||
| File read failure | Partial lease aborts; no `Blob` is returned |
|
||||
| Arena exhausted | `ArenaExhausted` |
|
||||
| Short or oversized transfer | Length failure; no `Blob` is returned |
|
||||
| Iroh failure | Partial lease aborts |
|
||||
| Python await cancellation | Waiting discovery, operation, transfer, and partial lease cancel |
|
||||
| Child session close | Active bindings terminate and leases reclaim |
|
||||
| Directory or orchestrator unavailable before resolution | Namespace-dependent operation waits for rediscovery; active independent transfers may finish |
|
||||
| Directory crash after committed mutation but before reply | Client retries the same operation ID and receives the committed result |
|
||||
|
||||
## Correctness guarantees
|
||||
|
||||
### Safety and consistency properties
|
||||
|
||||
1. Namespace operations are linearizable through the singleton authority: each mutation takes effect at one revision in one total order, and each successful resolve observes one revision.
|
||||
2. Each path has at most one current binding at every revision.
|
||||
3. A read observes one binding snapshot. Once resolved, rebinding and authority restart cannot redirect it.
|
||||
4. Blob visibility is all-or-nothing: no partial, failed, short, oversized, or unsealed transfer produces a `Blob`.
|
||||
5. A source exposes one fixed extent for the lifetime of a selected binding.
|
||||
6. Authorization precedes namespace discovery, arena allocation, and transfer.
|
||||
7. Disk and Python-produced blobs have the same publication and replacement semantics.
|
||||
|
||||
### Durability and recovery guarantees
|
||||
|
||||
1. Acknowledged namespace mutations survive orchestrator restart.
|
||||
2. A mutation retried with the same operation ID has at-most-once logical effect and returns its recorded result.
|
||||
3. Recovery never substitutes a different source for an already-selected read.
|
||||
4. A file binding is republished only after its private recovery descriptor reopens at the registered length.
|
||||
|
||||
### Conditional liveness and availability guarantees
|
||||
|
||||
1. An unresolved namespace operation remains pending while the authority is unavailable and resumes once the singleton authority is discoverable again.
|
||||
2. Callers may cancel or impose a deadline while waiting.
|
||||
3. Already-selected transfers may complete without the directory when their source and destination processes remain live.
|
||||
4. No progress guarantee is made while the singleton authority is unavailable, or when a selected source process has failed.
|
||||
|
||||
### Resource and lifetime invariants
|
||||
|
||||
1. A live `Blob` or `BlobView` prevents destination lease reclamation.
|
||||
2. Cancellation and failure reclaim candidate and partial leases.
|
||||
3. A retired source remains alive until its pinned reads finish while its source process remains live.
|
||||
4. The directory contains no payload bytes or transfer buffers.
|
||||
5. Python observes paths, fixed bytes, waiting, and typed failures—not actor, node, file, recovery, or transfer identities.
|
||||
|
||||
Safety assumes one orchestrator namespace authority. Concurrent authorities, partition recovery, and automatic failover are outside the MVP.
|
||||
|
||||
## Verification methodology
|
||||
|
||||
Verification combines deterministic conformance tests, stateful property testing, deliberate fault injection, and real-system integration.
|
||||
|
||||
1. **Deterministic guarantee tests (conformance scenarios)** encode each safety, durability, liveness, and lifetime guarantee as a small reproducible scenario over controlled actor runtimes.
|
||||
2. **Stateful model-based property tests (model-based fuzzing)** maintain a simple reference namespace model, generate bounded sequences of legal actions, apply each action to the model and implementation, and check the guarantees after every transition. Actions include register, replace, unregister, begin/resolve/finish read, publish/abort write, cancel, retire, lose authority, and recover authority. Seeds are replayable and failing sequences are shrunk.
|
||||
3. **Failpoint and fault-injection tests** deliberately fail storage before commit, after durable commit, and before reply; drop directory routes; fail file reads; break Iroh streams; cancel operations in each pre-grant state; and close child sessions. Each test checks both the reported result and the absence of leaked leases or unintended namespace changes.
|
||||
4. **Crash-recovery tests** reopen real temporary durable stores and run an actual orchestrator stop/restart to verify acknowledged state recovery, idempotent retry, and pending-operation resumption.
|
||||
5. **Real-adapter integration tests** use Iroh loopback to prove network buffers fill the final `FillingRead` lease directly and use the compiled Python extension to verify `BlobView` buffer and lifetime semantics.
|
||||
6. **End-to-end acceptance** registers the real weights file, runs the Myelin/Tinygrad job, and verifies `CUDA -> [2.75, -8.75]`.
|
||||
|
||||
The reference model covers externally observable namespace and operation state, not implementation fields. Stateful generation produces only legal commands for the current model state; targeted invalid-input tests remain separate.
|
||||
|
||||
The build loop for each vertical slice is:
|
||||
|
||||
```text
|
||||
state the guarantee
|
||||
-> add deterministic conformance case and model transition
|
||||
-> implement the narrow production path
|
||||
-> run bounded generated action sequences
|
||||
-> run relevant failpoints
|
||||
-> run directly affected existing tests
|
||||
```
|
||||
|
||||
Mocks are limited to deterministic scheduling and failpoint control at storage and transport ports. Final transport, Python, process-restart, and CUDA verification use real implementations.
|
||||
|
||||
## Code architecture
|
||||
|
||||
### New data-plane modules
|
||||
|
||||
| Location | Ownership |
|
||||
|---|---|
|
||||
| `crates/data-plane/src/control.rs` | Reusable `DataNamespaceService` recovery and public `DataPlaneControl` file registration, ensure, and unregister API. |
|
||||
| `crates/data-plane/src/namespace.rs` | `DataDirectoryActor`, runtime `BlobBinding`, revisions, operation IDs, register/resolve/unregister network protocol, and typed replies. |
|
||||
| `crates/data-plane/src/namespace_store.rs` | Versioned durable schema, opaque source recovery records, committed operation results, load/recovery, and crash-safe temporary-write/sync/rename persistence. |
|
||||
| `crates/data-plane/src/source.rs` | `FileBlobSourceActor`, recovery-time reopening and length validation, retirement, and one operation-specific source binding per read. |
|
||||
| `crates/data-plane/src/blob_transfer.rs` | Transport-neutral transfer offers, source/destination ports, completion/failure events, and transfer identifiers. It contains no Iroh types. |
|
||||
|
||||
`crates/data-plane/src/lib.rs` exports those modules. `crates/data-plane/src/protocol.rs` registers their wire codecs and adds only shared typed failures. `crates/data-plane/Cargo.toml` gains only dependencies required by the durable schema; distribution and Iroh remain outside this crate.
|
||||
|
||||
### Existing data-plane modules
|
||||
|
||||
| Location | Change |
|
||||
|---|---|
|
||||
| `crates/data-plane/src/host.rs` | Remove session-local application namespace ownership. Refactor `IncomingBlobSourceActor` into destination transfer state and route host reads/writes through the namespace client. Preserve the arena allocator as sole lease allocator/releaser. |
|
||||
| `crates/data-plane/src/data_plane.rs` | Preserve child session and operation APIs; extend cancellation and pending-publication handling for namespace rediscovery. |
|
||||
| `crates/data-plane/src/blob.rs` | Rename the public read view to `BlobView` without changing sealing, mapping, or lease lifetime. |
|
||||
|
||||
### Transport and Myelin composition
|
||||
|
||||
| Location | Change |
|
||||
|---|---|
|
||||
| `crates/iroh-driver/src/blob_transfer.rs` | Implement the data-plane transfer ports using Iroh streams and forward each received network buffer directly to the destination binding. |
|
||||
| `apps/myelin/src/data_namespace.rs` | New composition layer: open the namespace store, reconstruct sources, spawn the orchestrator directory, publish `swactor.data-directory`, and run the node-local namespace client/proxy that discovers and retries against the current authority epoch. |
|
||||
| `apps/myelin/src/orchestration/distribution_stack.rs` | Expose the existing `RegistryActor`/`RegistryView` to the namespace composition. It remains generic distribution infrastructure and gains no blob semantics. |
|
||||
| `apps/myelin/src/codecs.rs` | Register namespace, source, and transfer actor codecs. |
|
||||
| `apps/myelin/src/job_data_plane.rs` | Inject the namespace client and Iroh transfer ports into host sessions; remove direct blob maps and manual source chunk methods. |
|
||||
| `apps/myelin/src/job_deploy.rs` | Remove `InputBlobAssignment`, model edge setup, and incoming manual chunk dispatch. Keep the temporary result-stream bridge. |
|
||||
| `apps/myelin/src/node/worker_node_runtime.rs` | Install the node-local namespace client and transfer receiver when constructing job services. |
|
||||
| `apps/myelin/src/orchestration/app.rs` and deployable orchestrator startup | Install the authoritative namespace service and its configured durable state path. |
|
||||
| `crates/bindings/python/src/job.rs` | Expose `BlobView`, retain typed error mapping, and replace the embedded manual test source with the real registration path. |
|
||||
|
||||
### Architectural invariants
|
||||
|
||||
1. `DataDirectoryActor` is the only in-memory writer of current logical bindings and revisions.
|
||||
2. `NamespaceStore` is the only durable namespace writer. One actor serializes calls into it.
|
||||
3. A mutation constructs the next complete state, durably commits it, swaps the runtime state, then replies—in that order.
|
||||
4. A repeated operation ID returns its recorded result without creating another source, revision, or retirement transition.
|
||||
5. The node-local namespace client owns discovery and retry. Host sessions never retain a directory actor address as durable configuration.
|
||||
6. A host read accepts one successful resolve reply. After that transition, directory changes cannot alter its source actor, length, or revision.
|
||||
7. `DataDirectoryActor` and `NamespaceStore` never receive payload bytes or transfer buffers.
|
||||
8. The data-plane crate defines transport ports; Iroh-driver implements them; Myelin wires them. Dependencies never point from data-plane into Iroh-driver, distribution, or Myelin.
|
||||
9. `FileBlobSourceActor` exclusively owns its opened file. Each transfer child owns one read attempt and reports completion before retirement can close the source.
|
||||
10. `ArenaAllocatorActor` remains the sole allocator and releaser. Destination bindings write only within their granted `FillingRead` lease and seal only at the exact advertised length.
|
||||
11. Cancellation is an explicit state transition that stops discovery or transfer and reclaims any candidate or partial lease.
|
||||
12. Python receives only `Blob`, `BlobView`, lengths, optional digests, and typed failures. Actor, node, file, recovery, and transfer identities remain internal.
|
||||
|
||||
## Implementation sequence
|
||||
|
||||
1. Add deterministic guarantee tests and a reference state model for namespace ordering, durable acknowledgement, idempotent retry, resolve-once reads, waiting, cancellation, and lease cleanup; then add bounded legal-action sequence generation over that model.
|
||||
2. Implement `namespace_store.rs` and make its persistence and crash-injection contracts pass.
|
||||
3. Implement `namespace.rs`, wire codecs, and directory actor tests over the durable store.
|
||||
4. Implement `source.rs`, disk registration, source recovery, retirement, and source lifecycle tests.
|
||||
5. Define `blob_transfer.rs`, implement the Iroh adapter, and pass real loopback exact-length and failure tests.
|
||||
6. Add Myelin namespace bootstrap, stable `RegistryActor` discovery, authority epochs, and the node-local retrying namespace client.
|
||||
7. Refactor `host.rs` so read authorization precedes discovery, resolution precedes allocation, and transfer fills and seals the final arena lease.
|
||||
8. Move `write_blob` publication into the directory with durable operation IDs and lease retention/cancellation while authority is unavailable.
|
||||
9. Remove session-local `blobs` and `published` ownership and delete the manual begin/chunk/finish protocol and its callers.
|
||||
10. Rename Rust and Python `ArenaView` to `BlobView`, migrate every caller, and run the Python buffer-lifetime contracts.
|
||||
11. Replace Myelin model edge assignment and `include_bytes!` with one disk registration.
|
||||
12. Run the process-level orchestrator stop/restart contracts, including waiting read, waiting publication, committed-before-reply retry, and no post-resolution redirection.
|
||||
13. Run all affected suites, then exercise Tinygrad CUDA and verify `CUDA -> [2.75, -8.75]`.
|
||||
|
||||
## Deferred
|
||||
|
||||
- content IDs and digest verification;
|
||||
- replicated or leaderless namespace authority;
|
||||
- concurrent-orchestrator fencing, partition recovery, and automatic failover;
|
||||
- multiple providers or replicas;
|
||||
- payload durability across source-node loss;
|
||||
- caching and spilling;
|
||||
- lazy page-fault-backed mappings;
|
||||
- partial range reads;
|
||||
- HTTP and object-store sources;
|
||||
- SPSC stream namespace matching;
|
||||
- topics, fanout, and SPMC streams;
|
||||
- GPU-native blob realizations.
|
||||
|
||||
## Final MVP boundary
|
||||
|
||||
The MVP is intentionally narrow:
|
||||
|
||||
> A restartable singleton orchestrator durably owns a centralized actor registry that maps pointer-like paths to current fixed-byte sources. Namespace-dependent operations wait while that authority restarts; already-selected independent transfers and returned blobs do not depend on it. Disk registration is one operation. Python dereferences by path. Swactor resolves one source, performs the complete transfer into the existing sealed arena backing, and returns a read-only blob without exposing placement or recovery mechanics.
|
||||
|
||||
## Completion status
|
||||
|
||||
The remaining MVP work is complete.
|
||||
|
||||
### Delivered behavior
|
||||
|
||||
- Public `DataPlaneControl` registers disk sources through `blob::file(...)`; Myelin supplies only cluster publication, discovery, and Iroh transport composition.
|
||||
- A file registered on the authority node is readable by a session on a remote node.
|
||||
- A blob committed by one session is readable by an independent remote session.
|
||||
- Committed publication ownership detaches from the producer session. Producer close does not unregister it.
|
||||
- Replacement or unregister retires the source after active transfers, then reclaims the source actor and arena lease.
|
||||
- Tinygrad CUDA output and the temporary result socket behavior remain unchanged.
|
||||
|
||||
### Verification coverage
|
||||
|
||||
- `crates/data-plane/tests/namespace_host_read_guarantees.rs` deterministically covers public file registration, committed publication lifetime across producer close, and source reclamation after unregister.
|
||||
- `apps/myelin/src/tests/data_namespace_guarantees.rs` composes two real Iroh-backed nodes and covers remote control registration reads, remote session publication reads, producer close, and unregister reclamation.
|
||||
- The data-plane and Iroh blob-transfer suites cover namespace, source, transfer, exact-length, cancellation, and lease behavior.
|
||||
- The compiled Python extension suite covers `BlobView` buffer lifetime, write publication, the temporary output socket, real process attachment, and the Tinygrad CUDA result `[2.75, -8.75]`.
|
||||
|
||||
### Final code architecture
|
||||
|
||||
```text
|
||||
crates/data-plane
|
||||
├── control
|
||||
│ Reusable namespace service and public control API
|
||||
├── namespace + namespace_store
|
||||
│ Namespace semantics and durability
|
||||
├── source + blob_transfer
|
||||
│ Source lifetime and transport-neutral movement
|
||||
└── host + data_plane + blob
|
||||
Reusable arena, transfer, publication, and mapping machinery
|
||||
|
||||
apps/myelin
|
||||
├── data_namespace
|
||||
│ Cluster-level discovery, publication, and Iroh composition
|
||||
└── job_data_plane
|
||||
Creates data-plane sessions for managed jobs
|
||||
```
|
||||
Loading…
Reference in a new issue