swactor/DATA_PLANE_VIRTUAL_BLOB_NAMESPACE_MVP.md
Zachery Aaron Shores-Chmielewski cd870d3141 feat(data-plane): complete virtual blob namespace
Add durable namespace control, Iroh-backed source transfer, session-independent publication lifetime, and wire managed Myelin jobs and bindings through the reusable data-plane API.
2026-08-22 21:16:56 +04:00

31 KiB

Data-plane virtual blob namespace MVP

Purpose

This document defines the MVP for registering, resolving, transferring, and exposing fixed-size blobs through Swactor.

The immediate acceptance path is Myelin loading apps/myelin/jobs/tiny_linear.weights from disk, registering it at /models/tiny-linear/weights, transferring it into a job's arena, and exposing it to Python as a read-only blob.

The design deliberately targets the centralized happy path:

  • one authoritative namespace actor;
  • one current blob target per path;
  • one disk-backed source implementation;
  • complete eager transfer into the current arena-backed Python view;
  • existing actor routing and Iroh byte transport.

Content identities, verification, replication, leaderless consistency, lazy paging, and additional source types are deferred.

Core abstraction

A blob is a fixed-length virtual byte space.

A DataPath behaves like a named pointer:

DataPath -> current blob target

The path is the stable identity. It is not tied to a content hash or immutable object ID.

/models/tiny-linear/weights
    -> current FileBlobSourceActor

A later registration may atomically replace the target:

/models/tiny-linear/weights
    -> replacement source

Dereference semantics

read_blob(path) dereferences the path once.

read begins while path -> target A
path is later changed -> target B
existing read continues against A
new reads resolve B

An in-flight operation is never redirected midway through transfer.

This is pointer replacement, not live shared mutation of bytes underneath an existing view. Supporting visible in-place mutation requires a separate coherence model and is outside the MVP.

Developer experience

A developer registers a disk-backed blob with one control-plane operation:

data.register(
    "/models/tiny-linear/weights",
    blob::file(weights_path),
)
.await?;

A job reads it by path:

weights = await ctx.data.read_blob(
    "/models/tiny-linear/weights",
)

with weights.map() as mapped:
    values = struct.unpack_from("<6f", mapped)

Swactor owns everything between these calls:

  • namespace consistency;
  • file opening and source lifetime;
  • actor and node resolution;
  • transfer negotiation;
  • destination arena allocation;
  • completion and failure;
  • lease and source cleanup.

The developer does not supply:

  • byte length;
  • digest;
  • provider actor;
  • directory actor;
  • node identity;
  • edge identity;
  • source chunks;
  • provenance metadata;
  • cleanup behavior.

Central namespace

One authoritative DataDirectoryActor owns the cluster namespace:

struct DataDirectoryActor {
    blobs: BTreeMap<DataPath, BlobBinding>,
}

struct BlobBinding {
    source: ActorAddress,
    length: u64,
    revision: u64,
}

The actor serializes all registration, replacement, lookup, and removal operations.

Conceptual protocol:

enum DataDirectoryIn {
    Register {
        path: DataPath,
        source: ActorAddress,
        operation_id: OperationId,
        length: u64,
        reply_to: ActorAddress,
    },
    Resolve {
        path: DataPath,
        reply_to: ActorAddress,
    },
    Unregister {
        path: DataPath,
        operation_id: OperationId,
        reply_to: ActorAddress,
    },
}

Register is an atomic durable store:

  • an absent path receives its first binding;
  • an existing path has its binding replaced;
  • the namespace revision advances once;
  • success is returned only after the new binding and operation result are durable and authoritative.

Unregister follows the same durable commit rule. A client-generated OperationId makes mutation retry idempotent across a crash after commit but before reply:

same operation ID + same request -> return the committed result
same operation ID + different request -> reject

The revision is actor-message fencing and recovery state. It is not a content identity and is not exposed to applications.

Availability and restart model

The orchestrator hosts the one authoritative directory for the MVP:

  • no consensus;
  • no replication;
  • no leader election;
  • exactly one orchestrator instance may own the namespace;
  • namespace mutations and current bindings are persisted locally.

If the orchestrator or directory is unavailable, operations that still require namespace authority wait for service rediscovery instead of failing merely because the authority restarted. Callers may cancel or impose their own deadline. Reads that already selected a source no longer depend on the directory and may complete while it is unavailable.

On restart, the orchestrator restores the durable namespace, reconstructs recoverable sources, starts a new DataDirectoryActor, increments its authority epoch, and publishes the actor through stable service discovery. Waiting clients discover the new actor and retry unresolved or idempotent operations.

Already-returned Blobs, mapped views, job arenas, and independently hosted source actors do not depend on the orchestrator process. A transfer whose source actor died with the orchestrator may fail; it is never silently redirected to a replacement target.

Concurrent orchestrators and recovery from network partitions are outside the MVP. Deployment must provide singleton process ownership; split-brain safety requires external fencing or a future consensus design.

Registration lifecycle

The control-plane registration call performs:

open file
-> determine fixed length
-> create disk source behavior
-> make its actor routable
-> durably commit its recovery descriptor and namespace binding
-> atomically install it in DataDirectoryActor

Registration persists until:

  • explicit unregister(path);
  • replacement by a later registration.

It survives orchestrator restart and does not depend on retaining an application-visible registration guard. A registration acknowledged before a crash is present after recovery.

Replacement

A later registration atomically replaces the target:

data.register(
    "/models/tiny-linear/weights",
    blob::file(new_weights_path),
)
.await?;
before swap:
    new reads -> old source

after swap:
    new reads -> new source

Existing reads retain the old source binding until completion or failure.

The retired source remains alive while active operations use it. Once it is no longer the namespace target and has no active transfers, Swactor closes it.

unregister(path) follows the same lifetime discipline:

remove namespace binding
-> reject new reads with PathNotFound
-> permit already-pinned reads to finish
-> retire source when unreferenced

Disk source behavior

The MVP implements one concrete source:

FileBlobSourceActor

It owns:

  • an opened read-only file;
  • the fixed length observed at registration;
  • private source information needed to read it;
  • active source transfer bindings.

The public namespace record contains no file path. The durable namespace store retains a private recovery descriptor sufficient to reopen the source after orchestrator restart; that descriptor is never sent to Python or returned by path resolution.

The trusted-cluster MVP contract is sufficient:

  • the registrant supplies a stable file at a restart-stable private location;
  • Swactor retains the opened file while the source actor is live;
  • recovery reopens the file and checks the fixed registration length before republishing the source;
  • source length is fixed;
  • digest and mutation verification are deferred.

If recovery cannot reopen or stat the file at the recorded length, the binding remains known but unavailable and reads fail with a typed source failure until control replaces or unregisters it. Recovery must not silently bind different bytes.

HTTP, object storage, caching, and remote-origin plugins are not implemented now. The source boundary and durable descriptor remain opaque so DataDirectoryActor does not depend on disk-specific fields.

Swactor service installation

The namespace and source plumbing are Swactor services, not application setup.

Internally:

Swactor/Myelin orchestrator runtime
-> opens durable namespace store
-> restores current bindings and idempotency records
-> reconstructs recoverable source actors
-> starts DataDirectoryActor with a new authority epoch
-> publishes the namespace service through stable discovery

Host job sessions discover the namespace service internally. They retain the logical service locator rather than treating one runtime actor address as permanent.

Application code does not:

  • spawn the directory;
  • publish or retain its actor address;
  • wire routes;
  • pass it through job configuration;
  • implement reconnect or mutation retry.

The existing distribution DirectoryActor remains responsible for locating persistent source actors on nodes. The data directory resolves logical paths; the distribution directory resolves actor locations; stable service discovery locates the current data-directory actor after restart.

Python API

read_blob

blob = await ctx.data.read_blob(path)

The public API guarantees:

  • the path was canonicalized and authorized;
  • one current source target was selected;
  • the returned Blob represents one fixed byte space;
  • namespace-authority restart before resolution stalls the awaitable rather than changing its meaning;
  • source, placement, and recovery details remain opaque.

The MVP implementation additionally:

  • allocates the complete destination lease;
  • transfers all bytes;
  • checks exact length;
  • seals the lease;
  • returns only after the complete blob is ready.

That eager behavior is not a permanent public placement promise. A future implementation may provide lazy virtual mappings without changing the Blob abstraction.

Blob

blob.length
blob.digest       # normally None in this MVP
blob.map()

BlobView

with blob.map() as mapped:
    consume(mapped)

BlobView is the public name. ArenaView exposes the current backing strategy.

The MVP BlobView:

  • wraps the existing arena-backed view;
  • exports Python's read-only buffer protocol;
  • holds the blob lease alive;
  • prevents close while buffer exports remain active;
  • never constructs Python bytes or bytearray.

Standard read path

Python:
    ctx.data.read_blob(path)

PyDataPlane:
    converts Rust future to Python awaitable

DataPlane:
    asks ChildDataPlaneSessionActor

ChildDataPlaneSessionActor:
    spawns ReadBlobOperationActor

ReadBlobOperationActor:
    sends OpenReadBlob to HostDataPlaneSessionActor

HostDataPlaneSessionActor:
    expands /runs/self
    checks JobContext read prefixes
    spawns HostReadBlobBindingActor

HostReadBlobBindingActor:
    discovers the current DataDirectoryActor
    waits and rediscovers while namespace authority is unavailable
    asks DataDirectoryActor to resolve path

DataDirectoryActor:
    returns one current source actor, fixed length, and binding revision

HostReadBlobBindingActor:
    pins that source for this operation
    allocates FillingRead lease in child arena
    asks source actor to begin transfer

FileBlobSourceActor:
    spawns one source transfer binding
    reads the opened file
    sends bytes through existing Iroh transfer

Destination binding:
    writes incoming chunks into final arena payload
    tracks exact received length
    seals the lease after source completion

HostReadBlobBindingActor:
    sends BlobOpened through child session

ReadBlobOperationActor:
    validates sealed lease
    constructs Blob and lease guard
    completes original local ask

PyDataPlane:
    returns PyBlob

If authority disappears before resolution completes, the binding returns to discovery and keeps the original awaitable pending. It does not allocate a destination lease while waiting. Cancellation stops discovery. After a source binding has been returned, that logical read is pinned and is never re-resolved to a newer path target.

The current Iroh API produces WireEvent::BytesRead(Vec<u8>), so the MVP performs one copy from each network buffer into the final arena payload. It must not add another blob-assembly buffer or pass blob bytes through the reusable byte ring.

Python write_blob

write_blob updates the same namespace:

async with ctx.data.write_blob(path, length=n) as blob:
    with blob.map() as mapped:
        mapped[:] = result

Flow:

allocate writable candidate
-> Python fills candidate
-> clean context exit seals candidate
-> host binding becomes a readable source
-> discover or wait for namespace authority
-> durably and atomically replace path binding

Exceptional exit:

abort candidate
-> publish nothing
-> retain previous path target

No separate host-session published namespace remains. Disk registrations and Python-produced blobs are both current targets in DataDirectoryActor.

A sealed candidate waiting for directory recovery retains its arena lease. Cancellation or child-session close aborts it and reclaims the lease. A committed publication is retried by operation ID if the directory crashes before its reply.

Reuse from the current implementation

Retain:

  • DataPath parsing and segment-safe prefix checks;
  • /runs/self expansion in JobContext;
  • host-side read/write authorization;
  • child session and per-operation actors;
  • host binding actors;
  • arena allocator actor;
  • AllocationKind::FillingRead;
  • incoming chunk writes into the final lease;
  • exact-length sealing;
  • Blob, lease guard, and Python buffer lifetime handling;
  • Iroh data transfer;
  • one inherited arena descriptor.

Refactor:

  • IncomingBlobSourceActor becomes destination transfer machinery rather than a session-local namespace owner;
  • current BeginBlobSource, BlobSourceChunk, FinishBlobSource, and FailBlobSource calls become internal transfer events;
  • HostDataPlaneSessionActor::blobs and published no longer define application paths.

Remove from application setup:

  • InputBlobAssignment edge/path registration;
  • manual source begin/push/finish calls;
  • application-managed model edge IDs;
  • include_bytes! as the production model source.

Myelin acceptance path

Control side:

data.register(
    "/models/tiny-linear/weights",
    blob::file("apps/myelin/jobs/tiny_linear.weights"),
)
.await?;

Job side:

weights = await ctx.data.read_blob(
    "/models/tiny-linear/weights",
)

with weights.map() as mapped:
    values = struct.unpack_from("<6f", mapped)

Swactor resolves the source actor, negotiates the transfer, fills the job arena, and returns the blob.

The final scenario remains:

CUDA -> [2.75, -8.75]

The temporary result-stream bridge remains until the later SPSC stream slice replaces it.

State machines

Namespace binding

Absent
-> Bound(revision)
-> Replaced(new revision)
-> Unbound(new revision)

Every mutation is durably committed before acknowledgement. Replacement installs the new source atomically. The previous source transitions independently toward retirement.

Destination read binding

DiscoveringDirectory
-> Resolving
-> Allocating
-> Negotiating
-> Filling
-> Sealing
-> Granted
-> Released

DiscoveringDirectory or Resolving
-> WaitingForDirectory
-> DiscoveringDirectory

any pre-grant state
-> Faulted
-> Releasing

File source

Opening
-> Bound
-> Serving
-> Retiring
-> Closed

Each read uses an operation-specific source binding. The persistent source actor does not require a request-ID hashmap.

Failure behavior

Failure Behavior
Path absent PathNotFound
Unauthorized read Unauthorized before discovery or transfer
File open/stat failure during registration Registration fails; existing binding remains
File recovery open/stat/length failure Binding remains known but unavailable; reads return a typed source failure
Source route unavailable after resolution Read fails with source or session failure; it is not redirected
File read failure Partial lease aborts; no Blob is returned
Arena exhausted ArenaExhausted
Short or oversized transfer Length failure; no Blob is returned
Iroh failure Partial lease aborts
Python await cancellation Waiting discovery, operation, transfer, and partial lease cancel
Child session close Active bindings terminate and leases reclaim
Directory or orchestrator unavailable before resolution Namespace-dependent operation waits for rediscovery; active independent transfers may finish
Directory crash after committed mutation but before reply Client retries the same operation ID and receives the committed result

Correctness guarantees

Safety and consistency properties

  1. Namespace operations are linearizable through the singleton authority: each mutation takes effect at one revision in one total order, and each successful resolve observes one revision.
  2. Each path has at most one current binding at every revision.
  3. A read observes one binding snapshot. Once resolved, rebinding and authority restart cannot redirect it.
  4. Blob visibility is all-or-nothing: no partial, failed, short, oversized, or unsealed transfer produces a Blob.
  5. A source exposes one fixed extent for the lifetime of a selected binding.
  6. Authorization precedes namespace discovery, arena allocation, and transfer.
  7. Disk and Python-produced blobs have the same publication and replacement semantics.

Durability and recovery guarantees

  1. Acknowledged namespace mutations survive orchestrator restart.
  2. A mutation retried with the same operation ID has at-most-once logical effect and returns its recorded result.
  3. Recovery never substitutes a different source for an already-selected read.
  4. A file binding is republished only after its private recovery descriptor reopens at the registered length.

Conditional liveness and availability guarantees

  1. An unresolved namespace operation remains pending while the authority is unavailable and resumes once the singleton authority is discoverable again.
  2. Callers may cancel or impose a deadline while waiting.
  3. Already-selected transfers may complete without the directory when their source and destination processes remain live.
  4. No progress guarantee is made while the singleton authority is unavailable, or when a selected source process has failed.

Resource and lifetime invariants

  1. A live Blob or BlobView prevents destination lease reclamation.
  2. Cancellation and failure reclaim candidate and partial leases.
  3. A retired source remains alive until its pinned reads finish while its source process remains live.
  4. The directory contains no payload bytes or transfer buffers.
  5. Python observes paths, fixed bytes, waiting, and typed failures—not actor, node, file, recovery, or transfer identities.

Safety assumes one orchestrator namespace authority. Concurrent authorities, partition recovery, and automatic failover are outside the MVP.

Verification methodology

Verification combines deterministic conformance tests, stateful property testing, deliberate fault injection, and real-system integration.

  1. Deterministic guarantee tests (conformance scenarios) encode each safety, durability, liveness, and lifetime guarantee as a small reproducible scenario over controlled actor runtimes.
  2. Stateful model-based property tests (model-based fuzzing) maintain a simple reference namespace model, generate bounded sequences of legal actions, apply each action to the model and implementation, and check the guarantees after every transition. Actions include register, replace, unregister, begin/resolve/finish read, publish/abort write, cancel, retire, lose authority, and recover authority. Seeds are replayable and failing sequences are shrunk.
  3. Failpoint and fault-injection tests deliberately fail storage before commit, after durable commit, and before reply; drop directory routes; fail file reads; break Iroh streams; cancel operations in each pre-grant state; and close child sessions. Each test checks both the reported result and the absence of leaked leases or unintended namespace changes.
  4. Crash-recovery tests reopen real temporary durable stores and run an actual orchestrator stop/restart to verify acknowledged state recovery, idempotent retry, and pending-operation resumption.
  5. Real-adapter integration tests use Iroh loopback to prove network buffers fill the final FillingRead lease directly and use the compiled Python extension to verify BlobView buffer and lifetime semantics.
  6. End-to-end acceptance registers the real weights file, runs the Myelin/Tinygrad job, and verifies CUDA -> [2.75, -8.75].

The reference model covers externally observable namespace and operation state, not implementation fields. Stateful generation produces only legal commands for the current model state; targeted invalid-input tests remain separate.

The build loop for each vertical slice is:

state the guarantee
-> add deterministic conformance case and model transition
-> implement the narrow production path
-> run bounded generated action sequences
-> run relevant failpoints
-> run directly affected existing tests

Mocks are limited to deterministic scheduling and failpoint control at storage and transport ports. Final transport, Python, process-restart, and CUDA verification use real implementations.

Code architecture

New data-plane modules

Location Ownership
crates/data-plane/src/control.rs Reusable DataNamespaceService recovery and public DataPlaneControl file registration, ensure, and unregister API.
crates/data-plane/src/namespace.rs DataDirectoryActor, runtime BlobBinding, revisions, operation IDs, register/resolve/unregister network protocol, and typed replies.
crates/data-plane/src/namespace_store.rs Versioned durable schema, opaque source recovery records, committed operation results, load/recovery, and crash-safe temporary-write/sync/rename persistence.
crates/data-plane/src/source.rs FileBlobSourceActor, recovery-time reopening and length validation, retirement, and one operation-specific source binding per read.
crates/data-plane/src/blob_transfer.rs Transport-neutral transfer offers, source/destination ports, completion/failure events, and transfer identifiers. It contains no Iroh types.

crates/data-plane/src/lib.rs exports those modules. crates/data-plane/src/protocol.rs registers their wire codecs and adds only shared typed failures. crates/data-plane/Cargo.toml gains only dependencies required by the durable schema; distribution and Iroh remain outside this crate.

Existing data-plane modules

Location Change
crates/data-plane/src/host.rs Remove session-local application namespace ownership. Refactor IncomingBlobSourceActor into destination transfer state and route host reads/writes through the namespace client. Preserve the arena allocator as sole lease allocator/releaser.
crates/data-plane/src/data_plane.rs Preserve child session and operation APIs; extend cancellation and pending-publication handling for namespace rediscovery.
crates/data-plane/src/blob.rs Rename the public read view to BlobView without changing sealing, mapping, or lease lifetime.

Transport and Myelin composition

Location Change
crates/iroh-driver/src/blob_transfer.rs Implement the data-plane transfer ports using Iroh streams and forward each received network buffer directly to the destination binding.
apps/myelin/src/data_namespace.rs New composition layer: open the namespace store, reconstruct sources, spawn the orchestrator directory, publish swactor.data-directory, and run the node-local namespace client/proxy that discovers and retries against the current authority epoch.
apps/myelin/src/orchestration/distribution_stack.rs Expose the existing RegistryActor/RegistryView to the namespace composition. It remains generic distribution infrastructure and gains no blob semantics.
apps/myelin/src/codecs.rs Register namespace, source, and transfer actor codecs.
apps/myelin/src/job_data_plane.rs Inject the namespace client and Iroh transfer ports into host sessions; remove direct blob maps and manual source chunk methods.
apps/myelin/src/job_deploy.rs Remove InputBlobAssignment, model edge setup, and incoming manual chunk dispatch. Keep the temporary result-stream bridge.
apps/myelin/src/node/worker_node_runtime.rs Install the node-local namespace client and transfer receiver when constructing job services.
apps/myelin/src/orchestration/app.rs and deployable orchestrator startup Install the authoritative namespace service and its configured durable state path.
crates/bindings/python/src/job.rs Expose BlobView, retain typed error mapping, and replace the embedded manual test source with the real registration path.

Architectural invariants

  1. DataDirectoryActor is the only in-memory writer of current logical bindings and revisions.
  2. NamespaceStore is the only durable namespace writer. One actor serializes calls into it.
  3. A mutation constructs the next complete state, durably commits it, swaps the runtime state, then replies—in that order.
  4. A repeated operation ID returns its recorded result without creating another source, revision, or retirement transition.
  5. The node-local namespace client owns discovery and retry. Host sessions never retain a directory actor address as durable configuration.
  6. A host read accepts one successful resolve reply. After that transition, directory changes cannot alter its source actor, length, or revision.
  7. DataDirectoryActor and NamespaceStore never receive payload bytes or transfer buffers.
  8. The data-plane crate defines transport ports; Iroh-driver implements them; Myelin wires them. Dependencies never point from data-plane into Iroh-driver, distribution, or Myelin.
  9. FileBlobSourceActor exclusively owns its opened file. Each transfer child owns one read attempt and reports completion before retirement can close the source.
  10. ArenaAllocatorActor remains the sole allocator and releaser. Destination bindings write only within their granted FillingRead lease and seal only at the exact advertised length.
  11. Cancellation is an explicit state transition that stops discovery or transfer and reclaims any candidate or partial lease.
  12. Python receives only Blob, BlobView, lengths, optional digests, and typed failures. Actor, node, file, recovery, and transfer identities remain internal.

Implementation sequence

  1. Add deterministic guarantee tests and a reference state model for namespace ordering, durable acknowledgement, idempotent retry, resolve-once reads, waiting, cancellation, and lease cleanup; then add bounded legal-action sequence generation over that model.
  2. Implement namespace_store.rs and make its persistence and crash-injection contracts pass.
  3. Implement namespace.rs, wire codecs, and directory actor tests over the durable store.
  4. Implement source.rs, disk registration, source recovery, retirement, and source lifecycle tests.
  5. Define blob_transfer.rs, implement the Iroh adapter, and pass real loopback exact-length and failure tests.
  6. Add Myelin namespace bootstrap, stable RegistryActor discovery, authority epochs, and the node-local retrying namespace client.
  7. Refactor host.rs so read authorization precedes discovery, resolution precedes allocation, and transfer fills and seals the final arena lease.
  8. Move write_blob publication into the directory with durable operation IDs and lease retention/cancellation while authority is unavailable.
  9. Remove session-local blobs and published ownership and delete the manual begin/chunk/finish protocol and its callers.
  10. Rename Rust and Python ArenaView to BlobView, migrate every caller, and run the Python buffer-lifetime contracts.
  11. Replace Myelin model edge assignment and include_bytes! with one disk registration.
  12. Run the process-level orchestrator stop/restart contracts, including waiting read, waiting publication, committed-before-reply retry, and no post-resolution redirection.
  13. Run all affected suites, then exercise Tinygrad CUDA and verify CUDA -> [2.75, -8.75].

Deferred

  • content IDs and digest verification;
  • replicated or leaderless namespace authority;
  • concurrent-orchestrator fencing, partition recovery, and automatic failover;
  • multiple providers or replicas;
  • payload durability across source-node loss;
  • caching and spilling;
  • lazy page-fault-backed mappings;
  • partial range reads;
  • HTTP and object-store sources;
  • SPSC stream namespace matching;
  • topics, fanout, and SPMC streams;
  • GPU-native blob realizations.

Final MVP boundary

The MVP is intentionally narrow:

A restartable singleton orchestrator durably owns a centralized actor registry that maps pointer-like paths to current fixed-byte sources. Namespace-dependent operations wait while that authority restarts; already-selected independent transfers and returned blobs do not depend on it. Disk registration is one operation. Python dereferences by path. Swactor resolves one source, performs the complete transfer into the existing sealed arena backing, and returns a read-only blob without exposing placement or recovery mechanics.

Completion status

The remaining MVP work is complete.

Delivered behavior

  • Public DataPlaneControl registers disk sources through blob::file(...); Myelin supplies only cluster publication, discovery, and Iroh transport composition.
  • A file registered on the authority node is readable by a session on a remote node.
  • A blob committed by one session is readable by an independent remote session.
  • Committed publication ownership detaches from the producer session. Producer close does not unregister it.
  • Replacement or unregister retires the source after active transfers, then reclaims the source actor and arena lease.
  • Tinygrad CUDA output and the temporary result socket behavior remain unchanged.

Verification coverage

  • crates/data-plane/tests/namespace_host_read_guarantees.rs deterministically covers public file registration, committed publication lifetime across producer close, and source reclamation after unregister.
  • apps/myelin/src/tests/data_namespace_guarantees.rs composes two real Iroh-backed nodes and covers remote control registration reads, remote session publication reads, producer close, and unregister reclamation.
  • The data-plane and Iroh blob-transfer suites cover namespace, source, transfer, exact-length, cancellation, and lease behavior.
  • The compiled Python extension suite covers BlobView buffer lifetime, write publication, the temporary output socket, real process attachment, and the Tinygrad CUDA result [2.75, -8.75].

Final code architecture

crates/data-plane
├── control
│   Reusable namespace service and public control API
├── namespace + namespace_store
│   Namespace semantics and durability
├── source + blob_transfer
│   Source lifetime and transport-neutral movement
└── host + data_plane + blob
    Reusable arena, transfer, publication, and mapping machinery

apps/myelin
├── data_namespace
│   Cluster-level discovery, publication, and Iroh composition
└── job_data_plane
    Creates data-plane sessions for managed jobs