2026-06-05 07:25:43 +00:00
|
|
|
//! The per-node telemetry **datastream** (see `DATASTREAM_SPEC.md`).
|
|
|
|
|
//!
|
enforce datastream telemetry-only invariant: ban frame types from control code
The datastream is metrics/logging only; control decisions must never branch
on a frame. This was a recurring cultural problem with no structural
enforcement. This change makes it a compile-time and CI-enforced fact.
datastream crate (lib.rs):
- Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root.
is now a compile error (E0425). These types live
only in datastream::frame::* and are documented as the observer surface.
- Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain
re-exported at root for producer-side callers.
orchestration/app.rs:
- Extracted all frame-touching code (CollectedDatastreamFrame,
drain_datastream_connections, update_load_progress_from_frame,
drain_frames, archive_collected_frame, pump, OrchDatastream,
DashboardSupport) into two new observability modules:
frame_collector.rs and orch_datastream.rs.
- The orchestrator now interacts through a FrameCollector whose
drain/drain_with_progress methods take closures; it never names Frame,
DatastreamEvent, or CollectedDatastreamFrame.
- StageLoadProgress (the one control-relevant signal previously scraped
from frame payloads) is extracted inside FrameCollector and handed to
the control loop as plain data.
xtask:
- New check-telemetry-isolation command scans control-plane modules
(orchestration/, distribution/, data-plane/, provisioning/) for
forbidden frame-type references and fails the build if any are found.
Verified: workspace builds (myelin + dashboard feature), datastream 29
tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes
clean.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
|
|
|
//! A deliberately dumb pipe: producers dump bytes tagged with a stream-local
|
2026-07-18 09:38:16 +00:00
|
|
|
//! channel id, a single per-node mux accepts those bytes and assigns canonical
|
|
|
|
|
//! positions during drain, the endpoint broadcasts catalog-aware events to
|
|
|
|
|
//! subscribers, ingest reconstructs streams by position, and views are
|
|
|
|
|
//! read-time projections over stored frames. Nothing between a producer and a
|
2026-06-05 07:25:43 +00:00
|
|
|
//! view interprets the payload.
|
|
|
|
|
//!
|
enforce datastream telemetry-only invariant: ban frame types from control code
The datastream is metrics/logging only; control decisions must never branch
on a frame. This was a recurring cultural problem with no structural
enforcement. This change makes it a compile-time and CI-enforced fact.
datastream crate (lib.rs):
- Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root.
is now a compile error (E0425). These types live
only in datastream::frame::* and are documented as the observer surface.
- Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain
re-exported at root for producer-side callers.
orchestration/app.rs:
- Extracted all frame-touching code (CollectedDatastreamFrame,
drain_datastream_connections, update_load_progress_from_frame,
drain_frames, archive_collected_frame, pump, OrchDatastream,
DashboardSupport) into two new observability modules:
frame_collector.rs and orch_datastream.rs.
- The orchestrator now interacts through a FrameCollector whose
drain/drain_with_progress methods take closures; it never names Frame,
DatastreamEvent, or CollectedDatastreamFrame.
- StageLoadProgress (the one control-relevant signal previously scraped
from frame payloads) is extracted inside FrameCollector and handed to
the control loop as plain data.
xtask:
- New check-telemetry-isolation command scans control-plane modules
(orchestration/, distribution/, data-plane/, provisioning/) for
forbidden frame-type references and fails the build if any are found.
Verified: workspace builds (myelin + dashboard feature), datastream 29
tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes
clean.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
|
|
|
//! ## Producer vs observer surface
|
|
|
|
|
//!
|
|
|
|
|
//! This crate has two surfaces:
|
|
|
|
|
//!
|
|
|
|
|
//! - **Producer** — re-exported at the crate root ([`DatastreamEndpoint`],
|
|
|
|
|
//! [`DatastreamProducer`], [`Record`], [`ChannelId`], [`StreamId`], …).
|
|
|
|
|
//! Everything control-plane and actor code needs to *emit* telemetry.
|
|
|
|
|
//!
|
|
|
|
|
//! - **Observer** — in submodules ([`frame::Frame`], [`frame::DatastreamEvent`],
|
|
|
|
|
//! [`store::Store`], [`views`], [`ingest::Consumer`]). Everything a sink
|
|
|
|
|
//! (dashboard, archive, transport) needs to *read* telemetry.
|
|
|
|
|
//!
|
|
|
|
|
//! The crate root deliberately does **not** re-export [`frame::Frame`] or
|
|
|
|
|
//! [`frame::DatastreamEvent`]. `use datastream::Frame` is a compile error; the
|
|
|
|
|
//! full path `datastream::frame::Frame` compiles but is banned in control-plane
|
|
|
|
|
//! modules by `cargo xtask check-telemetry-isolation`.
|
|
|
|
|
//!
|
2026-06-05 07:25:43 +00:00
|
|
|
//! ```text
|
2026-06-23 15:42:28 +00:00
|
|
|
//! producers (caller-owned records + text)
|
2026-07-18 09:38:16 +00:00
|
|
|
//! │ bytes tagged by registered ChannelId
|
2026-06-05 07:25:43 +00:00
|
|
|
//! ▼
|
2026-07-18 09:38:16 +00:00
|
|
|
//! endpoint / catalog → [`endpoint`]
|
|
|
|
|
//! │ channel metadata + producer handles
|
2026-06-05 07:25:43 +00:00
|
|
|
//! ▼
|
2026-07-18 09:38:16 +00:00
|
|
|
//! per-node MUX → [`mux::Mux`]
|
|
|
|
|
//! │ positioned [`frame::Frame`]s
|
2026-06-05 07:25:43 +00:00
|
|
|
//! ▼
|
2026-07-18 09:38:16 +00:00
|
|
|
//! endpoint fanout → [`endpoint::DeliveryFanout`]
|
|
|
|
|
//! │ catalog-aware events, maybe dropped per subscriber
|
2026-06-05 07:25:43 +00:00
|
|
|
//! ▼
|
2026-07-18 09:38:16 +00:00
|
|
|
//! ingest / store → [`ingest`], [`store`]
|
|
|
|
|
//! │ position-keyed frame truth
|
2026-06-05 07:25:43 +00:00
|
|
|
//! ▼
|
2026-07-18 09:38:16 +00:00
|
|
|
//! views → [`views`]
|
2026-06-05 07:25:43 +00:00
|
|
|
//! ```
|
|
|
|
|
//!
|
2026-07-18 09:38:16 +00:00
|
|
|
//! The data model ([`frame`]), extension contract ([`record`]), endpoint/fanout
|
|
|
|
|
//! seam ([`endpoint`]), and compatibility wire helpers ([`wire`]) are the seams
|
|
|
|
|
//! tests observe. Channel meanings live in producer/consumer crates, not in a
|
|
|
|
|
//! datastream-wide global registry.
|
2026-06-05 07:25:43 +00:00
|
|
|
|
2026-06-06 17:53:25 +00:00
|
|
|
pub mod emit;
|
2026-06-25 12:30:18 +00:00
|
|
|
pub mod endpoint;
|
2026-06-05 07:25:43 +00:00
|
|
|
pub mod frame;
|
2026-07-07 10:40:02 +00:00
|
|
|
pub mod hardware;
|
2026-06-23 15:42:28 +00:00
|
|
|
pub mod health;
|
2026-06-05 07:25:43 +00:00
|
|
|
pub mod ingest;
|
|
|
|
|
pub mod mux;
|
2026-07-12 06:14:34 +00:00
|
|
|
pub mod publisher_actor;
|
2026-06-23 15:42:28 +00:00
|
|
|
pub mod record;
|
2026-06-09 09:29:07 +00:00
|
|
|
pub mod sink_actor;
|
2026-06-05 07:25:43 +00:00
|
|
|
pub mod store;
|
|
|
|
|
pub mod transport;
|
|
|
|
|
pub mod views;
|
|
|
|
|
pub mod wire;
|
|
|
|
|
|
enforce datastream telemetry-only invariant: ban frame types from control code
The datastream is metrics/logging only; control decisions must never branch
on a frame. This was a recurring cultural problem with no structural
enforcement. This change makes it a compile-time and CI-enforced fact.
datastream crate (lib.rs):
- Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root.
is now a compile error (E0425). These types live
only in datastream::frame::* and are documented as the observer surface.
- Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain
re-exported at root for producer-side callers.
orchestration/app.rs:
- Extracted all frame-touching code (CollectedDatastreamFrame,
drain_datastream_connections, update_load_progress_from_frame,
drain_frames, archive_collected_frame, pump, OrchDatastream,
DashboardSupport) into two new observability modules:
frame_collector.rs and orch_datastream.rs.
- The orchestrator now interacts through a FrameCollector whose
drain/drain_with_progress methods take closures; it never names Frame,
DatastreamEvent, or CollectedDatastreamFrame.
- StageLoadProgress (the one control-relevant signal previously scraped
from frame payloads) is extracted inside FrameCollector and handed to
the control loop as plain data.
xtask:
- New check-telemetry-isolation command scans control-plane modules
(orchestration/, distribution/, data-plane/, provisioning/) for
forbidden frame-type references and fails the build if any are found.
Verified: workspace builds (myelin + dashboard feature), datastream 29
tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes
clean.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
|
|
|
// ── Producer surface (re-exported at root; safe for control-plane code) ──
|
|
|
|
|
|
2026-06-25 12:30:18 +00:00
|
|
|
pub use endpoint::{
|
2026-07-12 06:14:34 +00:00
|
|
|
CatalogSnapshot, ChannelRegistrationError, DatastreamEndpoint, DatastreamProducer,
|
|
|
|
|
DatastreamSnapshot, DatastreamSubscription, DeliveryFanout, EndpointTick, SubscriberSnapshot,
|
enforce datastream telemetry-only invariant: ban frame types from control code
The datastream is metrics/logging only; control decisions must never branch
on a frame. This was a recurring cultural problem with no structural
enforcement. This change makes it a compile-time and CI-enforced fact.
datastream crate (lib.rs):
- Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root.
is now a compile error (E0425). These types live
only in datastream::frame::* and are documented as the observer surface.
- Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain
re-exported at root for producer-side callers.
orchestration/app.rs:
- Extracted all frame-touching code (CollectedDatastreamFrame,
drain_datastream_connections, update_load_progress_from_frame,
drain_frames, archive_collected_frame, pump, OrchDatastream,
DashboardSupport) into two new observability modules:
frame_collector.rs and orch_datastream.rs.
- The orchestrator now interacts through a FrameCollector whose
drain/drain_with_progress methods take closures; it never names Frame,
DatastreamEvent, or CollectedDatastreamFrame.
- StageLoadProgress (the one control-relevant signal previously scraped
from frame payloads) is extracted inside FrameCollector and handed to
the control loop as plain data.
xtask:
- New check-telemetry-isolation command scans control-plane modules
(orchestration/, distribution/, data-plane/, provisioning/) for
forbidden frame-type references and fails the build if any are found.
Verified: workspace builds (myelin + dashboard feature), datastream 29
tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes
clean.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
|
|
|
SubscriptionId,
|
2026-07-12 06:14:34 +00:00
|
|
|
};
|
|
|
|
|
pub use frame::{
|
|
|
|
|
ChannelContent, ChannelContentKind, ChannelDescriptor, ChannelFilter, ChannelId, ChannelRef,
|
enforce datastream telemetry-only invariant: ban frame types from control code
The datastream is metrics/logging only; control decisions must never branch
on a frame. This was a recurring cultural problem with no structural
enforcement. This change makes it a compile-time and CI-enforced fact.
datastream crate (lib.rs):
- Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root.
is now a compile error (E0425). These types live
only in datastream::frame::* and are documented as the observer surface.
- Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain
re-exported at root for producer-side callers.
orchestration/app.rs:
- Extracted all frame-touching code (CollectedDatastreamFrame,
drain_datastream_connections, update_load_progress_from_frame,
drain_frames, archive_collected_frame, pump, OrchDatastream,
DashboardSupport) into two new observability modules:
frame_collector.rs and orch_datastream.rs.
- The orchestrator now interacts through a FrameCollector whose
drain/drain_with_progress methods take closures; it never names Frame,
DatastreamEvent, or CollectedDatastreamFrame.
- StageLoadProgress (the one control-relevant signal previously scraped
from frame payloads) is extracted inside FrameCollector and handed to
the control loop as plain data.
xtask:
- New check-telemetry-isolation command scans control-plane modules
(orchestration/, distribution/, data-plane/, provisioning/) for
forbidden frame-type references and fails the build if any are found.
Verified: workspace builds (myelin + dashboard feature), datastream 29
tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes
clean.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
|
|
|
Lifetime, NodeId, Position, SourceFilter, StreamDescriptor, StreamId, StreamOrigin,
|
|
|
|
|
SubscriptionRequest,
|
2026-06-25 12:30:18 +00:00
|
|
|
};
|
2026-06-05 07:25:43 +00:00
|
|
|
pub use mux::Mux;
|
2026-07-12 06:14:34 +00:00
|
|
|
pub use publisher_actor::{
|
|
|
|
|
DATASTREAM_PUBLISHER_NAME, DatastreamPublisherActor, DatastreamPublisherMsg,
|
|
|
|
|
DatastreamSubscribe, register_datastream_publisher_codec,
|
|
|
|
|
};
|
2026-06-23 15:42:28 +00:00
|
|
|
pub use record::{ChannelKind, ChannelRegistry, Record};
|
|
|
|
|
pub use sink_actor::{DATASTREAM_SINK_NAME, DatastreamSink};
|
enforce datastream telemetry-only invariant: ban frame types from control code
The datastream is metrics/logging only; control decisions must never branch
on a frame. This was a recurring cultural problem with no structural
enforcement. This change makes it a compile-time and CI-enforced fact.
datastream crate (lib.rs):
- Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root.
is now a compile error (E0425). These types live
only in datastream::frame::* and are documented as the observer surface.
- Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain
re-exported at root for producer-side callers.
orchestration/app.rs:
- Extracted all frame-touching code (CollectedDatastreamFrame,
drain_datastream_connections, update_load_progress_from_frame,
drain_frames, archive_collected_frame, pump, OrchDatastream,
DashboardSupport) into two new observability modules:
frame_collector.rs and orch_datastream.rs.
- The orchestrator now interacts through a FrameCollector whose
drain/drain_with_progress methods take closures; it never names Frame,
DatastreamEvent, or CollectedDatastreamFrame.
- StageLoadProgress (the one control-relevant signal previously scraped
from frame payloads) is extracted inside FrameCollector and handed to
the control loop as plain data.
xtask:
- New check-telemetry-isolation command scans control-plane modules
(orchestration/, distribution/, data-plane/, provisioning/) for
forbidden frame-type references and fails the build if any are found.
Verified: workspace builds (myelin + dashboard feature), datastream 29
tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes
clean.
Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
|
|
|
|
|
|
|
|
// ── Observer surface (in submodules; NOT re-exported at root) ──
|
|
|
|
|
//
|
|
|
|
|
// frame::Frame, frame::DatastreamEvent, frame::FrameDelivery,
|
|
|
|
|
// store::Store, ingest::Consumer, views::*, transport::Delivery
|
|
|
|
|
//
|
|
|
|
|
// Access these via their module paths (e.g. `datastream::frame::Frame`).
|
|
|
|
|
// Control-plane modules must not import them — enforced by CI.
|