swactor/crates/datastream/src/emit.rs

171 lines
4.7 KiB
Rust
Raw Normal View History

//! Generic datastream emission helpers.
use std::sync::Arc;
use std::sync::OnceLock;
use swactor::actor::ActorAddress;
use swactor::process_observer::ProcessOutputObserver;
use swactor::runtime::Runtime;
enforce datastream telemetry-only invariant: ban frame types from control code The datastream is metrics/logging only; control decisions must never branch on a frame. This was a recurring cultural problem with no structural enforcement. This change makes it a compile-time and CI-enforced fact. datastream crate (lib.rs): - Stop re-exporting Frame, DatastreamEvent, FrameDelivery at crate root. is now a compile error (E0425). These types live only in datastream::frame::* and are documented as the observer surface. - Safe identity types (ChannelId, StreamId, Position, Record, etc.) remain re-exported at root for producer-side callers. orchestration/app.rs: - Extracted all frame-touching code (CollectedDatastreamFrame, drain_datastream_connections, update_load_progress_from_frame, drain_frames, archive_collected_frame, pump, OrchDatastream, DashboardSupport) into two new observability modules: frame_collector.rs and orch_datastream.rs. - The orchestrator now interacts through a FrameCollector whose drain/drain_with_progress methods take closures; it never names Frame, DatastreamEvent, or CollectedDatastreamFrame. - StageLoadProgress (the one control-relevant signal previously scraped from frame payloads) is extracted inside FrameCollector and handed to the control loop as plain data. xtask: - New check-telemetry-isolation command scans control-plane modules (orchestration/, distribution/, data-plane/, provisioning/) for forbidden frame-type references and fails the build if any are found. Verified: workspace builds (myelin + dashboard feature), datastream 29 tests pass, myelin 64 lib tests pass, check-telemetry-isolation passes clean. Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-08-12 12:14:57 +00:00
use crate::frame::{ChannelId, Frame, Lifetime, NodeId, StreamId};
use crate::record::Record;
use super::mux::Mux;
use super::wire::{DatastreamFrame, encode_delivery};
2026-07-18 09:38:16 +00:00
/// Legacy sink for frames after mux drain has assigned positions.
pub trait FrameSink: Send {
2026-07-18 09:38:16 +00:00
/// Ship one positioned frame for `stream`. Best-effort: a sink may drop.
fn ship(&mut self, stream: &StreamId, frame: &Frame);
}
/// Static identity a node needs to build its mux.
pub struct EmitterConfig {
pub node_hex: String,
pub life: u64,
pub mux_capacity: usize,
}
struct MuxProcObserver {
mux: Arc<Mux>,
channel_for: Arc<dyn Fn(&str, bool) -> ChannelId + Send + Sync>,
}
impl ProcessOutputObserver for MuxProcObserver {
fn on_output(&self, label: &str, is_stderr: bool, data: &[u8]) {
self.mux
.submit((self.channel_for)(label, is_stderr), data.to_vec());
}
}
/// Legacy per-node emitter retained while runtime callsites move to
/// [`crate::DatastreamEndpoint`]. New code should register channels on the
/// endpoint and submit through [`crate::DatastreamProducer`].
pub struct DatastreamEmitter {
stream_id: StreamId,
mux: Arc<Mux>,
sink: Box<dyn FrameSink>,
}
impl DatastreamEmitter {
pub fn new(cfg: EmitterConfig, sink: Box<dyn FrameSink>) -> Self {
let stream_id = StreamId::new(NodeId::new(&cfg.node_hex), Lifetime(cfg.life));
let mux = Arc::new(Mux::new(stream_id.clone(), cfg.mux_capacity));
Self {
stream_id,
mux,
sink,
}
}
pub fn stream_id(&self) -> &StreamId {
&self.stream_id
}
pub fn mux(&self) -> &Arc<Mux> {
&self.mux
}
pub fn assigned(&self) -> u64 {
self.mux.assigned()
}
pub fn dropped(&self) -> u64 {
self.mux.dropped()
}
2026-07-18 09:38:16 +00:00
pub fn submit_record<R: Record>(&self, channel: ChannelId, record: &R) -> bool {
self.mux.submit(channel, record.encode())
}
2026-07-18 09:38:16 +00:00
pub fn submit_text(&self, channel: ChannelId, text: impl AsRef<[u8]>) -> bool {
self.mux.submit(channel, text.as_ref().to_vec())
}
2026-07-18 09:38:16 +00:00
pub fn submit_bytes(&self, channel: ChannelId, bytes: Vec<u8>) -> bool {
self.mux.submit(channel, bytes)
}
2026-07-18 09:38:16 +00:00
pub fn submit_text_owned(&self, channel: ChannelId, text: String) -> bool {
self.mux.submit(channel, text.into_bytes())
}
refactor(process): process-manager cleanup Replace the driver/session/action/event abstraction with a single OS-process supervisor thread and a minimal lifecycle-only public API. - supervisor: add a dedicated swactor-process-supervisor thread that owns the child, runs it with null stdio, wakes via an eventfd plus poll(2), reaps with waitpid(WNOHANG), and escalates SIGTERM to SIGKILL after a deadline, reporting only lifecycle ThreadEvents over a SegQueue plus wake channel - actor: collapse ProcessActor<D> into a non-generic state machine (Spawning/Running/Stopping/Done) that owns the supervisor handle, drains events on SupervisorWake, forwards lifecycle as ProcessOutput, and triggers shutdown_now in on_stop - lifecycle: add ProcessOutputConfig (Disabled/DatastreamMirror) with a JSON proc.<label>.lifecycle mirror (schema swactor_process.lifecycle.v1), command-basename label derivation/sanitization, and an RAII reservation registry preventing duplicate channels - message/types/spawn/lib: trim the API — ProcessCommand is now only Stop { kill_after }, ProcessOutput covers Started/SpawnFailed/Exited/Error, ProcessSpec keeps command/args/env/working_dir/label; re-export spawn_local_process/send_process_command and drop the custom-driver spawn_process - removed: delete the action/event/local/mock/session modules and the ProcessDriver/ProcessWaker/EventQueue/PtySize/ProcessMode types plus the old test suite (actor_scenarios, e2e_process, local_driver, proptest_session, session_scenarios); add public_api_stage1/2 tests and the SWACTOR_MANAGED_PROCESS_SPEC.md - swactor core: demote ProcessOutputObserver to a legacy/custom adapter (no longer auto-attached), remove Runtime::set_process_output_observer and Ctx::process_output_observer, and add the datastream dependency to the process crate for the mirror Signed-off-by: Zachery Aaron Shores-Chmielewski <zacheryasc@gmail.com>
2026-07-18 11:21:34 +00:00
/// Build a legacy/custom process-output observer that writes chunks into this mux.
pub fn process_observer_with<F>(&self, channel_for: F) -> Arc<dyn ProcessOutputObserver>
where
F: Fn(&str, bool) -> ChannelId + Send + Sync + 'static,
{
Arc::new(MuxProcObserver {
mux: self.mux.clone(),
channel_for: Arc::new(channel_for),
})
}
pub fn tick(&mut self) {
for frame in self.mux.drain() {
self.sink.ship(&self.stream_id, &frame);
}
}
pub fn event_sink(&self) -> DatastreamEventSink {
DatastreamEventSink {
mux: self.mux.clone(),
}
}
}
/// A thread-safe submit handle. Legacy; prefer [`crate::DatastreamProducer`].
#[derive(Clone)]
pub struct DatastreamEventSink {
mux: Arc<Mux>,
}
impl DatastreamEventSink {
2026-07-18 09:38:16 +00:00
pub fn submit_record<R: Record>(&self, channel: ChannelId, record: &R) -> bool {
self.mux.submit(channel, record.encode())
}
2026-07-18 09:38:16 +00:00
pub fn submit_text(&self, channel: ChannelId, text: impl AsRef<[u8]>) -> bool {
self.mux.submit(channel, text.as_ref().to_vec())
}
2026-07-18 09:38:16 +00:00
pub fn submit_bytes(&self, channel: ChannelId, bytes: Vec<u8>) -> bool {
self.mux.submit(channel, bytes)
}
2026-07-18 09:38:16 +00:00
pub fn submit_text_owned(&self, channel: ChannelId, text: String) -> bool {
self.mux.submit(channel, text.into_bytes())
}
}
/// A sink that drops everything.
pub struct NoopSink;
impl FrameSink for NoopSink {
fn ship(&mut self, _stream: &StreamId, _frame: &Frame) {}
}
/// Legacy swactor frame sink retained until MVP runtime cutover removes it.
pub struct ClusterFrameSink {
rt: Runtime,
sink: Arc<OnceLock<ActorAddress>>,
}
impl ClusterFrameSink {
pub fn new(rt: Runtime, sink: Arc<OnceLock<ActorAddress>>) -> Self {
Self { rt, sink }
}
}
impl FrameSink for ClusterFrameSink {
fn ship(&mut self, stream: &StreamId, frame: &Frame) {
if let Some(addr) = self.sink.get() {
let _ = self.rt.send_to(
*addr,
DatastreamFrame {
payload: encode_delivery(stream, frame),
},
);
}
}
}