@causal-order/monitor

Health-aware buffering, replay, and operator monitoring for the causal-order stack.

The supported integration path is:

@causal-order/transport -> @causal-order/monitor -> @causal-order/dedupe -> causal-order

@causal-order/monitor owns buffering, degraded routing, and controlled replay between transport ingestion and downstream delivery. This lets the pipeline keep accepting events and preserve backlog when @causal-order/dedupe or causal-order becomes unavailable. Buffered events re-enter the downstream path through dedupe before causal ordering.

The monitor also owns the SQLite event reservoir behind SQLiteReservoir. This is separate from any SQLite adapter offered by @causal-order/persistence: persistence-owned adapters serve persistence concerns such as WALs, checkpoints, and component state, while the monitor reservoir serves bounded ingress buffering and replay. They have independent data, schemas, lifecycles, and APIs; @causal-order/persistence does not absorb or abstract the monitor reservoir.

Install

npm install @causal-order/monitor causal-order @causal-order/dedupe @causal-order/transport

@causal-order/monitor now uses the built-in node:sqlite module, so no separate SQLite package is required. causal-order, @causal-order/dedupe, and @causal-order/transport are expected alongside this package as peers.

What It Does

• buffers ingress events in SQLite so downstream outages do not immediately become data loss
• tracks health for transport, dedupe, and causal-order
• switches routing behavior when parts of the stack degrade or go offline
• replays buffered backlog through @causal-order/dedupe after recovery
• exposes snapshots and derived inspection state for operators and automation

Stability

Published version: v0.2.3.

When To Use It

Use this package when you already have a causal-order pipeline and want a monitor layer that can:

• keep ingesting while causal-order is offline
• throttle or bypass parts of the path when dedupe is unhealthy
• coordinate replay after recovery instead of mixing backlog and live flow loosely
• give operators a compact view of backlog, replay, retry, and health state

Package Model

The package gives you two main integration styles:

• TransportMonitorAdapter if you want a higher-level wrapper that calls your delivery handlers and manages replay pumping through that adapter surface
• createMonitorRuntime() or MonitorRuntime if you want direct control over ingress, health updates, replay, and storage

For most application integrations, prefer TransportMonitorAdapter.

Use MonitorRuntime when you intentionally want the lower-level/manual orchestration path.

These are now enforced as separate replay-orchestration styles on the same runtime instance:

• adapter-managed replay through TransportMonitorAdapter
• manual replay control through MonitorRuntime

If a runtime is adapter-managed, direct manual replay commands on that runtime now fail fast with ReplayOwnershipError instead of allowing mixed control ownership.

Advanced, inspection, and testing surfaces are documented in the sections below; they are secondary to the normal runtime and adapter paths.

Subpath Imports

The package exposes these official subpath entrypoints:

| Subpath | Purpose | | --- | --- | | @causal-order/monitor/config | Configuration loading and helpers; lightweight entrypoint. | | @causal-order/monitor/health | Health tracking; lightweight entrypoint. | | @causal-order/monitor/inspect | Snapshot inspection; lightweight entrypoint. | | @causal-order/monitor/replay | Replay coordination contracts. | | @causal-order/monitor/routing | Delivery routing; lightweight entrypoint. | | @causal-order/monitor/runtime | Lower-level manual runtime integration. | | @causal-order/monitor/storage | SQLite reservoir primitives. | | @causal-order/monitor/testing | Harness metadata; lightweight entrypoint. | | @causal-order/monitor/throttle | Throttle decisions; lightweight entrypoint. | | @causal-order/monitor/transport | Preferred higher-level adapter integration. | | @causal-order/monitor/types | TypeScript contracts; lightweight entrypoint. |

Root Migration Notes

In v0.1.3, selected advanced root exports are deprecated but remain available for compatibility. Their supported replacement is the corresponding published subpath.

This is a non-breaking migration path: existing root imports still work.

v0.1.3 root-to-subpath migrations:

• HealthTracker: @causal-order/monitor -> @causal-order/monitor/health
• ReplayCoordinator and ReplayBatch: @causal-order/monitor -> @causal-order/monitor/replay
• DeliveryRouter: @causal-order/monitor -> @causal-order/monitor/routing
• ThrottleController: @causal-order/monitor -> @causal-order/monitor/throttle

Example:

import { ReplayCoordinator } from "@causal-order/monitor/replay";

Quick Start

import { TransportMonitorAdapter } from "@causal-order/monitor";

const monitor = new TransportMonitorAdapter(
  {
    async deliverToDedupe(event, context) {
      await sendToDedupe(event, context);
    },
    async deliverToOrder(event, context) {
      await sendToCausalOrder(event, context);
    },
    async onBuffered(event, decision, rowId) {
      console.log("buffered", { rowId, action: decision.action, eventId: event.id });
    },
  },
  {
    reservoir: {
      databasePath: "./.causal-order-monitor/monitor.sqlite",
    },
  },
);

monitor.observeHeartbeat("transport");
monitor.observeHeartbeat("dedupe");
monitor.observeHeartbeat("causal-order");

await monitor.ingest({
  id: "evt-1001",
  nodeId: "transport-a",
  clock: {
    physicalTimeMs: BigInt(Date.now()),
  },
  payload: {
    entityId: "order-42",
    operation: "created",
  },
});

const snapshot = monitor.getInspectedSnapshot();
console.log(snapshot.operationalState);

Runtime Example

If you want lower-level control, use the runtime directly:

import { createMonitorRuntime } from "@causal-order/monitor";

const runtime = createMonitorRuntime({
  reservoir: {
    databasePath: "./.causal-order-monitor/monitor.sqlite",
  },
});

runtime.observeHeartbeat("transport");
runtime.observeHeartbeat("dedupe");
runtime.observeHeartbeat("causal-order");

const { rowId, decision } = runtime.ingestTransportEvent({
  id: "evt-1001",
  nodeId: "transport-a",
  clock: {
    physicalTimeMs: BigInt(Date.now()),
  },
  payload: {
    entityId: "order-42",
    operation: "created",
  },
});

if (decision.action === "accept" && decision.deliveryMode === "normal") {
  await sendToDedupe();
  runtime.acknowledgeIngressDelivery([rowId]);
}

Routing And Recovery Behavior

The monitor can move between a few main operating modes:

• normal: live flow goes through dedupe as usual
• dedupe_bypass_throttled: live flow can bypass dedupe with throttling when dedupe is unhealthy
• order_buffer_only: events are buffered while causal-order is offline
• full_outage_buffer: events stay buffered during broader outage conditions
• replay_through_dedupe: buffered backlog drains through dedupe during recovery

Recovery is intentionally conservative:

• the rolling SQLite buffer defaults to 4h
• the maximum dual-outage retention window defaults to 6h
• replay returns through @causal-order/dedupe
• replay waits for configured recovery confirmation before it starts draining after a real downstream recovery transition
• replay failure applies a retry backoff before trying again
• live flow stays gated during pre-replay confirmation, failed replay backoff, active replay drain, and post-replay confirmation

Ingress HTTP Semantics

If /monitor is sitting behind an HTTP ingress boundary, the intended status mapping is:

• accept -> 202 Accepted
• buffer_only -> 202 Accepted
• pause -> 503 Service Unavailable

That means:

• if the event was accepted into the monitor, even if it was only buffered in SQLite, the ingress contract should return 202
• if the monitor is refusing admission because it is in a true protective stop state, the ingress contract should return 503

The current monitor semantics do not use 429 Too Many Requests for ordinary buffering or protective-stop behavior.

Retention Semantics

The retention boundary is prune-driven, not ingest-driven.

In practical terms:

• the monitor does not immediately hard-reject ingress when the reservoir reaches the fullOutageMaxWindowMs ceiling
• it keeps accepting and buffering ingress while routing decisions such as buffer_only or pause express backpressure posture
• live forwarding stops when the active routing mode requires it
• older unacknowledged rows age out when pruning runs and are marked dead_letter once they pass the hard cutoff

So the current behavior is closer to “drop older buffered rows once they age past the ceiling” than “reject every new ingress immediately at the ceiling.”

When a pending row ages out, it transitions to dead_letter and starts its own evidence-retention clock. Delivered evidence is retained independently. Each call to pruneReservoir() marks at most pruneBatchSize pending rows and deletes at most pruneBatchSize eligible terminal rows, so larger cleanup sets require repeated calls.

Schema Compatibility

v0.2.0 established SQLite schema version 1 as an explicit persistence contract. v0.2.1 added deterministic restart and upgrade recovery. v0.2.2 migrates that format to schema version 2 so terminal evidence retention can be measured from the delivered or dead-letter transition.

v0.2.3 keeps schema version 2 and the v0.2.2 public surface unchanged while hardening crash, storage-failure, concurrency, payload-serialization, and shutdown behavior. Close is idempotent, post-close mutable work is rejected consistently, and accepted rows remain recoverable when shutdown overlaps delivery.

• new reservoirs record their schema version during transactional initialization
• compatible unversioned databases created by earlier monitor releases migrate transactionally without discarding rows
• current databases reopen without schema mutation
• newer, incomplete, and incompatible schemas fail before mutation with typed errors
• SQLiteReservoir.getSchemaInfo() and MonitorRuntime.getSchemaInfo() expose the current and latest supported schema versions
• file-backed reservoirs use WAL journaling with synchronous=FULL for higher write concurrency without reducing transaction synchronization

On process restart, persisted row state is authoritative:

• pending backlog automatically restores a queued replay posture and keeps live flow gated
• interrupted replaying claims return immediately to pending because their in-memory owner ended with the previous process
• an entirely retry-waiting backlog restores its failed/retry-waiting posture using the persisted absolute retry deadline and replay-attempt evidence
• delivered and dead-letter rows remain terminal and are not returned to normal replay eligibility
• replay ordering continues to use monitor ingest time and row identity
• persisted replay_sessions remain audit history; they do not override current row state when reconstructing the coordinator

Schema constants, information types, and compatibility errors are available through @causal-order/monitor/storage.

Package and schema versions are independent: a package update does not imply a database migration unless the schema version changes.

Schema version 2 adds terminal_at_ms. Existing terminal rows receive their original monitor-ingest time during migration; new delivered and dead-letter transitions record the transition time.

Configuration

For most deployments, the preferred bootstrap path is:

1. loadMonitorConfigFile() or loadMonitorConfigFromEnvironment() if you want an explicit config-loading step
2. createMonitorRuntimeFromFile() or createMonitorRuntimeFromEnvironment() if you want direct runtime bootstrapping
3. createTransportMonitorAdapterFromFile() or createTransportMonitorAdapterFromEnvironment() if you want the preferred adapter integration path with file/env config

The lower-level resolveMonitorConfig() and resolveMonitorConfigFromEnvironment() helpers are still public, but they are better treated as advanced composition tools than the first thing most deployers should reach for.

createDefaultMonitorConfig() returns the full configuration shape. Common settings:

• reservoir.databasePath: SQLite path, default ./.causal-order-monitor/monitor.sqlite
• reservoir.rollingBufferWindowMs: normal rolling retention window
• reservoir.fullOutageMaxWindowMs: hard retention ceiling during deeper outages
• transport.heartbeatGraceMs: heartbeat grace period before transport is considered stale
• reservoir.pruneBatchSize: maximum number of rows to dead-letter or delete in each SQLite prune batch
• reservoir.deliveredRetentionMs: delivered evidence retention, default 24h
• reservoir.deadLetterRetentionMs: dead-letter evidence retention, default 168h (7 days)
• reservoir.walAutoCheckpointPages: SQLite automatic WAL checkpoint threshold, default 1000 pages; set 0 to disable automatic checkpoints
• health.*.degradedAfterMs and health.*.offlineAfterMs: thresholds for each component
• throttle.*: ingress limits for each throttle tier
• replay.healthConfirmationHeartbeats: heartbeats required before replay resumes
• replay.pauseLiveFlowDuringReplay: whether live flow stays gated during drain
• replay.retryBackoffMs: delay before retrying a failed replay attempt

If you do not override reservoir.databasePath, the package now creates ./.causal-order-monitor/monitor.sqlite automatically on the host where the monitor runs.

By default, the monitor's internal now() clock is wall-clock anchored at startup and then advanced from a monotonic source so it does not move backward during the lifetime of the process. If you provide a custom now, you are taking responsibility for that time behavior.

On a server, prefer a host-local writable path such as /var/lib/causal-order-monitor/monitor.sqlite or another directory your service account owns. The monitor expects SQLite to run on a normal local filesystem. Avoid read-only mounts, synced workspace folders, and network filesystems unless you have validated them for SQLite locking and durability.

If the reservoir cannot start, the monitor now fails fast with a startup error that includes the resolved SQLite path and deployment guidance instead of only surfacing the raw SQLite exception.

Deep-Outage Disk I/O Sizing

During a sustained full_outage_buffer period, accepted ingress is persisted into SQLite and peak ingress can become sustained local write pressure for hours.

Plan host storage and throughput around the outage worst case, not the healthy-path average.

1. Capacity floor

Treat required local storage as at least:

peak accepted events/sec * average stored event bytes * 21,600 seconds

That 21,600 second window is the full 6h hard-outage horizon.

Example:

• 1,000 accepted events/sec
• 1 KB average stored event size
• minimum floor of about 21.6 GB

Treat that as a floor, not a full estimate. Real sizing should add headroom for SQLite metadata, indexes, journaling, filesystem slack, and recovery overlap.

2. Throughput and media choice

• size for peak accepted ingress writes per second across the full 4h to 6h outage window
• include SQLite and filesystem overhead such as journaling, checkpoint activity, and indexed writes
• prefer local SSD-class storage such as NVMe or provisioned SSD volumes
• avoid NFS, SMB, synced cloud folders, or other shared/network-backed storage for the live SQLite file
• leave headroom for prune work, replay recovery, and other host processes instead of sizing exactly to the expected steady-state ingress rate
• assume reconnect bursts and uneven node jitter can temporarily raise write pressure above the long-run average

3. Prune and expiry behavior

The 6h ceiling does not make /monitor immediately hard-reject new ingress just because older buffered rows have reached the maximum retention window.

Instead:

• routing decisions still determine whether the monitor continues to accept or buffer ingress
• older buffered rows age out when pruning runs
• rows that have crossed the hard cutoff are marked dead_letter
• in v0.1.1, prune performs this work in indexed batches instead of one large sweep

That means large expiry cliffs can still create predictable background write spikes. A reasonable operational target is to leave at least 20% I/O headroom so prune and live ingress can overlap without destabilizing the host, but that is a sizing guideline, not a guarantee.

Operationally, deployers should watch for:

• SQLite file growth that outpaces expected backlog math
• backlog age increasing faster than downstream recovery can drain it
• host-level disk queueing, latency spikes, or noisy-neighbor contention
• prune or replay phases overlapping with continued ingress during recovery

If the host cannot sustain that write profile, the monitor may still preserve correctness semantics while becoming operationally fragile under deep outage pressure.

You can also load deployer-facing settings from a JSON file:

import { loadMonitorConfigFile, createMonitorRuntime } from "@causal-order/monitor";

const config = loadMonitorConfigFile("monitor.config.json");
const runtime = createMonitorRuntime(config);

For deployments that do not want to hardcode the config path, the package also supports CAUSAL_ORDER_MONITOR_CONFIG:

import {
  createMonitorRuntime,
  loadMonitorConfigFromEnvironment,
} from "@causal-order/monitor";

const config = loadMonitorConfigFromEnvironment();
const runtime = createMonitorRuntime(config);

If you want to skip the separate load step entirely, the package now exposes convenience creators for both the runtime and the transport adapter:

import {
  createMonitorRuntimeFromEnvironment,
  createTransportMonitorAdapterFromFile,
} from "@causal-order/monitor";

const runtime = createMonitorRuntimeFromEnvironment();

const adapter = createTransportMonitorAdapterFromFile(
  {
    async deliverToDedupe(event, context) {
      await sendToDedupe(event, context);
    },
    async deliverToOrder(event, context) {
      await sendToCausalOrder(event, context);
    },
  },
  "/etc/causal-order/monitor.config.json",
);

Config precedence is:

1. explicit in-code config passed to resolveMonitorConfigFromEnvironment(inlineConfig, ...)
2. config file path from CAUSAL_ORDER_MONITOR_CONFIG
3. default monitor.config.json in the current working directory, if present
4. package defaults

If CAUSAL_ORDER_MONITOR_CONFIG is set, the monitor treats that as authoritative and fails clearly if the file cannot be loaded.

Supported duration values in JSON can be either integer milliseconds or strings like 15000ms, 30s, 5m, or 4h.

If you need the package's monotonic-backed default clock factory directly, createDefaultMonitorNow() is also available. Most consumers do not need to call it explicitly unless they are composing lower-level runtime config.

Example monitor.config.json:

{
  "reservoir": {
    "databasePath": "/var/lib/causal-order-monitor/monitor.sqlite",
    "rollingBufferWindowMs": "4h",
    "fullOutageMaxWindowMs": "6h",
    "pruneIntervalMs": "60s",
    "pruneBatchSize": 1000
  },
  "transport": {
    "heartbeatGraceMs": "15s",
    "reconnectBurstWindowMs": "30s",
    "sourceLabel": "@causal-order/transport"
  },
  "health": {
    "transport": {
      "degradedAfterMs": "10s",
      "offlineAfterMs": "30s"
    },
    "dedupe": {
      "degradedAfterMs": "10s",
      "offlineAfterMs": "30s"
    },
    "causalOrder": {
      "degradedAfterMs": "10s",
      "offlineAfterMs": "30s"
    }
  },
  "throttle": {
    "defaultTier": "open",
    "open": {
      "maxEventsPerSecond": 5000,
      "batchSize": 500
    },
    "slow": {
      "maxEventsPerSecond": 1000,
      "batchSize": 200
    },
    "verySlow": {
      "maxEventsPerSecond": 250,
      "batchSize": 50
    },
    "paused": {
      "maxEventsPerSecond": 0,
      "batchSize": 0
    }
  },
  "replay": {
    "healthConfirmationHeartbeats": 2,
    "pauseLiveFlowDuringReplay": true,
    "retryBackoffMs": "5s"
  }
}

Inspection And Operator State

For raw state, use:

• getSnapshot()
• getReplaySnapshot()
• getReservoirStats()
• getReservoirLifecycleStats() for delivered and dead-letter evidence counts
• checkpointReservoirWal() for explicit WAL maintenance results

For operator-facing state, use:

• getInspectedSnapshot()
• inspectMonitorSnapshot(snapshot)

The inspected snapshot is the easiest entry point when you want a quick read on:

• current operational posture
• whether live flow is gated
• whether replay is waiting on pre-replay confirmation, retry-waiting, actively draining, or in post-replay confirmation
• backlog size and replay progress
• whether operator attention is needed

Event Timing Evidence

v0.2.2 exports the type-only MonitorEventTimingEvidence contract for handing neutral timing facts to application-owned policy code:

import type { MonitorEventTimingEvidence } from "@causal-order/monitor/types";

const monitorIngestTimeMs = event.ingestedAt;
if (monitorIngestTimeMs === undefined) {
  throw new Error("Monitor ingest time must be captured before producing timing evidence");
}

const observedTimeMs = BigInt(Date.now());
const timing: MonitorEventTimingEvidence = {
  eventTimeMs: event.clock.physicalTimeMs,
  monitorIngestTimeMs,
  observedTimeMs,
  latenessMs: observedTimeMs - event.clock.physicalTimeMs,
  causalMetadata: {
    eventId: event.id,
    nodeId: event.nodeId,
  },
};

The same type is also available from the package root:

import type { MonitorEventTimingEvidence } from "@causal-order/monitor";

latenessMs is signed so negative values retain clock-skew evidence instead of being silently classified. The interface introduces no runtime behavior, storage requirement, processing horizon, threshold, or policy preset.

Monitor supplies a shared evidence shape; it does not decide what "too late" means. A company's development team or consultants remain responsible for accepting, flagging, quarantining, compensating, or discarding an event. Monitor's SQLite dead_letter state is a separate operational recovery outcome for buffered rows that exceed retention, not a business-lateness decision.

Public Types

Key exported types include:

• MonitorConfig
• MonitorIngressEvent
• MonitorIngressDecision
• MonitorEventTimingEvidence
• MonitorSnapshot
• ReplaySessionSnapshot
• ReservoirStats
• InspectedMonitorSnapshot

Advanced And Testing Surfaces

The package also publishes specialist surfaces that are supported but secondary to the main runtime and adapter path. Deprecated advanced root exports and their replacements are listed once under Root Migration Notes.

Non-deprecated advanced runtime-building surface:

• SQLiteReservoir

Testing-oriented surfaces:

• monitorHarnessArtifacts
• monitorHarnessScenarios
• harness metadata types through @causal-order/monitor/testing

Compatibility-only metadata surfaces:

• monitorPackageVersion
• monitorImplementationStatus

These remain part of the v0.2.2 public compatibility contract and continue to follow the package's semantic-versioning guarantees.

Node Support

• Node.js >=22.13.0
• ESM package output

Release And Repository Documentation

• Release history
• Roadmap
• Validation guide and evidence
• Persistence operations

License

MIT