sqlite-recovery-envelope

An in-process, health-aware active architectural shield and local SQLite recovery envelope for fragile downstream systems.

               ┌──────────────────────────────┐
               │   sqlite-recovery-envelope   │
               │                              │
 Ingress ─────►│   ┌────────┐    ┌────────┐   │─────► Target Handler
  / Jobs /     │   │ Ingress│    │ Gated  │   │        / API / Queue /
  Webhooks     │   │ Router │    │ Replay │   │        Workflow
               └───┴───┬────┴────┴────▲───┴───┘
                       │              │
                       ▼              │
                 ┌────────────────────┴┐
                 │ Local SQLite Buffer │
                 │  (Zero-Dependency)  │
                 └─────────────────────┘

sqlite-recovery-envelope sits directly on an ingress or handoff boundary. It transparently intercepts application work, watches one or more downstream dependencies, decides whether work should pass through, buffer locally, or pause, and preserves accepted work in an embedded SQLite reservoir when immediate delivery is unsafe.

Imagine your application relies on a shaky third-party API or an internal microservice that goes down from time to time. When that downstream service crashes while you are still receiving webhooks or processing user API requests, you usually end up choosing between two bad options:

• hold work in memory with a retry loop and lose it on process restart or crash
• bring in heavier infrastructure such as SQS, RabbitMQ, or Redis-backed job queues just to get basic durability and replay

sqlite-recovery-envelope is the third option. It is a zero-dependency, local-disk buffer that lives inside your Node.js process and behaves like an intelligent toll booth at your application boundary.

What It Does

sqlite-recovery-envelope is for applications that need to keep accepting ingress even when a downstream API, queue, workflow, or handler becomes unsafe to call directly.

It gives you:

• a host-local SQLite reservoir with fail-fast WAL startup validation
• routing decisions for normal, degraded_bypass, buffer_only, gated_replay, and protective_stop
• replay safety gates based on heartbeat confirmation, stabilization windows, and bounded retry backoff
• a low-level runtime for custom orchestration and a small high-level adapter for common embedding cases

How It Works

1. inspectIngress: When a request, job, or webhook hits your boundary, you pass the event details to the library.
2. Decision engine: The library checks the real-time health of your monitored downstream dependencies.
3. normal: If everything is healthy, work can pass straight through.
4. buffer_only: If a downstream dependency is unsafe, the payload is written to a local SQLite file through Node 22's native node:sqlite engine so your application can safely return 202 Accepted while the event remains durable on disk.
5. gated_replay: When the downstream path recovers, the library does not immediately dump the entire backlog and create a stampede. It waits for healthy confirmation heartbeats, enforces stabilization windows when configured, and drains stored work in controlled batches.
6. protective_stop: If local safety thresholds are breached, the library signals that your application should stop accepting work and return 503 Service Unavailable to protect the host machine and the recovery path.

Why It Is Different

• Zero infrastructure overhead: No separate broker, queue service, or sidecar is required. The reservoir stays on host-local disk in SQLite WAL mode, and startup fails fast if the configured filesystem cannot actually support WAL safely.
• Separated timestamps: Internal replay and retention bookkeeping run on a monotonic clock, so wall-clock drift or manual system time changes do not corrupt recovery sequencing.
• Built for modern Node: The package requires Node.js >= 22.13.0 and relies on the built-in node:sqlite engine, so installation avoids native addon compilation steps.
• Broker-style recovery behavior without broker-style weight: You get local durability, bounded replay, and operational safety gates in a lightweight embedded library instead of a separate distributed system.

Install

npm install sqlite-recovery-envelope

Requires Node.js >= 22.13.0 for built-in, native SQLite engine support.

Quick Start

If you want a wrapper around your delivery hook, start with DeliveryRecoveryAdapter:

import { DeliveryRecoveryAdapter } from "sqlite-recovery-envelope";

const shield = new DeliveryRecoveryAdapter(
  {
    async deliver(event) {
      await fetch("https://internal-fragile-api/v1/work-items", {
        method: "POST",
        body: JSON.stringify(event.payload),
        headers: { "Content-Type": "application/json" },
      });
    },
    async onBuffered(event, decision, rowId, result) {
      console.warn(
        `Buffered ${event.id} as ${result.reasonCode} with handle row ${rowId}: ${result.reason}`,
      );
    },
    async onPaused(event, decision, result) {
      console.error(`Refused ${event.id}: ${result.reason}`);
    },
  },
  {
    monitoredServices: ["primary-api", "analytics-sink"],
    reservoir: {
      databasePath: "./.recovery-reservoir/storage.sqlite",
      rollingBufferWindowMs: "4h",
      fullOutageMaxWindowMs: "6h",
    },
  },
);

shield.observeHeartbeat("primary-api");
shield.observeHeartbeat("analytics-sink");

const result = await shield.ingest({
  id: "evt_10024A",
  timestamp: BigInt(Date.now()),
  payload: { userId: "usr_99X", action: "checkout_completed" },
});

if (result.outcome === "paused") {
  // translate directly to a 503 boundary response
}

const httpStatus = result.statusCode;
const ingressHandle = result.handle;

await shield.pumpReplay();

Boundary Integration Patterns

The high-level adapter is meant to let outer boundary code translate envelope posture without rebuilding the same decision shell in every controller or worker.

For adapter-facing code, treat result.handle as the public accepted-work reference. result.rowId still exists for compatibility, but new boundary code should avoid depending on the raw storage field directly. The adapter hooks also receive the structured boundary result, so callback code can use result.reasonCode, result.statusCode, and result.handle without re-deriving them from lower-level arguments.

HTTP/API handler shape:

const result = await shield.ingest({
  id: requestId,
  timestamp: BigInt(Date.now()),
  payload: req.body,
});

if (result.outcome === "delivered") {
  return res.status(result.statusCode).json({
    accepted: true,
    delivery: "live",
    handle: result.handle,
  });
}

if (result.outcome === "buffered") {
  return res.status(result.statusCode).json({
    accepted: true,
    delivery: "buffered",
    handle: result.handle,
  });
}

return res.status(result.statusCode).json({
  accepted: false,
  delivery: "paused",
  reason: result.reason,
  reasonCode: result.reasonCode,
});

Webhook receiver shape:

const result = await shield.ingest({
  id: webhookEvent.id,
  timestamp: BigInt(Date.now()),
  payload: webhookEvent,
});

if (result.outcome === "paused") {
  return new Response("temporarily unavailable", { status: result.statusCode });
}

return new Response("accepted", { status: result.statusCode });

Worker or queue-consumer shape:

const result = await shield.ingest({
  id: job.id,
  timestamp: BigInt(Date.now()),
  payload: job.payload,
});

if (result.outcome === "paused") {
  throw new Error(`Envelope paused ingress: ${result.reasonCode}`);
}

if (result.outcome === "buffered") {
  logger.info({ handle: result.handle }, "job buffered for replay");
}

Lifecycle note:

• close() is idempotent on DeliveryRecoveryAdapter
• adapter methods throw AdapterClosedError after close so shutdown bugs fail fast instead of touching a closed runtime

If you want direct control over delivery, acknowledgements, and replay loops, use the runtime:

import { createEnvelopeRuntime } from "sqlite-recovery-envelope";

const runtime = createEnvelopeRuntime({
  monitoredServices: ["payment-gateway"],
  reservoir: { databasePath: "./.recovery-reservoir/storage.sqlite" },
});

const { rowId, decision } = runtime.inspectIngress({
  id: "evt_10024B",
  timestamp: BigInt(Date.now()),
  payload: { amount: 500, currency: "USD" },
});

if (decision.action === "accept" && decision.routingMode === "normal") {
  await sendToPaymentGateway();
  runtime.acknowledgeDelivery([rowId]);
} else if (decision.action === "buffer_only") {
  // Delivery is unsafe right now; the item is already persisted locally.
}

Operator Snapshot

The v0.5.7 release exposes a versioned, JSON-safe operator snapshot through the runtime or the decoupled inspection subpath:

import { createEnvelopeRuntime } from "sqlite-recovery-envelope/runtime";

const runtime = createEnvelopeRuntime({
  monitoredServices: ["payment-gateway"],
  reservoir: { databasePath: "./.recovery-reservoir/storage.sqlite" },
});

const snapshot = runtime.getInspectedSnapshot();

logger.info({
  schemaVersion: snapshot.schemaVersion,
  status: snapshot.status,
  recommendedAction: snapshot.recommendedAction,
  backlog: snapshot.backlog,
  replay: snapshot.replay,
  storagePressure: snapshot.storagePressure,
});

schemaVersion is currently the literal 1. The structured ingress, services, backlog, and replay fields are the preferred operator contract; the earlier flattened inspection fields remain available for compatibility.

Operator status and action semantics are deterministic:

| Envelope condition | Operator status | Recommended action | Boundary meaning | | --- | --- | --- | --- | | Normal delivery | healthy | none | Ingress is accepted with 202. | | Degraded bypass | degraded | investigate_services | Ingress is accepted through the reduced delivery path. | | Buffer only | degraded | monitor_buffer_capacity | Downstream failure is contained; ingress remains accepted and persisted with 202. | | Gated replay or retry wait | recovering | monitor_recovery | Backlog recovery is controlling live-flow posture. | | Terminal replay failure | critical | restart_replay | An operator must explicitly restart replay after addressing the delivery failure. | | Protective stop | critical | relieve_storage_pressure | Ingress is refused with 503 and is not persisted until pressure is relieved. |

Backlog age is derived from the runtime's injected monotonic clock. When no pending row exists, backlog.oldestPendingAgeMs is null; it never falls back to 0 or the current time. A value of 0 therefore means a real pending row has a non-positive computed age, not an empty reservoir.

storagePressure exposes the configured pendingRowLimit, current pendingRows, remaining capacity, and exact utilizationRatio without inventing an opinionated warning threshold. When no protective-stop limit is configured, the limit, remaining capacity, and utilization fields are null. atOrAboveLimit reports the numeric boundary, while protectiveStopActive reports the actual routing posture; operator tooling should use both rather than inferring admission behavior from a percentage alone.

Public Subpaths

The package exposes stable ESM entry points so consumers can import only the layer they need:

| Subpath | Public responsibility | | --- | --- | | sqlite-recovery-envelope | Complete supported root surface. | | sqlite-recovery-envelope/adapter | Boundary-facing delivery adapter. | | sqlite-recovery-envelope/config | Configuration defaults, parsing, and resolution. | | sqlite-recovery-envelope/health | Dependency health tracking. | | sqlite-recovery-envelope/inspect | Versioned operator snapshot synthesis and types. | | sqlite-recovery-envelope/keep-watch | Reference target-watch adapter built on the generic boundary and operator contracts. | | sqlite-recovery-envelope/replay | Replay controller and ownership errors. | | sqlite-recovery-envelope/routing | Routing assessment and ingress decisions. | | sqlite-recovery-envelope/runtime | Low-level envelope runtime and factories. | | sqlite-recovery-envelope/storage | Native SQLite reservoir primitives. | | sqlite-recovery-envelope/types | TypeScript-only public contracts; the runtime module is intentionally empty. | | sqlite-recovery-envelope/package.json | Published package metadata. |

npm run test:phase5:subpaths packs the actual npm tarball, installs it offline into a clean ESM consumer, imports every runtime entry point, reads the exported package manifest, and compiles TypeScript against every declaration subpath. This prevents repository-local source or build state from hiding a broken published export.

Reference Keep Watch Adapter

KeepWatchAdapter is a deliberately small reference integration showing how a domain-specific target monitor can translate its vocabulary onto the generic envelope without adding target topology to the core runtime:

import { KeepWatchAdapter } from "sqlite-recovery-envelope/keep-watch";

const watch = new KeepWatchAdapter(
  {
    async deliver(event, context) {
      await deliverMonitorEvent(event.payload, context.source);
    },
  },
  {
    targets: ["payment-gateway", "analytics-sink"],
    reservoir: {
      databasePath: "./.recovery-reservoir/keep-watch.sqlite",
      protectiveStopPendingRowLimit: 50_000,
    },
  },
);

watch.observe({
  target: "payment-gateway",
  state: "degraded",
  reason: "elevated latency",
});

const status = watch.getStatus();
// status.state is "ok", "watch", or "alert".
// status.operator retains the complete versioned operator snapshot.

Healthy observations clear any manual override and record a fresh heartbeat. Degraded or offline observations become explicit service-state overrides with the supplied reason. Ingress, replay, reset/restart, reservoir settings, and idempotent shutdown continue to use DeliveryRecoveryAdapter semantics. The simplified status is derived as follows:

| Operator status | Keep Watch state | | --- | --- | | healthy | ok | | degraded or recovering | watch | | critical | alert |

The adapter is a reference composition, not a new topology embedded in the storage, routing, health, replay, or inspection layers.

Deployment Model

sqlite-recovery-envelope is meant to be embedded directly inside the process that owns the ingress boundary.

That means:

• your API worker, webhook receiver, job runner, or queue consumer loads this package directly
• the same process decides whether to pass through, buffer locally, replay later, or pause
• the SQLite reservoir lives on the same host's local filesystem as that process

Recommended deployment looks like:

• config from inline parameters, recovery.config.json, or SQLITE_RECOVERY_ENVELOPE_CONFIG
• a host-local SQLite path such as ./.recovery-reservoir/storage.sqlite during development
• a server-local writable path owned by the service account in production
• process supervision through your normal runtime such as systemd, PM2, Docker, or another service manager

Avoid treating this package as:

• a separate central recovery service that other applications call over the network
• a required standalone daemon
• a substitute for validating local-disk requirements around SQLite durability and locking

Validation

Library validation commands:

npm run check
npm run test:phase1
npm run test:phase2

Fastest operational smoke check:

npm run test:phase2:smoke10m

That smoke contract simulates ten minutes of runtime behavior in a few seconds and walks through healthy ingress, outage buffering, replay confirmation, retry backoff, and clean recovery drain.

Replay hardening suite:

npm run test:phase3

Phase 4 boundary adapter suite:

npm run test:phase4

Phase 5 operator snapshot suite:

npm run test:phase5

The populated-reservoir snapshot benchmark can also be run directly:

npm run test:phase5:benchmark

It loads 25,000 rows across pending, replaying, delivered, and dead-letter states, warms the real runtime inspection path, and reports median, p95, and maximum snapshot latency over 100 reads. Results are environment-specific evidence for regression comparison, not a universal latency guarantee; the benchmark deliberately has no flaky wall-clock pass threshold.

Packed-subpath consumer contract:

npm run test:phase5:subpaths

Run npm run test:phase1 and npm run test:phase2 sequentially, not in parallel. Both suites rebuild into the shared ./.build output directory, so parallel execution can produce transient build/output contention.

Core Features

• Generic Monitor Core: Tracks arbitrary named dependencies instead of hard-coding a single topology.
• Zero-Dependency Storage Core: Uses native Node.js node:sqlite with no external database service.
• WAL-Backed Concurrency Posture: Boots the reservoir with SQLite journal_mode=WAL and synchronous=NORMAL so live ingress reads and replay bookkeeping can coexist without rollback-journal lock contention.
• Policy-Driven Routing: Flips between pass-through, degraded handling, buffering, and protective pause based on real-time health signals with explicit routing assessments and reason codes.
• Downstream Safety Gating: Keeps live flow gated when required so backlog recovery does not stampede a freshly recovered downstream.
• Fail-Fast Safety: Prevents manual replay control and adapter-managed replay from colliding on the same runtime.
• Automatic Housekeeping: Prunes old buffered rows in bounded batches to keep disk usage constrained during longer outages.
• Separated Internal Timing: Uses a monotonic-backed internal runtime/storage clock while preserving event timestamps separately for payload timing.
• Operator Replay Reset & Restart: Lets callers return replay state to idle explicitly and gives the adapter-owned surface an explicit restart path after terminal replay failure.
• Replay Health Confirmation Gate: Holds queued replay in waiting_confirmation until monitored services satisfy the configured healthy-heartbeat threshold.
• Boundary Outcome Contract: The high-level adapter now returns explicit delivered, buffered, and paused ingress outcomes, plus top-level statusCode, routing/reason metadata, and ingress-handle fields, so outer HTTP or worker layers can translate envelope posture without digging through lower-level internals.

Integration Model

This package exposes:

• a generic runtime core for topology-agnostic monitoring and recovery
• a small high-level adapter for callers who want a delivery-hook wrapper

You choose the monitored service names, payload shape, and delivery hook. The package does not assume a specific queue, API topology, or domain workflow.

State Machine & Posture Mapping

The envelope continuously balances system load and ingress contracts by pivoting between five generic postures:

| Operational Mode | Pipeline Condition | Ingress Behavior | HTTP Boundary Status | | --- | --- | --- | --- | | normal | Dependencies are active. | Direct pass-through delivery. | 202 Accepted | | degraded_bypass | Some dependencies are unhealthy but a reduced path still exists. | Throttled or alternate delivery. | 202 Accepted | | buffer_only | Immediate downstream delivery is unsafe. | Ingress accepted and written to disk. | 202 Accepted | | gated_replay | Delivery path recovered and backlog is draining. | Live flow can remain clamped while replay finishes. | 202 Accepted | | protective_stop | SQLite reservoir limits are breached or local storage pressure is unsafe. | Admission refused to shield the host or dependency. | 503 Service Unavailable |

Current runtime behavior:

• health transitions now drive online, degraded, and offline dependency states
• runtime routing decisions now expose reason codes, affected services, and aggregate health summaries
• deterministic Phase 2 routing validation now covers normal, degraded_bypass, buffer_only, gated_replay, and threshold-based protective_stop
• replay gating currently takes precedence over protective-stop admission while a replay gate is still active
• replay now honors healthConfirmationHeartbeats and waits in waiting_confirmation until monitored services are healthy enough to start draining backlog
• replay confirmation now depends on fresh recovery heartbeats after stale or overridden health periods instead of reusing old healthy counts
• replay failures now enter retry_wait, keep live flow gated during backoff, and still require fresh health confirmation before retry draining resumes
• replay-gated ingress now exposes distinct reason codes for waiting confirmation versus retry backoff instead of flattening every gated replay condition into one generic reason
• replay queuing is now a no-op when no backlog exists, avoiding misleading replay-gate state without pending work
• replay reset now releases actively claimed rows back to pending storage instead of only clearing controller state
• replay batch release now keeps active row tracking and replay readiness aligned when only part of an active claim is released or when the full active batch is handed back to pending storage
• stale replay claims reclaimed during housekeeping now resynchronize the in-memory replay session instead of leaving controller state stranded after interruption-style recovery
• a deterministic 10-minute operational smoke contract now exercises the main Phase 2 lifecycle end to end without requiring a real-time wait
• deterministic wall-clock contracts now validate 4h, 6h, and replay-backoff boundary behavior through injected now() control rather than real waiting
• active replay clamping now stays in force while replay-owned rows are still draining, so fresh ingress is buffered instead of slipping through direct delivery during the last claimed replay batch
• replay can now require a minimum continuous healthy stabilization window before backlog drain begins, helping avoid replay starts during flapping recovery
• replay retry backoff can now escalate across repeated failures with a bounded exponential policy instead of a fixed delay only
• replay can now stop at a terminal retry ceiling and hold ingress behind an explicit operator-visible replay-failed gate instead of retrying forever
• adapter-owned replay now acknowledges rows incrementally, so a mid-batch replay failure only re-queues the unfinished remainder instead of duplicating already-delivered rows on retry
• adapter-owned replay now has explicit restart control after terminal replay failure instead of leaving the adapter surface stranded behind its own ownership lock

Retention & Storage Realities

The sqlite-recovery-envelope uses an expiry-driven retention model rather than immediately hard-rejecting every new item once an outage ceiling is crossed.

• Reaching the maximum outage cutoff does not automatically imply instant hard rejection.
• The runtime can continue to accept and buffer ingress while surfacing backpressure posture to the caller.
• Older unacknowledged rows age out through bounded background pruning and move into a deterministic dead_letter state.

The current storage layer now also separates:

• event time carried on event.timestamp
• internal monitor-ingest/runtime time used for retention and replay bookkeeping

For the current runtime line, the reservoir also initializes SQLite in WAL mode during startup. That is the intended default for the Phase 2 and early Phase 3 concurrency model, where live ingress can continue appending while replay control is simultaneously claiming and updating rows inside the same local database.

Safety Invariants

To avoid silent races and replay drift, the architecture enforces these technical boundaries:

1. Mutually Exclusive Control Ownership: A runtime instance cannot have mixed replay governors. If a high-level adapter owns replay, manual replay commands on the same runtime must fail fast with a typed ReplayOwnershipError.
2. Stale-Claim Recovery Engine: If a background worker node or processing thread crashes or drops offline mid-drain while handling an allocated block of replaying records, the core engine automatically executes a sweep to reclaim abandoned tracking lines. This stops events from becoming permanently trapped in a phantom replaying state.

Configuration Guide

The package can be booted from a recovery.config.json file, an environment-provided config path, or inline parameters.

Current exported helpers:

• loadEnvelopeConfigFile()
• loadEnvelopeConfigFromEnvironment()
• createEnvelopeRuntimeFromFile()
• createEnvelopeRuntimeFromEnvironment()

Current runtime replay helpers:

• queueReplay()
• claimReplayBatch()
• acknowledgeReplayBatch()
• releaseReplayBatch()
• resetReplay()

Current replay-start behavior:

• queueReplay() can move replay into waiting_confirmation when backlog exists but monitored services have not yet met the configured healthy-heartbeat threshold
• claimReplayBatch() returns no rows until that confirmation threshold is satisfied
• the adapter now queues replay before attempting a pump so the same gate semantics apply at both runtime and adapter level
• stale health periods and manual overrides reset healthy-heartbeat confirmation so recovery replay requires fresh observed heartbeats
• failed replay now enters retry_wait, releases the claimed batch back to pending, and blocks replay claims until backoff has elapsed and health has been reconfirmed
• releasing part of an active replay batch now shrinks activeRowIds in place, while releasing the full active claim re-queues backlog and re-applies replay readiness gates before draining can resume
• stale replaying rows reclaimed by housekeeping now move the runtime back into a coherent queued or confirmation-gated replay posture, and fully expired interrupted claims are cleared back to idle
• npm run test:phase2:smoke10m now provides a short deterministic end-to-end confidence pass over the main replay lifecycle
• claimReplayBatch() now requires replay to be explicitly queued first, and active replay keeps live ingress clamped until the replay-owned batch is fully cleared
• replay.recoveryStabilizationMs can now enforce a minimum continuously healthy window before queued replay is allowed to start or resume
• replay snapshots now expose retryCount, and replay.retryBackoffMultiplier plus replay.maxRetryBackoffMs can shape bounded exponential retry timing
• replay.maxReplayAttempts can now force a terminal replay-failed posture that requires explicit manual re-queue before draining resumes

Current adapter replay helpers:

• pumpReplay()
• resetReplay()
• restartReplay()

A complete generic configuration shape looks like this:

{
  "monitoredServices": ["target-api-endpoint"],
  "reservoir": {
    "databasePath": "./.recovery-reservoir/storage.sqlite",
    "rollingBufferWindowMs": "4h",
    "fullOutageMaxWindowMs": "6h",
    "pruneIntervalMs": "60s",
    "pruneBatchSize": 1000
  },
  "replay": {
    "healthConfirmationHeartbeats": 3,
    "recoveryStabilizationMs": "10s",
    "pauseLiveFlowDuringReplay": true,
    "retryBackoffMs": "5s",
    "retryBackoffMultiplier": 2,
    "maxRetryBackoffMs": "1m",
    "maxReplayAttempts": 5
  }
}

replay.maxReplayAttempts is optional. Set it to null if you want replay retries to continue without a terminal failure cutoff.

Example: boot from a file

import {
  createEnvelopeRuntimeFromFile,
  loadEnvelopeConfigFile,
} from "sqlite-recovery-envelope";

const config = loadEnvelopeConfigFile("./recovery.config.json");
const runtime = createEnvelopeRuntimeFromFile("./recovery.config.json");

Example: boot from environment

import {
  createEnvelopeRuntimeFromEnvironment,
  loadEnvelopeConfigFromEnvironment,
} from "sqlite-recovery-envelope";

process.env.SQLITE_RECOVERY_ENVELOPE_CONFIG = "./recovery.config.json";

const config = loadEnvelopeConfigFromEnvironment();
const runtime = createEnvelopeRuntimeFromEnvironment();

If you do not override reservoir.databasePath, the package creates ./.recovery-reservoir/storage.sqlite on the host where the process runs.

On a server, prefer a host-local writable path owned by the service account. Avoid read-only mounts, synced workspace folders, and network filesystems unless you have explicitly validated them for SQLite locking and durability.

License

MIT