Sentinel

Long-lived, leader-elected, health-gated background workers for Node.js. A Sentinel runs a function repeatedly on a schedule with crash-recovery, distributed locking, graceful shutdown, structured observability, and process-manager-agnostic deployment.

What you get

• Single-leader by default — distributed lock with TTL self-healing. One instance per name runs at a time; standbys take over on leader death.
• Health-gated execution — preflight + per-iteration checks; pause and retry until dependencies recover.
• AbortSignal cancellation — modern signal-aware libraries unwind cleanly on timeout or stop.
• Pluggable everything — state store, metrics sink, logger. Zero required dependencies; sane defaults for each.
• Process telemetry — memory, CPU, event-loop, FDs sampled per heartbeat and emitted as gauges.
• Process-manager-agnostic — PM2, systemd, Kubernetes, Docker. The HTTP health handler is opt-in so PM2 deployments stay zero-overhead.

Quick start

import { runSentinel, LockContendedError } from '@quicore/sentinel';

runSentinel('invoice-poller', async (ctx) => {
  if (ctx.isStopping()) return;

  const items = await fetchInvoices({ signal: ctx.signal });
  ctx.metric.counter('invoices_fetched_total', 'Invoices fetched').inc(items.length);

  for (let i = 0; i < items.length; i++) {
    if (ctx.isStopping()) break;
    if (ctx.timeRemaining() < 1000) {
      ctx.log.warn('Approaching deadline, committing partial batch');
      break;
    }
    await processInvoice(items[i], { signal: ctx.signal });
    ctx.progress({ done: i + 1, total: items.length, message: items[i].id });
  }
}, {
  waitTime: 30000,
  executionTimeout: 25000,
  retry: { maxAttempts: 3 },
  version: process.env.GIT_SHA,
}).catch(err => {
  if (err instanceof LockContendedError) process.exit(0);  // clean — not a crash
  console.error(err);
  process.exit(1);
});

WorkerContext

Every worker invocation receives a WorkerContext:

| Field | Purpose | |-------------------|------------------------------------------------------------------| | ctx.worker.name | Stable identifier (visibility — logs, metrics, alerts) | | ctx.worker.id | Per-process instance ID (internal — tracing across hosts) | | ctx.run.number | 1-based iteration counter | | ctx.run.id | Per-iteration opaque ID (correlation across logs/services) | | ctx.signal | AbortSignal — aborted on timeout or stop | | ctx.log | Scoped structured logger (auto-labels every emission) | | ctx.metric | Scoped metrics sink (auto-labels {processor, hostname}) | | ctx.isStopping()| Returns true if stop requested — poll in long loops | | ctx.deadline() | Unix ms when executionTimeout will fire (or null) | | ctx.timeRemaining() | ms left in the timeout budget (or Infinity) | | ctx.progress(s) | Report progress; surfaces in heartbeat + iteration:end event |

Pass ctx.signal to any I/O that accepts it. Without the signal, an executionTimeout only unblocks the loop — the underlying I/O keeps running and holding its connection. With the signal, the connection is released.

Metrics

All Sentinel-internal metrics are labelled {processor, hostname} by default.

Lifecycle metrics

| Metric | Type | Extra labels | |-------------------------------------------|-----------|-----------------| | sentinel_runs_total | counter | outcome | | sentinel_attempts_total | counter | outcome | | sentinel_run_duration_seconds | histogram | | | sentinel_lock_held | gauge | | | sentinel_paused | gauge | | | sentinel_last_success_timestamp_seconds | gauge | | | sentinel_consecutive_failures | gauge | |

Health metrics

| Metric | Type | Extra labels | |----------------------------------------------|-----------|--------------| | sentinel_health_check_probes_total | counter | check | | sentinel_health_check_failures_total | counter | check | | sentinel_health_check_latency_seconds | histogram | check |

Process telemetry (auto-emitted each heartbeat)

| Metric | Type | |-----------------------------------------------------|-----------| | sentinel_process_memory_rss_bytes | gauge | | sentinel_process_memory_heap_used_bytes | gauge | | sentinel_process_memory_heap_total_bytes | gauge | | sentinel_process_memory_heap_limit_bytes | gauge | | sentinel_process_memory_external_bytes | gauge | | sentinel_process_cpu_user_seconds_total | counter | | sentinel_process_cpu_system_seconds_total | counter | | sentinel_process_cpu_utilization_ratio | gauge | | sentinel_process_event_loop_utilization_ratio | gauge | | sentinel_process_event_loop_lag_p50_ms | gauge | | sentinel_process_event_loop_lag_p99_ms | gauge | | sentinel_process_open_fds | gauge |

sentinel_process_cpu_utilization_ratio can exceed 1.0 on multi-core machines if the process uses workers/threads — it's CPU-seconds per wall-second, not a percentage.

Recommended alerts

• sentinel_last_success_timestamp_seconds not advancing for N × waitTime
• sentinel_lock_held{processor="X"} == 0 for any X expected to be running
• sentinel_consecutive_failures > 5
• rate(sentinel_health_check_failures_total) > 0 sustained
• sentinel_process_event_loop_lag_p99_ms > 100 sustained (event loop blocked)
• sentinel_process_memory_rss_bytes trending up over hours (leak)

Health checks

Health checks return { healthy, message, details? }. Sentinel times each probe and emits a health:probe event with latencyMs regardless of outcome.

Convention for custom checks: populate details.latencyMs with measured dependency latency (DB ping time, API response time). This lets dashboards graph dependency health from the worker's perspective.

import { HealthCheck } from '@quicore/sentinel';

class MongoHealthCheck extends HealthCheck {
  constructor(conn) { super('mongo'); this.conn = conn; }
  async check() {
    const t0 = Date.now();
    try {
      await this.conn.db.admin().ping();
      return {
        healthy: true,
        message: 'reachable',
        details: { latencyMs: Date.now() - t0 },
      };
    } catch (err) {
      return {
        healthy: false,
        message: err.message,
        details: { latencyMs: Date.now() - t0 },
      };
    }
  }
}

State stores

Three implementations behind one interface — same semantics, different deployment fits.

| Store | Use when | |--------------------|-----------------------------------------------------| | FileStateStore | Single host. No Redis/Mongo in scope. Default. | | RedisStateStore | Multi-host. Lowest-latency lock ops. | | MongoStateStore | Multi-host. Mongo already in stack, prefer reuse. |

All three honour identical ownership semantics: renewLock returns false when this instance has lost the lock. Sentinel uses this as the split-brain detection signal — on false, it emits lock:lost, stops itself, and exits cleanly.

Process-manager deployment

PM2

Use fork mode, not cluster mode. Cluster mode races N workers for one lock and N-1 lose every restart cycle — even with the clean-exit behaviour, it's wasted work.

{
  "name": "invoice-poller",
  "script": "dist/workers/InvoicePoller.js",
  "exec_mode": "fork",
  "instances": 1,
  "autorestart": true,
  "max_restarts": 10,
  "min_uptime": "30s",
  "kill_timeout": 30000
}

kill_timeout should exceed your longest expected iteration so SIGTERM has time to drain. Default 1600ms is too aggressive for any non-trivial worker.

Hot standby (HA across hosts)

Deploy the same worker name on multiple hosts with RedisStateStore or MongoStateStore. One wins the lock and runs; the others exit cleanly with LockContendedError (code 0) and PM2 restarts them periodically. When the leader dies, one of the standbys takes over within lockTtl ms.

systemd

Works as-is. SIGTERM handling and structured logging map naturally.

[Unit]
Description=Invoice Poller Sentinel
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/node /opt/app/dist/workers/InvoicePoller.js
Restart=on-failure
RestartSec=10s
TimeoutStopSec=60
User=workers
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

TimeoutStopSec should exceed worst-case iteration duration.

Kubernetes / Nomad / Docker

Mount the optional HTTP health handler so the orchestrator can probe liveness and readiness:

import { createHealthHandler, runSentinel } from '@quicore/sentinel';

const sentinel = new Sentinel({ ... });
createHealthHandler(sentinel).listen(8080);
await sentinel.start();

Then in your pod spec:

livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  initialDelaySeconds: 30
  periodSeconds: 10
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
terminationGracePeriodSeconds: 60

• /livez returns 200 while the process is alive and ticking. K8s restarts the pod on 503.
• /readyz returns 200 only when the lock is held and not paused. K8s removes the pod from rotation on 503 (useful for monitoring even though the worker doesn't take traffic).
• / returns the full status JSON, always 200 (for human consumption).

You can also mount the handler on an existing Express/Fastify server:

import { createHealthHandler } from '@quicore/sentinel';

const { liveness, readiness } = createHealthHandler(sentinel);
app.get('/livez', liveness);
app.get('/readyz', readiness);

Control plane (REST API)

For operator workflows — pause for maintenance, resume, force a run now, inspect state, stop a worker remotely — Sentinel ships an optional control handler. Same shape as the health handler: a composable function, not a built-in.

The control handler ships without authentication. This is deliberate. Sentinel doesn't try to be an identity provider; callers wire their own policy. The HTTP listener should be bound to localhost or behind an auth middleware in production. Don't bind the control surface to a public interface without auth in front of it — POST /stop from a curl on the internet is a bad day.

import { createControlHandler } from '@quicore/sentinel';

const sentinel = new Sentinel({ ... });
const ctrl = createControlHandler(sentinel);

// Standalone — defaults to 127.0.0.1:9090
ctrl.listen(9090);

// Or compose onto an existing server with your auth middleware
app.get('/sentinel/status',   requireAuth, ctrl.status);
app.post('/sentinel/pause',   requireAuth, ctrl.pause);
app.post('/sentinel/resume',  requireAuth, ctrl.resume);
app.post('/sentinel/run-now', requireAuth, ctrl.runNow);
app.post('/sentinel/stop',    requireAuth, ctrl.stop);

await sentinel.start();

Endpoints

| Method | Path | Action | |--------|-------------|------------------------------------------------------------| | GET | /status | Full state snapshot (running, paused, lock, stats) | | GET | /stats | Just the stats object | | GET | /progress | Current in-flight progress snapshot (or null) | | POST | /pause | Operator pause (graceful — current iteration finishes) | | POST | /resume | Clear operator pause | | POST | /run-now | Wake the inter-iteration sleep, advance now | | POST | /stop | Graceful stop. Requires ?confirm=true |

Pause semantics

Sentinel tracks pause sources as a set, not a flag. Two sources exist today:

• health — added by the runtime health gate when a check fails; removed when the check recovers.
• operator — added by POST /pause; removed by POST /resume.

Both can be active at once. POST /resume only clears the operator source — if a health check is still failing, the sentinel remains paused. Symmetric: health recovery doesn't override an operator pause.

While paused, the lock stays held and heartbeats continue. Standbys on other hosts do not take over. sentinel_paused gauge is 1; status.paused is true; status.pauseSources lists which sources are active.

run-now

Calling runNow() wakes the inter-iteration sleep so the next iteration starts on the next event-loop tick. Cannot interrupt a running iteration (it's advance-to-next, not fire-additional). Returns 409 if the sentinel is paused or stopping.

Common use case: you just deployed a fix and don't want to wait the rest of waitTime to see if it worked.

Response shape

// GET /status
{
  "ok": true,
  "status": {
    "processorName": "invoice-poller",
    "instanceId": "host01:1234:1716036000:a3f9",
    "hostname": "host01",
    "running": true,
    "paused": false,
    "pauseSources": [],
    "hasLock": true,
    "lockAcquiredAt": "2026-05-16T10:00:00.000Z",
    "currentProgress": { "done": 487, "total": 2000 },
    "stats": { "totalRuns": 1234, "successfulRuns": 1233, ... }
  }
}

// POST /pause
{ "ok": true, "action": "pause", "changed": true, "state": <status> }

// POST /run-now while paused
{ "ok": false, "error": "sentinel is paused — resume before run-now",
  "pauseSources": ["operator"] }

Pluggable metrics / logger

Both interfaces are simple. Wire your preferred backend by implementing the interface.

Prometheus example

import client from 'prom-client';
import { IMetricsSink } from '@quicore/sentinel';

export class PrometheusMetricsSink extends IMetricsSink {
  counter(name, help, labelNames = []) {
    const c = new client.Counter({ name, help, labelNames });
    return { inc: (n = 1, labels) => c.inc(labels || {}, n) };
  }
  histogram(name, help, labelNames = [], buckets) {
    const h = new client.Histogram({ name, help, labelNames, buckets });
    return { observe: (v, labels) => h.observe(labels || {}, v) };
  }
  gauge(name, help, labelNames = []) {
    const g = new client.Gauge({ name, help, labelNames });
    return {
      set: (v, labels) => g.set(labels || {}, v),
      inc: (n = 1, labels) => g.inc(labels || {}, n),
      dec: (n = 1, labels) => g.dec(labels || {}, n),
    };
  }
}

Pino example

import pino from 'pino';
import { ILogger } from '@quicore/sentinel';

export class PinoLogger extends ILogger {
  constructor(baseFields = {}) {
    super();
    this.log = pino().child(baseFields);
  }
  debug(msg, fields) { this.log.debug(fields || {}, msg); }
  info(msg, fields)  { this.log.info(fields || {}, msg); }
  warn(msg, fields)  { this.log.warn(fields || {}, msg); }
  error(msg, fields) { this.log.error(fields || {}, msg); }
}

File layout

@quicore/sentinel
├── Sentinel.js              — main lifecycle manager (EventEmitter)
├── runSentinel.js           — runSentinel(name, worker, options) one-call helper
├── WorkerContext.js         — per-iteration context handle passed to workers
├── TelemetryCollector.js    — process telemetry sampled each heartbeat
├── createHealthHandler.js   — opt-in HTTP liveness/readiness handler
├── createControlHandler.js  — opt-in HTTP control plane (pause/resume/stop)
├── errors.js                — LockContendedError, PreflightFailedError, LockLostError
├── RetryPolicy.js           — configurable backoff strategies
├── HealthCheck.js           — base class for health probes
├── IStateStore.js           — state store interface
├── FileStateStore.js        — local FS lock store (default)
├── RedisStateStore.js       — Redis lock store (Lua-script CAS)
├── MongoStateStore.js       — MongoDB lock store (TTL indexes)
├── IMetricsSink.js          — metrics interface + NoopMetricsSink default
└── ILogger.js               — logger interface + ConsoleStructuredLogger default