@pagermon/ingest-core

Shared ingest core runtime for PagerMon.

Looking to run PagerMon Ingest with RTL-SDR?
You probably want the multimon adapter repository instead.
This repository is the shared core runtime library and is only relevant if you're developing custom adapters or contributing to the core.

What This Is

This repository contains the stable core pipeline shared by all PagerMon ingest adapters:

• config parsing and validation
• queue and worker processing
• API client and health monitor
• adapter orchestration and lifecycle management

It does not contain a concrete source adapter implementation (RTL-SDR, SMTP, etc.).

Who This Is For

• Adapter developers: building custom ingest sources
• Core contributors: improving shared runtime behavior
• Not for: end users who just want to run a ready-made adapter

If you just want to run PagerMon Ingest with RTL-SDR hardware, use a concrete adapter repository instead of this one.

Architecture At A Glance

@pagermon/ingest-core separates reusable runtime concerns from source-specific logic.

• Core runtime responsibilities:
  • read and validate config
  • initialize queue/API/health/worker services
  • start an adapter and consume emitted messages
  • enqueue normalized messages for API delivery
• Adapter responsibilities (in adapter repo):
  • read from a concrete source (SDR, SMTP, polling, etc.)
  • parse source-specific payloads
  • emit normalized Message objects

This split keeps source integration complexity out of the core and allows multiple adapter repos to share one stable runtime.

Runtime Flow

On startup, the core follows this sequence:

1. Validate INGEST_CORE__* configuration.
2. Initialize API client, queue manager, health monitor, and worker.
3. Load/create adapter instance.
4. Start adapter stream processing.
5. For each emitted message:
   • set source label

• enqueue one delivery job per configured API target
• worker submits to PagerMon API

6. On signal/error: stop adapter pipeline and core services gracefully.

This behavior is orchestrated in lib/runtime/service.js and lib/runtime/pipeline.js.

Repository Structure

Important paths in this repository:

• index.js: default entrypoint (loader mode)
• bootstrap.js: bootstrap API used by adapter repos
• lib/config.js: env parsing and validation
• lib/core/: queue, API, worker, health services
• lib/runtime/: adapter loader and runtime orchestration
• lib/message/Message.js: shared normalized message model
• test/unit/, test/integration/: core runtime tests

Adapter Convention

A concrete adapter image must provide this module:

• /app/adapter/adapter.js

The core loads that module at startup and validates the runtime adapter contract (getName, start, stop, isRunning).

Configuration Prefixes

• Core: INGEST_CORE__*
• Adapter: INGEST_ADAPTER__*

The core forwards adapter keys as structured config (adapter) and raw env map (rawEnv) to the selected adapter.

Example mapping:

• Env: INGEST_ADAPTER__SMTP__HOST=smtp.example.org
• In adapter: this.config.adapter.smtp.host === 'smtp.example.org'
• Raw fallback: this.config.rawEnv.INGEST_ADAPTER__SMTP__HOST

Runtime Modes

@pagermon/ingest-core supports two startup modes:

• Default loader mode: node index.js
• Bootstrap mode: adapter repo entrypoint calls bootstrapWithAdapter(AdapterClass)

Default loader mode expects an adapter entry module at /app/adapter/adapter.js (override with INGEST_CORE__ADAPTER_ENTRY).

Bootstrap mode lets adapter repos pass the adapter class directly and avoids path conventions.

Use loader mode when your container layout already provides /app/adapter/adapter.js. Use bootstrap mode when your adapter repo wants explicit startup control in code.

Adapter Developer Docs

This README intentionally stays high-level for runtime and operations.

If you are building a custom adapter, use ADAPTER_DEVELOPMENT.md as the primary source. It contains the full contract, end-to-end implementation flow, test patterns, metrics guidance, and deployment options.

Development

npm ci
npm run check
npm test

Container

Pre-built Images

Pre-built images are available from two registries:

• Docker Hub: shutterfire/pagermon-ingest-core
• GitHub Container Registry: ghcr.io/eopo/pagermon-ingest-core

Both registries contain identical images. Choose based on your preference:

# Docker Hub
docker pull shutterfire/pagermon-ingest-core:latest

# Or GitHub Container Registry
docker pull ghcr.io/eopo/pagermon-ingest-core:latest

Building Locally

Build core image:

docker build -t shutterfire/pagermon-ingest-core:latest .

Build Exactly From A Release Tag

For reproducible self-builds, build from a Git tag instead of a moving branch:

git fetch --tags
git checkout <release-tag>   # e.g. v1.3.1
docker build -t shutterfire/pagermon-ingest-core:<release-tag> .

Using Ingest with PagerMon Server

Ingest delivers to one or multiple PagerMon API targets using ENV configuration.

Required configuration:

# Target 1 via alias variables
INGEST_CORE__API_NAME=pm-prod-a
INGEST_CORE__API_URL=http://pagermon:3000
INGEST_CORE__API_KEY=key_a

# Additional targets via enumerated variables
INGEST_CORE__API_2_NAME=pm-prod-b
INGEST_CORE__API_2_URL=http://pagermon-b:3000
INGEST_CORE__API_2_KEY=key_b
INGEST_CORE__API_3_NAME=pm-prod-c
INGEST_CORE__API_3_URL=http://pagermon-c:3000
INGEST_CORE__API_3_KEY=key_c

Docker secrets are supported for keys via *_KEY_FILE:

INGEST_CORE__API_KEY_FILE=/run/secrets/pagermon_api_key
INGEST_CORE__API_2_KEY_FILE=/run/secrets/pagermon_api2_key

Rules:

• INGEST_CORE__API_URL/INGEST_CORE__API_KEY/INGEST_CORE__API_KEY_FILE map to target 1
• INGEST_CORE__API_NAME/INGEST_CORE__API_<n>_NAME define stable target names for metrics labels
• INGEST_CORE__API_<n>_URL with INGEST_CORE__API_<n>_KEY or INGEST_CORE__API_<n>_KEY_FILE define target n
• when alias target variables (API_URL/API_NAME/API_KEY[_FILE]) are used, start additional targets at API_2_* - or move the API_* ones to API_1_*
• per target, KEY and KEY_FILE are mutually exclusive (do not set both)
• if *_KEY_FILE is unreadable or points to an empty file, startup fails with a configuration error
• alias variables and enumerated variables can be combined in one configuration
• each incoming message is enqueued once per configured target
• delivery is queued per target, so retries/failures are isolated per API target

Metrics & Observability

Ingest supports optional Prometheus metrics collection for monitoring and observability.

Metrics are registered through the shared runtime and, when enabled, exposed via an HTTP endpoint. Custom adapters can also register their own counters, gauges, and histograms through the injected config.metrics object.

Enabling Metrics

To enable metrics, set:

INGEST_CORE__METRICS_ENABLED=true

Metrics will be exposed at http://<host>:<port>/metrics in Prometheus text format.

Configuration

| Environment Variable | Default | Description | | -------------------------------------- | ------------------ | ----------------------------------------------------------------- | | INGEST_CORE__METRICS_ENABLED | false | Enable/disable metrics collection | | INGEST_CORE__METRICS_PORT | 9464 | HTTP port for metrics endpoint | | INGEST_CORE__METRICS_HOST | 0.0.0.0 | HTTP host for metrics endpoint | | INGEST_CORE__METRICS_PATH | /metrics | HTTP path for metrics endpoint | | INGEST_CORE__METRICS_PREFIX | pagermon_ingest_ | Prefix for all metrics | | INGEST_CORE__METRICS_COLLECT_DEFAULT | true | Collect Node.js default metrics (process, gc, etc.) | | INGEST_CORE__METRICS_DEFAULT_LABELS | (empty) | Parsed from config, but currently not applied to exported metrics |

Available Metrics

With the default prefix pagermon_ingest_, the runtime exposes the following core metrics.

If you change INGEST_CORE__METRICS_PREFIX, prepend that value instead.

Messages

• pagermon_ingest_messages_enqueued_total – Total messages added to the queue
• pagermon_ingest_messages_processed_total{status="success",target_name="..."} – Total messages processed successfully per API target
• pagermon_ingest_messages_failed_total{reason="http_<code>|network_error",target_name="..."} – Total failed message submissions per API target
• pagermon_ingest_message_process_duration_seconds{status="success|failure",target_name="..."} – Message processing duration histogram per API target
• pagermon_ingest_last_message_timestamp_seconds{target_name="..."} – Unix timestamp of the last successfully processed message per API target

Queue

• pagermon_ingest_queue_depth_messages – Current queue depth (number of pending messages)

Health

• pagermon_ingest_api_up{target_name="..."} – API health (1=healthy, 0=unhealthy) per target
• pagermon_ingest_health_check_failures_total{target_name="..."} – Total failed health checks per target

Adapter

• pagermon_ingest_adapter_up – Adapter status (1=running, 0=stopped)

Optional (if enabled)

• Node.js process metrics (memory, CPU, GC, event loop, etc.)

Custom Adapter Metrics

When developing adapters, ingest-core injects a metrics registry as config.metrics.

• Register adapter metrics via counter, gauge, and histogram.
• Define metric names without the global prefix.
• Keep names stable and express dimensions via labels.

Detailed implementation examples, test assertions, and naming guidance are documented in ADAPTER_DEVELOPMENT.md.

Example Usage

# Enable metrics on default port
INGEST_CORE__METRICS_ENABLED=true

# Scrape with curl
curl http://localhost:9464/metrics

# Use with Prometheus (add to scrape_configs)
- job_name: 'pagermon-ingest'
  static_configs:
    - targets: ['localhost:9464']

Developing Your Own Adapter

You can always build your own adapter to support other sources, such as PDW, incoming emails, polling from websites and so on.

• Full adapter contract and implementation guide: ADAPTER_DEVELOPMENT.md
• Metrics registration and observability guidance for adapter authors: ADAPTER_DEVELOPMENT.md

Rule of thumb:

• change this repo when runtime behavior should be shared by all adapters
• change adapter repo when behavior is source-specific

If you only need to run Ingest, you can ignore this section.

Contribution

If you plan to change code in this repository, use CONTRIBUTING.md as the primary guide.

Quick local quality check:

npm run lint
npm test

Detailed testing conventions and adapter integration test behavior are documented in CONTRIBUTING.md.