Status: current Owns: the OpenTelemetry stack behind every log, metric and trace on the platform

Uniform: library/connector

Duty sections that apply:

• all: L-MAIN, L-ENGINES, L-TESTS, L-TEST-SCRIPT, L-PACK-TESTS, L-PINS, L-NO-FILE-RANGE, L-CHANGELOG, L-README, L-README-REGION, L-CONSUMER
• connector: L-CONNECTOR-ENV

@onlineapps/monitoring-core

Core monitoring infrastructure for OA Drive microservices using OpenTelemetry.

Architecture Pattern

This library uses a Factory Pattern instead of singleton to allow:

• Multiple independent instances with different configurations
• Better testability and isolation
• Parallel running of services
• Clean separation between service instances

Installation

npm install @onlineapps/monitoring-core

Usage

Factory Pattern (Recommended)

Create independent instances for each service:

const { init } = require('@onlineapps/monitoring-core');

// Create new instance with factory
const monitoring = await init({
  serviceName: 'my-service',
  serviceVersion: '1.0.0',
  mode: 'light', // off | light | full | debug
  logLevel: 'INFO'
});

// Use the instance
monitoring.logger.info('Service started');
monitoring.workflow.start('wf-1', 'process');

Multiple Instances

Each service can have its own configuration:

// Service 1
const service1 = await init({
  serviceName: 'api-gateway',
  mode: 'full'
});

// Service 2 - independent configuration
const service2 = await init({
  serviceName: 'worker',
  mode: 'light'
});

Direct Module Usage

For simple logging without full telemetry:

const { createLogger, WorkflowTracker } = require('@onlineapps/monitoring-core');

// Standalone logger
const logger = createLogger({
  serviceName: 'simple-service',
  mode: 'off'
});

// Standalone workflow tracker — the logger is a required dependency
const { createProcessStdoutSink } = require('@onlineapps/monitoring-core/src/stdoutSink');
const tracker = new WorkflowTracker('my-service', { logger: createProcessStdoutSink() });

The last output channel

This package IS the logging library of every service on the platform, so when its own RabbitMQ transport is down it cannot report that through "the service's logger" — that logger is this package. Until 2026-09-07 it reached for an ambient console in 39 places: an undeclared channel nobody injected and no test covered.

Owner decision: one explicit end channel, recorded in api/docs/governance/confirmations/connector-logger-contract.md, confirmation 005. src/stdoutSink.js builds an object satisfying @onlineapps/logger-contract (info/warn/error/debug) that writes one NDJSON line per call:

{"timestamp":"2026-09-07T05:31:14.648Z","level":"INFO","source":"monitoring-core","message":"[Telemetry] Disabled by configuration","meta":{"serviceName":"hello-service"}}

Docker collects stdout and Loki parses those lines, so a service whose telemetry transport is gone still leaves a machine-readable record. The channel is injected, never ambient, and createProcessStdoutSink() is the single place in the package that names process.stdout.

It is NOT a default for a missing parameter: every constructor below still validates what it is handed and throws when it is not a complete logger (confirmation 001).

Constructors that require it

| Constructor | Signature | Who passes the channel | |---|---|---| | TelemetryCore | new TelemetryCore() | builds its own via createProcessStdoutSink(); it is the composition root | | RabbitMQExporter | new RabbitMQExporter(config, type, { logger }) | TelemetryCore.setupOpenTelemetry() | | RabbitMQMetricExporter | new RabbitMQMetricExporter(config, { logger }) | TelemetryCore.setupOpenTelemetry() | | TelemetryLogger | new TelemetryLogger(config, { logger }) | createLogger(); held as this.sink, so no field inside a logger is called this.logger | | WorkflowTracker | new WorkflowTracker(serviceName, { logger }) | TelemetryCore.init(), or the caller for standalone use |

init(), createLogger() and createInfrastructureLogger() keep their signatures — they build the channel for you.

The sink applies no level threshold: filtering is TelemetryLogger's job, and by the time a line reaches the sink the decision to log it has been made. Dropping it here would hide the emergency the channel exists for.

The service's own records use the same channel — stdoutEnabled

Confirmation 006 closed the last exception. Until 2026-09-07 a record accepted by TelemetryLogger.log() was additionally printed by an ambient console as

[INFO] Order accepted  { orderId: 42 }

— a second, unstructured shape of the same stdout, switched by a config key called consoleEnabled. There is now one channel and one line shape: the record goes to the injected sink, as NDJSON.

{"timestamp":"2026-09-07T08:03:19.850Z","level":"INFO","source":"monitoring-core","message":"Order accepted","meta":{"service":"smoke-service","severity_number":9,"trace_id":null,"span_id":null,"version":"1.2.3","environment":"smoke","pid":82891,"hostname":"Igor--MacBook-Air.local","data":{"orderId":42}}}

The key survives the change under the name that says what it does — stdoutEnabled (env MONITORING_STDOUT_ENABLED, default true) — because it still decides something the sink cannot decide for itself:

| stdoutEnabled | what reaches stdout | |---|---| | true (default) | every record the logger accepts, plus the library's own transport failures | | false | the library's own transport failures only — the channel of confirmation 005 |

consoleEnabled and MONITORING_CONSOLE_ENABLED are retired; nothing reads them, and a config still setting them changes nothing (no backward compat before production, architecture principle 11).

The debug preset's own line — which trace provider the SDK ended up with — goes to the same channel, under the option traceDebugToStdout (retired name: consoleTrace). It is the only other switch in this package that decides whether something reaches stdout, and after 006 no name here mentions a console, because there is none left in src/.

Two details of the shape, both deliberate:

• source is always monitoring-core — this library wrote the line. What distinguishes a service record from a library-internal line is meta.service: a record has it, an internal line ([Logger] RabbitMQ not available …) does not.
• One level, one severity_number, one channel method. The record levels are the four of @onlineapps/logger-contract, so "level":"ERROR" always carries "severity_number":17 and no severity has to borrow another's method. A fifth level, FATAL (severity 21), existed here until d.372 and did borrow error; nothing on the platform ever wrote one.

Human readability in development is a tool's job, not a second format:

docker logs <service> | jq -r 'select(.meta.service) | "[\(.level)] \(.message) \(.meta.data)"'

Configuration Modes

| Mode | Use Case | Features | Performance Impact | |------|----------|----------|-------------------| | off | Testing, Development | stdout (and file) logging only, no transport | Minimal | | light | Production | Logs + Basic metrics | Low | | full | Staging | Logs + Metrics + Traces | Medium | | debug | Debugging | Everything + Debug info | High |

API Reference

init(options)

Factory function to create new monitoring instance.

const monitoring = await init({
  serviceName: 'my-service',      // Required
  serviceVersion: '1.0.0',         // Optional
  serviceInstanceId: 'my-service-3', // Optional: which running copy this is
  mode: 'light',                   // Optional (default: 'off')
  logLevel: 'INFO',                // Optional: DEBUG|INFO|WARN|ERROR|NONE (default: 'INFO')
  environment: 'production',       // Optional
  sampling: 1.0,                   // Optional (0-1)
  traceDebugToStdout: true,        // Optional (default: only in mode 'debug')
  rabbitmq: {                      // Optional
    url: 'amqp://127.0.0.1:5672',
    exchange: 'telemetry.exchange'
  }
});

A failed initialization throws — there is no quieter service. If anything in init() fails (an unknown logLevel, an instrumentation that cannot be loaded, a transport that refuses the configuration), the error propagates to the caller with its original cause attached, and the instance hands out no logger at all. It used to be swallowed: the failure was printed once and the instance handed back a logger of four empty methods, so a service booted, logged into nothing and nobody was told. A wrapper boots on that promise — a service that cannot log must not run (api/docs/governance/confirmations/wrapper-boot-logger.md).

Silence is available, but only as a decision: mode: 'off' is a declared disabling and keeps the four-method no-op surface. A failure is not a declaration.

A failed init() also stops what it had already started: the OpenTelemetry SDK comes up first, and since the caller gets the exception instead of the instance, an SDK left running would be a telemetry stack nobody can reach to shut down.

shutdown() — and who owns the process

shutdown() closes everything this instance opened: the OpenTelemetry SDK, its exporters, and the logger's own RabbitMQ connection, file stream and timers. After it returns, nothing from this package is left holding the event loop, so a service whose work is finished exits on its own.

Including a connection that is still opening. shutdown() may be called before the logger's amqp socket has come up — a boot that fails a step later, a short-lived job — and it ends that attempt as well: it waits for the attempt in flight and the attempt closes the socket it is handed. Until d.509 it did not: this.connection was still null, so nothing was closed, and the socket arrived afterwards with nobody left to close it. The process then printed "[Telemetry] Shutdown complete" and kept running until something killed it.

A record written after shutdown() is not a transport failure. The summary line a service writes last — after the logger step of its own shutdown, so that a failure of that step still reaches somebody — finds the transport gone, because shutdown() closed it. From the moment shutdown() begins, such a record is no longer published to the broker and nothing is reported about it: the same doctrine as the pre-connect window, a state the process walks through on purpose rather than a fault (d.626; until then every orderly end of a service left one [Logger] RabbitMQ not available … ERROR in the log, and that ERROR carried the shutdown summary). The record itself is not lost: the end channel (stdout) has it. The file sink does not — shutdown() closes the declared file stream, which is its own announced job. An outage during the run — a connection that failed, fell over or was refused — is still an ERROR, unchanged.

The service owns its process; this library owns none of it. It registers no SIGTERM/SIGINT handler, takes no uncaughtException / unhandledRejection, and never calls process.exit() — confirmation mq-client-lifecycle-contract 001 verdict 1. Call shutdown() from your own handler, in your own order:

process.on('SIGTERM', async () => {
  await server.close();
  await mqClient.disconnectAll();
  await monitoring.shutdown();   // last: everything above still logs
  process.exit(0);
});

A crash is the service's to decide too. Log it with the logger this instance hands out, then fail fast — do not let a library turn a fatal condition into a log line the process survives.

Logger Methods

monitoring.logger.debug('Debug message', { data });
monitoring.logger.info('Info message', { data });
monitoring.logger.warn('Warning message', { data });
monitoring.logger.error('Error message', { error });

These four ARE the platform logger contract (@onlineapps/logger-contract), and they are the whole surface: every logger this package hands out carries exactly them, in every mode. A process on its way down reports with error() and then fails fast in its own code — this library has no fifth level to say it with.

Level names are a contract

log(level, message, data) takes the level as its FIRST argument, as a string. The four record levels are DEBUG, INFO, WARN, ERROR (case-insensitive); anything else — a name this package does not know, the retired FATAL, or a winston-shaped { message, level } object — throws, naming the allowed values. It used to become an INFO record with an undefined message instead.

config.logLevel accepts those four plus NONE: explicit silence, where no record reaches the file, stdout or the transport. The library's own last-resort channel is not gated by it — a transport failure is still reported, which is the whole point of confirmation 005. An unknown logLevel throws in the constructor rather than quietly becoming INFO.

createInfrastructureLogger(config)

The logger for infrastructure services (gateway, registry, validator, dispatcher, delivery endpoint) — TelemetryLogger with child labels, without the OpenTelemetry SDK.

const logger = createInfrastructureLogger({
  serviceName: 'api-services-registry',  // REQUIRED (or env SERVICE_NAME)
  serviceVersion: pkg.version,           // REQUIRED (or env SERVICE_VERSION)
  logLevel: 'INFO',                      // Optional (default: 'INFO')
  file: {
    enabled: true,
    directory: 'logs'
    // maxSize + maxFiles: REQUIRED, no default in code. Both travel the same
    // chain as the keys above — explicit config, then LOG_MAX_SIZE_BYTES /
    // LOG_MAX_FILES. WHICH values the platform declares is an owner decision,
    // recorded once in api/docs/governance/confirmations/log-file-bounds.md 001;
    // a number repeated here would be a second copy of it.
  },
  rabbitmq: { url: 'amqp://...' }        // REQUIRED (or env RABBITMQ_URL)
});

serviceName and serviceVersion have no default, deliberately. They are the identity every record is attributed by: meta.service and, on the wire the monitoring consumer reads, resource_attributes['service.version']. Until 2026-09-08 a config omitting them was answered with 'unknown-infra-service' and '1.0.0' — the second is the more expensive of the two, because a stamped version is not a gap an operator notices in Loki, it is a wrong value they believe. Both keys follow this package's priority chain (src/config.js): explicit config, then the environment; there is no third step, so a config with neither throws in the constructor, naming the key and the fix.

Every file.* key travels the same single chain — explicit config, then the environment, then this package's defaults (src/config.js) — and it is that chain which also gives the key its type, so maxDays: '3' reaches the sink as the number 3. Nothing composes those values a second time afterwards.

logLevel is genuinely optional, and its default (INFO) has exactly one owner: DEFAULT_LOG_LEVEL in src/logger.js. An invalid level — including an empty string — is refused by the same validator that refuses one for createLogger(); this constructor no longer repairs it into INFO.

The local file sink has two bounds, and both are declared

Rotation here is this package's own (no winston-daily-rotate-file), and it is bounded twice:

| Key | Env | Unit | Bounds | |---|---|---|---| | file.maxDays | LOG_MAX_DAYS | days | how OLD a file may get — the value is owned by src/defaults.js (fileMaxDays) | | file.maxSize | LOG_MAX_SIZE_BYTES | bytes | how LARGE one file may get — required, no default | | file.maxFiles | LOG_MAX_FILES | files | how many files of one row are kept — required, no default | | file.onFatal | — | callback | WHO is told when the row can no longer be written — required, no default |

file.onFatal(err) is the third required key and the only one no environment variable can carry. A write that fails at RUNTIME — the disk fills, the directory goes away under a rotation, the permissions change — used to switch the channel off after one warning, so a service that had asked for a file went on running and wrote none for the rest of the day. It now reports one ERROR on the library's end channel and calls onFatal once, with an error carrying the path, the cause (error.cause) and the fix. The library does not end the process: that is the lifecycle owner's decision (@onlineapps/service-wrapper answers it with _shutdownFatally). Owner decision: api/docs/governance/confirmations/logger-runtime-write-failure.md 001.

A file that reaches maxSize is not rewritten: the row continues in the next part, app.<date>.1.log, app.<date>.2.log, … while the day's first file keeps the name it always had, app.<date>.log. maxFiles then bounds that row, oldest part deleted first — a size bound alone fills the disk exactly as fast, only in many files instead of one.

Both bounds apply to that row and to nothing else. A log directory belongs to nobody in particular — services share LOG_DIRECTORY, and an operator drops an archived file beside the live ones — so a .log this sink did not write is never deleted by it, however old it is. Membership of the row is decided by the name this sink writes: app.<date>.log and its parts, from file.filename.

Neither of the two size bounds has a default, deliberately. How many bytes a log file may occupy and how many of them a directory keeps are facts about the DISK the service runs on, which this library cannot know; a number invented here would be the fallback architecture-principles.md §3 forbids. With file logging enabled and either key missing, the constructor throws, naming the key, the env variable and the fix. The measured cost of having no size bound at all: hello service wrote 2.4 GB of log in one day (d.383) — one such day fills a biz box, and the age bound would have removed it a day later at the earliest.

An enabled sink needs all five keys, and TelemetryLogger holds a default for none of them. directory, filename and maxDays do have owner defaults — in src/defaults.js, read through the one chain above — and both entry points deliver them; what ended in d.509 is the SECOND copy of those three values inside the logger, which bypassed the environment layer (a caller reaching it got logs however LOG_DIRECTORY was set). A config built by hand that omits one is now told which key is missing and where it comes from.

A declared sink that cannot be opened stops the constructor. An unwritable directory, or a file.directory that turns out to be a file, throws — naming the path, the underlying cause and the fix. Until d.509 the same situation printed one line on stdout and set file.enabled = false: a service that had ASKED for a local log ran without one, nothing on disk and nothing failing. Writing no files remains available as a decision, file: { enabled: false }, which opens nothing and raises nothing.

Workflow Tracking

// The tracker is the instance's `workflow` property, a `WorkflowTracker`.

// Start workflow
monitoring.workflow.start('wf-123', 'process-type', {
  userId: '456',
  source: 'api'
});

// Add workflow step
monitoring.workflow.step('wf-123', 'validation', {
  status: 'success'
});

// End workflow
monitoring.workflow.end('wf-123', 'completed', {
  recordsProcessed: 100
});

// Mark a workflow as failed
monitoring.workflow.error('wf-123', new Error('Database unreachable'), {
  retryCount: 1
});

Metrics

createMetric(name, type) is the only metric method, and it returns the meter's own instrument — so the instrument's own method records: add on a counter, record on a histogram. type is counter (the default), histogram or gauge.

// Counter
const counter = monitoring.createMetric('requests_total', 'counter');
counter.add(1, { endpoint: '/api/users' });

// Histogram
const histogram = monitoring.createMetric('response_time', 'histogram');
histogram.record(235, { endpoint: '/api/users' });

Wire shape — what value is for each metric_type

The RabbitMQ metric exporter publishes ONE envelope per data point, with exactly these fields: timestamp, service_name, metric_name, metric_type, value, unit, attributes, resource. The exporter does not convert the value: whatever the OpenTelemetry aggregation produced for that data point is what leaves the channel, serialized as JSON. So the shape of value is decided by the instrument, and it is a contract for every consumer of telemetry.metrics.*:

| metric_type | instruments behind it | value on the wire | |---|---|---| | counter | Counter, UpDownCounter and their observable variants | a number — the Sum of the aggregation | | gauge | Gauge, ObservableGauge | a number — the LastValue of the aggregation | | histogram | Histogram | an OBJECT — the Histogram value type of @opentelemetry/sdk-metrics (build/src/aggregator/types.d.ts) |

A histogram value therefore looks like this, and a consumer storing value as a single number has to aggregate it itself — it never arrives pre-reduced:

{
  "buckets": { "boundaries": [0, 5, 10, 25], "counts": [0, 2, 0, 0, 1] },
  "sum": 40,
  "count": 3,
  "min": 5,
  "max": 30
}

• buckets.boundaries are inclusive upper bounds, and buckets.counts always has one member more than boundaries — the last one counts everything above the top boundary.
• count and buckets are always present. sum, min and max are declared optional by the SDK: min/max are written only while the aggregation records them (the default explicit-bucket aggregation does), and sum is omitted for an instrument that allows negative values (build/src/aggregator/Histogram.js § toMetricData). A member the SDK leaves undefined does not reach the wire at all — JSON.stringify drops it.
• unit is the instrument's unit, or an empty string when it declares none, and attributes is {} when the measurement carried none. Both fields are always present.

Held by tests/unit/metric-wire-shape-is-a-contract.test.js, which drives a real MeterProvider through this exporter and asserts the JSON that left the channel.

Testing

The factory pattern makes testing much easier:

describe('MyService', () => {
  let monitoring;

  beforeEach(async () => {
    // Fresh instance for each test
    monitoring = await init({
      serviceName: 'test-service',
      mode: 'off' // No external connections
    });
  });

  afterEach(async () => {
    await monitoring.shutdown();
  });

  test('should log messages', () => {
    monitoring.logger.info('Test message');
    // No singleton state pollution
  });
});

Environment Variables

• TELEMETRY_MODE - Override configuration mode
• LOG_MAX_SIZE_BYTES - Size in bytes at which the local log file is rotated. REQUIRED wherever file logging is on; no default (see § The local file sink has two bounds).
• LOG_MAX_FILES - How many files of one log row the directory keeps. REQUIRED wherever file logging is on; no default.
• SERVICE_INSTANCE_ID - Which RUNNING COPY of the service this process is. Reaches the OpenTelemetry resource as service.instance.id and travels on every exported metric envelope, so two replicas of one service are two samples rather than one overwritten twice. Composed from hostname and pid when not declared; HOSTNAME is reported separately, as host.name.

Migration from Singleton

If you have old code using singleton pattern:

// Old (singleton)
const monitoring = require('@onlineapps/monitoring-core');
monitoring.init({ serviceName: 'old-way' });

// New (factory)
const { init } = require('@onlineapps/monitoring-core');
const monitoring = await init({ serviceName: 'new-way' });

API Design Decisions

activeWorkflows Map Encapsulation

The WorkflowTracker class maintains an internal activeWorkflows Map for tracking currently running workflows. This is intentionally NOT exposed directly in the public API.

Why it's private:

• Internal implementation detail that may change
• Direct access breaks encapsulation
• Could lead to state inconsistencies if modified externally

Public API alternatives provided:

// Check if specific workflow is active
tracker.isWorkflowActive(workflowId) // returns boolean

// Get count of active workflows (for monitoring)
tracker.getActiveWorkflowCount() // returns number

// Get workflow info (read-only copy)
tracker.getWorkflowInfo(workflowId) // returns object or null

These methods provide controlled access for testing and monitoring without exposing internal state.

Performance Considerations

• Use mode: 'light' for production (5MB overhead)
• Use mode: 'off' for unit tests (0MB overhead)
• Each instance has its own telemetry pipeline
• Batching reduces network overhead

Tests

Two tiers, and the second one is not optional:

npm run test:unit          # hermetic; amqplib is mocked
npm run test:integration   # the live broker; RABBITMQ_URL required, never skipped
npm test                   # both
npm run test:coverage      # the current numbers, from the runner

The integration tier runs real node processes against api_services_queuer: it watches a service exit after shutdown(), watches a service with a broken telemetry configuration refuse to boot at all, and reads back from the exchange the metric and the span that an SDK export really produced. A missing RABBITMQ_URL or an unreachable broker aborts the run rather than skipping it — a skipped suite reports green and leaves the regression loop unnoticed.

Coverage figures and test counts are not written here. They are a fact about the code on the day it is measured, nothing regenerates a number typed into this page, and the one that used to stand here (45.26%, "77 passing tests") had drifted to roughly a third of the real suite before anybody noticed. Ask the runner.

License

MIT