@super-oar/timeback-tracelog

Drop-in observability for Node services in the TimeBack ecosystem.

Install the package, set two environment variables, initialize at startup — and you're done. No code changes to your handlers, routes, or workers. Every HTTP request, outbound fetch, Postgres query, Redis call, BullMQ job, AWS SDK call, and console.log in your service becomes a trace + structured log, correlated across services, and shipped to SigNoz.

If you want richer signals — manual spans for custom operations, structured event/outcome fields, the opt-in audit log, or business-entity context on every log — those are separate steps you can add later. The baseline works on its own.

Concepts in 60 seconds

If you've never used OpenTelemetry before, here are the only terms you need:

• Trace — one request's full journey across every service it touches, identified by a shared trace_id.
• Span — one unit of work inside a trace (an HTTP call, a DB query, a handler). A trace is a tree of spans.
• Baggage — small key/value pairs (e.g. studentEmail, courseId) that travel alongside the trace across service boundaries. Set it once at the boundary; it shows up on every child span and log line downstream.
• OTLP — the OpenTelemetry wire protocol the package uses to ship signals.
• Collector — a separate process (not part of this package) that receives OTLP and forwards it to a backend.
• SigNoz — the TimeBack observability backend. It stores traces, logs, and metrics, and provides the UI you query.

What you get without writing any code

Once the package is installed and initialized at startup:

• Traces — auto-instrumented for HTTP server + client, fetch, Postgres (pg), Redis (ioredis), AWS SDK v2/v3, BullMQ, Express, Fastify, Mongoose, and Node.js built-ins. Every external call your service makes becomes a timed span, linked across services via W3C Trace Context.
• Logs — Pino is pre-configured with a span-aware mixin. Your existing console.log / console.error calls are monkey-patched so they flow through the same pipeline. Every log line carries trace_id, span_id, baggage fields, and AWS runtime identity automatically.
• Cross-service context propagation — the trace ID and baggage values ride W3C headers on outbound HTTP, AWS SDK calls, and BullMQ jobs, so the next service downstream picks them up with no work on your side.
• Error capture — uncaught exceptions, unhandled promise rejections, and errors thrown inside spans are recorded with stack traces and exported. You don't need a try/catch to make an error visible.
• AWS runtime identity — Lambda function ARN + request ID, ECS task ARN, region, and git SHA are auto-detected and stamped on every signal.
• Metrics — HTTP request rate/duration, DB pool saturation, BullMQ queue throughput, and Node.js runtime metrics are exported on a 30-second interval by default.

Where your signals end up

The package emits OTLP and stops there. It does not know or care what backend you're using.

A typical deployment points OTLP traffic from each service at a central OTel collector (Bearer-token auth), which forwards traces, logs, and metrics to a backend such as self-hosted SigNoz (ClickHouse-backed). You query everything from the backend's UI — service maps, trace waterfalls, log search, dashboards.

Switching the backend (e.g. SigNoz → AWS-native CloudWatch + X-Ray) is a collector-config change. Your service code and this package do not change.

Install

The package is published to the public npm registry:

pnpm add @super-oar/timeback-tracelog
# or: npm install @super-oar/timeback-tracelog

(Releases prior to public hosting were served from GitHub Packages under the same name; consumers migrating from that setup can delete the @super-oar:registry lines from their .npmrc.)

Quick start

Set two environment variables:

export OTEL_SERVICE_NAME=my-service
export OTEL_EXPORTER_OTLP_ENDPOINT=https://collector.example.com
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer ${OTLP_TOKEN}"

Then initialize the package at startup. Two ways, equivalent:

// package.json — preload flag, no code change
"scripts": {
  "start": "node --import @super-oar/timeback-tracelog/register dist/server.js"
}

// or, one line at the top of your entry file
import { initObservability } from '@super-oar/timeback-tracelog'
initObservability()
import express from 'express' // and everything else after this line

That's it. Every framework you load after init is automatically instrumented.

For ECS one-shot tasks, Lambda (with or without the ADOT layer), and custom bundling setups, see docs/CONFIGURATION.md §3 (recipes) and §16 (Lambda integration).

Tailored setup — point your AI assistant at CONFIGURATION.md

Quick start gets you baseline observability in any standard Node service. Real services have details — Lambda with or without the ADOT layer, ECS one-shot tasks, BullMQ workers, audit-log wiring, custom metrics, span allowlists, Morgan migration, bundling for esbuild — that depend on your stack and aren't worth working through by hand.

The fastest way to get your service fully wired is to hand docs/CONFIGURATION.md to your AI coding assistant (Claude Code, Cursor, etc.) along with your service's entry file and package.json, and ask:

"I want to fully adopt @super-oar/timeback-tracelog in this service. Read docs/CONFIGURATION.md end-to-end, then walk through the changes needed for this codebase: init module, env vars, framework adapters, and any audit / span-attribute config that applies. Make the edits."

CONFIGURATION.md is the package's authoritative reference (every InitOptions field, every env var, every recipe, troubleshooting catalog, the Aurora audit schema, and the "where data should live" decision matrix). It's written so an agent can read it once and produce a correct setup tailored to your service without further guidance.

Verify it's working

1. Init ran — start your service and look for a tracelog init log line on stdout. If you don't see one, the package isn't initializing (preload flag missing, or initObservability() not being reached).
2. Traces are leaving the process — set OTEL_DIAG_LOG_LEVEL=info and watch for BatchSpanProcessor export messages. No exports = wrong OTEL_EXPORTER_OTLP_ENDPOINT or missing auth header.
3. Records in SigNoz — open SigNoz, select your service.name, and set the time window to at least 6 hours (the default 30 minutes often misses recent ingest). Then search by trace_id or browse the service map.

Two adoption profiles

Most services need the baseline above and nothing else. A small number of canonical-state services do more.

• Stamper — every service except the two below. Install + init only. Get traces, logs, context propagation, error capture, metrics. Optionally call runWithContext (next section) to enrich logs with business-entity fields.
• Writer — only TimeBack API and MasteryTrack. Same install + init, plus the opt-in business event audit module that records canonical state changes (enrollments, test resets, assignments) to a dedicated Aurora audit_log table.

If you're not building TimeBack API or MasteryTrack, you're a stamper. You can stop reading here unless you want the optional enhancements below.

Optional: add business-entity context to your logs

Without any extra code, every log line carries trace_id and AWS runtime fields. To also stamp business-entity context (which student, which course, which tenant, who initiated the action), wrap your handler in runWithContext:

import { runWithContext } from '@super-oar/timeback-tracelog'

app.post('/enroll', async (req, res) => {
  await runWithContext(
    {
      studentEmail: req.body.studentEmail,
      courseId: req.body.courseId,
      subject: req.body.subject,
      tenantId: req.user.tenantId,
      actor: req.user.email,
      actor_type: 'guide',
      reason: req.body.reason,
    },
    async () => {
      // your handler logic
    }
  )
})

These nine fields are the canonical baggage set — they're the only fields that propagate across service boundaries. Don't invent new ones; if your service needs to record a field that doesn't appear in this list, log it directly instead of putting it on baggage. See docs/CONFIGURATION.md §25 for the "baggage vs span attribute vs log field" decision matrix.

addContextFields({ ... }) adds fields mid-handler when you only learn them after a DB lookup.

Optional: the structured-log convention

For org-wide dashboards to work, every service uses the same three field names at decision points:

logger.info({ event: 'ENROLLMENT_CREATED', outcome: 'success' }, 'student enrolled')
logger.warn({ event: 'PROGRESSION_DECISION', outcome: 'skipped', reason: 'no_active_test' }, 'skipped')

• event — SCREAMING_SNAKE_CASE, format {AREA}_{ACTION} (e.g. INGESTION_RECEIVED, LISTENER_FILTERED).
• outcome — one of success | failure | skipped | partial.
• reason — free-form qualifier, only when outcome isn't success.

This is recommended, not required. Your logs work without it; the convention just makes cross-service queries trivial.

Optional: framework adapters

The baseline auto-instruments most things. A few cases need one-line wiring:

• Express + audit middleware — app.use(getAuditMiddleware()) (no-op when audit is off).
• BullMQ — auto-instrumentation works, but to get full trace propagation from web → worker you must pass BullMQOtel in your QueueOptions.telemetry and WorkerOptions.telemetry. Roughly three lines per host. The package can't inject this for you because BullMQ's API requires the caller to opt in.
• Lambda — wrap with wrapLambdaHandler(handler). See Quick Start above.
• ECS one-shot task — use /register-ecs preload, or call wrapEcsTask(fn) from inside your entry point.
• Morgan — use createOtelMorganStream() per call, or enable fleet-wide auto-instrumentation via initObservability({ morganInstrumentation: { enabled: true, parse: parseCombinedMorganFormat } }).
• Mongoose — auto-instrumented; tune dbStatementMaxBytes if your queries are large.
• Postgres via porsager postgres — wrap your sql client with wrapPorsagerSql(sql). The standard pg driver is auto-instrumented; only porsager needs this.

Optional: manual spans for custom operations

When auto-instrumentation can't see what you're doing (a pure computation, an in-memory pipeline stage, a batch step), wrap it in a span:

import { withSpan, addSpanAttributes } from '@super-oar/timeback-tracelog'

await withSpan('progression.evaluate-test-out', async () => {
  addSpanAttributes({ 'progression.kind': 'test_out' })
  // your logic
})

addSpanEvent, recordErrorOnActiveSpan, and getActiveTraceId are also exported for finer-grained needs.

Business event audit (writer profile, opt-in)

When AUDIT_ENABLED=true and the audit config is populated, the package records one row to a centralized Aurora audit_log table for every state-changing call at whitelisted HTTP routes, BullMQ workers, and Lambda handlers. Each row captures:

• actor / actor_type / reason (from baggage, or derived from JWT / service identity if absent)
• trace_id (so the audit row joins to the SigNoz trace)
• action (HTTP method + route, worker name, or function name)
• touched_tables, request_payload (first N bytes, configurable), http_status, duration_ms

Audit writes are fire-and-forget with a bounded in-memory retry buffer; they never block the main request and an audit-DB outage cannot fail your service.

Configuration:

// tracelog-init.js — the module pointed to by TRACELOG_INIT_MODULE
export default {
  audit: {
    enabled: true,
    auroraConn: process.env.AUDIT_DB_URL,
    http: ['/api/enrollments', '/api/tests'],
    workers: ['progression', 'caliper-ingest'],
    lambdas: ['progression-worker'],
    tables: ['enrollments', 'test_attempts', 'progression_decisions'],
  },
}

// in your Express app, mount the middleware (no-op when audit is off)
import { getAuditMiddleware } from '@super-oar/timeback-tracelog'
app.use(getAuditMiddleware())

See docs/CONFIGURATION.md §6 for the full Aurora schema, validation rules, and wiring patterns for BullMQ + Lambda + ECS task writers.

Configuration reference

The must-know environment variables:

| Variable | Required? | Purpose | |---|---|---| | OTEL_SERVICE_NAME | yes | Service name in SigNoz | | OTEL_EXPORTER_OTLP_ENDPOINT | yes | OTel collector URL | | OTEL_EXPORTER_OTLP_HEADERS | usually | Bearer token for the collector | | OTEL_RESOURCE_ATTRIBUTES | optional | Extra service identity (service.env=prod,...) | | GIT_SHA | optional | Build SHA stamped on every signal | | TRACELOG_INIT_MODULE | optional | Path to a JS file exporting InitOptions (audit config, span allowlists, etc.) | | OTEL_SDK_DISABLED=true | emergency | Kill switch — disables all instrumentation |

For the full surface — every InitOptions field, every environment variable, every recipe, every troubleshooting case — see docs/CONFIGURATION.md. It is the authoritative reference; this README is the onboarding.

Public API at a glance

import {
  // Lifecycle
  initObservability, wrapLambdaHandler, flushTelemetry,

  // Logging
  logger, registerMorganTraceIdToken, createOtelMorganStream, parseCombinedMorganFormat,

  // Context propagation
  runWithContext, addContextFields, setAuthContext,

  // Manual spans
  withSpan, addSpanEvent, recordErrorOnActiveSpan, addSpanAttributes,
  getActiveTraceId, getActiveSpanContext,

  // Business event audit
  getAuditMiddleware, wrapAuditWorker, wrapAuditLambdaHandler, wrapAuditEcsTask,
  validateAuditConfig,

  // BullMQ
  BullMQOtel, attachQueueJobMetrics, attachQueueJobFailureLogging, registerQueueGauges,

  // Metrics
  getMeter, registerPgPoolGauges, registerMongoosePoolGauges,

  // Drivers
  wrapPorsagerSql,

  // ECS cross-process
  installFlushOnExitHooks, injectActiveContextAsEnvEntries, extractContextFromEnv, wrapEcsTask,

  // Lint
  auditDriftRule,
} from '@super-oar/timeback-tracelog'

Subpath entry points:

• @super-oar/timeback-tracelog/register — preload that initializes the package via --import (no code change in your entry file).
• @super-oar/timeback-tracelog/register-ecs — same as /register, plus upstream trace-context extraction from env and a flush-on-exit hook for one-shot ECS tasks.
• @super-oar/timeback-tracelog/cdk — TracelogAspect and withTracelogBundling for CDK Lambda fleets.

Troubleshooting

| Symptom | Likely cause | Fix | |---|---|---| | No traces in SigNoz | Package never initialized; modules loaded unpatched | Add the --import preload to your start command, or call initObservability() before any other imports | | "No logs for trace" in SigNoz | UI time window too narrow | Change "Last 30 minutes" → "Last 6 hours" first, before diagnosing ingest | | HTTP works, BullMQ doesn't | bullmq-otel not wired into Queue/Worker options | Pass BullMQOtel in QueueOptions.telemetry and WorkerOptions.telemetry | | Audit rows not appearing | Table or boundary not in whitelist, or response wasn't 2xx/3xx | Check tables / http / workers arrays in your init module | | pnpm add 401 | GITHUB_TOKEN missing or lacks read:packages | Set the token; verify with npm whoami --registry https://npm.pkg.github.com | | Preload no-ops on Lambda with ADOT | Layer already initialized OTel; expected | Package detects this and skips its own SDK setup — instrumentation still runs |

See docs/CONFIGURATION.md §21 for the full troubleshooting catalog.

Further reading

docs/CONFIGURATION.md — the complete reference. Every InitOptions field, every environment variable, every recipe (Express, Lambda Mode A and B, ECS one-shot, BullMQ writer, Morgan migration, custom metrics), the full Aurora audit schema, troubleshooting catalog, and the "where data should live" decision matrix.