heimdall-node-logger

Node.js log agent for the Heimdall API Platform. It ships your application logs to a Heimdall gateway's ingestion endpoint, where they are correlated with the gateway's own request events and made searchable side-by-side in OpenSearch.

Relationship to the platform This package is the client-side half of Phase 4 — Observability Agents of the Heimdall API Platform. The gateway side (the POST /ingest/logs endpoint, the x-heimdall-gateway-id propagation and the request-event indexing) lives in the core repo:

• Gateway / platform: https://github.com/protonss4fun/heimdall-api-platform
• Phase 4 design doc: docs/roadmap/phase-4-observability-agents.md
• This agent (Node): https://github.com/protonss4fun/heimdall-node-logger
• Java/Logback agent (planned): heimdall-logback-appender

The agent and the gateway are versioned and released independently; they share only the ingestion contract and the correlation headers described below.

Why

The gateway already knows what happened at the edge (request → flow → response, indexed with correlationId/traceId). This agent adds what happened inside your application during that same request — joined by the very id the gateway round-trips to you. The result is an end-to-end trace (gateway + app) in a single index.

How correlation works

For every request, the Heimdall gateway:

1. round-trips an x-request-id (your correlationId), and
2. propagates its own identity to upstreams as x-heimdall-gateway-id.

The agent's Express middleware captures both from the inbound request into an AsyncLocalStorage context; the Winston transport then stamps every log line emitted during that request with the correlationId and gatewayId — and routes the log to the gateway that originated the request.

client → gateway A → your service
                       │  (logs tagged correlationId + gatewayId=A)
                       └─ heimdall-node-logger → gateway A /ingest/logs → cluster A (joins request A)

Multi-gateway

A single service is often fronted by several gateways, each with its own OpenSearch cluster, ingestion endpoint and token. The agent holds one entry per gateway and routes each log to the gateway that handled its request, authenticating with that gateway's token:

gateways: {
  "gw-br-prod-a": { endpoint: "https://gw-a.example/ingest/logs", token: "<tokenA>" },
  "gw-br-prod-b": { endpoint: "https://gw-b.example/ingest/logs", token: "<tokenB>" }
}

Logs whose gatewayId has no configured entry (and no defaultGatewayId) are dropped rather than misrouted — keeping the correlationId join intact in every cluster.

Install

npm install heimdall-node-logger winston

Requires Node ≥ 20 (uses global fetch). winston is a peer dependency.

Quick start — init() (recommended, dd-trace style)

One call wires every capture surface: server spans per inbound request, client spans for outbound http/https calls (with traceparent + x-request-id + x-heimdall-gateway-id propagated to the next hop — multi-service waterfalls), log shipping and process error capture (uncaughtException/unhandledRejection).

Secure by default. Correlation headers are propagated downstream only to internal hosts (private/RFC1918, localhost, *.svc/*.internal, bare service names) — your trace and gateway ids never leak to third-party APIs. Widen or narrow with propagateTraceTo. A fatal process error is logged, flushed, then the process exits(1) (crash semantics preserved); opt out with exitOnFatal: false.

const express = require("express");
const winston = require("winston");
const { init } = require("heimdall-node-logger");

// Single-gateway shorthand: no gateway map needed.
const heimdall = init({
  service: "billing-api",
  endpoint: "https://gw.example.com/ingest/logs",
  token: process.env.HEIMDALL_INGEST_TOKEN,
});

const logger = winston.createLogger({
  transports: [new winston.transports.Console(), heimdall.transport],
});

const app = express();
app.use(heimdall.middleware()); // context + server spans

app.get("/charge", async (req, res) => {
  logger.info("charge started");          // auto-correlated
  // any outbound http/https call here becomes a client span automatically
  res.sendStatus(200);
});

process.on("SIGTERM", async () => { await heimdall.close(); process.exit(0); });

Multi-gateway fleets keep the explicit map (one token per gateway):

init({
  service: "billing-api",
  gateways: {
    "gw-a": { endpoint: "https://gw-a/ingest/logs", token: "<tokenA>" },
    "gw-b": { endpoint: "https://gw-b/ingest/logs", token: "<tokenB>" },
  },
});

init() extra options:

| Option | Default | Purpose | |---|---|---| | propagateTraceTo | 'internal' | Which outbound hosts receive the correlation headers. 'internal' (private/cluster only), 'all' (everywhere — legacy), or an allowlist ['api.acme.internal', /\.svc$/] (string exact/suffix or RegExp). | | instrumentHttp | true | Set false to skip the outbound http/https patch. | | captureProcessErrors | true | Capture uncaughtException/unhandledRejection. Idempotent across repeated init(). | | exitOnFatal | true | exit(1) after a fatal is shipped (preserves crash semantics). | | shutdownFlushMs | 2000 | Upper bound on the pre-exit flush. |

Plus every HeimdallTransport option below (incl. maxBatchBytes, default 1 MB — batches also flush on a byte budget so a burst of large records can't exceed the gateway body limit; under back-pressure the oldest buffered event is evicted so the most recent are retained). Spans ship to POST /ingest/spans (endpoint derived from the logs one, or set spansEndpoint per gateway). Out of scope: fetch/undici outbound calls are not instrumented (which also keeps the agent's own shippers out of the trace).

Manual wiring (transport only)

const express = require("express");
const winston = require("winston");
const { HeimdallTransport, heimdallContext } = require("heimdall-node-logger");

const logger = winston.createLogger({
  level: "info",
  format: winston.format.combine(winston.format.timestamp(), winston.format.json()),
  transports: [
    new winston.transports.Console(),
    new HeimdallTransport({
      service: "payments-api",
      gateways: {
        "gw-br-prod-a": { endpoint: "https://gw-a.example/ingest/logs", token: process.env.HEIMDALL_TOKEN_A },
        "gw-br-prod-b": { endpoint: "https://gw-b.example/ingest/logs", token: process.env.HEIMDALL_TOKEN_B }
      },
      // optional
      env: process.env.NODE_ENV,
      labels: { region: "br" },
      batchSize: 100,
      flushIntervalMs: 2000,
      onError: (err) => console.warn("[heimdall-logger]", err.message)
    })
  ]
});

const app = express();
app.use(heimdallContext()); // capture x-request-id / x-heimdall-gateway-id into context

app.get("/charge", (req, res) => {
  logger.info("charge started");           // auto-tagged with correlationId + gatewayId
  // ...
  logger.error("charge declined", { error: new Error("insufficient funds"), userId: "u-9" });
  res.sendStatus(402);
});

On graceful shutdown, flush buffered logs:

process.on("SIGTERM", async () => { await transport.close(); process.exit(0); });

AWS Lambda

A Lambda is ephemeral — the runtime freezes right after the handler returns, so the transport's in-memory buffer would be lost before it ships. The Express middleware also doesn't apply (no long-lived server). Wrap your handler with withHeimdall: it opens the correlation context per invocation and awaits a flush in a finally, so buffered logs reach the gateway before the freeze — covering normal returns, early returns and errors (the pattern used by Powertools/dd-trace/Middy).

Configure init() with the two Lambda tweaks: instrumentHttp: false (no long-lived server to instrument) and exitOnFatal: false (never kill the Lambda runtime).

HTTP Lambda (API Gateway / ALB)

The default extract reads event.headers (x-request-id / x-heimdall-gateway-id and the W3C traceparent, case-insensitive) — the traceId is parsed from traceparent, so the function's logs land in the same trace as the gateway request, automatically.

const winston = require("winston");
const { init, withHeimdall } = require("heimdall-node-logger");

const heimdall = init({ ...config.logger.Heimdall, instrumentHttp: false, exitOnFatal: false });
const logger = winston.createLogger({
  transports: [new winston.transports.Console(), heimdall.transport],
});

exports.handler = withHeimdall(async (event) => {
  logger.info("processando", { path: event.rawPath });
  return { statusCode: 200 };
}, { heimdall });          // flush por invocação embutido

Event Lambda (SQS / SNS / EventBridge / cron)

No headers → pass a custom extract that reads the correlation id from the payload. When it's absent, withHeimdall starts a root trace so the invocation still gets its own traceId and its logs group together.

exports.handler = withHeimdall(async (event) => {
  logger.info("mensagem recebida", { records: event.Records?.length });
}, {
  heimdall,
  extract: (event) => ({
    correlationId: event.Records?.[0]?.messageAttributes?.correlationId?.stringValue,
    // traceId ausente → o wrapper inicia uma raiz via ids.js
  }),
});

The only difference between the two cases is extract. Transport, shipping and the finally flush are identical.

withHeimdall(handler, options) options:

| Option | Default | Purpose | |---|---|---| | heimdall | — (required) | The object returned by init(...). Its flush() chain is reused — no duplicate shipping. | | extract | headers (incl. traceparent) → payload → root trace | (event, context) => { correlationId?, traceId?, gatewayId? }. Fail-soft: if it throws, the wrapper falls back to a root trace. | | flushMode | perInvocation (always) | perInvocation flushes every invocation — the default whenever you use withHeimdall, so logs are never lost outside Lambda or in tests. batch is an explicit opt-in that keeps the buffer between invocations (discouraged — the freeze loses it). |

Fail-soft. A shipping error during flush is swallowed (console warning) and never leaks into the handler result. A handler error is re-thrown after the flush, so the error's own logs still reach the Heimdall.

Options

| Option | Default | Description | |---|---|---| | service | — | Logical service name stamped on every event. | | gateways | {} | Map gatewayId → { endpoint, token }. The token is the gateway's static ingest token. | | defaultGatewayId | — | Used when an event has no gatewayId (e.g. logs outside a request). | | env | process.env.NODE_ENV | Environment label. | | host | os.hostname() | Host label. | | labels | {} | Static labels merged into every event. | | batchSize | 100 | Flush when this many events are buffered. | | flushIntervalMs | 2000 | Periodic flush interval (0 disables the timer). | | maxQueue | 10000 | Hard cap; events beyond it are dropped (reported via onError). | | timeoutMs | 5000 | Per-request HTTP timeout. | | maxRetries | 3 | Retries for 5xx / 429 / network errors (exponential backoff). | | onError | no-op | Called with non-fatal errors. The agent never throws into your app. | | fetch | global fetch | Override the HTTP implementation (tests / custom agents). |

Authentication

Each gateway exposes a static ingest token per client (configured on the gateway under application.observability.ingest.tokens). The agent sends it as X-Ingest-Token. In a multi-gateway fleet, the same logical client holds a different token per gateway — exactly the per-gateway entries in gateways.

Event contract

Each event POSTed to /ingest/logs:

{
  "timestamp": "2026-06-09T12:00:00.123Z",
  "level": "error",
  "message": "charge declined",
  "logger": "payments-api",
  "correlationId": "9b2f…",   // = x-request-id (join key)
  "gatewayId": "gw-br-prod-a", // = x-heimdall-gateway-id (routing)
  "traceId": "4bf9…",          // W3C, when present
  "service": "payments-api",
  "env": "production",
  "host": "pod-7c9f",
  "labels": { "region": "br" },
  "mdc": { "userId": "u-9" },
  "exception": { "type": "Error", "message": "insufficient funds", "stack": "…" }
}

The gateway validates/sanitizes and bulk-indexes into <gatewayId>-client-logs.

Design guarantees

• Fail-soft — a slow/unreachable gateway never blocks, throws into, or crashes your app. Failures surface via onError only.
• Non-blocking — log() buffers and returns immediately; shipping is async/batched.
• Bounded — a maxQueue cap prevents unbounded memory growth under backpressure.

License

MIT © P4F