@epoch-protocol/observability

Shared logging, secret-redaction, tracing, and metrics for every Epoch service (epoch-sio, smallocator, solvers). One source of truth: change a redaction rule or trace format here and it changes everywhere.

Three docs, three jobs — read the right one: | Doc | Use it when you want to… | |-----|--------------------------| | README.md (this file) | call the library from a service — what to import, and the rules behind it | | OBSERVABILITY.md | run/operate the whole pipeline (LGTM), onboard a service, ship to prod | | stack/README.md | spin up the local Grafana stack and query it |

Quickstart

import {
  createLogger, safeIntent, setIntentTrace, traceHeaders,
  expressTraceMiddleware, // or fastifyTraceHook
} from "@epoch-protocol/observability";

const logger = createLogger({ svc: "epoch-sio" });

app.use(expressTraceMiddleware() as RequestHandler); // mount BEFORE routes

app.post("/solve", (req, res) => {
  const intent = req.body;
  logger.info({ msg: "received", intent });          // auto-projected via safeIntent
  setIntentTrace({ nonce: intent.nonce, user: intent.sender }); // promote trace id
  // ... await axios.post(url, body, { headers: { ...traceHeaders() } })
});

That's the whole day-to-day surface: make a logger, mount the middleware, log domain objects under their named key (intent/err/req/res), and forward traceHeaders() on internal calls. Everything below is why it's shaped this way and the rules you must not break.

For OpenTelemetry traces and metrics (initTelemetry, withIntentTrace, recordClaim, …), see OBSERVABILITY.md — they're part of the same package but only matter once you're wiring up the LGTM pipeline.

1. Logging + secret redaction

Two layers with opposite failure modes — an allowlist that fails closed, backed by a denylist that fails open:

logger.info({ intent, err, foo })
   │
   ▼  pino pipeline (createLogger)
 ┌──────────────────────────────────────────────────────────────┐
 │ serializers    ALLOWLIST — fails CLOSED (intended path)       │
 │   intent → safeIntent()   err/req/res → pino std              │
 │   per-service: createLogger({ serializers: { compact: … } })  │
 ├──────────────────────────────────────────────────────────────┤
 │ mixin          stamps trace ctx on EVERY line                 │
 │   { traceId, user, nonce, protocolType, chainId } from ALS    │
 ├──────────────────────────────────────────────────────────────┤
 │ formatters.log → redact()   DENYLIST — fails OPEN (safety net)│
 │   • DENYLIST key (case/separator-insensitive) → <redacted>    │
 │   • MASK_KEYS wallet key → 0x1234…cdef                        │
 │   • HEX_RE: any 0x hex ≥130 chars → <redacted:hex:LEN>        │
 └──────────────────────────────────────────────────────────────┘
   │
   ▼  JSON → stdout (no transport: pkg-binary-safe) → log drain

Allowlist is the real defense. Log a domain object under a serialized key (intent, err, req, res) and it's projected to a curated safe view before redaction — only named fields survive, so a newly-added secret field can't leak. Register your service's shapes with createLogger({ serializers }).

Denylist is the backstop, for whatever skipped the safe path. It's a thin net, not the primary guard:

| Rule | What dies | What survives | |------|-----------|---------------| | DENYLIST keys | key material (privatekey, mnemonic, seed), signatures, credentials (apikey, password, bearer, tokens), policy keys (task, mandate, approvals, balances — fund sizes, OpSec) | everything else, incl. generic keys like data | | MASK_KEYS | wallet addresses → masked 0x1234…cdef | token/contract addresses (debug-useful) | | HEX_RE value scrub | any 0x hex ≥130 chars (signatures, calldata blobs) under ANY key | short hex: selectors (10), addresses (42), hashes (66) |

logFailure(logger, code, stage, ctx) emits a structured failure line (+ optional Sentry, lazily required), with ctx redacted before either sink.

Generic keys (data, calldata, orderdata, prices) are intentionally not in the DENYLIST — the HEX_RE value scrub kills their dangerous blob values while the structured debug data underneath stays readable. Don't re-add them to the denylist; if a generic key carries something sensitive, add a serializer instead.

2. Trace context propagation

Every log line for one intent shares one id, so you can grep the whole story:

inbound HTTP
  │  expressTraceMiddleware / fastifyTraceHook   (mount earliest)
  │    reuse inbound X-Epoch-Trace-Id, else mint a UUID
  │    echo on the response header + open ALS context
  ▼
handler parses intent → setIntentTrace({ nonce, user, protocolType, chainId })
  │    promotes traceId → deriveTraceId(nonce, user)  (deterministic domain id)
  │    re-echoes the response header so header == logs
  ▼
every log line stamped via the pino mixin (reads ALS)
  │
  ▼ outbound internal call:  { headers: { ...traceHeaders() } }

Two ids, two jobs:

• traceId (ep1) — the intent's identity. deriveTraceId(nonce, user) returns ep1-<user12>-<fragment>: a pure function of (sender, nonce), both already in the intent. So every service, retry, and background leg recomputes the same id with no shared state and no secrets. Grep it to see one intent end-to-end.
• legId (8 hex) — one attempt. Quote attempts deliberately share a traceId (same wallet + nonce slot = one logical intent), so per-attempt separation lives here: minted per inbound request, forwarded via X-Epoch-Leg-Id, returned in error bodies. Filter legId to isolate one try.

Edge cases:

• setIntentTrace with an empty nonce or missing user keeps the minted UUID (a nonce-only id would collide across wallets) but still annotates nonce/protocolType/chainId.
• Background workers: re-establish context per job with runWithContext({ traceId: deriveTraceId(nonce, sender) }, () => …) (or withIntentTrace, see OBSERVABILITY.md).
• Never forward traceHeaders() to third parties (socket.tech, infura, …).

• user12 = first 12 hex (48 bits) of sha256(lowercased sender) — the user is embedded only as a truncated hash, never a raw address substring, because traceId is the one field passed through redaction raw.
• fragment = the nonce reduced to its low 96 bits. Why not String(nonce): Compact nonces pack sponsor(20B) || frag(12B), so the raw decimal nonce is the wallet address in disguise (a redaction leak), and nonces are per-sponsor (raw values collide across users). Masking to 96 bits strips the sponsor and makes full-composite and fragment-only call sites converge on the same value.
• Why hash the user at all / why 48 bits is enough: gap-filled nonces cluster at 0/1/2…, so the user hash carries nearly all the separating entropy. 48 bits ⇒ ≈1.8e-5 expected colliding pairs at 100k wallets (≈0.18 at 10M).
• Why the ep1- prefix: keeps any future scheme regex-distinguishable in logs (ep2 = HMAC-keyed if the threat model hardens; OTel 128-bit packing per the AWS X-Ray precedent in Phase 3).
• Why legId is never folded into the trace id: an attempt counter isn't re-derivable from the intent, so baking it in would break reconstruction.

Old ids (raw decimal) and ep1 ids are format-disjoint — the legacy Grafana regex is [0-9]{1,78}. The durable join across the cutover is the nonce log field (fragment form) plus the masked user. Bump this package in all consumers within one release window: an old-package service re-deriving mid-chain clobbers an inbound ep1 id (the trace splits and only rejoins via the nonce field).

Module map

| Area | Files | Key exports | |------|-------|-------------| | Logging | logger.ts, serializers.ts, safe-intent.ts, failure.ts | createLogger, serializers, safeIntent, logFailure | | Redaction | redact.ts, mask.ts, constants.ts | redact, maskAddress, DENYLIST/HEX_RE | | Tracing | context.ts, http-trace.ts, propagation.ts, middleware/ | setIntentTrace, deriveTraceId, traceHeaders, expressTraceMiddleware/fastifyTraceHook | | OTel traces | telemetry/ | initTelemetry, withIntentTrace, withIntentSpan | | Metrics | metrics.ts | recordIntent/recordClaim/recordQuote/recordFailure/recordSettleDuration | | Errors | error.ts, http-error.ts | AppError, toExternalBody, expressErrorHandler (two-tier: internal logs vs external body) |

OTel traces and metrics are documented in OBSERVABILITY.md — their usage only makes sense alongside the running pipeline.

Invariants (do not break)

• Logging must never throw — redact() survives cycles, BigInt, throwing getters, and functions; safeIntent() returns {} on junk.
• No pino transports in prod — worker threads crash under @yao-pkg/pkg binaries. LOG_FORMAT=pretty is local-dev only.
• traceId is the only field passed through redaction raw — it must contain no raw wallet bytes, including via composite nonces (ep1 strips to the low 96 bits and hashes the user). Same rule for the nonce log field: emit the 12-byte fragment, never the raw composite.
• Fund sizes (amounts, balances) stay out of logs — OpSec policy. Reverse this consciously in safe-intent.ts / constants.ts, never ad hoc.

Dev

pnpm typecheck && pnpm test && pnpm build   # tsc / jest / tsup (esm+cjs+dts)

Consumers pick up changes only after pnpm build here and a dependency refresh in the service (a plain install reuses the stale store copy — see OBSERVABILITY.md → Operational notes).