@cat-factory/observability-otel

Opt-in OpenTelemetry (OTLP) trace + metrics + logs publisher for the Agent Architecture Board.

It implements the runtime-neutral LlmTraceSink port from @cat-factory/kernel, so when wired into a facade every LLM call: container-agent calls (through the LLM proxy) and inline calls (requirements review, document planner, fragment selector, inline agent): is exported to any OTLP/HTTP backend (Grafana Tempo/Mimir, Honeycomb, Datadog OTLP, Jaeger, an OpenTelemetry Collector, …) as:

• a trace span per generation, plus a span per container tool call, arranged into a per-run hierarchy (run → agent kind → generations + tool calls); and
• metrics: a gen_ai.client.token.usage counter (input/output) and a gen_ai.client.operation.duration histogram: following the OpenTelemetry GenAI semantic conventions.

Two further exporters ship beside it, each its own opt-in: the deployment-level platform-operator metrics, and the log export that carries the platform's own structured log lines to the same endpoint.

Span hierarchy

A run's spans form a tree, not a flat set of siblings sharing a trace id:

run                                 (root; the whole execution, INTERNAL)
├── invoke_agent coder              (one per agent KIND that ran, INTERNAL)
│   ├── chat claude-sonnet-4-5      (one per LLM call, CLIENT)
│   ├── execute_tool edit_file      (one per container tool call, INTERNAL)
│   └── execute_tool run_command
└── invoke_agent ci                 (a gate step)
    └── invoke_agent ci-fixer       (a HELPER the gate escalated to, nested under it)
        └── chat claude-sonnet-4-5

Nothing is buffered to build it. Every parent id is a pure function of the run id (deriveRunSpanId / deriveStepSpanId in src/mapping.ts), so a generation recorded by the LLM proxy on one isolate and a tool span drained by the engine on another both name the same parent without either having seen it, and a durable replay re-derives the identical tree rather than a duplicate one. The parents themselves are emitted once, when the run settles (LlmTraceSink.recordRunSpans, driven from the engine's single terminal hook), which is when its boundaries are first known. Children therefore export before their parent: the ordinary OpenTelemetry ordering, where a parent outlives what it contains, just over a longer span of time than usual.

Two consequences worth knowing:

• The step level's grain is (run, agent kind), not (run, step index), because the agent kind is the finest thing an LlmGenerationEvent can name — a per-step parent would be unaddressable from the very spans meant to hang under it. It is also the grain the rest of the LLM telemetry buckets by (the prompt-chain key, the (agentKind, phase) rollup). A pipeline running two coder steps folds them into one span carrying cat_factory.step_count: 2, so a reader is told about the fold rather than seeing one long step.
• A standalone inline call stays a root. Requirements review, the doc planner and the fragment selector run outside any execution, so they have no run to hang under and no parent is claimed — rather than pointing at one that will never be emitted.
• A HELPER kind gets its own span, nested under the step that dispatched it. A step often runs work under a kind that is not its own: a gate escalating to ci-fixer / conflict-resolver / on-call, a Tester handing off to the fixer, a two-phase coder's fork-proposer. Every telemetry row that work produces is tagged with the HELPER's kind, so without a span of its own each of those would name a parent nobody emits and dangle inside its own run's trace. The run records what it dispatched (PipelineStep.dispatches), and the helper inherits its host step's window: a parent is required to contain its children, and the host's window is the tightest bound the run actually recorded.

Cycles: counted, not separated

The repetition these runs actually contain is cyclical rather than duplicated. A pipeline with two steps of one kind is rare (no built-in has one); a review that spawns fixes four times, a Ralph loop that iterates seven, a bounced step re-run on a judge's verdict are all ordinary.

A span cannot separate those rounds, because the events hanging under it carry no attempt ordinal to be separated BY — threading one reaches the proxy URL contract and the harness, so it is a deliberate non-goal here. What is NOT acceptable is letting the fold pass silently, so each step span states both of its collapses: cat_factory.step_count (steps of this kind) and cat_factory.attempt_count (dispatches inside them). Converging in one round and thrashing through six are otherwise the same picture, and that picture is what an operator is looking for.

The extent is stated honestly across rounds too: a re-run step's span starts at PipelineStep.firstStartedAt, which survives the reset that re-stamps startedAt. Without it a bounced step's parent would begin AFTER the generations of its own earlier attempts, which still name it.

Two transports, one behaviour

The Cloudflare Worker runtime (workerd) cannot run the official @opentelemetry/* SDK (it relies on Node-only APIs), so this package ships two exporters behind the same port:

| Entry | Export | Transport | Used by | | -------------------------------------- | -------------------- | ------------------------------------------- | ----------------- | | @cat-factory/observability-otel | createOtelSink | hand-rolled OTLP/HTTP JSON over fetch | Cloudflare Worker | | @cat-factory/observability-otel/node | createNodeOtelSink | the official @opentelemetry/* SDK | Node / local |

Both map events through the same src/mapping.ts layer, so they emit identical span names, attributes, trace-id grouping and metric names/units. src/conformity.test.ts feeds the same events through both and asserts the emitted telemetry matches: the guard that the transports never drift.

The Worker entry (.) never imports @opentelemetry/*, so the SDK is kept out of the workerd bundle; it depends only on the fetch/crypto globals.

Behaviour

• Never throws into the caller: every method swallows its own errors (logging at most a warning). Observability must never break agent work.
• Honours the same LLM_RECORD_PROMPTS privacy switch as the local metric store: when prompt recording is off, spans carry usage/timing/attributes but no prompt or response bodies.
• Composes alongside the Langfuse sink (@cat-factory/observability-langfuse) via the kernel composeTraceSinks fan-out: a deployment can export to both at once.

Usage

// Cloudflare Worker (workerd-safe fetch exporter)
import { createOtelSink } from '@cat-factory/observability-otel'

const sink = createOtelSink({
  endpoint: env.OTEL_EXPORTER_OTLP_ENDPOINT, // e.g. http://collector:4318
  headers: { 'x-api-key': env.OTEL_KEY }, // optional
  serviceName: 'cat-factory', // optional; defaults to 'cat-factory'
})

// Node / local (official @opentelemetry/* SDK exporter)
import { createNodeOtelSink } from '@cat-factory/observability-otel/node'

const sink = createNodeOtelSink({ endpoint: process.env.OTEL_EXPORTER_OTLP_ENDPOINT! })

Wired into a facade via its container's buildTraceSink(config); absent config (no OTEL_ENABLED=true + endpoint) ⇒ the sink is never built and there is no external emission or behaviour change.

GenAI semantic-convention coverage

The OpenTelemetry GenAI semantic conventions are still experimental, so this section states exactly what is claimed rather than leaving a reader to diff the emitted attributes against a moving spec. src/mapping.ts's ATTR object and this table are edited together.

Emitted, per the convention:

| Attribute / signal | Where | Notes | | ---------------------------------- | ------------------------- | --------------------------------------------------------------- | | gen_ai.operation.name | generation / tool / step | chat / execute_tool / invoke_agent | | gen_ai.system | generation span + metrics | the provider (anthropic, openai, workers-ai, …) | | gen_ai.request.model | generation span + metrics | the model asked for | | gen_ai.usage.input_tokens | generation span | FRESH input only; the cache classes are separate (see below) | | gen_ai.usage.output_tokens | generation span | | | gen_ai.response.finish_reasons | generation span | a one-element list; omitted when upstream reported none | | gen_ai.agent.name | step span | the agent kind | | gen_ai.tool.name | tool span | | | gen_ai.tool.arguments | tool span EVENT | the call's arguments; only when body capture is permitted | | gen_ai.tool.result | tool span EVENT | what the tool returned; same gate | | gen_ai.client.token.usage | counter, {token}, DELTA | split by gen_ai.token.type | | gen_ai.client.operation.duration | histogram, s, DELTA | | | span name {operation} {model} | generation span | e.g. chat claude-sonnet-4-5; other names stay low-cardinality |

Extended beyond the convention, because the convention has no equivalent and the fact is load-bearing here:

| Attribute | Why it exists | | ------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | | gen_ai.usage.cache_read_input_tokens | The convention lumps input into one count. These are priced ~0.1x and ~1.25–2x | | gen_ai.usage.cache_creation_input_tokens | base input, so summing them hides whether a loop rides a warm cache or rewrites it. | | gen_ai.token.type values cache_read / cache_write | The same split on the counter's dimension. | | cat_factory.workspace_id | Tenant scope. Spans only; unbounded, so never a metric dimension. | | cat_factory.execution_id | The run. Searchable, since the trace id is a hash of it rather than the id itself. | | cat_factory.agent_kind | Kept beside gen_ai.agent.name because it is also a bounded METRIC dimension. | | cat_factory.pipeline / cat_factory.step_count | The run span's pipeline, and the step span's fold size. | | cat_factory.attempt_count | Dispatches folded into a step span: the rounds of a loop, stated since not split. | | cat_factory.tool_call.seq | The call's ordinal in its dispatch. Start time cannot order a tool loop: several | | | calls routinely share one millisecond. | | cat_factory.tool_call.arguments_dropped_chars | What the harness's capture cap dropped, so a truncated body is legible AS | | cat_factory.tool_call.result_dropped_chars | truncated rather than read as the whole of a short one. | | Deliberately NOT emitted, each for a reason rather than an oversight: |

• COST, in any form. A span or a counter carrying money would be DERIVED data (tokens x rates) in a store that cannot reprice it: a corrected rate table leaves the history permanently wrong with nothing marking it, and the figure would sit beside the platform's own spend ledger (SpendService / priceRollupCells) as a second, un-reconcilable answer, at a grain that deliberately drops workspace_id, so it could never be checked against what anyone is billed. What this exporter emits are the OBSERVED facts a downstream consumer prices FROM: gen_ai.request.model plus the three input classes and the output count, kept apart rather than lumped, which is exactly the input a rate table needs. Money belongs where it can be recomputed.
• gen_ai.request.* sampling parameters (temperature, top_p, max_tokens, …). The event that reaches this package carries none of them: the proxied path records what the upstream returned, and the subscription harnesses lift metrics off a CLI's event stream, which never reports the ceiling it applied. Emitting defaults would state a request nobody made.
• gen_ai.response.model / gen_ai.response.id. A single model field rides the event, and it is the model the provider says SERVED the call where the transport reports one. Emitting it as both request and response would fabricate agreement between two facts we only have one of.
• gen_ai.conversation.id. cat_factory.execution_id already scopes it, and the run is the unit the rest of the platform threads a conversation by.
• The log-based gen_ai.client.inference.operation.details event. Prompt and completion bodies ride the older span events gen_ai.content.prompt / gen_ai.content.completion, which every OTLP backend in the compatibility list above renders today. They are emitted only when LLM_RECORD_PROMPTS is on; an empty body is omitted rather than sent blank.
• gen_ai.operation.name on the RUN span. A pipeline run is orchestration: it spends most of its wall clock waiting on CI and on humans. Claiming a GenAI operation for it would put that wait onto an operator's GenAI dashboards.
• The pipeline name in the run span's NAME. The root span is named the bare run, with the pipeline as cat_factory.pipeline. A span name is the one field a backend treats as a low-cardinality class (it keys the RED metrics a span-metrics connector derives), which is the trace-side counterpart of the rule that a metric dimension must be bounded. Every other name here is a closed vocabulary (an operation, a model, an agent kind, a tool); a pipeline is workspace-authored free text, so interpolating it would let a tenant mint unbounded series on an operator's backend just by renaming pipelines.

Replaying a settled run

The parents are emitted from the engine's terminal hook, which fires again for a run that has already settled (a durable re-drive, a decision resolved against a finished run). That is safe by construction rather than by a claim table, and both halves are load-bearing:

• The ids are derived, so a re-emission names the same spans rather than minting a second tree beside the first.
• The extent is folded from stamps the run recorded (buildRunTraceSpans in orchestration), never from a clock read at emit time. Both are set-once so a replay cannot move them: a done run is bounded by its last step's finishedAt, a failed one by AgentFailure.occurredAt, and a step still in flight by its heartbeat. Pairing stable span ids with a duration that changed between emissions would give a backend a contradiction to store where it can collapse a byte-identical duplicate.

Platform-operator metrics (deployment health)

The per-call sink above answers "what did THIS run do". The PlatformMetricsOtelExporter (createPlatformMetricsOtelExporter, the . entry) answers "how is the WHOLE deployment doing": a periodic sweep (Worker scheduled cron ⇄ Node interval, runtime-symmetric) computes the platform-observability projection per account and this exporter pushes it to the same OTLP endpoint as OpenTelemetry gauge metrics, so an operator watches deployment health in their own metrics backend, the dual of the post-release-health gate that watches the user's release.

Metrics (cat_factory.platform.*, all gauges: the OTel backend trends the series over time):

| Metric | Unit | Split dimension | | --------------------------------------- | ------- | --------------------------- | | cat_factory.platform.runs | {run} | cat_factory.run_status | | cat_factory.platform.run_success_rate | 1 | — | | cat_factory.platform.run_failures | {run} | cat_factory.failure_kind | | cat_factory.platform.live_runs | {run} | cat_factory.run_state | | cat_factory.platform.run_duration | s | cat_factory.duration_stat |

Every point carries cat_factory.account_id (the bounded tenant scope: safe on a metric, unlike the unbounded workspace id excluded from the per-call metrics); the windowed gauges also carry cat_factory.window. Null aggregates (a success rate / percentiles with no terminal runs) are omitted rather than emitted as a misleading zero.

Unlike the per-call LLM path, the platform exporter is the fetch transport on both runtimes (there is no SDK counterpart): the push is a stateless, low-frequency snapshot POST with no need for the SDK's async instruments / periodic reader, so one workerd-safe exporter serves both facades and is tested once.

Opt-in on top of the base exporter (it adds recurring DB rollup load): off unless OTEL_ENABLED=true + an endpoint AND OTEL_PLATFORM_METRICS=true. OTEL_PLATFORM_METRICS_WINDOW (1h/24h/7d, default 1h) sets the trailing window; on Node OTEL_PLATFORM_METRICS_INTERVAL_MS (default 60s) sets the sweep cadence (the Worker is cron-driven). The runtime-neutral sweepPlatformMetrics driver + distinctAccountIds account enumeration live in @cat-factory/orchestration.

Log export (the third signal)

The two exporters above answer "what did this run do" and "how is the deployment doing". The OtelLogExporter (createOtelLogExporter, the . entry) carries the platform's own structured log lines to {endpoint}/v1/logs, so an operator reads logs beside the traces and metrics they already correlate with, in one backend, instead of tailing a Worker and grepping a container's stdout.

It implements the kernel LogSink port, and @cat-factory/server's logging adapter (observability/logger.ts, the one place a logging library is named) copies every emitted line to whichever sink a facade installed. Nothing in the domain changes: packages keep logging through the one Logger port and never learn a second destination exists.

import { createOtelLogExporter } from '@cat-factory/observability-otel'
import { setLogSink } from '@cat-factory/server'

setLogSink(createOtelLogExporter({ endpoint, headers, serviceName, maxBatchSize, logger }))

What a record carries. The message is the OTLP body; the level maps onto the base SeverityNumber of its range (debug 5, info 9, warn 13, error 17), so a backend's severity >= WARN filter means what an operator reading LOG_LEVEL expects. The line's fields become attributes under the names they already have (workspaceId, executionId, jobId, scope, …) rather than being renamed into the cat_factory.* namespace the spans use: one key works against stdout and the collector, which is worth more than symmetry with a signal whose attribute names this package chose in the first place. child-bound fields are folded in, so a line keeps the correlation the emitting scope gave it.

Logs and traces meet structurally. A line naming an executionId is stamped with the SAME derived trace id that run's spans carry, so a backend links them without either side matching an attribute, and the record marks that trace SAMPLED so a backend knows the id it is pointed at is one it should have. A line with no run claims no trace (and no flags), rather than pointing at one nobody will emit (the same rule the standalone inline call follows above).

That sharing is a call to the same deriveTraceId the spans go through, never a second copy of the derivation, and the test asserts a log record's trace id against a span's. Both matter: the two signals agreeing is the whole feature, nothing else enforces it, and a re-derivation plus a test comparing one log record to another would let the span side drift with every test green.

An INBOUND traceparent fills in the rest. A caller already collecting a distributed trace (one of the four SDK clients, a gateway, a sibling service) sends W3C trace context on the request; mountRequestLogging parses it once through kernel's parseTraceparent and binds traceId + spanId onto the request-scoped child logger, so every line the request emits is exported into the caller's trace, pointing at the caller's own span. Malformed or absent means ignored: a bad correlation header is not a reason to refuse real work, and the request falls back to the correlation it always had.

The run wins when a line has both, and that precedence is load-bearing. The derived trace id is the only thing joining a run's lines to the spans this package emits for it; re-pointing them at the caller's trace would leave the run's spans with no logs and its logs with no spans. The two are true statements at different scopes (the caller's trace covers the REQUEST, the derived one covers the RUN it touched) and the run is the narrower answer. Nothing is lost either way: the adopted ids stay queryable as attributes. Only an adopted context names a spanId: a run-derived trace has no single span a line belongs to, so it attaches to the trace and lets the backend place it.

Deliberately not here: emitting a traceparent of our own, and carrying tracestate. Both are things a producer of spans owes its callees, and the request boundary produces no span: its evidence is the log line. Adding them is the slice that adds a request span.

Draining is the facade's job, and the two runtimes differ only there:

| Runtime | When the buffer is drained | | ----------------- | ----------------------------------------------------------------------------------------------------- | | Node / local | An interval (OTEL_LOGS_FLUSH_INTERVAL_MS, default 5s), plus a final flush on shutdown bounded to 5s | | Cloudflare Worker | The end of every invocation, as a waitUntil after the response |

A Worker's module state is per ISOLATE and an isolate is discarded whenever the runtime decides, so a buffered line has no later tick guaranteed to reach it: the same reasoning (and the same per-invocation cost, stated rather than assumed) as the operational-metrics flush beside it. A full batch also sends on its own, so a burst does not wait for the next tick.

What it does with what it cannot deliver, each of which was a way to make observability the new failure class:

• A failed POST is logged and dropped, never thrown: record only buffers and flush resolves even when the delivery failed.
• The exporter refuses to export its own failure reports. They are logged through a logger bound with otelLogExport: true and records carrying that field are dropped, or a collector outage becomes an ever-growing batch of lines about a collector outage. The warning still reaches the local writer, which is where an operator whose collector is down can read it.
• The buffer is bounded (maxBatchSize × 8) and drops the OLDEST beyond it, because during an outage the newest lines are the ones being looked at. The drop count rides the next batch out as its own record: a silently short stream reads exactly like a quiet one.
• An oversized field is capped at 8 KiB and says how much it cut. A collector rejects an oversized batch whole, so one field carrying captured command output would otherwise drop every line beside it.
• A field that cannot be serialised degrades to a note naming the problem, never taking the line with it. Absent and empty stay distinct: a null/undefined field is omitted rather than exported as "".
• A field that cannot even be READ degrades the same way, per field. LogFields is Record<string, unknown>, so a value can throw on access (an accessor property, a Proxy trap). This runs on the DRAIN path, past the try/catch the logging adapter wraps record in, so the mapping owes its own guards: an unreadable value is reported under its own key (the key's presence is what says the emitter had something there) and its neighbours are unaffected.
• The send chain is terminated, so flush cannot reject however the drain path fails. Sends are serialised behind one promise tail, and an escaped rejection there is not one lost batch: every later send would inherit it, the exporter would go permanently silent, and on Node the flush interval's un-awaited call would become an unhandled rejection that the process failure guards answer by exiting. Observability may not take the deployment down.

Opt-in on top of the base exporter (it adds an egress POST per batch of lines): off unless OTEL_ENABLED=true + an endpoint AND OTEL_LOGS=true. OTEL_LOGS_MAX_BATCH_SIZE (default 128) sets the lines per POST and, with the multiplier above, the buffer bound; on Node OTEL_LOGS_FLUSH_INTERVAL_MS (default 5s) sets the cadence (the Worker flushes per invocation).

LOG_LEVEL governs both destinations. The export sits behind the same threshold as the local writer, so an operator has one dial rather than two that can disagree; there is deliberately no separate export level.

Two things are NOT exported, and both are stated here rather than left to be discovered:

• Lines emitted before config is resolved (the LOG_LEVEL read, the process guards, config validation, migrations) reach the local writer only. The endpoint IS config, and buffering against one that may never arrive would be a leak on exactly the boots that fail.
• Nothing is scrubbed here. A field carrying command output, a URL or model text goes through redactSecrets at its emit site (backend/docs/logging.md); this exporter carries what the local writer already got, and adding a second scrub would state a guarantee the stdout stream does not have.