@crvouga/mockingbird-service-tempo

Local emulators. Real API contracts. Part of Mockingbird.

Stateful emulator of Grafana Tempo for test suites: the OTLP/HTTP trace receiver (POST /v1/traces, JSON and protobuf) and the query routes a trace reader calls (TraceQL search, trace by id, tag names), over one store. A test exports spans the way its application does and reads them back with the requests its production reader sends, with no Docker and no Tempo.

• Operation coverage: SUPPORT.md
• Status: work in progress. Tempo publishes no OpenAPI document and no instance was available while this was written, so the contract (openapi.yaml) follows Grafana's HTTP API reference and Tempo's source at v3.1.0. What was verified, and how lists every detail that could not be confirmed.

Install

npm install -D @crvouga/mockingbird-service-tempo

ESM only. Node >= 22 or Bun >= 1.2. No native dependencies. Serve it with npx mockingbird-tempo serve, createServer from ./server (Node), or createRuntime with any Fetch server.

Usage

Tempo listens on two ports (4318 for OTLP, 3200 for queries); the emulator serves both from one origin. Point the exporter's endpoint and the reader's Tempo URL at it:

npx mockingbird-tempo serve --port 8930
# OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:8930   (the exporter posts /v1/traces)
# TEMPO_URL=http://127.0.0.1:8930                     (the reader calls /api/…)

# Grafana Cloud style credentials, and tenants:
npx mockingbird-tempo serve --basic-auth 123456:fixture-token --multitenancy

import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http"
import { resourceFromAttributes } from "@opentelemetry/resources"
import { BasicTracerProvider, SimpleSpanProcessor } from "@opentelemetry/sdk-trace-base"
import { createServer } from "@crvouga/mockingbird-service-tempo/server"

const server = await createServer()

// The OpenTelemetry SDK your application already uses. Only the exporter's URL changes
// (the protobuf exporter works the same way).
const provider = new BasicTracerProvider({
  resource: resourceFromAttributes({ "service.name": "checkout" }),
  spanProcessors: [new SimpleSpanProcessor(new OTLPTraceExporter({ url: `${server.url}/v1/traces` }))],
})
const span = provider.getTracer("checkout").startSpan("POST /checkout", {
  startTime: new Date(1_700_000_000_000),
  attributes: { "user.id": "user-1" },
})
span.end(new Date(1_700_000_001_500))
await provider.forceFlush()

// What the trace reader sends (Tempo's query API is plain HTTP).
const q = encodeURIComponent('{ .user.id = "user-1" }')
const search = await fetch(`${server.url}/api/search?q=${q}&start=1699999990&end=1700000010&limit=50`)
const { traces } = (await search.json()) as { traces: { traceID: string }[] }
const trace = await fetch(`${server.url}/api/traces/${traces[0]?.traceID}`)

await provider.shutdown()
await server.close()

Routes

| Route | Behaviour | | --- | --- | | POST /v1/traces | An ExportTraceServiceRequest as application/json or application/x-protobuf (gzip accepted). Success is 200 with {"partialSuccess":{}} for JSON and an empty protobuf message for protobuf. Errors are google.rpc.Status in the request's encoding: 400 code 3 for a payload that does not decode, trace ids must be 128 bit, received N bits and span ids must be 64 bit and not all zero, received N bits for ids Tempo's distributor refuses (the whole export is refused, nothing is stored). Any other content type is 415 415 unsupported media type, supported: [application/json, application/x-protobuf] as plain text. An export without spans is accepted before any other check. A span id already stored for the trace is replaced. | | GET /api/search | q (TraceQL, see below), start/end (epoch seconds), limit (default 20), spss (spans per span set, default 3, at most 100). Answers {traces: [...], metrics: {...}}; each trace has traceID, rootServiceName, rootTraceName, startTimeUnixNano (nanoseconds, a decimal string), durationMs (a number), spanSet/spanSets (the matched spans: spanID, startTimeUnixNano, durationNanos, and the attributes the query compared; matched counts all of them) and serviceStats. No match is {"traces": []}. Newest trace first. With both start and end (non-zero), a trace is returned when it overlaps the window: it starts at or before end and ends at or after start, both inclusive to the nanosecond; with either missing no window applies, as in Tempo. With no q every trace in the window is returned. | | GET /api/traces/{traceId} | The trace as Tempo marshals it: {"batches": [{resource, scopeSpans: [{scope, spans}]}]}. This is not OTLP/JSON: traceId, spanId and parentSpanId are base64, kind and status.code are enum names (SPAN_KIND_SERVER, STATUS_CODE_ERROR), 64-bit integers (times, intValue) are decimal strings, zero values are omitted, and every span has a status object. A trace id shorter than 32 hex characters is left-padded. Unknown trace: 404 with an empty body. Not hex: 400 trace IDs can only contain hex characters: invalid character 'x' at position N. | | GET /api/v2/search/tags | scope = span, resource, event, link, instrumentation, intrinsic, or absent/none for all of them. Answers {scopes: [{name, tags}], metrics}: attribute names seen on any stored span, each list sorted, a scope with no names left out. A name seen on a single span is listed. limit caps each list. Unknown scope: 400 invalid scope: <scope>. |

Every 400 from the query routes is Tempo's: the error text as the whole body, text/plain. Search validates in Tempo's order: invalid start: strconv.ParseUint: parsing "abc": invalid syntax, invalid limit: must be a positive number, http parameter start must be before end. received start=10 end=5, limit N exceeds max limit M, then the TraceQL query, the 168 h maximum range and spans per span set exceeds 100. received N.

TraceQL

One spanset filter is evaluated:

• { }, { true }, { false }
• attribute equality: .name, span.name or resource.name (quoted parts such as span."user id" included) = a string, integer, float or boolean
• && between those; every condition has to hold on the same span

An unscoped .name reads the span's attribute and falls back to the resource's. Types compare exactly, except that integers and floats compare as numbers. An array attribute matches when any element does.

Anything else is a 400 and never a looser match:

• Input Tempo's parser rejects answers with Tempo's wording and position, for example invalid TraceQL query: parse error at line 1, col 3: syntax error: unexpected IDENTIFIER for { user.id = "u" } (the dot is missing).
• Valid TraceQL outside the subset (=~, !=, >, ||, intrinsics such as status or duration, event./link./parent. scopes, pipelines, spanset operators, hints) answers invalid TraceQL query: <construct> not yet supported (Mockingbird emulates a TraceQL subset). The first part is the wording Tempo's validator uses for features it lacks; a real Tempo evaluates these queries.

Authentication and tenants

Tempo itself authenticates nobody; a gateway in front of it does. The emulator plays both:

| Setting | Effect | | --- | --- | | none (default) | A local Tempo: no credentials needed, stray ones ignored. | | bearerTokens (--bearer-token) | Authorization: Bearer <token> on ingest and query. | | basicUsers (--basic-auth user:password) | Authorization: Basic …, Grafana Cloud style (instance id and access token). | | unauthorized | The response for missing or wrong credentials: {status, body, contentType} with status 401 or 403, default 401 Unauthorized as plain text. Set it to what your gateway answers. | | multitenancy (--multitenancy) | Tempo's multitenancy_enabled: X-Scope-OrgID is required and a tenant reads only what it wrote. A query route without the header is 401 no org id; an export without it is 503 {"code":14,"message":"no org id"}. a\|b on a query reads both tenants. Off, the header is ignored. | | leftPadTraceIds (--left-pad-trace-ids) | Tempo's left_pad_trace_ids: search answers 32-character trace ids. Off (Tempo's default), leading zeros are trimmed. | | defaultLimit, maxLimit | Search's default (20) and maximum (262144) result count. |

Bearer tokens and Basic users can be configured together; a request needs one that matches.

Admin (beyond the standard contract)

| Route | Effect | | --- | --- | | POST /__admin/traces | {resourceSpans: [...], tenant?}: store an OTLP/JSON export exactly as given, with no credentials, faults or tenant header in the way. A span without startTimeUnixNano starts at the emulator clock (POST /__admin/clock), and without endTimeUnixNano ends there. Answers {accepted, tenant, traceIds}. | | GET /__admin/traces | {traces: [{tenant, traceID, spanCount, rootServiceName, rootTraceName, startTimeUnixNano, durationMs}]}, oldest first. ?tenant= narrows it. | | GET /__admin/spans | {spans: [...]}: the stored spans (hex ids beside the wire shape). ?traceId=, ?tenant=. | | GET /__admin/settings, PUT /__admin/settings | The settings above, for the calling namespace. POST /__admin/reset returns them to their defaults. |

Snapshots (POST /__admin/snapshots, POST /__admin/snapshots/:id/restore), checkpoints, reset, the clock, the request journal and metrics are the standard contract. The journal records that an export happened and which trace ids it touched, never span contents.

Fault presets (POST /__admin/faults {"preset": "<name>", "count"?: n}; "count": 1 is a one-shot; GET /__admin/faults/presets): ingest_unavailable (503, exporters retry), ingest_rate_limited (429 RATE_LIMITED: …, retried), ingest_server_error (500, dropped), ingest_dropped (the connection dies), query_unavailable (503), query_server_error (500 internal error), query_rate_limited (429 too many outstanding requests), query_timeout (504 context deadline exceeded), query_dropped, unauthorized (the configured 401 on every route). Ingest faults answer google.rpc.Status in the request's encoding.

Namespaces

Parallel workers each get their own store by x-mockingbird-namespace, by a /__admin/ns/<name> prefix on the base URL, or by credential: PUT /__admin/credentials {"credentials": {"<bearer token or Basic username>": "<namespace>"}}. Tenants (X-Scope-OrgID) are a separate wall inside a namespace.

What was verified, and how

No Tempo instance was available. Each behaviour was read from Grafana's documentation or Tempo's source (v3.1.0) and the OpenTelemetry Collector's OTLP receiver (v0.153.0, which Tempo embeds):

• Search parameters, defaults, validation order and messages: pkg/api/http.go (ParseSearchRequest), modules/frontend/search_handlers.go, search_sharder.go.
• Search and tags response fields and JSON types: pkg/tempopb/tempo.proto marshalled with gogo jsonpb defaults (modules/frontend/combiner/common.go), and the documented examples.
• Newest-first ordering, the overlap window, trimmed trace ids, <root span not yet received>: pkg/traceql/combine.go, tempodb/encoding/vparquet5/block_traceql.go, pkg/util/traceid.go, modules/frontend/combiner/search.go.
• batches, base64 ids, enum names and span ordering in trace by id; the bodiless 404: pkg/tempopb/trace_utils.go, pkg/model/trace/sort.go, modules/frontend/combiner/trace_by_id.go, and the example in "Push spans with HTTP".
• Tag scopes, sorting and the intrinsic list: pkg/api/search_tags.go, modules/frontend/tag_handlers.go, pkg/search/util.go, pkg/collector.
• TraceQL tokens, attribute lexing and error positions: pkg/traceql/lexer.go, parse.go, and the expected errors in parse_test.go.
• OTLP responses and errors: the collector's receiver/otlpreceiver/otlphttp.go and internal/errors/errors.go; id checks in modules/distributor/distributor.go.
• no org id: dskit middleware.AuthenticateUser for queries, Tempo's receiver middleware for ingest; cross-tenant reads: "Cross-tenant query federation".

Unverified (derived, or chosen where the sources say nothing):

• The default 401 for wrong credentials. It belongs to whatever gateway fronts Tempo (Grafana Cloud's body is not documented); configure unauthorized to match yours.
• Parse-error text beyond the cases in Tempo's parser tests. The emulator never appends goyacc's , expecting … list except where a test confirms it, and some invalid input that sits outside the subset is reported as "not yet supported" instead of as a syntax error.
• The message text of a 400 for an undecodable export (the status, code 3 and encoding are the receiver's; its decoder's wording is not reproduced), and for an id of the wrong length, which the receiver rejects before Tempo's distributor would.
• 503 no org id for an export without X-Scope-OrgID under multi-tenancy: derived from the receiver mapping a plain error to UNAVAILABLE, not observed.
• metrics values in search and tags (inspectedBytes, inspectedTraces): synthetic.
• serviceStats, and which attributes a matched span lists: taken from the engine's source and the documented example, not from a live response.
• Scope order in an unscoped tags response (Tempo's is unordered), the order of traces that start at the same nanosecond (the emulator uses the trace id), and which traces a limit keeps: Tempo "takes the first N results" of a parallel search, the emulator always keeps the newest.
• The 404 for a path Tempo does not serve (404 page not found).
• Bodies of the query fault presets other than internal error, too many outstanding requests and context deadline exceeded.

Deliberately not modelled

• The Grafana UI, dashboards, PromQL and metrics; TraceQL metrics (/api/metrics/query_range) and the metrics-generator.
• Full TraceQL: only the subset above is evaluated, everything else is a 400.
• The tag search that predates TraceQL (tags=, minDuration, maxDuration), filtered tag names (q on the tags route), tag values, and the v1 tags route: refused or absent.
• GET /api/v2/traces/{id} and Accept: application/protobuf on query routes: answers are JSON.
• Block-granular time ranges: start/end on trace by id and tags are validated, then ignored (the emulator is one block that always overlaps).
• Tail sampling, ingestion limits, retention, compaction, distributed storage, and the delay between ingest and searchability: a stored span is visible at once.
• OTLP/gRPC, Jaeger and Zipkin receivers; OTLP logs and metrics.
• Administration, provisioning, overrides and cloud resource creation.
• Span-id de-duplication for Zipkin-style shared ids, and Tempo's root-span and service statistics for traces spread over several tenants.

Send 64-bit OTLP/JSON values (timestamps, intValue) as strings, as exporters do: a JSON number past 2^53 loses precision before the emulator sees it. Values that do not fit their protobuf width, ids of the wrong length and attribute values nested more than 100 levels deep are refused with a 400.

API

| Export | Kind | Description | | --- | --- | --- | | TempoAPI | class | The in-process emulator: fetch(request), reset(), inject(payload, tenant?), spans(filter?), traces(tenant?). Options: sqlite, now, namespace, settings. | | createRuntime | function | The emulator with the full service contract (health, admin, namespaces, credentials, presets, clock, checkpoints). Options: settings, clock, seed, adminPrefix, adminKey, onLog, sqlite. | | TEMPO_PRESETS | object | Every named fault preset. | | TEMPO_NAMESPACE | string | The service name, "tempo". | | DEFAULT_SETTINGS | object | The settings a namespace starts with. | | SINGLE_TENANT | string | Tempo's tenant when multi-tenancy is off, "single-tenant". | | ORG_ID_HEADER | string | "x-scope-orgid". | | ROOT_SPAN_NOT_YET_RECEIVED | string | rootServiceName of a trace without a root span. | | INTRINSIC_TAGS | array | The names under the intrinsic tag scope. | | tempoCredential | function | The bearer token or Basic username of a request (how credentials map to namespaces). | | parseTraceQL, matchesSpan, TraceQLError | functions, class | The TraceQL subset: parse a query, test one span's attributes against it, and the error whose message follows invalid TraceQL query: . | | decodeSpans, validateIds, OtlpError | functions, class | An OTLP/JSON export as spans in Tempo's wire shape, Tempo's id checks, and the error a refused export raises. | | decodeTraceRequest, encodeRpcStatus, decodeRpcStatus, ProtobufError | functions, class | The protobuf ExportTraceServiceRequest decoder and google.rpc.Status codec. | | hexToBase64, base64ToHex | functions | Convert ids between OTLP's hex and Tempo's base64. | | document, operationIds, supportedOperationIds | values | The OpenAPI contract and its operation ids. | | createServer, serveTarget, DEFAULT_PORT (./server) | Node | Serve over node:http; the serve CLI target; port 8930. |

Part of mockingbird.