Observability stack

A Docker Compose bundle that wires Grafana, Tempo, Loki, and an OpenTelemetry Collector together with a lightweight Node CLI for managing the stack from anywhere in the workspace.

Getting started

1. Bootstrapping the monorepo so the CLI is available:

   pnpm install

2. Copy the sample environment file into place near the stack so you can override the default ports/credentials:

   cp packages/obs-stack/.env.example packages/obs-stack/.env

3. Tweak any ports or credentials in packages/obs-stack/.env if you hit conflicts. The compose stack already declares persistent volumes (grafana-data, tempo-data, loki-data) so dashboards, traces, and logs survive restarts.
4. Bring the stack up from the repository root (the CLI auto-resolves the stack directory for you):

   pnpm --filter @emeryld/obs-stack obs-stack up

   The CLI sets OBS_STACK_DIR to the packages/obs-stack directory so every docker compose run sees the Grafana provisioning and config files no matter where you invoke the command. If you manage the stack by calling docker compose directly, either run the command from packages/obs-stack or set OBS_STACK_DIR (for example OBS_STACK_DIR=packages/obs-stack docker compose up) so the bind mounts keep pointing at grafana/provisioning/dashboards/files.
5. Inspect the endpoints reported by the CLI:

   pnpm --filter @emeryld/obs-stack obs-stack urls

The CLI loads packages/obs-stack/.env and also merges in any .env file that lives beside the directory where you run the command, so you can keep machine-specific overrides outside of version control.

CLI commands

• obs-stack up [...] – starts the stack (docker compose up -d plus any extra args you pass).
• obs-stack down [...] – stops the containers without deleting persistent volumes.
• obs-stack reset [...] – runs docker compose down -v so you can start from scratch.
• obs-stack status [...] – shows docker compose ps for the stack so you can see container health.
• obs-stack logs [...] – tails all services (docker compose logs --follow --tail 100 [...]). Pass service names or additional flags to limit the stream.
• obs-stack urls – prints the Grafana, Tempo, Loki, and OTLP host endpoints using your current .env overrides.

Each command ensures both docker and docker compose are available before delegating to Compose. Run the CLI from the workspace root via pnpm --filter @emeryld/obs-stack obs-stack <command> so it can find the Compose file and configs, or cd packages/obs-stack and run the same command if you prefer.

Interactive menu

Run the CLI with no arguments to launch the helper menu from a terminal:

pnpm --filter @emeryld/obs-stack obs-stack

You can also pass menu, --menu, --interactive, or -m to open it explicitly. The menu lets you pick bring-up, stop, reset, status, logs, or URLs by pressing the numbered keys.

The legacy obs-stack-menu binary is still published (and available via pnpm --filter @emeryld/obs-stack obs-stack-menu) for backwards compatibility.

Environment variables

| Name | Description | Default | | --- | --- | --- | | GRAFANA_PORT | Host port forwarded to Grafana | 3000 | | GRAFANA_USER | Grafana admin user (provisioned) | admin | | GRAFANA_PASS | Grafana admin password (provisioned) | admin | | OTEL_OTLP_HTTP_PORT | Host port forwarded to OTLP HTTP (4318 inside the collector) | 4318 | | OTEL_OTLP_GRPC_PORT | Host port forwarded to OTLP gRPC (4317 inside the collector) | 4317 | | LOKI_PORT | Host port forwarded to Loki query API (3100) | 3100 | | TEMPO_HTTP_PORT | Host port forwarded to Tempo query API (3200) | 3200 |

Override any of these by editing packages/obs-stack/.env, creating another .env alongside your working directory, or exporting the variables before running the CLI.

Example workflows

Bringing the stack up, checking status, and watching URLs

cp packages/obs-stack/.env.example packages/obs-stack/.env
pnpm obs-stack up
pnpm obs-stack status
pnpm obs-stack urls
pnpm obs-stack logs   # add service names or flags if you need to scope the output

obs-stack urls prints the host ports for Grafana, Tempo, Loki, and both OTLP endpoints so you can plug them into clients or instrumentation.

Running a backend against the stack

1. Install @emeryld/otel-bootstrap into your backend workspace:

   pnpm add @emeryld/otel-bootstrap

   Then either import "@emeryld/otel-bootstrap"; as early as possible in your entry point or preload it via NODE_OPTIONS="--require ./node_modules/@emeryld/otel-bootstrap/dist/otel.cjs" when running node so traces + logs are configured automatically.
2. Point the backend at the collector depending on its network scope:
   • Backend running inside the Compose network (see the override example below):

     OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
     OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

   • Backend running outside the Docker stack (e.g., on your host or in another service):

     OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:${OTEL_OTLP_HTTP_PORT}
     OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

3. Keep OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES defined so traces/logs carry meaningful service metadata.

Once telemetry is flowing, Grafana Explore (Tempo + Loki) will start showing traces and logs with matching trace_id values.

Running the backend + stack together

If you want your backend inside the same Docker network as the observability stack (so it can target otel-collector by name and share volumes), use the override example:

cd packages/obs-stack
docker compose -f docker-compose.yml -f examples/docker-compose.override.yml up -d

Adjust the backend service inside examples/docker-compose.override.yml to match your build image, ports, and runtime command. The override already makes the service depend on otel-collector and sets its OTLP exports to http://otel-collector:4318.

Grafana provisioning

Grafana relies on the datasources defined in grafana/provisioning/datasources/datasources.yaml to wire up Loki and Tempo automatically. You never need to configure the data sources manually—just open Grafana at the URL reported by obs-stack urls. Each datasource now exposes a stable uid (loki and tempo) so the shipping dashboards can reference them reliably.

We also preload three starter dashboards via grafana/provisioning/dashboards/. Each JSON file targets the Loki datasource (uid loki) so they work out of the box:

• Logs overview: log volume by log type, application log level breakdown, top application event names, and a recent log stream.
• Error spotlight: app error counts, request 5xx counts, schedule failures, an error-rate trend, and a recent error stream with top failing routes.
• Service telemetry: request throughput + latency percentiles, top request paths, cache hit ratio/latency, schedule status counts, and socket event volume.

Edit the JSON in grafana/provisioning/dashboards/files/ (or export updates from Grafana) and restart the stack or reload dashboards to see your changes automatically.

Troubleshooting

• Ports already in use: edit packages/obs-stack/.env (or your local override), then run pnpm --filter @emeryld/obs-stack obs-stack reset to recreate the stack with the new settings.
• Grafana dashboards missing: ensure the OBS_STACK_DIR environment variable points at packages/obs-stack (the CLI sets it automatically) so Grafana sees grafana/provisioning/dashboards/files. Running docker compose without OBS_STACK_DIR from another directory binds an empty host path and Grafana keeps logging stat /etc/grafana/provisioning/dashboards/files: no such file or directory; once the variable points at the stack, restart with obs-stack reset.
• Collector fails to start: make sure the collector can resolve tempo and loki. The CLI always runs Compose inside the package directory so no host networking is required.
• docker compose not on PATH: install Docker Desktop (Mac/Windows) or Docker Engine with the Compose plugin (Linux) so both docker and docker compose can run.

Tests

The package ships with a lightweight Node test suite that enforces that the Compose stack is wired up the way the README describes. Run it with:

pnpm --filter @emeryld/obs-stack test

The tests check that the services/all volumes exist, host networking is avoided, Grafana data sources point at the local Loki/Tempo, the collector exports to both Loki and Tempo, configs keep 24h retention, and the env example still documents the exposed ports.