tracecase

Version-controlled, replayable eval cases for AI agents — catch regressions in CI.

AI agents regress silently: a prompt tweak, a model bump, or a refactor quietly changes behavior, and you find out in production. tracecase turns a recorded agent run into a checked-in test you can replay forever — deterministic, offline, and CI-gated.

record a run  →  ingest  →  YAML case  →  replay (mock tools)  →  score  →  exit ≠0 on regression

How it works

You record one good run of your agent. tracecase normalizes it into a vendor-neutral Trace, saves a version-controlled case (the input, the tool calls the agent made, and the assertions that must hold), then replays that case against your current agent with the tools mocked from the recording. Pluggable evaluators score the result, and tracecase ci exits non-zero on regression so a PR that breaks the agent fails the build.

Two ideas make it work:

• Vendor-neutral trace model — OpenInference, OTel GenAI, and LangChain all normalize to one Trace; nothing downstream knows the source format.
• Replay tool-mocking — a case answers each tool call from the recording, so tests are deterministic and offline, and evaluators check what the agent did, not just what it said.

Quickstart

npm install --save-dev tracecase

Then the loop:

npx tracecase init                       # scaffold tracecase.config.mjs + a sample case
npx tracecase ingest trace.json          # normalize a recorded trace into the store
npx tracecase flag <traceId>             # draft a case from it (LLM-assisted)
npx tracecase run cases/my-case.yaml     # replay + score against your agent
npx tracecase ci                         # gate CI: exit non-zero on regression

Working on tracecase itself instead:

git clone https://github.com/vdeshmukh1697/tracecase && cd tracecase
npm install && npm run build && npm test

tracecase run prints one line per assertion and exits non-zero on any failure:

refund-checks-policy
  ✓ contains "order #4471"
  ✗ tool_called check_refund_policy   (tools called: lookup_order)

1 case, 1 passed, 1 failed — FAIL

Features

• 3 ingest formats, auto-detected — OpenInference · OTel GenAI (gen_ai.*) · LangChain run-trees
• Replay (deterministic, offline) or live mode against your real tools
• 6 evaluators — contains, tool_called, tool_order, json_schema, no_fabricated_numbers, and an LLM judge
• Baseline-diff CI gate with a ready-made GitHub Action
• Live OTLP receiver (tracecase serve), a self-contained HTML report, and a local web UI — run a case and see ✓/✗ per check, a CI regression diff, good-vs-buggy trace comparison, live trace arrival, and in-browser case editing
• Flag — draft a case from a real trace with LLM assistance
• Three worked example agents, one per ingest format — examples/{meal-plan,intake,content}-agent/ show the whole pattern end to end

Connect your agent

tracecase init scaffolds a working config; point runAgent at your agent, or use the bundled reference agent:

// tracecase.config.mjs
import { defineConfig, createReferenceAgent, createAnthropicModel } from "tracecase";

export default defineConfig({
  runAgent: createReferenceAgent({ model: createAnthropicModel() }), // needs ANTHROPIC_API_KEY
  tools: {
    async lookup_order(args) {
      // your real tool, used in `--mode live`
      return { status: "delivered" };
    },
  },
});

Replay mode (the default) answers tool calls from the recording; --mode live runs them against the tools above. run and ui auto-detect tracecase.config.* in the working directory, or take an explicit --config <path> (handy for pointing the UI's Run button at a specific agent). See Writing evals for the full config and case format.

In use: gating a real content pipeline

examples/content-engine-guard/ is tracecase running against a live workflow, not a toy. A nutrition coaching practice drafts Instagram posts with an LLM; every draft has to clear a safety bar (a coach may not claim to diagnose, treat, cure, or reverse anything, name supplement doses, or promise outcomes) and a house-style bar before it goes out. That review used to be a human re-reading their own drafts — the step most likely to be skipped when you're shipping daily.

The guard makes it a gate. It uses tracecase as a library rather than a CLI — there are no tool calls to replay, so it wraps the post text in a minimal Trace and runs the evaluators over it:

npm run lint-post -- drafts/11am-energy-crash.md --topic "the 11am energy crash"

Three checks, mixing deterministic and LLM evaluators: contains for the handle (a must-have, no judgment required), then two judge rubrics for safety and brand voice. Non-zero exit on any failure, so it drops into a pre-publish step.

The interesting part is the test strategy. LLM judges are non-deterministic and cost money, which usually means "untested." Here the judge is an injected seam, so guard.test.ts swaps in a deterministic offline judge and proves the discrimination that matters — a real on-brand post passes all three checks; an unsafe variant with disease claims, a supplement dose, and a guaranteed outcome is caught by both judges. 4 tests, no API key, no network, runs in milliseconds. Production swaps the real Anthropic judge back in.

That seam is the whole argument for the design: the evaluator you can't make deterministic is the one you most need to be able to test.

Documentation

• Instrument your agent — emit traces tracecase can ingest
• Writing evals — the case format and every evaluator
• Contributing — dev setup and the four core principles
• Architecture — the pipeline, code layout, and design principles
• Postmortem: the 0.1.0 no-op CLI — why a green suite didn't mean a working package

Status

The core tool is feature-complete — 439 tests, fully offline and deterministic: three ingest formats, six evaluators, the CI gate, a full web-UI workbench, and three worked example agents. Published on npm; install 0.1.1 or later — 0.1.0's CLI silently no-oped when run through npx. See NEXT.md for what shipped and what's next.

License

MIT