@duskel/cassette
v0.1.0
Published
Deterministic record/replay for LLM and tool calls, recorded at the SDK boundary instead of the socket.
Maintainers
Readme
cassette
Deterministic record/replay for LLM and tool calls. Record your agent once against a live model, then replay it forever: the same prompts return the same completions, the same tool calls run against recorded results, and a readable diff tells you the moment a prompt or a tool signature drifts away from its recording.
Agent tests become fast, free and reproducible. No API key in CI, no flakes, no bill.
Model calls served from the cassette 3
Tools executed for real 2
Completed in 2.7msThe problem
Testing an agent means running a loop that calls a model, executes tools, feeds the results back, and calls the model again. Every run of that loop is slow, costs money and returns something slightly different. So agent test suites end up in one of two bad places: nobody runs them, or everybody mocks the model so heavily that the test no longer resembles the thing being shipped.
Why HTTP mocking is the wrong layer
The obvious fix is the one HTTP fixtures libraries have used for a decade: intercept the
socket, replay the response body. vcrpy and friends do this well, for the problem they
were designed for.
It is structurally wrong for agents.
When you mock at the HTTP layer, you are replacing the transport. But an agent's interesting behaviour is not in the transport, it is in the loop around it:
- The model returns a
tool_useblock. Your code has to parse it, dispatch it, run the tool, and format the result back into the next request. That is where agents break. - Mock the HTTP response and your tool never actually runs. The dispatch path, the argument coercion, the error handling in the tool itself: all untested.
- SSE framing, retries and base-URL changes are transport details. Recording them couples your fixtures to the SDK's internals, so an SDK upgrade invalidates cassettes that describe conversations that did not change.
cassette records at the SDK boundary, not the socket. It patches
client.messages.create, not fetch. The consequence is the important part:
Under replay, your agent loop, your tool dispatch and your tools all still execute for real. Only the model's answers come off the tape.
If convertTemperature has an off-by-one, the replayed test fails, exactly as production
would. An HTTP-level mock cannot tell you that, because it never called the function.
You can still record the tool boundary when you want to, for tools that are slow, paid
or non-deterministic. That is what the tools adapter is for, and it is opt-in per tool.
Install
npm install --save-dev @duskel/cassetteNode 20.11+, ESM, TypeScript types included, zero runtime dependencies. The Anthropic and OpenAI SDKs are optional peer dependencies; you only need the one you actually use.
Quickstart
import Anthropic from '@anthropic-ai/sdk';
import { withCassette, anthropicAdapter } from '@duskel/cassette';
import { expect, test } from 'vitest';
test('the agent converts the temperature it looked up', async () => {
const client = new Anthropic();
const run = await withCassette('weather-agent', async (tape) => {
tape.use(anthropicAdapter, client);
return runAgent(client, 'What is the weather in Paris, in fahrenheit?');
});
expect(run.answer).toContain('69.8');
expect(run.toolCalls.map((c) => c.name)).toEqual(['get_weather', 'convert_temperature']);
});The default mode is auto: the first run has no cassette, so it records against the live
API and writes cassettes/weather-agent.cassette.json. Commit that file. Every run after
that replays it, offline, in milliseconds.
Run the example
The repository ships a working agent with two tools and a committed cassette. It uses
the real @anthropic-ai/sdk and needs no API key:
git clone <this repo> && cd cassette
npm install
npm run exampleReal output:
Replaying from the committed cassette. No network, no API key.
Question What is the weather in Paris right now, in fahrenheit?
Answer It is currently 69.8 degrees fahrenheit and clear in Paris.
Model calls served from the cassette 3
Tools executed for real 2
get_weather({"city":"Paris"}) -> {"city":"Paris","temperature":21,"unit":"celsius","conditions":"clear"}
convert_temperature({"from":"celsius","to":"fahrenheit","value":21}) -> {"value":69.8,"unit":"fahrenheit"}
Completed in 2.7msThree model calls served from disk. Both tools executed for real, in the replayed run. That is the difference from HTTP mocking, in one command.
Drift detection
When a request stops matching its recording, you get a diff, not a shrug. Change the example's question and re-run:
cassette: no recording matched this request.
cassette weather-agent
adapter anthropic
operation messages.create
model claude-sonnet-4-5
recorded 3 interactions
Closest recording is #1 (98% payload match, 1 field differs):
~ payload.messages[0].content
recorded: "What is the weather in Paris right now, in fahrenheit?"
current: "What is the weather in Lisbon right now, in fahrenheit?"
Re-record with CASSETTE_MODE=record, or relax matching with
`ignorePaths` / `fuzzy` if the difference is not meaningful.This is the feature that makes recorded fixtures survive contact with a real codebase. Edit a system prompt six months from now and the failing test tells you which prompt, on which line, changed how.
Modes
| Mode | Behaviour |
| ------------- | ---------------------------------------------------------------- |
| auto | Replay if a cassette exists, otherwise record it. The default. |
| record | Always call through and rewrite the cassette from scratch. |
| replay | Never call through. An unmatched request throws. |
| passthrough | Call through and record nothing. Useful for a one-off live check. |
Set per call site, or globally with CASSETTE_MODE:
CASSETTE_MODE=record npm test # refresh every cassette
CASSETTE_MODE=replay npm test # what CI should run: fails rather than dialling outreplay is the mode you want in CI. A miss throws CassetteMissError instead of
falling through to the network, so a stale fixture can never turn into a surprise API
bill.
Matching
By default a request matches a recording when the adapter, operation, model, tool name and full payload all agree. Loosen it when something legitimately varies:
await withCassette('agent', run, {
// Drop volatile fields before comparing.
ignorePaths: ['metadata.request_id', 'messages[*].timestamp'],
// Normalize non-deterministic values on both sides.
fuzzy: { timestamps: true, ids: true, whitespace: true, numberPrecision: 2 },
});Or replace the matcher stack entirely:
import { matchers } from '@duskel/cassette';
await withCassette('agent', run, {
matchers: [
matchers.adapter,
matchers.operation,
matchers.messages, // conversation only; temperature and max_tokens may vary
matchers.custom((request, recorded) => request.payload.tenant === recorded.payload.tenant),
],
});Built-in matchers: adapter, operation, model, toolName, payload, messages,
toolArguments, field(path), payloadIgnoring(paths) and custom(fn).
Redaction
Redaction is on by default and runs before a cassette touches disk. A leaked key in
a committed fixture is the failure mode that would make this library not worth using, so
the built-in rules err towards over-redacting: Anthropic, OpenAI, Stripe, GitHub, Slack,
Google and AWS credentials, JWTs, bearer tokens, private key blocks, e-mail addresses,
and the value of any field named like a secret (authorization, api_key, cookie,
password, and others).
await withCassette('agent', run, {
redact: {
patterns: [/CUST-\d{6}/, 'acme-internal-tenant'],
keys: ['x-internal-trace'],
transform: (value) => value.replaceAll(realCustomerName, 'CUSTOMER'),
},
});Redaction also runs on the incoming request during replay, so a redacted recording still matches a live request carrying the real credential.
CLI
npx cassette list
npx cassette inspect weather-agent
npx cassette prune --delete
npx cassette verifyCASSETTE CALLS ADAPTERS SIZE RECORDED
------------- ----- --------- ------- ----------
weather-agent 3 anthropic 11.2 KB 2026-07-24
1 cassette in /path/to/examples/cassetteslist- every cassette, its call count and which adapters it used.inspect <name>- the interactions in one cassette;--index nfor a full payload.prune- cassettes no longer referenced from any source file. Dry run by default.verify- fails if any committed cassette contains something that looks like a credential. Put this in CI:
- run: npx cassette verifyChecked 1 cassette in /path/to/examples/cassettes.
No credentials found.Adapters
The adapter registry is the architecture. One file per SDK, and every adapter is validated by the same shared contract suite.
| Adapter | Covers |
| ----------- | ---------------------------------------------------------------------------- |
| anthropic | @anthropic-ai/sdk: messages.create (streaming and not), countTokens |
| openai | openai: chat.completions.create, responses.create, embeddings.create |
| tools | Any tool registry: bare functions or { execute } descriptors |
| fetch | Generic HTTP fallback for providers without a dedicated adapter |
Adapters use structural typing against the SDK surfaces, so the SDKs stay optional peer dependencies and the whole test suite runs with neither installed, with no network and no API key.
Adding one is one file plus one call to runAdapterContract. See
CONTRIBUTING.md.
Test output
Run from a clean checkout with npm test, offline:
RUN v2.1.9 /Users/.../cassette
✓ test/matchers.test.ts (20 tests) 3ms
✓ test/redact.test.ts (23 tests) 5ms
✓ test/drift.test.ts (18 tests) 3ms
✓ test/store.test.ts (26 tests) 39ms
✓ test/example.test.ts (10 tests) 16ms
✓ test/session.test.ts (18 tests) 44ms
✓ test/contract/anthropic.test.ts (33 tests) 80ms
✓ test/cli.test.ts (24 tests) 110ms
✓ test/contract/tools.test.ts (33 tests | 3 skipped) 84ms
✓ test/contract/openai.test.ts (33 tests) 134ms
✓ test/contract/fetch.test.ts (33 tests | 3 skipped) 136ms
Test Files 11 passed (11)
Tests 265 passed | 6 skipped (271)
Duration 440msThe four test/contract/* files are the same 33 assertions applied to four different
adapters. The six skips are streaming assertions on the two adapters that do not stream.
Documentation
- docs/FORMAT.md - the versioned cassette file format.
- CONTRIBUTING.md - writing an adapter, and how the contract validates it.
- docs/GOOD-FIRST-ISSUES.md - scoped starting points.
Known limitations
Stated plainly, because finding these out later is worse:
client.messages.stream(), the Anthropic event-emitter helper, is not intercepted yet.messages.create({ stream: true })is. See the good-first-issues list.- Recording a stream drains it fully before returning, so record mode does not reproduce live token timing. Replay and record then return the same shape, which is deliberate.
- The
fetchadapter buffers response bodies, so it is not suitable for very large downloads, and it stores binary bodies as a size marker rather than base64. prunefinds cassette names written as string literals. Names built at runtime cannot be detected, which is why it dry-runs by default.
Built and maintained by Duskel
We build and audit production AI agents. cassette came out of needing our own agent test suites to run in CI without an API key and without flaking, and we maintain it because we use it.
If you are shipping an agent and want a second pair of eyes on it, or you want one built properly the first time, talk to us: duskel.com.
Issues and adapter pull requests are welcome regardless.
License
MIT, copyright Duskel. See LICENSE.
Built and maintained by Duskel — an AI automation & software studio. We build LLM apps and needed deterministic tests that didn't depend on the network to run. Work with us.
