llm-wire-stub

A scripted Anthropic Messages API at the network boundary, for Playwright tests. Real SDK, real SSE decode, real tool loop — only the bytes are scripted.

Why: green theater

This was extracted from the test suite of a private product. That suite's gate tests used to stage model output straight into application state — write a pending proposal card, click approve, assert. Every test green. It proved the consumer of that state worked, and never once exercised the path that produces it: a regression that wrote proposals pre-approved (and dropped the approval guard) left the whole suite green.

The honest fix is to stub the wire, and nothing above it. The app under test runs unmodified: a real @anthropic-ai/sdk client, a real MessageStream, a real SSE decode, a real tool loop. The stub intercepts api.anthropic.com at Playwright's network layer and answers with the documented streaming envelope:

message_start → (content_block_start → …delta… → content_block_stop)* →
message_delta → message_stop

built strictly enough that the SDK's own accumulator accepts it — and fails loudly on a wrong shape. That claim is demonstrated, not asserted: tests/envelope.test.ts feeds the stub's bytes through the real SDK and shows finalMessage() reconstructing text and tool_use input exactly, then feeds it deliberately broken streams and shows the SDK throw.

Quickstart

npm install -D github:egnaro9/llm-wire-stub   # npm registry publish pending

The package ships compiled ESM plus type declarations in dist/ (with src/ alongside for reading), so it imports cleanly from node_modules — no transpiling of package internals required.

import { test, expect } from "@playwright/test";
import { stubAnthropic } from "llm-wire-stub";

test("the agent answers from the scripted wire", async ({ context, page }) => {
  const stub = await stubAnthropic(context, [
    { text: "First scripted answer." },
    {
      text: "Filing a card now.",
      tool: { name: "create_card", input: { title: "prove the loop" } },
    },
    { text: "Card filed. Done." },
  ]);

  await page.goto("/");                       // your app, unmodified
  // …drive the UI; the app's real SDK client hits the stub…

  expect(stub.requests).toHaveLength(3);      // what the app actually sent
  expect(stub.overflow).toBe(0);              // no unscripted model calls
});

Your app authenticates however it normally does — the stub answers every request regardless of key, and records the x-api-key header it saw. Seed a fake key into your app the way your app reads it; the stub deliberately does not touch storage.

Fixture expressiveness IS coverage

A fixture that cannot produce a second turn makes "requests == quoted" pass vacuously. A scripted turn with a tool entry streams a tool_use block as input_json_delta fragments (the way the real API does) and stops with stop_reason: "tool_use" — so the SDK's tool loop sends a tool_result back and takes a second request. The spec a tool turn makes the SDK loop take a second request in e2e/stub.spec.ts drives a real browser-side SDK loop through both round trips and asserts on the second request body.

Scripting failure: errors and stop reasons

A stub that can only say yes cannot test the app's bad day. A turn with an error answers with the documented Anthropic error JSON at the given HTTP status, and the real SDK surfaces it as a catchable APIError subclass (429 → RateLimitError), exactly as in production:

const stub = await stubAnthropic(context, [
  { error: { status: 429, type: "rate_limit_error", message: "Rate limited." } },
]);
// …drive the UI; assert the app shows its rate-limit state, not a crash…

(The real SDK retries 429/5xx by default — an error-turn spec either scripts the retries too or runs the client with maxRetries: 0.)

A success turn can also override its stop reason — stopReason: "max_tokens" scripts a truncated turn, and finalMessage() reports it, so the app's truncation handling is testable:

{ text: "an answer that was cut of", stopReason: "max_tokens" }

Assert on request bodies

stub.requests records what the app sent — model, system prompt, messages (including tool_result blocks), tool names, api key. A string system is recorded verbatim; the block-array form (the prompt-caching shape) is recorded as the blocks' text joined with newlines, with the original under systemRaw. The response side of a test says the app can render what it was given; the request side says the app asked the right question. That is where context-plumbing bugs live: a missing message in requests[n].messages is a node that never saw upstream output, provable in one assertion.

hold() / release(): concurrency observed, not assumed

const stub = await stubAnthropic(context, (req) =>
  JSON.stringify(req.messages).includes("alpha")
    ? { text: "for alpha" }
    : { text: "for beta" }
);

stub.hold();                 // responses now block
// …trigger two sends in the UI…
await expect.poll(() => stub.requests.length).toBe(2);  // both IN FLIGHT
stub.release();              // both complete

If the producer serialized its calls, the second request could never reach the wire while the first is still pending — so requests.length == 2 under hold is proof of concurrency, not a timing accident. Note the function script: an array keys answers to arrival order, which is a lottery under concurrency; a function keys them to who asked.

API surface

• stubAnthropic(context, script, options?) → AnthropicStub
  • script: ScriptedTurn[] (answers by arrival order) or (request, index) => ScriptedTurn | undefined (answers by content)
  • options.url: route pattern, default https://api.anthropic.com/**
  • Only POST …/v1/messages (and its CORS preflight) is fulfilled; anything else under the pattern falls through via route.fallback()
• ScriptedTurn: text?, tool?, stopReason? (success override, e.g. "max_tokens"), error? ({ status, type, message } — answers with the Anthropic error JSON instead of a stream)
• AnthropicStub: requests (recorded bodies), overflow (off-script call count — assert it is 0), hold(), release(). One outstanding hold at a time: a second hold() releases the previous gate's waiters before installing the new gate.
• sseBody(turn, model?): one complete text/event-stream body, exported for use outside Playwright
• errorBody(error): the Anthropic error JSON for a ScriptedError, likewise framework-free

Limits (read before adopting)

• Anthropic envelope only. One provider, tested end to end, is a stronger claim than multi-provider support with one tested path. No OpenAI or Gemini framing.
• Playwright-oriented. stubAnthropic wants a Playwright BrowserContext. (sseBody and errorBody are framework-free — the vitest suite uses them with a plain custom fetch.) The declared peer range is @playwright/test >=1.40.0; this repo tests against 1.62. The APIs used (context.route, route.fulfill, route.fallback) long predate 1.40, but versions below 1.62 are not exercised in CI.
• Messages API v1 streaming only. No batch API, no Files API, no extended-thinking blocks, no citations content, no server tool use. Success turns are text and/or one tool_use block; failures are the error variant. Nothing else is expressible, on purpose.
• Scripted, not simulated. The stub never invents behavior; if your script runs out, the overflow counter says so loudly.

Provenance

Extracted from the Playwright suite of a private product, where it replaced a state-staging fixture and caught real defects a green suite had been hiding — including a producer that dropped context between agents (visible only in the recorded request bodies) and a UI price chip that disagreed with what was actually sent over the wire. Those anecdotes are provenance history from the private suite; they are not reproducible from this repository. Every claim about what the stub does, though, is demonstrated by a test here: npm test (envelope through the real SDK) and npm run test:e2e (browser fixture through the real SDK's tool loop).

License

MIT