@app-studio/qa-harness

The shape of a test harness, and a record/replay transport whose missing fixtures always fail.

The harness

Two small types and a helper, deliberately:

import { defineActors, type TestApp } from '@app-studio/qa-harness';

export async function createTestApp(): Promise<TestApp<Client>> { /* your bootstrap */ }

const actors = defineActors(app, (app) => ({
  signUp: (email: string) => /* … */,
  createWorkspace: (owner: Actor) => /* … */,
  addMember: (workspace: string, actor: Actor, role: Role) => /* … */,
}));

What a harness must be is a rule, not a library, and this package cannot enforce it — only say it clearly:

The application under test has to be the application that ships. Same composition, same HTTP configuration, same startup. A hand-assembled test app validates something that does not exist in production, and the resulting green suite is a statement about a program nobody runs. Write the factory against your real bootstrap.

Prefer listening on a real port if the application attaches anything to the HTTP server — WebSockets, most commonly. On a server that was never started, the socket library's takeover of the request listeners races the framework's own handler, and the occasional request falls through as a 404 on a route that plainly exists. That failure is intermittent, which is the worst kind.

A fresh client per actor. Sharing one client across two roles is how an authorization test quietly starts proving nothing.

Actors are what make authorization testing cheap enough to actually do. When "an owner, a member and an outsider" costs three lines, every endpoint gets its cross-tenant probe; when it costs thirty, the probe is what gets skipped — and cross-tenant access is the single most common real-world API defect.

Record and replay

For anything whose answer is neither stable nor free: a model, a payment provider, a third-party search.

import { assertReplayBootSafety, createReplayTransport } from '@app-studio/qa-harness/replay';

assertReplayBootSafety({ mode: process.env.AI_MODE ?? 'live' });

const ask = createReplayTransport<Request, Response>({
  mode: process.env.AI_MODE as ReplayMode,
  dir: 'tests/fixtures/ai',
  live: (request) => realProvider.complete(request),
  source: 'gpt-4o-2026-01',
});

Three rules, each the fix for a specific way this goes wrong.

A missing fixture always fails. Never a fallback, never a default answer. A transport that quietly substitutes something plausible turns "this test no longer asks the question it was written for" into a green run — the exact failure the fixtures exist to prevent.

The key is a digest of the request. Not a filename somebody chose, and not the test's name. Change what you ask and the fixture stops matching, so a suite cannot keep replaying the answer to a question it no longer asks. Key order does not affect the digest, so refactoring the request object does not invalidate every recording.

A fixture says where it came from. A recording captured against a version of a service that has since changed is indistinguishable from a fresh one, and tests replaying it assert last year's behaviour with total confidence. The provenance block is required on read; qa-guard denies committing a fixture without one.

The boot guard

assertReplayBootSafety refuses to start in live or record mode under NODE_ENV=test. A guard at the transport is too late and too easy to bypass; the point is that a process which could reach the live service never starts inside a test run in the first place — a run that reaches a real service is slow, costs money, and is not reproducible.

Housekeeping

unusedFixtures(dir, usedDigests) lists recordings nothing asked for. Worth checking after a recording session: a fixture nobody replays is either dead weight or evidence that a code path stopped being exercised, and both are things to know before they are a year old.

Requirements

Node.js 20 or newer. No runtime dependencies beyond @app-studio/qa-core.