@nage-api/testing

The shared test kit (PLAN.md §19, §24 Phase 9).

Depends on @nage-api/contracts and nothing else — no test runner, no driver, no Docker. That constraint is what makes it usable from every other package without creating a cycle, and it shapes most of the design decisions below.

What's in it

| | | | ----------------------- | --------------------------------------------------------------------------------------- | | Security packs | Cases a whole class of implementation must pass, parameterised over an injected harness | | expectNoSecrets | The leak check, done properly: non-enumerable Error fields, encodings, cycles | | Fixtures | Deterministic builders — a suite cannot fail once a month at midnight | | Port doubles | Real implementations of the framework's ports | | Envelope assertions | The three things every e2e test asserts about a response | | Container harness | Pinned images, retry, parallel-safe — and a clean skip with no Docker |

Security packs

A pack is the case list, not the wiring. You supply a harness that knows how to drive your implementation; the pack supplies the attacks.

import { describeQueryAllowList, type TestApi } from '@nage-api/testing';
import { afterEach, beforeEach, describe, expect, it } from 'vitest';

import { parseQuery } from '@nage-api/data';
import { policy } from './widget.policy.js';

// The injected runner — see "Why the runner is injected" below.
const api: TestApi = { describe, it, beforeEach, afterEach, expect };

describeQueryAllowList(api, {
  name: 'parseQuery',
  parse: (query) => parseQuery(query, policy),
  forbiddenField: 'cost_price',
  allowedField: 'name',
  maxLimit: 50,
  allowedOperators: ['eq', 'gt'],
  // Only operators the policy really refuses. `ne` is in `DEFAULT_OPERATORS`, so
  // listing it here against a policy that does not narrow `operators` fails the
  // case — and the failure is the pack's, not the parser's.
  forbiddenOperators: ['regexp', '$where'],
});

The four packs:

• describeQueryAllowList — a forbidden field at the top level, nested in and/or, and nested twice; limit: -1, above the ceiling, at the ceiling (must pass), non-numeric; a negative offset; __proto__ and constructor as keys; each allowed and forbidden operator.

• describeAuthzBypass — no credential, the wrong grant, empty roles and permissions, and look-alike grants that differ only by case, substring or whitespace. Also asserts a refusal names neither the missing grant nor the target, since a precise "403 you lack billing:admin on invoice 41" is an enumeration oracle.

  The look-alike cases substitute into the authorised principal, not the unauthorised one. It reads backwards until you try it the other way: a guard that also consults a store refuses an unauthorised id on the store check, never reaches the grant comparison, and passes the case while still granting admin to a stored principal whose role is administrator-readonly. Conversely, a surface that re-derives grants from a store — a resolver in front of a guard — cannot be attacked through the credential at all; the pack detects that shape by stripping the grants first, and covers it with forgedClaims instead.

• describeUploadValidation — the nine files in MALICIOUS_UPLOADS: PHP bytes behind a .png name, a double extension, an SVG, HTML behind image/png, an empty file, a traversal filename, a name whose real extension hides behind a separator, a truncated magic number, and a type outside the allow-list. Then JPEG bytes declared as image/png, and shell.PHP in capitals for a denied-extension check written case-sensitively. Plus a control case, because a validator that refuses everything passes an all-negative suite.

• describeErrorNonLeakage — the operator detail, error metadata and cause chain must not reach the client; a non-empty safe message and a stable code must. Given toLogRecord, it also asserts the operator does get the detail — redaction that eats the diagnostic is its own defect.

Every pack includes at least one case that must pass. An assertion suite made only of refusals is satisfied by an implementation that refuses everything.

Why the runner is injected

The framework standardises on Vitest; generated apps may use Jest (§27.3). A pack that imported vitest would be unusable in a Jest suite and would put a runner in this package's dependencies. So the runner arrives as a TestApi — five members: describe, it, beforeEach, afterEach and an expect returning the small Expectation matcher surface the packs use.

Vitest's and Jest's real APIs both satisfy that structurally, so the whole adapter is one object literal:

import { afterEach, beforeEach, describe, expect, it } from 'vitest';
import type { TestApi } from '@nage-api/testing';

export const api: TestApi = { describe, it, beforeEach, afterEach, expect };

Pass all five, since a pack is free to use the hooks even if today's ones mostly do not. Declaring the constant as TestApi rather than passing the literal inline is worth the extra line: the assignment is where a runner that has drifted from the interface is reported, at the file that owns the adapter rather than at every pack call.

expectNoSecrets

The same defect appears in every package: a password in a validation error, a connection string in a health response, a token in a log line. The obvious check misses it three ways:

import { expect } from 'vitest';

declare const error: Error;
declare const password: string;

expect(JSON.stringify(error)).not.toContain(password); // passes on a leak

• Non-enumerable fields. Error.message, Error.stack and cause do not survive a JSON round trip — exactly the fields most likely to hold the leak.
• Encodings. A secret survives as base64, base64url or URL-encoded text.
• Cycles. A serialiser that throws on a cyclic object turns a security assertion into an unrelated failure, and the leak goes unchecked.

import { RecordingLogger, expectNoSecrets } from '@nage-api/testing';

// The response the test just made, and the credentials it was built from.
declare const response: { body: unknown };
declare const password: string;
declare const refreshToken: string;
declare const logger: RecordingLogger;

expectNoSecrets(response.body, [password, refreshToken], 'client payload');
expectNoSecrets(logger.entries, [password], 'log');

The failure message masks what it found — via the exported mask() — including the excerpt of scanned text. A CI log is where secrets go to live forever, and a failing leak assertion must not be the thing that puts one there.

Fixtures

Deterministic, and counter-based rather than stateful-random: hash32(seed, index) means inserting a fixture at the top of a suite does not renumber every id below it, so a diff stays readable.

import { authUser, fixtureDate, fixtures, paginated } from '@nage-api/testing';

declare const records: readonly { readonly id: string }[];

const user = authUser({ roles: ['editor'] }); // overrides say what the test is about
const page = paginated(records); // count derived from the records
const oneMinuteIn = fixtureDate(60_000); // fixed epoch + offset
const scoped = fixtures(7); // an independent sequence per suite

Dates are anchored to FIXTURE_EPOCH_MS, so no test depends on the wall clock. Emails use example.test (RFC 2606), so a fixture can never send real mail. paginated derives count from the records it was given: a fixture that accepted a blind count could describe an impossible page, and a test asserting on pagination would be asserting on the lie.

Port doubles

FakeClock, RecordingLogger, MemoryKeyValueStore, MemoryRateLimitStore, StubSecretProvider. Mocked at the port boundary, never at ORM internals (§19).

Each is a complete implementation, not a stub that returns undefined. That distinction is the point: a stub agrees with whatever the code under test does, so a test built on one passes even when the code is wrong. MemoryRateLimitStore is a real fixed-window counter that actually refuses — a guard tested against a permissive double is a guard nobody has tested. StubSecretProvider.require rejects for a missing name, exactly as the env and AWS providers do.

RecordingLogger.child() shares its parent's entries array, so a test can assert nothing in the whole logger tree leaked a credential without having to know which children the code created.

Container harness

testcontainers is not a dependency — this package must stay installable without Docker. The runtime is a port:

import { describe, it } from 'vitest';
import { ContainerHarness, describeWithContainers, type ContainerRuntime } from '@nage-api/testing';

// The adapter over `testcontainers` — or over anything else that starts a
// container — that the project supplies. `unavailableRuntime` is the default.
declare const dockerRuntime: ContainerRuntime;

const harness = new ContainerHarness({ runtime: dockerRuntime });

await describeWithContainers(
  { describe, it },
  {
    name: 'postgres driver conformance',
    harness,
    suite: () => {
      /* … */
    },
  },
);

Three mitigations for the flakiness §24 names, all structural rather than remembered: images are pinned to exact patch versions in IMAGES (a floating postgres:16 is a different image next month); start retries with exponential backoff; and every container gets a process-wide unique name plus an ephemeral port, so two suites running at once cannot collide.

Without a runtime the suite skips loudly — the suite title says it skipped and why, and it leaves a passing marker test behind, because a silently skipped suite is indistinguishable from a passing one. required: true turns the skip into a failure, for the CI job that is supposed to have Docker.

Two deliberate deviations from §19

No in-memory repository. §19 lists an in-memory driver as part of this package. @nage-api/data already ships MemoryRepository, and it is the one the driver conformance suite runs against — so it is the only in-memory repository whose behaviour is proven to match a real driver's. A second copy here would have no such guarantee, and this package sits below @nage-api/data in the dependency graph and may not import it. Use MemoryRepository from @nage-api/data.

The container adapter has never been exercised against Docker. There is no container runtime in this development environment. ContainerHarness is unit-tested against a fake runtime — retry, backoff, naming, teardown, the skip and the required-failure paths are all covered — but no test in this repository has started a real container. The first project to supply a testcontainers adapter is exercising that path for the first time.

Testing the kit

174 tests, and the most valuable of them are in test/packs.spec.ts, which runs every pack twice: against a compliant subject, where each case must pass, and against subjects broken in one named way each, where the pack must fail on the case that names the defect. A pack that quietly passes a broken implementation is worse than no pack, because it produces a green suite as evidence — and only this kind of test can tell the two apart. It is also what proves each case is not decorative: the off-by-one-ceiling subject exists to show why the pack asserts that a limit at the maximum is accepted, and the refuses-everything subject exists to show why every pack needs a control case.

The suite has caught three real defects in the kit's own code so far: expectNoSecrets printing the raw secret into its own failure message, ContainerHarness handing two containers the same name when both started inside one millisecond, and the authorization pack's look-alike cases passing for the wrong reason against any store-backed guard.