acceptance-runner

acceptance-runner is a local-first acceptance testing runner for binary project criteria. Criteria live in YAML, checks run sequentially, and the runner writes a deterministic JSON score report.

Install

npm install --save-dev acceptance-runner

The package is ESM-only and requires Node.js 22.18 or newer.

Quickstart

Create acceptance/criteria.yaml:

criteria:
  - id: has_cli_entrypoint
    description: "Package exposes CLI entrypoint"
    check: "node ./dist/cli.js --help"
    weight: 2

  - id: parses-tsconfig
    description: "TypeScript config can be parsed"
    check: "parses-tsconfig"

Run:

npx acceptance-runner run

The default output is acceptance/results.json.

Criteria

Each criterion has:

• id: unique string identifier
• description: human-readable criterion
• check: shell command or registered programmatic check id
• weight: optional positive number, default 1

Shell checks pass when the command exits with code 0. Non-zero exits fail. The runner captures stdout, stderr, and duration for every criterion.

Custom Checks

Programmatic checks are registered in a checks module next to the criteria file. The CLI auto-loads these file names from the criteria directory:

• checks.ts, checks.js, checks.mjs
• criteria.checks.ts, criteria.checks.js, criteria.checks.mjs
• <criteria-file-name>.checks.ts, .js, or .mjs

Example acceptance/checks.ts:

import { readFile } from "node:fs/promises";
import { join } from "node:path";
import { registerCheck } from "acceptance-runner";

registerCheck("parses-tsconfig", async (ctx) => {
  const source = await readFile(join(ctx.cwd, "tsconfig.json"), "utf8");
  JSON.parse(source);
  return { pass: true, reason: "tsconfig.json is valid JSON." };
});

Use a registered check id as the criterion check value.

Scoring

The report includes:

• passed: number of passing criteria
• total: total criteria
• score: passed / total
• weighted_score: passing weight divided by total weight

If no weights are defined, each criterion has weight 1.

CLI

npx acceptance-runner run \
  --criteria acceptance/criteria.yaml \
  --output acceptance/results.json \
  --prompt-id task-123 \
  --fail-on-error

--fail-on-error exits non-zero when any criterion fails. Without it, failures are recorded in JSON while the CLI exits 0.

CI

Run the package after building the project under test:

- run: npm ci
- run: npm run build
- run: npx acceptance-runner run --fail-on-error

The JSON report can be uploaded as a CI artifact or used by later automation.

Local Example

This repository includes:

• acceptance/example.criteria.yaml
• acceptance/example.checks.ts

Build and run it locally:

npm run build
npm run run:example