failfirst

A CI gate that proves new tests actually test the change.

A new test that passes on the base commit would pass without your change. It is not testing your change. failfirst catches it.

The problem

A pull request adds a feature and a test. The test passes. CI is green. Reviewer approves.

But did the test ever depend on the feature? If you reverted the code change, would the test notice? With AI-generated PRs this failure mode went from occasional to routine: plausible-looking tests that assert things the old code already did. They pass before the change, they pass after the change, they pass after the next regression too. Green forever, guarding nothing.

The classic discipline is "watch the test fail first." failfirst turns that discipline into a CI gate: it runs your branch's new tests against the code from before your branch. Any new test that passes there is flagged vacuous, and the gate fails.

See it

A PR implements multiply() and adds two tests. One imports multiply and checks it. The other looks multiply-related but only exercises the old add() function. Both are green in a normal test run. failfirst tells them apart:

$ failfirst main
failfirst v0.1.0
  base     main (merge-base 4dbdb3a)
  runner   node-test
  changed  2 test file(s)

  running 2 file(s) against base 4dbdb3a...
  running 2 file(s) against HEAD...

  TEST                                                                           BASE    HEAD   VERDICT
  test/multiply-props.test.js > multiply: adding a number to itself doubles it   pass    pass   VACUOUS
  test/multiply.test.js > multiply multiplies two numbers                        error   pass   GOOD

  1 good, 1 vacuous

FAIL: 1 new test passes on the base commit.
A test that passes without your change is not testing your change.

$ echo $?
1

That transcript is a real run against the fixture repository used in this project's integration tests.

Install

No install needed:

npx failfirst

Or add it to the project:

npm install --save-dev failfirst

Requires Node 18 or newer and git. Zero dependencies.

Usage

failfirst [base-ref] [options]

Arguments:
  base-ref          Base to compare against (default: origin/main, main,
                    origin/master, or master, first one that exists)

Options:
  --runner <name>   node-test | vitest | jest (default: auto-detect)
  --json            Machine-readable JSON output
  --color           Force colored output
  --no-color        Disable colored output
  -h, --help        Show help
  -v, --version     Show version

Exit codes:
  0  no vacuous tests (or no changed test files)
  1  at least one vacuous test
  2  usage or environment error

GitHub Actions

jobs:
  failfirst:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # failfirst needs the merge-base commit
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci        # only needed for the vitest/jest runners
      - run: npx failfirst origin/${{ github.base_ref }}

How it works

1. Find the changed tests. git diff --name-status <merge-base>..HEAD, filtered to test files (*.test.*, *.spec.*, *_test.*, test-*.*, and files under test/, tests/, __tests__/, spec/ directories).
2. Rebuild the world before your change. A temporary detached git worktree is created at the merge-base of the base ref and HEAD.
3. Overlay only the new tests. The added and modified test files (plus changed support files from test directories) are copied into that worktree. Production code stays old; tests are new.
4. Handle modified test files fairly. Before the overlay, failfirst runs the base version of each modified test file and records its test names. Those pre-existing tests are reported but never gated; only tests your branch actually added are judged.
5. Run twice, compare per test. The changed test files run against the base worktree and against HEAD. Each new test gets a verdict:

| Verdict | Base | HEAD | Meaning | |---|---|---|---| | GOOD | fail / error | pass | The test fails without your change. It proves something. | | VACUOUS | pass | pass | The test passes without your change. It proves nothing. Gate fails. | | BROKEN | fail | fail | The test fails everywhere. Your normal CI run will catch it. | | PRE-EXISTING | any | any | Already existed in the base version of a modified file. Not gated. | | SKIPPED | any | skip | Skipped on HEAD. Nothing to judge. |

6. Clean up. The worktree is removed even when a run blows up.

A BASE of error means the test file could not even load on the old code, typically because it imports something your branch introduced. That is strong evidence the test targets the change, so it counts as failing on base.

Runners

| Runner | Detection | Notes | |---|---|---| | node-test | default | Built into Node. Parses TAP from node --test, including nested describe/it suites. | | vitest | test script or dependency mentions vitest | Uses the JSON reporter. | | jest | test script or dependency mentions jest | Uses --json with --runTestsByPath. |

All three adapters are validated against real fixture repositories (base commit, then a PR adding one good and one vacuous test) as part of this project's development. For vitest and jest, the base worktree borrows the main checkout's node_modules via symlink, so run npm ci first in CI.

Why not X

Why not tdd-guard or other live TDD enforcers? Those hook into an agent's or developer's editing session and enforce red-green discipline while the code is being written. Great when you control the session. failfirst works at the other end: it gates the finished PR in CI, no matter what tool, agent, or human produced it, with nothing installed on the author's side.

Why not mutation testing (Stryker and friends)? Mutation testing mutates your production code and checks that existing tests notice. It measures the strength of the whole suite, costs minutes to hours, and does not know what a specific PR changed. failfirst answers one narrow question per PR in seconds: do the new tests depend on the new code? The two are complementary; mutation testing for depth, failfirst for the PR loop.

Why not SWE-bench style fail-to-pass checks? The fail-to-pass concept is exactly right, and SWE-bench uses it to build benchmark datasets with internal tooling tied to their harness and container images. failfirst is that concept packaged as a generic, zero-dependency CLI for any git repo with a supported runner.

Why not just review the tests? Vacuity is invisible in review. The test imports the right module, asserts plausible things, and passes. The only way to know whether it would pass without the change is to run it without the change, which is tedious by hand and trivial for a tool.

Scope and limitations

• Tests are matched between base and HEAD runs by file path plus full test name. A renamed test in a modified file is treated as new.
• A pre-existing test whose body was edited (same name) is not re-gated. Gating it would re-flag every harmless refactor of an old test.
• New tests in modified files that pass on base are flagged even if they sit next to legitimate ones; the table tells you exactly which test to fix.
• Support files outside test directories (e.g. a changed helper in src/) are not overlaid onto the base worktree. If a new test needs them, it will fail or error on base, which is the safe direction: failfirst never produces a false VACUOUS from a missing dependency, only a conservative GOOD.
• For vitest and jest, the base run uses the head checkout's node_modules. If your PR also changed dependency versions, the base run sees the new versions.
• Flaky tests that pass on base by luck will be flagged. They deserve it.

Roadmap

• --require-tests: fail PRs that change source code but add no tests.
• Adapters: pytest, mocha, bun test.
• GitHub Action wrapper with inline PR annotations on vacuous tests.
• Optional gating of edited pre-existing tests.

Contributing

See CONTRIBUTING.md. The short version: zero dependencies, pure logic stays pure and unit-tested, and every new behavior needs a test that fails without it.

License

MIT, Ben Malaga.