testdelta

Prove that changed tests fail before a patch and pass after it.

testdelta is a local-first, language-independent CLI for regression-test proof. It runs the same changed tests against two isolated Git worktrees:

changed tests + base code  -> must fail
changed tests + head code  -> must pass

If both runs pass, the test is green but does not prove the patch. That distinction matters when tests are generated quickly, written after the implementation, or produced by a coding agent optimizing for a green CI signal.

$ npx testdelta verify --base main --command "node --test {tests}"

testdelta PROVEN
The changed tests fail before the patch and pass after it.

✓ before  exit 1  438ms
✓ after   exit 0  301ms

tests   math.test.js
command node --test math.test.js
base    1fcf8510a27a

No API key, service, account, container, or language-specific plugin is required. The published package has zero runtime dependencies.

Why this exists

Normal CI answers one question: does the current branch pass?

For a regression test, reviewers need a second answer: would this test have caught the bug before the fix? A test that passes on both revisions may exercise code, but it cannot demonstrate that the production change caused the behavior to become correct.

Fail-to-pass testing is standard in software-engineering benchmarks, but it is still awkward to run on ordinary pull requests. testdelta turns it into one command.

Quick start

Requirements: Node.js 20+ and Git.

npx testdelta init

Edit the generated .testdelta.json, then commit your production and test changes before running:

npx testdelta verify --base main --setup "npm ci"

Or skip configuration:

npx testdelta verify \
  --base origin/main \
  --command "pytest {tests}" \
  --setup "python -m pip install -e ."

The {tests} placeholder expands to shell-quoted paths for the added or modified test files. Commands without {tests} are also supported, which is useful for go test ./... and cargo test.

Try from source

git clone <your-fork-url>/testdelta.git
cd testdelta
npm install
npm test
npm run demo

Verdicts

| Verdict | Before | After | Exit | Meaning | |---|---:|---:|---:|---| | PROVEN | fail | pass | 0 | The changed tests distinguish the patch from its base. | | WEAK | pass | pass | 2 | The tests pass without the production change. | | BROKEN | fail | fail | 3 | The tests still fail on the new revision. | | REGRESSION | pass | fail | 4 | The patch made previously passing behavior fail. |

Tool/configuration errors exit with 1. This makes every result except PROVEN fail closed in CI.

How it works

1. Resolve the merge base between --base and --head.
2. Find added or modified files matching the test patterns.
3. Create detached temporary worktrees for the merge base and head revision.
4. Apply only the changed-test patch to the base worktree.
5. Run the optional setup command and identical test command in both worktrees.
6. Remove both worktrees and emit a deterministic verdict.

The production patch is never applied to the before worktree. The original checkout is never modified.

Configuration

.testdelta.json:

{
  "command": "npm test -- {tests}",
  "setupCommand": "npm ci",
  "testPatterns": [
    "**/*.test.*",
    "**/*.spec.*",
    "tests/**"
  ],
  "timeoutMs": 120000
}

CLI options override configuration. Use repeated --pattern flags for custom layouts:

testdelta verify --pattern "spec/**" --pattern "packages/*/checks/**"

JSON and receipts

testdelta verify --json
testdelta verify --report .testdelta-report.json

The JSON report includes refs, merge base, test paths, command, exit codes, durations, output, timeout state, and schema version.

CI

The important GitHub Actions setting is fetch-depth: 0; the merge base must exist locally. A complete workflow is in docs/github-actions.md.

For high-trust CI, pass --command in the protected workflow rather than allowing a pull request to redefine its own verifier.

What testdelta is not

• It is not a test framework; it wraps the command you already trust.
• It does not claim that a fail-to-pass test fully specifies the feature.
• It does not semantically judge assertions or detect every form of test weakening.
• It currently verifies committed changes between Git refs, not unstaged files.
• A new test that cannot even load on the base revision counts as a failure. Review the captured before output to distinguish a meaningful assertion failure from missing infrastructure.

Why not just inspect the diff?

Static test linters can detect .skip, .only, deleted assertions, and suspicious patterns. Command receipts can prove that a test command ran. Those are useful but answer different questions. testdelta executes the changed test in both realities and checks whether the production patch changes the outcome.

Project status

The MVP supports cross-platform Git worktrees, conventional test discovery, custom globs, setup commands, timeouts, four verdicts, JSON output, and a public TypeScript API. See ROADMAP.md for the path to v1.

Contributions are welcome. Start with CONTRIBUTING.md and the good-first-issue items in the roadmap.

License

MIT © TestDelta contributors