DiffFence

Differential Fence Test Generation — Catch regressions before they ship.

DiffFence generates tests that are designed to fail when your code change introduces unintended regressions. Unlike traditional test generators that confirm your code works, DiffFence proves your change broke something.

Inspired by Meta's research: Just-in-Time Catching Test Generation (FSE '26).

Quick Start

# Install
npm install -D difffence

# Set your API key
export OPENAI_API_KEY=sk-...

# Initialize config (optional)
npx difffence init

# Make a code change on a feature branch, then:
npx difffence run

How It Works

git diff → Diff Analyzer → Context Collector → LLM Test Generator
                                                        ↓
PR Comments ← Delivery ← Assessor Pipeline ← Differential Executor
     ↑                          ↑
  Feedback Loop ────────────────┘

1. Diff Analyzer — Parses your git diff to find changed functions and their semantic changes (via tree-sitter AST)
2. Context Collector — Gathers callers, types, existing tests, and module interfaces
3. Test Generator — LLM synthesizes catching tests using intent-aware mutant generation
4. Differential Executor — Runs tests on base branch (should pass) and PR branch (should fail)
5. Assessor Pipeline — RubFake rules + LLM assessment filter false positives (~70% reduction)
6. Delivery — Surfaces high-confidence catches via CLI, PR comments, or exported test files

Commands

| Command | Description | |---------|-------------| | difffence run | Run catching test generation on current diff | | difffence run --export | Also export catching tests as files | | difffence run --pr-comment | Generate PR comment markdown | | difffence run --post-to-pr 42 | Auto-post results to GitHub PR #42 | | difffence run --json | Machine-readable JSON output | | difffence run --verbose | Show detailed pipeline output | | difffence ci | CI mode with structured output and exit codes | | difffence init | Initialize .difffencerc.json config | | difffence history | List previous runs | | difffence show <run_id> | View detailed results from a past run | | difffence feedback <catch_id> thumbs_up\|thumbs_down | Rate a catch for auto-tuning | | difffence tune | View auto-tuning recommendations | | difffence usage | LLM usage summary and costs |

Configuration

Create .difffencerc.json in your repo root (or run difffence init):

{
  "provider": {
    "name": "openai",
    "model": "gpt-4o",
    "apiKeyEnv": "OPENAI_API_KEY"
  },
  "analysis": {
    "baseBranch": "main",
    "maxFunctions": 20
  },
  "generation": {
    "framework": "jest",
    "maxCandidatesPerFunction": 5,
    "temperature": 0.3
  },
  "assessment": {
    "threshold": 0.6,
    "preset": "balanced"
  }
}

CLI Flags

Options:
  -b, --base <branch>      Base branch to diff against (default: auto-detect)
  -p, --provider <name>    LLM provider: openai, anthropic
  -m, --model <model>      LLM model to use
  -f, --framework <name>   Test framework: jest, vitest, mocha
  -t, --threshold <number> Assessment threshold 0-1 (default: 0.6)
  -k, --api-key <key>      API key (overrides env var)
  --export                 Export catching tests as hardening test files
  --pr-comment             Generate PR comment markdown
  --post-to-pr <number>    Auto-post results to a GitHub PR
  --no-ensemble            Use single LLM scoring instead of 3-vote ensemble
  --json                   Output results as JSON
  --verbose                Show detailed pipeline output

CI Integration

GitHub Actions

name: DiffFence
on: [pull_request]

jobs:
  difffence:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-node@v4
        with:
          node-version: '20'

      - run: npm ci

      - name: Run DiffFence
        run: npx difffence ci --output-format github-actions
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

DiffFence CI mode will:

• Auto-detect the PR branch and base branch
• Run the full pipeline
• Post results as PR comments (when GITHUB_TOKEN is set)
• Exit with code 1 if regressions are found

Web Dashboard

npx tsx web-ui/server.ts
# Opens at http://localhost:4200

The web UI provides a visual interface for running the pipeline, viewing results, and browsing run history.

License

MIT