gitgauge

gitgauge mines fail-to-pass evaluation tasks straight from your own repository's git history. It walks recent commits, finds bugfixes whose tests fail before the fix and pass after, and verifies each candidate by actually running the test suite in a throwaway git worktree at both commits. The result is a set of tasks that are native to your codebase, not a hand-curated benchmark repo.

Those tasks exist to answer one question: does a change to your agent's harness actually help? gitgauge runs one or more named agent configurations against every mined task and reports resolve rate, cost, wall time, and diff size per configuration. This is a different axis than SWE-bench (a fixed, research-oriented benchmark over a handful of famous open-source repos) or agent-vs-agent tools like repogauge. gitgauge instead compares harness configurations on your codebase: the same underlying model with a different CLAUDE.md, a different skill set, different CLI flags, or a different prompt, run against the same fail-to-pass tasks so you can see whether the change moved the needle.

Because the tasks come from your own history, they track the kind of bugs your codebase actually produces, and they cost nothing to source beyond git log. The tradeoff is that gitgauge is not a standardized cross-repo benchmark: numbers from one repo's mined tasks are not comparable to another repo's, and old commits can fail to build if their dependency versions have drifted. See Limitations below.

Requirements

• Node.js >= 20
• git
• A repository with a runnable test command (e.g. npm test, pytest)

Quickstart

npm install -g gitgauge
cd your-repo
gitgauge init                 # writes gitgauge.yaml and .gitgauge/
# edit gitgauge.yaml: set testCommand (and setupCommand if needed)
gitgauge mine                 # mines and verifies fail-to-pass tasks from git history
# write one agent config YAML per harness configuration you want to compare
gitgauge run -c a.yaml -c b.yaml
gitgauge report

gitgauge init creates gitgauge.yaml in the repo root and a .gitgauge/.gitignore (marking .gitgauge/* — where mined tasks and run results are stored — as untracked). It will not overwrite files that already exist.

gitgauge.yaml reference

testCommand: "npm test"        # required: how to run the test suite
setupCommand: "npm install"    # optional: run before testCommand in each worktree
mine:
  last: 200                    # optional, default 200: how many recent commits to scan
timeouts:
  test: 300000                 # optional, default 300000ms: timeout for testCommand
  setup: 600000                # optional, default 600000ms: timeout for setupCommand
  adapter: 900000              # optional, default 900000ms: timeout for the agent adapter run
worktree:
  link:                        # optional, default []: dirs to symlink into throwaway worktrees
    - node_modules
    - .venv

worktree.link symlinks the listed directories from the main repo into every worktree gitgauge creates (for mining verification and for agent runs), instead of reinstalling dependencies from scratch each time. This is recommended whenever setupCommand is slow or old commits may not resolve the same dependency versions — linking node_modules or .venv from the current checkout sidesteps both problems. Linked directories are also excluded from the diff-size metric so they don't inflate an agent's diff stats.

Agent config YAML reference

Each harness configuration you want to benchmark is a separate YAML file:

name: baseline            # required: label shown in the report
adapter: claude-code       # required: "claude-code" or "script"
model: sonnet              # optional: passed to the adapter
args:                      # optional: extra CLI args appended to the adapter invocation
  - "--permission-mode"
  - "acceptEdits"

• adapter: claude-code invokes the claude CLI with -p --output-format json --permission-mode acceptEdits (the prompt itself is piped to stdin rather than passed as an argv argument, so it does not appear in ps output), plus --model <model> if set, plus any args appended after. Cost and token usage are parsed from the JSON output.
• adapter: script requires a command field (a shell command) and runs it with the task prompt available in the GITGAUGE_PROMPT environment variable. Use this to plug in any other agent or harness. No cost/token metrics are collected for this adapter (both are reported as null).

Pass one or more configs to gitgauge run with repeated -c/--config flags: gitgauge run -c a.yaml -c b.yaml. Use --task <id> to run a single mined task instead of the full set.

How mining works

gitgauge mine walks the last mine.last commits (or --last <n> on the command line) and classifies each one by its subject line only: if the subject follows the Conventional Commits format (type: ... or type(scope): ...), the type must be fix, bugfix, or hotfix; otherwise the subject is checked against a fallback pattern of bug-related words (fix, bug, error, crash, defect, regression, and Korean equivalents 수정, 오류, 버그). Commits are further required to touch at least one test file and at least one non-test, non-Markdown source file.

Each candidate is then verified in throwaway git worktrees (created with git worktree add --detach and removed afterward):

1. Check out the fix commit, run setupCommand (if set) and testCommand — the suite must pass.
2. Check out the fix commit's parent, run setupCommand (if set), overlay just the candidate's test files from the fix commit on top, and run testCommand — the suite must fail.

Only candidates that fail before the fix and pass after it are saved as tasks (in .gitgauge/tasks/), each with an id derived from the commit sha and subject, the base and fix commit shas, a prompt built from the commit subject and body, and the list of test files to overlay when scoring an agent's attempt.

Prompt quality classification

Each candidate commit body is classified as either rich (40 or more non-whitespace characters) or title-only (fewer than 40). The classification is stored on the saved task as promptQuality and reported in the mine command output so you can see how many tasks have substantive commit bodies versus minimal ones.

The --require-body flag tells gitgauge mine to skip any candidate whose body is title-only before verification, saving the time and worktree overhead of verifying commits whose body text would make for a thin task prompt. Candidates skipped this way are still counted in the total candidates found; they are not counted as verified, rich, or title-only.

Mined tasks are worth a quick human review — delete task YAML files from .gitgauge/tasks/ whose fix required information that was unavailable from the source alone (e.g. live API responses), since no agent can solve those from the prompt.

LLM prompt rewriting (experimental)

By default a task prompt is simply Fix the following bug:\n\n<subject>\n\n<body>, which can contain spoilers — the commit message may describe exactly what changed and why. --rewrite-prompts passes the commit's subject, body, source diff, and test file contents to an LLM that produces a rewritten prompt describing only the symptom and expected behavior without revealing how the fix was implemented. This gives a more realistic evaluation of an agent's ability to diagnose a bug from scratch.

When rewriting succeeds, the original (unrewritten) prompt is preserved on the task as originalPrompt. If the rewrite fails (the LLM call errors or returns unexpected output), the original prompt is kept as-is, a warning is logged, and mining continues — a rewrite failure is never fatal.

Usage:

gitgauge mine --rewrite-prompts --rewrite-model haiku

--rewrite-model defaults to haiku (the fastest/cheapest Claude model) and is passed directly to the claude CLI's --model flag.

Trust boundary: mining — and prompt rewriting in particular — is designed for repositories whose history you trust. Commit messages and diffs are attacker-controlled input in an adversarial repo, and rewritten prompts are fed to the benchmarked agent, which runs with real permissions. The rewrite prompt wraps repository content in untrusted-data delimiters and instructs the model to ignore embedded instructions, but this is a mitigation, not a guarantee — do not point --rewrite-prompts at untrusted repositories.

Requirements and costs: This feature requires the claude CLI (npm install -g @anthropic-ai/claude-code) to be installed and available on PATH. Each rewrite invocation consumes tokens on the model you specify, so rewriting a large batch of tasks will incur token costs. Overriding the binary path is possible via the GITGAUGE_CLAUDE_BIN environment variable — this is primarily intended for internal testing and should not be needed in normal use.

Metrics table

gitgauge report prints one row per agent configuration (the latest run by default; pass --all to include every run recorded in .gitgauge/):

| Column | Meaning | | --- | --- | | CONFIG | Agent config name | | RESOLVED | Tasks where the agent's patch made the suite pass, out of total tasks run | | ERRORS | Tasks where setup or the adapter itself failed (crash/timeout), not counted as resolved or unresolved | | REGRESS | Count of tasks where the agent's final state fails the base commit's own original test suite (a regression relative to pre-existing behavior), shown as - when the task's base-suite-passing status is unknown/unrecorded (e.g. tasks mined before this feature existed, or where the base suite already failed at mine time) | | AVG COST | Mean total_cost_usd across tasks with a reported cost (- if none) | | AVG TIME | Mean wall-clock time of the adapter run | | AVG DIFF | Mean of (insertions + deletions) in the agent's diff, excluding worktree.link paths |

Limitations

• No container sandboxing. Agent runs execute directly in local throwaway git worktrees, with your OS user's permissions — there is no isolation beyond the worktree's own file boundary. The claude-code adapter defaults to --permission-mode acceptEdits; if you want tighter or looser permissions, set them explicitly via args in the agent config and be deliberate about it.
• Task prompts come verbatim from commit messages. The prompt is the fix commit's subject and body, which may describe or hint at the actual fix (a "spoiler"). This is acceptable for comparing configurations against each other, since every configuration sees the exact same prompt — but it means resolve rates should not be read as a measure of an agent's ability to diagnose bugs from scratch.
• Old commits may not build. Mining and scoring both check out historical commits into worktrees and run setupCommand/testCommand there; if dependencies have moved on since that commit, setup or the test run can fail for reasons unrelated to the bug itself. Mitigate this by keeping mine.last reasonably small (recent history is more likely to still build) and by using worktree.link to reuse your current node_modules/.venv instead of reinstalling per-commit.
• Regression detection is suite-level. The REGRESS column tells you that something regressed, not which test, and the tail of the failing suite's output is available in the run output for manual inspection but is not parsed per-test. To prevent agents from gaming this signal, gitgauge reverts agent edits to pre-existing files that match either the test-file heuristic or the test-infrastructure heuristic (e.g. package.json, vitest.config.ts, jest.config.js, pytest.ini, Makefile) to their committed state before running the regression check. Brand-new test-infrastructure files added by the agent are deleted before scoring (unlike brand-new plain test files, which are left alone). Files that match neither heuristic (e.g. shared test helpers outside recognized test directories or naming patterns) remain unprotected; this is a known container-free tradeoff, not a bug. The flip side: a fix that legitimately needs to change a protected file (e.g. adding a dependency to package.json or requirements.txt) is also reverted and will likely score as unresolved — protected-file changes are outside what gitgauge can currently evaluate.