AIBench Arena

Public benchmarks test AI coding agents on someone else's code. AIBench Arena tests them on yours.

AIBench Arena turns your repository's own git history into a private benchmark, then runs Claude Code, Codex CLI, and Gemini CLI against it and shows you a leaderboard — for your codebase, not Django.

Real results, two repos, two different answers:

• honojs/hono, 17 tasks (live report →): Claude Code, Codex and Gemini tied at 7/17 — same 7 solved, same 4 failed, different implementations. The standout difference was speed: Gemini ~9× slower per task.
• axios/axios, 14 tasks (live report →): Codex fixed 12/14 vs Claude's 10/14 and Gemini's 9/14 (overlapping intervals — no statistical verdict) and was ~8× faster than both.

Same tool, same method, different repo — different answer. That's exactly why a public leaderboard can't pick your agent for you.

Why

Every engineering team is deciding which AI coding agent to standardize on — and doing it based on Twitter hype and one public benchmark run on a dozen famous open-source Python repos. That number says nothing about your 7-year-old Java monolith or your Next.js monorepo with a custom build.

Worse: the same model scores wildly differently inside different harnesses — one independent test found Claude Opus at 77% inside Claude Code vs 93% inside Cursor on identical tasks. Public leaderboards literally cannot tell you which agent wins for you.

AIBench Arena answers that with data from the one codebase you actually care about.

How it compares

| | AIBench Arena | SWE-bench | Aider leaderboard | Terminal-Bench | |---|---|---|---|---| | Runs on your repo | ✅ | ❌ (12 public Python repos) | ❌ (Exercism puzzles) | ❌ (curated terminal tasks) | | Works on private code | ✅ never leaves your machine | ❌ | ❌ | ❌ | | Tests the full agent harness (not just the model) | ✅ | partial (scaffold-dependent) | ❌ model-only | ✅ | | Reports confidence intervals | ✅ Wilson 95% CI, refuses to over-claim | ❌ | ❌ | ❌ | | Cost | your own agent subscriptions | n/a | n/a | n/a | | Setup | npx aibench | research harness | benchmark suite | Docker harness |

Install

npm install -g aibench     # or run everything through npx aibench …

From source (or before an npm release):

npm install -g github:Elnino0009/aibench-arena   # builds automatically via `prepare`

Requires Node 20+, git, and at least one agent CLI installed (macOS/Linux; Windows via WSL). gh (GitHub CLI) is optional but improves task descriptions.

Quickstart

aibench doctor        # preflight: agents installed + authenticated + can edit
aibench init          # detect your stack, write .aibench.yml
aibench check         # how benchmarkable is this repo? (free, any repo)
aibench mine          # find benchmark tasks in your git history (free)
aibench validate      # prove they're gradeable (free)
aibench run           # benchmark your agents (spends YOUR tokens)
aibench report --format html

Supported agents: Claude Code (claude), Codex CLI (codex), Gemini CLI (gemini), Cursor CLI (cursor), opencode, and Aider (aider). Pick a model per agent with --agents claude:opus-4-8,codex:gpt-5.5.

How it works

Every merged PR that fixed a bug and shipped with tests is an exam question with an answer key:

1. Mine — scan your git history for those PRs.
2. Validate — rewind to just before each fix, confirm its tests fail without the fix and pass with it (fail-to-pass). Unreliable tasks are thrown out. No tokens spent.
3. Run — for each task, drop the agent into an isolated checkout that ends at the parent commit (it can't see the future fix), hand it the original issue, and let it work.
4. Grade — apply the real tests. Pass = the agent genuinely solved a problem your team once solved.

You get a leaderboard specific to your repo.

Who pays for tokens?

You do — via your own agent subscriptions. AIBench Arena is an orchestrator: it invokes the agent CLIs you already have installed and authenticated (claude, codex, gemini). AIBench Arena itself makes zero LLM calls. It always prints a cost estimate and asks before spending anything, and --budget-usd is a hard cap.

Agent authentication

Run aibench doctor first — it verifies each agent is installed, authenticated, and can actually edit a file (a tiny self-test), and prints the exact fix for anything broken. By default it probes your configured agents (or claude, codex, and gemini if there's no config yet) — scope it with aibench doctor --agents claude to test just one. The common setups:

| Agent | How to authenticate | |---|---| | Claude Code | run claude once and log in (or set ANTHROPIC_API_KEY) | | Codex CLI | printf %s "$OPENAI_API_KEY" \| codex login --with-api-key | | Gemini CLI | set GEMINI_API_KEY, then aibench doctor --fix (writes the CLI's auth-type setting — without it headless runs silently do nothing) | | opencode | set the provider key (e.g. GEMINI_API_KEY) or opencode auth login |

Configuration

aibench init auto-detects most stacks. The generated .aibench.yml looks like:

# .aibench.yml
setup: npm ci                          # run once per sandbox before grading
test: npx vitest run {test_files}      # {test_files} = the task's own test files
test_all: npx vitest run               # full suite (regression check)
typecheck: npx tsc --noEmit            # optional: enables --oracle typecheck
agents:
  - claude
  - codex
# docker_image: node:20                # optional: containerize setup + tests
# sandbox_exclude:                     # optional: paths stripped from sandboxes
#   - .env.local

If detection fails, copy the block above and adapt setup/test to your stack — {test_files} is required in test so each task grades only its own tests.

Contamination control

Agents will look for the answer, so AIBench Arena stops them:

• No git worktree. Sandboxes are truncated clones whose history physically ends at the parent commit — git log reveals nothing about the fix.
• Web tools disabled where the CLI supports it (Claude Code today; Codex blocks network at its own sandbox layer). Agents whose CLIs expose no such flag could still reach the web — for public repos, treat cross-agent comparisons accordingly.
• Golden tests are hidden during the run and applied fresh for grading; edits to test files are discarded, so test-tampering can't game the score.
• Sandboxes live in a temp directory outside your repo, so agents won't stumble on .aibench/ or the answer key. (The agent still runs as your user on your machine — full host isolation is on the roadmap; --docker already containerizes setup and test execution.)

Public repos: models may have trained on your actual fixes. AIBench Arena's headline use case is private repos; for public ones it defaults to recent tasks and flags the contamination risk in the report.

Docker mode

Pass --docker to run setup and test execution inside a container for reproducible environments and to isolate untrusted repo build/test code from your host:

aibench validate --docker              # default image: node:20
aibench run --agents claude --docker node:20

Set a custom image in .aibench.yml (docker_image:) for non-JS stacks. The agent itself still runs on the host, where its CLI is installed and authenticated; only the environment-sensitive steps go in the container.

Reading the results honestly

• Every pass rate ships with a 95% Wilson confidence interval. With 18 tasks, "72%" really means "somewhere around 49–88%."
• AIBench Arena refuses to declare a winner when the top agents' intervals overlap. Small samples produce noise; it won't dress noise up as a verdict.
• Rows are labeled harness + model + config, never bare "Claude" — because the harness matters as much as the model.
• Infrastructure failures (sandbox or setup errors) are reported separately and excluded from pass rates — an npm flake is not the agent's fault.
• Feature PRs are hard to grade fairly (their tests import brand-new symbols by name), so AIBench Arena defaults to bug-fix tasks only. Use --type all to include features (experimental; ungradeable rate reported separately).

Commands

| Command | What it does | Tokens? | |---|---|---| | aibench doctor | Preflight: agents installed, authenticated, can edit | ~cents | | aibench init | Detect stack, write .aibench.yml (--yes for non-interactive) | No | | aibench check | Score how benchmarkable any repo is | No | | aibench mine | Extract candidate tasks from git history | No | | aibench show <id> | Inspect a task (statement, files, --diff for the fix) | No | | aibench validate | Prove tasks are fail-to-pass gradeable | No | | aibench run | Benchmark agents on validated tasks | Yes | | aibench report | Render terminal / HTML / md / json report | No | | aibench clean | Remove AIBench Arena data + orphaned sandboxes | No |

Run any with --help for options (--since, --max, --trials, --budget-usd, --agents, --resume, …).

Beyond unit tests: repos whose fixes ship without tests can still be benchmarked with a command oracle — aibench mine --oracle typecheck (or build, or --oracle command --oracle-cmd '<repro>') grades fixes by a command that fails before the fix and passes after. Unit tests remain the strongest oracle; command oracles are best-effort and gaming-checked (see docs/METHODOLOGY.md).

Roadmap

• Opt-in anonymized community leaderboard — the state of coding agents on real-world repos, aggregated from private runs
• Full agent-in-container isolation (v1 Docker mode containers setup + tests)
• TDD-mode study (give the agent the failing tests up front)
• Scheduled re-runs on every new model release

Contributing

Adapters for new agent CLIs are the highest-value contribution — see CONTRIBUTING.md. The whole pipeline is testable without tokens via the mock adapter and a seeded fixture repo (npm test).

License

MIT