agentbench

Lighthouse for AI coding harnesses. Benchmark your Claude Code setup and get a score out of 100.

Why

You spend hours tweaking CLAUDE.md, skills, hooks, and rules — but have no way to measure if your changes actually help. People share configs on X and GitHub with no evidence they work better than the defaults.

Superpowers has 89K stars. Everyone has a different CLAUDE.md. Nobody can prove theirs is better.

agentbench gives you a number. Run it before and after every harness change. Share your score. Compete.

Install

npx agentbench

Or install globally:

npm install -g agentbench

Or clone and run locally:

git clone https://github.com/naman10parikh/agentbench.git
cd agentbench
pnpm install
pnpm dev

Quick Start

# Run the full benchmark (10 tasks, ~2 min, ~$0.50 API cost)
npx agentbench

# Run a single task to test quickly
npx agentbench --task 1

# Get JSON output for CI/scripts
npx agentbench --json

# Compare your harness against another CLAUDE.md
npx agentbench --compare ~/other-project/CLAUDE.md

# Re-run baseline from scratch (ignores cache)
npx agentbench --no-cache

# Use a specific model
npx agentbench --model claude-sonnet-4-6

# See detailed output for each task
npx agentbench --verbose

Example Output

  ┌─────────────────────────────────────────────┐
  │                 agentbench                   │
  │                                              │
  │  HARNESS SCORE: 73 / 100                    │
  │                                              │
  │  Completion    ████████░░  82%              │
  │  Efficiency    ██████░░░░  61%              │
  │  Tool Use      ████████░░  78%              │
  │  Recovery      █████░░░░░  54%              │
  │  Quality       █████████░  91%              │
  │                                              │
  │  vs. baseline: +31 points                   │
  │  Top tip: Add error recovery skills          │
  └─────────────────────────────────────────────┘

  Full report: .agentbench/report-2026-03-17.json

What It Measures

Your harness is scored across 5 dimensions (0-100 each):

| Dimension | What It Measures | How It's Scored | | -------------- | ----------------------------------------- | -------------------------------------- | | Completion | Did the agent finish each task correctly? | Pass/fail per task, averaged | | Efficiency | How many tokens did it use vs. baseline? | Ratio against bare defaults | | Tool Use | Did it pick the right tools for the job? | Expected vs. actual tool selection | | Recovery | Did it recover from injected errors? | Error recovery rate on seeded failures | | Quality | How good is the output code? | LLM-as-judge evaluation (via Haiku) |

The overall score is the average across all 5 dimensions.

The 10 Benchmark Tasks

Each task runs in an isolated temporary workspace — your actual code is never touched.

| # | Task | Difficulty | What It Tests | | --- | -------------------------------------------- | ---------- | ----------------------------------- | | 1 | Fix a typo in a TypeScript file | Easy | Basic navigation + edit | | 2 | Add a function with a specific signature | Easy | Code generation accuracy | | 3 | Refactor a function to reduce complexity | Medium | Code understanding + simplification | | 4 | Write unit tests for an existing module | Medium | Test generation quality | | 5 | Fix a failing test by reading error output | Medium | Error comprehension + debugging | | 6 | Add error handling to a throwing function | Medium | Recovery pattern knowledge | | 7 | Resolve a merge conflict in git | Hard | Multi-file coordination | | 8 | Find and fix a SQL injection vulnerability | Hard | Security awareness | | 9 | Refactor 3 files to extract a shared utility | Hard | Cross-file reasoning | | 10 | Add a REST endpoint with validation + tests | Hard | Full-stack completion |

Tasks 1-6 use automated checks (compilation, test pass, diff comparison). Tasks 7-10 use LLM-as-judge (Claude Haiku evaluates against a rubric).

How It Works

1. Detect harness  →  Reads CLAUDE.md, settings.json, .claude/skills/, .claude/rules/
2. Run tasks       →  10 coding tasks in isolated temp workspaces via Claude API
3. Evaluate        →  Automated checks + LLM-as-judge scoring
4. Compare         →  Score vs. cached baseline (bare Claude Code defaults)
5. Report          →  Terminal scorecard + JSON report + recommendations

Harness Detection

agentbench auto-detects your harness by reading:

| File/Dir | What It Extracts | | ----------------------- | ------------------------------------- | | CLAUDE.md | System prompt, rules, operating model | | .claude/settings.json | Hook definitions, permissions | | .claude/skills/*.md | Skill count and content | | .claude/rules/*.md | Rule count and content |

This config is injected into the Claude API system prompt when running each task, exactly as Claude Code would use it.

Baseline Caching

The first time you run agentbench, it benchmarks bare Claude Code defaults and caches the result. Subsequent runs only benchmark your harness and compare against the cache. The cache key is sha256(task_suite_version + model_version).

Clear cache: npx agentbench --no-cache

Cost Estimate

| Component | Cost | | -------------------------------- | ----------- | | 10 tasks via Sonnet | ~$0.30-0.60 | | LLM-as-judge (4 tasks via Haiku) | ~$0.05-0.10 | | Baseline run (first time only) | ~$0.30-0.60 | | Typical run | ~$0.50 |

Configuration

| Flag | Default | Description | | ------------------ | ------------------- | ------------------------------------------- | | --task <n> | all | Run a single task by number (1-10) | | --json | false | Output results as JSON | | --compare <path> | none | Compare against another CLAUDE.md | | --no-cache | false | Ignore cached baseline, re-run from scratch | | --model <id> | claude-sonnet-4-6 | Model to benchmark with | | --verbose | false | Show detailed per-task output |

JSON Output

With --json, agentbench outputs a structured report:

{
  "version": "0.1.0",
  "timestamp": "2026-03-17T10:30:00Z",
  "model": "claude-sonnet-4-6",
  "overallScore": 73,
  "baselineScore": 50,
  "dimensions": [
    { "name": "completion", "score": 82, "details": "8/10 tasks completed" },
    {
      "name": "efficiency",
      "score": 61,
      "details": "12,400 tokens (baseline: 18,200)"
    },
    { "name": "toolUse", "score": 78, "details": "7 unique tools used" },
    { "name": "recovery", "score": 54, "details": "2/4 errors recovered" },
    { "name": "quality", "score": 91, "details": "LLM judge average" }
  ],
  "recommendations": [
    "Add error recovery skills or an error escalation protocol to your harness"
  ],
  "harnessConfig": {
    "hasClaudeMd": true,
    "claudeMdLines": 142,
    "skillCount": 8,
    "ruleCount": 3,
    "hookCount": 5
  }
}

Adding Custom Tasks

Each task is a directory under tasks/ with:

tasks/NN-task-name/
├── task.json       # Task definition, prompt, scoring rubric
├── workspace/      # Initial repo state (the broken/incomplete code)
└── expected/       # Reference solution (for automated comparison)

task.json Format

{
  "id": 11,
  "name": "Your task name",
  "category": "bug-fix",
  "difficulty": "medium",
  "description": "What the task tests",
  "prompt": "The exact prompt sent to the agent",
  "expectedTools": ["Read", "Edit", "Bash"],
  "scoring": {
    "automated": true,
    "checks": ["tsc --noEmit exits 0", "specific check here"]
  }
}

Improving Your Score

Common harness improvements by dimension:

| Low Score In | What to Add | | ------------ | ------------------------------------------------- | | Completion | Clearer task completion criteria in CLAUDE.md | | Efficiency | Model routing rules (Haiku for simple subtasks) | | Tool Use | Tool preference rules ("Use Read instead of cat") | | Recovery | Error escalation protocol, /troubleshoot skill | | Quality | Code style rules, pre-commit quality gate hook |

Comparison to Existing Benchmarks

| Benchmark | What It Tests | Our Difference | | --------------- | ------------------------------ | -------------------------------------- | | SWE-bench | Model ability on GitHub issues | We test the harness, not the model | | HumanEval | Code generation | Single-function, no tool use | | MBPP | Python programming | No harness awareness | | Aider benchmark | Edit accuracy | Doesn't score efficiency or recovery |

agentbench is the only benchmark that holds the model constant and measures the harness. Same model, different scaffold: 42% vs 78%. The harness IS the product.

Requirements

• Node.js 18+
• ANTHROPIC_API_KEY environment variable
• ~$0.50 per benchmark run

Contributing

See CONTRIBUTING.md. We especially welcome:

• New benchmark tasks — the more diverse, the better the signal
• Scoring improvements — better rubrics, tighter automated checks
• Bug reports — if a score feels wrong, tell us

License

MIT