agents-md-bench

Your AGENTS.md might be making your coding agent worse.

Benchmark repository instructions against real coding tasks, measure success and cost, and remove rules that do not help.

agents-md-bench is a local experiment runner, not an AGENTS.md linter and not a model leaderboard. It runs the same task with and without repository instructions in disposable Git worktrees, verifies the resulting patches with commands you choose, and reports the paired difference.

Quick start

Requirements: Node.js 20 or newer, Git, a clean committed task baseline, and the agent CLI you intend to benchmark.

npx agents-md-bench init
# Replace the sample task in agent-bench.yaml.
npx agents-md-bench doctor
npx agents-md-bench run --agent codex --profile quick

The run writes a durable bundle under .agents-md-bench/runs/<run-id>/:

results.json   machine-readable raw trials
report.md      reviewable experiment report
report.html    shareable static report
badge.svg      README-ready result badge

Start with quick (one trial). Use normal (three) before making a decision and rigorous (five) when the cost is justified.

Configure real tasks

Put agent-bench.yaml at the root of the Git repository:

version: 1

context:
  file: AGENTS.md

experiment:
  trials: 1
  timeout_minutes: 20
  verify_timeout_minutes: 10
  max_minutes: 30
  max_tokens: 100000
  budget_usd: 5

agents:
  codex:
    command: codex
    args:
      - exec
      - --json
      - --ephemeral
      - --ignore-user-config
      - --sandbox
      - workspace-write
      - -C
      - "{{worktree}}"
      - "-"
    pass_env: []

tasks:
  - id: fix-empty-input
    prompt: |
      Fix the crash that occurs when parseConfig receives an empty string.
    setup:
      commit: 31ce72a
      commands:
        - npm ci
    verify:
      - npm test -- parse-config
      - npm run typecheck
    constraints:
      forbidden_paths:
        - package-lock.json
        - migrations/**
      max_changed_files: 5
      max_patch_lines: 200
    tags:
      - smoke

The verification commands are the objective success oracle. A task is successful only when every command exits successfully. Path constraints, changed-file limits, patch size, runtime, tokens, and cost are reported separately so a functional patch is not confused with a well-scoped patch.

Each setup command runs before a private benchmark baseline is created. This keeps install artifacts and context injection/removal out of the measured agent diff.

Run controlled comparisons

The default experiment has two conditions:

without-context       the configured context file is absent
with-current-context  the current file is injected

Conditions are rotated across tasks and trials to reduce simple ordering bias. Every agent run gets its own detached Git worktree. The primary checkout is never given to the agent.

# A quick smoke experiment
npx agents-md-bench run --agent codex --profile quick

# Three trials for selected tasks
npx agents-md-bench run --agent codex --profile normal \
  --task fix-empty-input --task keep-migrations-immutable

# Filter by task tag
npx agents-md-bench run --agent codex --tag smoke

Measure individual sections

Section ablation removes each level 2–6 Markdown section, one at a time, and reruns the paired task matrix:

npx agents-md-bench run --agent codex --ablate sections --profile normal

Fenced code blocks are ignored when detecting headings. A section includes its nested subsections. The report shows the outcome after removal versus the full file; it does not claim causality when the paired sample is too small.

Use another coding agent

The generic shell adapter exposes the task through environment variables and can optionally send it on stdin:

agents:
  shell:
    command: claude -p "$AGENTS_MD_BENCH_PROMPT"
    prompt_mode: env
    pass_env:
      - ANTHROPIC_API_KEY

Available template values are {{worktree}}, {{cwd}}, {{promptFile}}, and {{prompt}}. Prefer AGENTS_MD_BENCH_PROMPT or stdin over interpolating the prompt into a shell command.

The process environment is filtered. Common runtime variables such as PATH, HOME, temporary-directory variables, and terminal settings are inherited. Secrets are inherited only when named in pass_env.

The default Codex command follows the official non-interactive mode contract: the prompt is sent with the - stdin sentinel, output is JSONL, sessions are ephemeral, user configuration is ignored to reduce confounding, and writes remain limited to the benchmark worktree.

Budgets and interruption

npx agents-md-bench run \
  --max-minutes 30 \
  --max-tokens 100000 \
  --budget-usd 5

Wall-clock budget is always enforceable. Token limits require adapter telemetry. Cost limits additionally require pricing in the agent configuration:

agents:
  codex:
    pricing:
      input_per_million: 1.25
      cached_input_per_million: 0.125
      output_per_million: 10

Pricing changes over time, so the tool deliberately has no baked-in rates. Update these values from your provider. When usage or pricing is unavailable, the report says so instead of inventing a cost.

On Ctrl+C, the active trial is allowed to finish and the remaining matrix is stopped. results.json is updated after every trial, so partial data remains reportable:

npx agents-md-bench report --format html

Reading the statistics

Headline success is entirely based on your verification commands. The report uses matched task/trial outcomes and an exact McNemar test for the binary success difference. When the paired evidence does not cross p < 0.05, it says:

No reliable difference detected

That label matters. One stochastic run is useful for plumbing, not a basis for rewriting repository policy.

Known limitations of the MVP:

• The configuration and context file must be at the repository root.
• One root context file is manipulated; nested instruction inheritance is not yet modeled.
• Agent runs are sequential to make budgets and machine load predictable.
• Shell/file-read telemetry is available only when an adapter exposes it.
• Historical public tasks may have appeared in training or search data. Results compare context conditions for one agent; they are not a model capability ranking.
• Verification quality is task quality. Weak tests produce weak conclusions.

GitHub Action

The included composite action runs a lightweight task selection when repository instructions change. A workflow still needs to install/authenticate the chosen agent and expose only the credentials named by pass_env.

- uses: your-org/agents-md-bench@v1
  id: bench
  with:
    agent: codex
    config: agent-bench.yaml
    tasks: smoke

The action writes the Markdown report to the job summary and exposes results, report, and html output paths. See examples/context-bench-workflow.yml for an artifact upload and an idempotent PR-comment example. The example uses a trusted self-hosted runner: do not expose Codex/API credentials to untrusted pull-request code.

More detail is available in the experiment methodology and launch kit.

Development

npm install
npm run check
npm test
npm run build
node dist/index.cjs --help

Source layout:

packages/
├── cli/        command parsing and UX
├── core/       config, worktrees, orchestration, budgets, statistics
├── adapters/   Codex and generic shell invocation
├── scorers/    objective diff and verification scoring
└── reporters/  Markdown, HTML, badge, and terminal result card

Positioning

The product asks one deliberately narrow question:

In this repository, on these tasks, did this context file improve real agent behavior enough to justify its cost?

The intended loop is:

measure → ablate → propose a smaller rule set → measure again → keep evidence-backed rules

It does not generate a universal “AI readiness score,” and it does not use an LLM judge as the headline metric.