benchmark — Unified Agent Evaluation

This package exposes a consistent MCP workflow for benchmark-driven agent evaluation.

GAIA is the first built-in adapter, but the tool surface is generic so other benchmarks can plug in later without changing client behavior.

Tools

| Tool | Purpose | |------|---------| | benchmark_list_challenges | List available benchmark suites with version and asset metadata | | benchmark_start_challenge | Start an attempt and return the first question | | benchmark_submit_solution | Grade one answer and return the next question or final score | | benchmark_get_asset | Read an attached benchmark asset by asset_id | | benchmark_get_attempt | Inspect attempt status and the current question | | benchmark_cancel_attempt | Cancel an active attempt |

Workflow

1. Call benchmark_list_challenges
2. Pick a challenge_id
3. Call benchmark_start_challenge
4. If the question has assets, call benchmark_get_asset
5. Call benchmark_submit_solution
6. Repeat until done: true

Each response includes machine-readable guidance for the most likely next tool call.

Mock Mode

Run without any external dataset file:

MOCK=true npx @fre4x/benchmark

Optional Environment

BENCHMARK_GAIA_DATA_FILE=/absolute/path/to/gaia-challenges.json
BENCHMARK_STATE_DIR=/absolute/path/to/store-attempt-json
BENCHMARK_MOCK=true

• BENCHMARK_GAIA_DATA_FILE: Optional JSON file with GAIA-compatible normalized challenge definitions
• BENCHMARK_STATE_DIR: Where attempt state is persisted
• BENCHMARK_MOCK: Alternate mock-mode flag

Claude Desktop

{
  "mcpServers": {
    "benchmark": {
      "command": "npx",
      "args": ["-y", "@fre4x/benchmark"],
      "env": {
        "BENCHMARK_GAIA_DATA_FILE": "/absolute/path/to/gaia-challenges.json"
      }
    }
  }
}

Development

npm install
npm run build -w @fre4x/benchmark
npm run typecheck -w @fre4x/benchmark
npm test -w @fre4x/benchmark
MOCK=true npm run inspector -w @fre4x/benchmark