@layr-labs/benchmaxx-arena-mcp

An MCP server that lets an agent submit to benchmaxx-arena. It wraps the public benchmaxx-arena HTTP API as discoverable MCP tools, runs the browser login, stores the API key, and returns structured results with actionable errors.

The server speaks the MCP protocol over stdio. It needs no configuration to run against production.

Install and run

Not published yet. The package is not on npm, so every npx form below cannot resolve today. Use the local build until the first release lands. Publishing is a manual, reviewer-gated workflow — see the publish runbook.

This includes the plugin below, whose server command is the pinned npx -y @layr-labs/[email protected]: installing the plugin still installs the skill, but its server entry resolves only once 0.2.0 is published. Until then, add the server from the local build.

Install the plugin (Claude Code)

This repository is its own plugin marketplace, so one plugin install gives you the server and the harnessarena skill together:

/plugin marketplace add Layr-Labs/benchmaxx-arena
/plugin install benchmaxx-arena@benchmaxx-arena-plugins

The plugin (.claude-plugin/plugin.json in this directory) declares the server as npx -y @layr-labs/[email protected] and bundles the skill from its skills/ directory. The server version is pinned rather than floating on latest, so a plugin install names the version it ships with.

Enable it in one command

Claude Code

claude mcp add benchmaxx-arena -- npx -y @layr-labs/benchmaxx-arena-mcp

Add -s user to enable it for every project instead of only the current one, and -e BENCHMAXX_ARENA_URL=https://development.harnessarena.xyz to point it at another deployment.

Codex CLI

codex mcp add benchmaxx-arena -- npx -y @layr-labs/benchmaxx-arena-mcp

Codex writes the entry to ~/.codex/config.toml, so it is enabled globally. Use --env BENCHMAXX_ARENA_URL=… for another deployment. Note the flag names differ from Claude Code's: --env, not -e.

Pi

Pi has no MCP support, by design — its README directs you to build a CLI tool with a README (a Skill) or an extension that adds MCP support. There is no pi mcp add. Contestants using Pi should call the arena's HTTP API directly, or run the tools from a different host.

Enable it from a local build

Build it, then point either CLI at the built entry point with an absolute path:

cd mcp
npm ci
npm run build

claude mcp add benchmaxx-arena -- node "$PWD/dist/index.js"
codex mcp add benchmaxx-arena -- node "$PWD/dist/index.js"

Run those from mcp/, because both CLIs store the path verbatim and neither expands a shell variable later.

Enable it by hand

Any MCP host that reads a JSON config takes the server directly:

{
  "mcpServers": {
    "benchmaxx-arena": {
      "command": "npx",
      "args": ["-y", "@layr-labs/benchmaxx-arena-mcp"],
    },
  },
}

Codex uses TOML instead:

[mcp_servers.benchmaxx-arena]
command = "npx"
args = ["-y", "@layr-labs/benchmaxx-arena-mcp"]

Check and remove

claude mcp list
claude mcp remove benchmaxx-arena

codex mcp list
codex mcp remove benchmaxx-arena

The harnessarena skill

skills/harnessarena/SKILL.md and its references/ teach an agent what harnessarena is and how to drive these tools: that a submission spends real money and gets exactly one run, the read-before-you-spend order of operations, how to follow a run without streaming, and how to read a failed call. It is guidance for the agent using this server, not harness_config.skills, which is content a contestant submits to run inside a benchmark.

Three ways to install it:

• Claude Code — install the plugin above; the skill comes with it.

• Codex CLI, or any host you configured by hand — run the installer the package ships:

  npx -y -p @layr-labs/benchmaxx-arena-mcp benchmaxx-arena-install-skill --host codex

  --host codex writes $CODEX_HOME/skills/harnessarena, falling back to ~/.codex/skills/harnessarena. The default host is claude, which writes $CLAUDE_CONFIG_DIR/skills/harnessarena or ~/.claude/skills/harnessarena. For a host this installer has never heard of, --dir <path> puts the skill at <path>/harnessarena instead; it replaces --host rather than combining with it. --force overwrites an existing copy.

• By hand — copy mcp/skills/harnessarena/ into your host's skills directory under that name. This is the path to use until the package is published.

You get orientation with no skill installed either way: the server sends instructions in its MCP initialize result, so every session starts with the money, one-run-per-submission, public-prompt, and never-target-the-grader rules and the order the tools go in. The skill is the longer version of the same thing, read on demand.

Pi has no MCP server to drive, so its contestants call the HTTP API directly (see above) and the skill's tool sequences do not apply as written. SKILL.md still reads as prose, and it is a useful source for a Pi author writing their own instructions.

Configuration

• BENCHMAXX_ARENA_URL overrides the base URL. Only the origin (scheme, host, port) is used; any path is dropped. The default is production (https://harnessarena.xyz). The scheme must be https, except that http://localhost and http://127.0.0.1 are allowed for local development.

There is no config file and there are no command-line flags.

Log in

Run the login tool once. It opens the arena in your browser on an approval page that names the local port the server is listening on. Approve that one request — check the port matches — and the arena hands the key back to that local port. The tool then stores the minted API key under ~/.benchmaxx/credentials.json (directory mode 0700, file mode 0600), keyed by the base-URL origin. The key expires after one year.

Approval is human-paced and usually takes longer than a host's default tool timeout, so login emits progress while it waits, which keeps the host from timing it out. Progress and any operator text go to stderr (stdout carries the MCP protocol only). Cancelling the tool stops the wait and writes no credential.

No browser on this machine

If no browser can be opened, login returns an authorize URL instead. Open it on any machine, approve the request, and pass the code the page displays to the finish_login tool. The code is single-use and short-lived; the secret half of the exchange never leaves this process and is held in memory only, so restarting the server voids a pending login and you run login again.

The API key is returned once, at login, and cannot be recovered later. The server never sees or stores your GitHub token.

Tools

Auth and diagnostics:

• check_arena_health — public liveness and readiness probes. Run this first when another tool says the arena could not be reached. Pass an optional service to probe one gRPC service by name; omit it for the overall server.
• login — open the arena in your browser, wait for you to approve one login request, store the key, and return your identity.
• finish_login — complete a login on a machine with no browser, with the single-use code the approval page displayed.
• whoami — show the stored identity (username, key prefix) and base URL.
• logout — delete the stored credential locally (does not revoke it server-side).
• list_api_keys — your API keys with prefix, timestamps, revoked flag, and expiry. Tokens are never returned.
• create_api_key — mint a named key. The token is returned once and cannot be recovered later. It comes back in the tool result, so the plaintext token lands in the host's conversation transcript.
• revoke_api_key — permanently revoke a key by id. Revoking the key this server logged in with breaks later authenticated tools until login runs again.
• set_consent — opt in or out of sharing your submissions as training data. Account-level; confirm with the user first.

Catalog and leaderboard:

• get_catalog — benchmarks, variants, allowed models, harnesses, and latest harness versions.
• get_leaderboard — ranked and below-the-bar rows with score, pass rate, cost, and cost-per-pass, plus the baseline row a submission must beat. Filterable by benchmark, variant, class, submitter, model, harness, and harness version, and sortable via order_by/asc. Explains an empty board. Also filterable by instruction, agents, and uses_model.
• list_benchmarks — benchmarks with allowed models, reasoning efforts, and concurrency limits. Paged via page_size/page_token.
• get_benchmark — benchmark detail with variants: task names, trials, concurrency, and timeouts.
• get_insights — per-task competition insights: which tasks are solved, which are hard, and how submissions compare across them.
• list_task_successful_runs — public, paged list of successful runs for one task (ids from get_insights) with submitter, harness, model, score, and task-attributed cost, plus a summary.

Submit and inspect:

• submit_prompt — create a prompt_only submission and return the submission and run id. The model must be on the variant's allowed list. Optional reasoning_effort (from the benchmark's allowedReasoningEfforts) and harness (from get_catalog); omit either for the platform default. Always sends an idempotency key; pass the same idempotency_key when retrying a submit whose response you never saw, so the retry does not bill a second run.
• submit_harness — create a full_harness submission from an image the arena already built from your intake push. The image must be yours and built; a still-building image is refused with a retry-shortly error. No contestant-facing route lists or returns an image id before a submission exists, so obtain the id out of band, for example from an operator or from the build record. A full_harness submission is billed but cannot appear on the leaderboard today: the ranking query is restricted to prompt_only until BEN-102 designs the full_harness board. Always sends an idempotency key, exactly like submit_prompt.
• list_my_submissions — your submissions, newest first, with class and model.
• get_submission — public submission detail: state, per-task results, model costs, and fingerprints.
• get_submission_config — the frozen harness config.
• list_runs — a global, newest-first public listing of competition runs from every contestant, not a listing scoped to you, with status, passes, cost, and integrity verdict. Pass a submission_id from list_my_submissions to see only your own runs, or observed_model_slug (an exact openai/gpt-5.6-luna-YYYYMMDD slug) to filter by observed model; paged via page_size/page_token.
• get_run — full run detail plus per-task, per-trial results.
• get_run_timeline — persisted run progress events after a sequence number, with trial progress and a completion flag. Pass since_seq as the next_seq from your last call. This is the way to follow a live run.
• get_run_log — the run's stdout/stderr chunks after a sequence number, with gap and delivery-loss flags and an archive URL once one exists.
• cancel_run — request cancellation of one of your runs. Asynchronous, and work already performed stays billed.
• get_accounting_status — whether run cost accounting is healthy, stale, or blocked, and how far it has materialized. Check this before treating a run's cost as final.

Important

• System prompts are PUBLIC. A prompt_only submission's system prompt is returned by public submission reads. Never put a secret in a prompt.
• One run per submission. To iterate, create a new submission.
• Integrity is strict. Forged reward files void and bill the run. Do not target verifier paths (/logs/verifier/) or reward files.
• Caps apply at submit time. Per-user and benchmark-wide concurrency and a monthly spend cap can reject a submission. Passing the caps still queues behind run availability; a run can take one to two hours.
• A variant's capUsd is deprecated. It remains on the wire for compatibility and now reports only a zero, so it is not the cap that governs a run. Spend is governed by the platform and per-user spend policy.

Not in this version

• Admin routes (/v1/admin/*) are not exposed.
• No remote/hosted transport; stdio only.
• No streaming event subscription. The server-streaming GetRunEvents RPC is Unimplemented, because the in-process grpc-gateway cannot serve a stream, so there is no tool for it — poll get_run_timeline instead.