⚔️ pi-arena

Run the same task across multiple models in parallel and compare their answers side by side — with an optional referee judge that ranks the anonymized answers.

Works with every model your pi installation knows about, including models released after this package: contestants are discovered at runtime from pi's model registry, nothing is hardcoded. New model out? pi update, and it's in the arena.

Install

pi install npm:pi-arena

Quick start

/arena-models                 # pick 2+ contestants from your configured providers
/arena Explain the tradeoffs of SQLite vs Postgres for a small CMS
/arena-history                # browse past runs of this session anytime

That's it. On your very first /arena the contestant picker opens automatically, so you can also just start with a prompt.

Commands

| Command | What it does | |---|---| | /arena <prompt> | Run the prompt against all selected models, side by side | | /arena-models | Pick which models compete — live multi-select with filtering | | /arena-judge [spec] | Pick the referee: interactive picker, current, off, or provider/id | | /arena-history | Reopen any past run of this session in the interactive browser |

/arena-models opens a live multi-select: ↑/↓ to move, space to toggle, type to filter, enter to save.

The live race

The moment you submit a prompt, an animated status panel appears above the editor: every contestant gets its own spinner, token/turn counters tick up as answers stream in, and the progress slots settle into solid blocks as models cross the finish line.

Each contestant runs in its own isolated pi subprocess (read-only tools by default), so answers never touch your files. Contestants that hang are killed on timeout — and no contestant process ever outlives its arena run, even if the run itself fails.

The scoreboard

When all runs finish, a compact scoreboard lands in your chat history: medals, judge scores as bars, and time / tokens / cost per model. Press ctrl+o on the message to expand it into the full answers and the referee's notes.

The report is injected into your session as a normal message, so you can keep chatting about the results with your main model.

The results browser

Right after the scoreboard (and any time later via /arena-history), an interactive browser opens:

| Key | Action | |---|---| | ↑ / ↓ | Switch between contestants (and the referee's verdict) | | enter | Toggle a full-height, distraction-free reading view | | pgup / pgdn | Scroll long answers | | esc | Leave the reading view, then close |

The referee

If a judge is configured (default: your current session model), the arena runs one extra round after the contestants finish: the answers are anonymized and shuffled — the judge sees "Answer 1..N" in random order, never model names — so it can't develop position or brand bias. It scores each answer out of 10 with a one-line reason, then the winner is revealed.

/arena-judge                  # interactive picker
/arena-judge current          # judge with the session's model (default)
/arena-judge openai/gpt-5.2   # or any explicit model
/arena-judge off              # skip judging entirely

Configuration

Settings live in ~/.pi/agent/arena.json (managed by the commands above, hand-editable):

{
	"models": ["anthropic/claude-opus-4-8:low", "anthropic/claude-opus-4-8:high"],
	"judge": "current",
	"tools": ["read"],
	"timeoutSec": 300
}

| Field | Meaning | |---|---| | models | Contestant specs as provider/id, with an optional :<thinking> suffix | | judge | "current" (session model), "off", or an explicit provider/id | | tools | Tool allowlist given to contestants ([] = no tools) | | timeoutSec | Per-contestant timeout in seconds |

Model specs pass through pi's provider/id:<thinking> syntax, so you can race the same model against itself at different thinking levels — the config above pits Opus-low against Opus-high.

[!WARNING] Beware of granting write / edit / bash: parallel contestants share your working directory.

Headless / CI benchmarking

PI_ARENA_TEST="Refactor suggestion for src/foo.ts?" PI_ARENA_OUT=report.md \
  pi --mode json -p --no-session "arena"

Runs the arena without a TUI and writes the markdown report to PI_ARENA_OUT (stderr if unset). Handy for nightly "which model handles our codebase best" checks.

Development

extensions/arena.ts     entry point — command registration and wiring
extensions/arena/       types, config, contestant (process management),
                        judge, report, run (orchestration), widget (live
                        animation), render, browser, picker (TUI)

npm run check type-checks, npm test runs the process-reaping and TUI test suites (Node ≥ 23). Both suites carry watchdogs: a hung test kills its own process tree and aborts instead of lingering.

License

MIT