pi-terminal-bench

68 coding tasks for pi. No Docker, no frameworks, no API keys — just the pi CLI and Python 3.10+. You watch the agent work in real time.

Install

pi install /path/to/pi-terminal-bench

Then restart pi or run /reload.

Requirements

• pi CLI
• Python 3.10+ and bash
• Optional: numpy, pandas, sympy, word2number — a handful of ported Terminal-Bench tasks use these. The verify scripts auto-install on demand, so you usually don't have to care.

Runs against any model pi has configured — local (OMLX, LM Studio, Ollama) or remote (Anthropic, OpenAI). Defaults to your active model; append provider/model to any command to override.

Commands

| Command | What it does | |---|---| | /bench-list [filter] | List tasks. filter matches name, category, or tag | | /bench-run <task\|category\|all> | Run one task, a whole category, or everything | | /bench-results [N] | Recent runs. With N, per-task detail for run N | | /bench-doctor | Check prerequisites | | /bench-cleanup | Kill stray benchmark processes |

Tasks — 68 across 11 categories

| Category | Count | Command | What it tests | |---|---|---|---| | QuixBugs | 40 | /bench-run quixbugs | Single-line Python bug fixes (upstream) | | Terminal-Bench ports | 8 | /bench-run terminal-bench | Tasks ported from Terminal-Bench, Docker-free | | Hard | 7 | /bench-run hard | Multi-step algorithms, parsing, concurrency | | Long Context | 6 | /bench-run long-context | Multi-file refactors, test generation, API migrations | | Code Generation | 3 | /bench-run codegen | Build CLIs, REST APIs, state machines from a spec | | Performance | 2 | /bench-run perf | Optimize O(n²) code | | Security | 2 | /bench-run security | Fix SQL injection and path traversal | | File Operations | 2 | /bench-run file-operations | Read/write/transform files | | Mathematics | 2 | /bench-run math | Symbolic math, arithmetic puzzles | | Games | 2 | /bench-run games | Game-logic and puzzle solvers | | Data Science | 1 | /bench-run data-science | pandas ETL | | Debugging | 1 | /bench-run debugging | Fix a diverging ML training loop |

Run /bench-list <category> to see individual task names.

Example

/bench-run quixbugs-python-bitcount                         # one task
/bench-run hard                                             # one category
/bench-run quixbugs anthropic/claude-sonnet-4-20250514      # override model
/bench-run all                                              # everything
/bench-results                                              # past runs
/bench-results 1                                            # per-task detail

Results are written as JSON to ~/.pi/agent/pi-terminal-bench/results/.

Adding tasks

Drop a JSON file in tasks/:

{
  "name": "my-task",
  "description": "What this tests",
  "instruction": "What the agent sees",
  "setup_files": { "buggy.py": "...", "test.py": "..." },
  "verify": "cd $BENCH_WORK_DIR && python3 test.py",
  "timeout": 180000,
  "tags": ["custom"]
}

$BENCH_WORK_DIR is replaced with the task's workspace. verify passes iff exit code is 0. Keep verifies fast (< 30s), deterministic, and scoped to the workspace.

Safety

Every task runs in an isolated temp directory with a pi-bench. prefix ($TMPDIR/pi-bench.XXXXXX). After each task — pass, fail, or abort — the runner kills lingering processes (including descendants reparented to launchd) and removes the workspace. Active workspaces are persisted to ~/.pi/agent/pi-terminal-bench/active-workdirs.txt, so /bench-cleanup can sweep orphans from crashed sessions.

Every cleanup path is scoped strictly to paths matching pi-bench. — Homebrew, Xcode, git, and any other tool's temp files are untouchable.

Timeouts

Each task has a timeout (default 180s; harder tasks use 240s or 360s). If a command hangs, the agent gets a 2× extended window to recover with a steer message explaining the timeout. If the agent makes no file changes, the task is recorded as FAIL — never a false PASS.

Contributing

PRs welcome. Terminal-Bench has 241 Docker-based tasks; we've ported a subset that runs without Docker and will expand over time.