PocketBench

A keyboard-driven terminal UI and CLI for benchmarking AI models against real public benchmarks, built to stay honest when the API stops cooperating.

Install and run

PocketBench is the public name of BenchTUI. Install the global command with npm:

npm install -g pocketbench
pocketbench

Requires Node.js 18+ and uv. On macOS, uv can be installed with brew install uv. PocketBench uses uv to download Python 3.12.11 when needed and install the locked Python dependencies in a private user-cache environment. The first launch needs network access for runtime setup; subsequent launches reuse it. Installation runs no postinstall scripts. The npm launcher checks for a newer published PocketBench version at most once per day and updates the global npm installation in the background when one is available. The check is best-effort and never blocks a command; set POCKETBENCH_NO_AUTO_UPDATE=1 to disable it. pocketbench --version never performs an update check.

pocketbench opens the TUI in an interactive terminal. With redirected input/output, it shows help. pocketbench tui explicitly opens the TUI. All CLI commands work under the same name:

pocketbench --version
pocketbench doctor --json
pocketbench list --json
pocketbench scenario success --events jsonl
pocketbench inspect RUN_ID --json
pocketbench replay RUN_ID --case synthetic/0 --offline --json

Existing BenchTUI data locations and the BENCHTUI_* configuration variables remain supported. The npm installation saves Setup credentials in the user's configuration directory, outside the installed package. To reuse a source checkout's .env, set BENCHTUI_DOTENV to that file's absolute path. POCKETBENCH_CACHE_DIR overrides the private runtime cache, and POCKETBENCH_UV selects an explicit uv executable.

The Python module and legacy benchtui/benchtui-tui entry points retain their names. The development instructions and recorded-results references below refer to the source checkout. Maintainers can run npm run release:check to validate the packed npm install; npm pack automatically rebuilds the runtime using an explicit file allowlist.

pocketbench run gsm8k --model openrouter/nvidia/nemotron-3-super-120b-a12b:free --limit 30

Every claim below is backed by a test in tests/ or a recorded run in RESULTS.md. If something is not implemented, it is not listed.

Managed harnesses

Choose the model and Harness separately in Settings: No harness (direct API), OpenCode, or Terminus 2. Clicking Run prepares missing dependencies and data with cancellable progress. Docker Desktop must already be installed and running; Recheck harness setup checks Docker without starting an evaluation. Managed execution currently requires macOS or Linux and uv. It uses its own Python 3.13.11 / Harbor 0.23.0 environment, OpenCode 1.18.30, and fresh Docker task sessions. It never uses host-installed agents, their configuration, login sessions, skills, or MCP servers.

pocketbench harnesses --json
pocketbench run gsm8k --model openrouter/PROVIDER/MODEL --harness opencode --limit 5
pocketbench run terminal_bench --model openai/MODEL --harness terminus-2 --limit 5 --events jsonl
pocketbench run terminal_bench --model openai/MODEL --harness opencode --repetitions 5

Use a real model ID and configure its OpenAI, Anthropic, OpenRouter, or Gemini API key in PocketBench. Only the selected connection is supplied to a trial. Preparation is not proof of working provider access. Downloads and image preparation happen before inference; setup can take several minutes. Trials currently run sequentially; native agent recovery is retained, but PocketBench does not automatically rerun a completed or failed agent attempt. Unresolved trials can be resumed explicitly.

Terminal-Bench uses the actual 2.0 task dataset and Harbor verifiers. It requires a harness; one attempt per selected task is the default. Five repeats are an explicit research preset, not a claim of leaderboard parity. Task-defined resources, timeouts, and network policy apply. Existing text benchmarks use a labeled agent-assisted adaptation: the original question plus an answer-file instruction, the original scorer, and container-local tools with only the model endpoint reachable during agent execution. Harbor applies that allowlist in the agent phase; environment setup stays public so packages can install. Docker allowlist/dynamic policy needs Linux nftables CONFIG_NFT_FIB_INET (OrbStack or a Linux host). Docker Desktop for macOS often lacks it; Harbor then rejects the trial instead of running with an open network. Chat token/temperature settings are not enforced by managed harnesses; their native defaults and task timeouts apply. OpenCode title generation and automatic updates are disabled, and helper models are bound to the selected model.

Run manifests and trial evidence record the harness/preset, exact task fingerprints, dependency lock, resolved execution configuration, and Docker image identities before inference. Unknown usage/cost is reported as unknown. inspect RUN --case CASE --json includes sanitized verifier and trajectory evidence. Resume checks saved task and runtime identity and skips scored trials. Text replay remains offline; Terminal-Bench regrading is unavailable without retained final environment artifacts. Logs alone are not replay.

Methodology references: Terminal-Bench §3, Harness-Bench §4.1, and Harbor.

Why this exists

Benchmark harnesses fail in two quiet ways, and both produce numbers that look fine.

They measure invented data. A benchmark named after a public one, populated with plausible hand-written questions, produces a score that resembles a real score and means nothing. This tool refuses to do that: a canonical benchmark either loads its real, revision-pinned, checksum-verified data or reports UNAVAILABLE. There is no fallback path that substitutes generated items.

They turn outages into scores. A free-tier model gets rate limited half-way through a run, forty requests fail, and the leaderboard shows 0%. That is not a fact about the model. Here, a case that never produced a gradeable answer is RATE_LIMITED or ERROR, is excluded from the denominator, and the run is marked PARTIAL everywhere it appears.

Benchmarks

Five public datasets (26,739 items) plus four original ForensicFin cases. Public data is pinned to upstream revisions and verified by SHA-256, schema and row count.

| Benchmark | Items | Split | Source | |---|---:|---|---| | GSM8K | 1319 | test | openai/grade-school-math | | MMLU | 14042 | test | cais/mmlu (57 subjects) | | HumanEval | 164 | test | openai/human-eval | | HellaSwag | 10042 | validation | rowanz/hellaswag | | ARC-Challenge | 1172 | test | allenai/ai2_arc | | ForensicFin | 4 | — | BenchTUI original — not a published benchmark |

HumanEval is scored by executing each problem's own check(candidate) harness in a sandbox, not by matching text. HellaSwag uses the validation split because the test split ships without labels and cannot be scored locally at all.

These adapters use a chat evaluation protocol, not an exact reproduction of the original papers. --protocol paper reports unavailable instead of substituting chat inference. The methodology audit maps the original papers to the implemented variants and their limitations. --max-tokens overrides the configured output budget for a new CLI run; the default is 2,048 per response. Results record budgets and truncation. Agent CLIs control their own generation settings.

ForensicFin is our own work and is labelled that way in the UI, the CLI, the exports and the leaderboard. Its scores are not comparable to published numbers.

Honesty rules

These are enforced by tests, not convention.

1. A benchmark named after a public benchmark uses that benchmark's real data, or reports UNAVAILABLE. It never invents items.
2. Non-canonical benchmarks carry is_canonical = False and are labelled everywhere.
3. A subset run is labelled with both counts: GSM8K [30/1319], never a bare GSM8K. Thirty questions is not GSM8K.
4. A run with unresolved cases is PARTIAL everywhere, with the unresolved count.
5. Unresolved cases are excluded from accuracy, never counted as wrong.

Rate limits

Free models get rate limited. The design goal is that this changes how long a run takes and never what it concludes.

• Retry on 408/409/429/5xx and timeouts: exponential backoff, ±25% jitter, per-sleep cap. A server Retry-After overrides the computed delay and is never jittered downward — sleeping less than the server asked for earns another 429.
• Adaptive concurrency (AIMD): halve on a 429 and cool down every worker, creep back up after a clean streak.
• Per-case wall-clock cap: a sleep that would overshoot the budget is refused rather than taken, so one dead endpoint cannot wedge the run.
• Resume: every case is committed to SQLite the moment it resolves. A run killed at any point resumes and re-runs only the cases that never produced an answer.
• Nothing is hidden: retries, backoff, concurrency changes and unresolved counts are visible live in the TUI and persisted for the case inspector.

Recorded live in RESULTS.md: a 30-case run interrupted with SIGINT at case 12, resumed, four cases hitting real rate limits during the resume and all four recovering, finishing complete and not partial.

Install

Needs Python 3.12+.

git clone <this repo> && cd benchtui
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python -e ".[dev]"

echo 'OPENROUTER_API_KEY=sk-or-v1-...' > .env   # or OPENAI_/ANTHROPIC_/GEMINI_
benchtui download                                # ~16MB, all five datasets

Ollama needs no key; BenchTUI discovers whatever you have pulled locally.

Use

benchtui list                          # benchmarks and dataset status
benchtui models --free                 # what you can run, from the live catalogue
benchtui download gsm8k mmlu           # or all of them with no arguments

benchtui run gsm8k --model openrouter/nvidia/nemotron-3-super-120b-a12b:free --limit 50
benchtui runs                          # run history with PARTIAL and subset flags
benchtui resume <run_id>               # re-run only the unresolved cases
benchtui export <run_id> --format json # full per-case detail incl. retry history

Exit codes: 0 success · 1 error · 2 completed but PARTIAL. The third exists so CI can decide for itself whether a rate-limited run counts as a pass.

The TUI

./run_tui.sh

| Key | Action | |---|---| | r / a | run the selected benchmark / all available ones | | c / s | cancel a run · resume the last unfinished one | | m / u | pick a model (searchable; type any id) · refresh the catalogue | | d | download missing datasets | | l / i | leaderboard · case inspector | | q | quit |

The runner pane shows per-case results as they land plus a live rate-limit panel (concurrency, in-flight, rate-limit hits, retries, unresolved, time spent waiting, ETA). The leaderboard is read from SQLite and survives restarts. The case inspector shows the prompt, the raw model output, the extracted answer and the retry history for that case.

Adding a benchmark

Drop a file in benchtui/benchmarks/. Nothing else needs editing.

from benchtui.core.benchmark import Benchmark, BenchmarkCase, Score
from benchtui.core.registry import register_benchmark
from benchtui.core.status import CaseStatus

@register_benchmark
class MyBenchmark(Benchmark):
    benchmark_id = "mine"
    name = "My Benchmark"
    category = "Custom"
    is_canonical = False        # True only if it loads real published data

    def load_cases(self, limit=None, seed=0):
        return [BenchmarkCase(case_id="mine/0", prompt="2+2?", expected="4")]

    def score(self, case, output):
        ok = output.strip() == case.expected
        return Score(CaseStatus.PASS if ok else CaseStatus.FAIL, 1.0 if ok else 0.0)

A benchmark is data, a prompt and a scorer. It does not call models — the runner owns retry, rate limiting, concurrency, cancellation and persistence, so every benchmark gets all of it for free.

Architecture

benchtui/
  core/        Settings, CaseStatus, the Benchmark interface, the registry
  providers/   async client, typed errors, retry policy, AIMD limiter, model catalogue
  datasets/    canonical dataset download, cache and row-count verification
  benchmarks/  one module per benchmark — auto-discovered
  execution/   run engine (workers, retries, resume) and the code sandboxes
  storage/     SQLite: runs, case_results, retry_events
  ui/          the Textual app

Sandboxing

HumanEval requires Docker and a cached Python image before generation. Execution uses the recorded immutable image ID, --network none, --read-only, --cap-drop ALL, --pids-limit, memory and CPU caps, and a non-root user. Host fallback and automatic image pulls are disabled. Candidate deadlines, bounded output and named-container cleanup apply on completion and cancellation. The controlled test runner prevents ordinary stdout-marker/early-exit false passes; it is not an adversarial security boundary against Python code tampering with its own interpreter.

lm-evaluation-harness bridge

ForensicFin can be run through EleutherAI's harness:

lm_eval --model local-completions \
  --model_args model=<model>,base_url=https://openrouter.ai/api/v1/chat/completions \
  --include_path ./eleuther_tasks --tasks forensic_fin

lm_eval is not a dependency of this project. See RESULTS.md for what was actually verified against it.

Tests

./scripts/verify.sh              # full offline verification
./scripts/verify.sh live         # explicit live provider and dataset checks

The gate proves it can fail before it reports success: it sabotages a canary test and requires pytest to go red, requires the same canary to be green unsabotaged, requires ruff to flag a known-bad file, and requires the placeholder scan to match a real TODO. If any probe fails it prints GATE IS NOT TRUSTWORTHY and refuses to evaluate the code.

Rate-limit behaviour is tested against a real local HTTP server that returns scripted 429s with Retry-After, 5xx, hangs that trip real read timeouts, gateway HTML, HTTP 200 with an error envelope, and the verbatim OpenRouter "agentic harnesses only" 403.

Agent and worktree workflows

Run ./scripts/bootstrap.sh once in every checkout or Git worktree. It creates that checkout's .venv using .python-version and the checked-in uv.lock. Updating the lock is deliberate (uv lock); bootstrap uses --frozen and does not resolve upgrades.

Use ./scripts/dev.sh instead of the installed command for isolated development. It keeps writable databases and caches in the checkout's ignored .benchtui/ directory and ignores personal dotenv/TOML files unless BENCHTUI_DOTENV or BENCHTUI_CONFIG_FILE is explicitly supplied. Provider keys may be supplied through the environment; do not put them in command arguments or commit them.

./scripts/dev.sh doctor --json
./scripts/dev.sh list --json
./scripts/dev.sh scenario rate-limit --events jsonl
./scripts/dev.sh inspect <run-id> --json
./scripts/dev.sh inspect <run-id> --case synthetic/0 --json
./scripts/dev.sh replay <run-id> --case synthetic/0 --offline --json

list, models, runs, run, and resume accept --json. New doctor, inspect, and replay commands support it too. Snapshots are one JSON object with schema_version: 1, command, data, and error. An error has code, message, and retryable; successful snapshots have error: null.

run, resume, and scenario accept --events jsonl. Stdout contains only JSON objects, one per line. The first event is run.started (or run.resumed) with the persisted run ID, emitted before any target request. Each run event carries schema_version, type, run_id, a per-invocation sequence, and at (Unix seconds). The event types are case.started, case.completed, retry.scheduled, run.waiting, and run.finished, plus the initial event. A preflight error emits type: error with no run ID. run.finished distinguishes completed, partial, cancelled, aborted, and failed. Exit codes remain 0 complete, 1 error, 2 partial/cancelled. Do not combine --json and --events; event streaming takes precedence.

The CLI and TUI both use TargetClient, including agent/<id> targets that manage their own authentication. doctor reads local configuration and runtime availability; a configured key is explicitly unverified. doctor --probe --model <target> opts into one small, bounded request, which can incur provider usage. Diagnostics never print credentials or provider response bodies.

New runs save immutable case inputs, effective generation requests, credential-free settings, dataset/sample checksums, and code fingerprints before contacting a target. export includes these inputs and the manifest. Resume uses saved questions and request parameters when available. Runs with existing scores refuse to resume when the scorer or executor policy changed or its provenance is unknown.

Replay uses the saved case and response with the currently installed scorer, without provider calls or dataset downloads. It reports the old/new scores and whether the scoring-module and declared-dependency fingerprints match. Older manifests without a scorer fingerprint report scorer_matches_recording: null (unknown). Replay does not overwrite history. Runs recorded before input snapshots were introduced report that replay is unavailable. Offline HumanEval replay requires Docker and an already cached python:3.12-slim image; image pulls and host fallback are disabled.

Synthetic scenarios are success, rate-limit, authentication, truncated, and cancel. They run scripted HTTP responses through the real parser, runner, retries, storage and event output. Their benchmark name and manifest explicitly say synthetic; they are never canonical results or evidence of live provider availability. Use a new scenario invocation to exercise another outcome; synthetic runs are inspectable and replayable, but are not provider-resumable jobs.

Verification scopes:

./scripts/verify.sh core
./scripts/verify.sh providers
./scripts/verify.sh scoring
./scripts/verify.sh tui
./scripts/verify.sh full          # default; offline
./scripts/verify.sh live          # explicit real provider calls/downloads

Every scope runs lint and gate self-checks. Offline pytest sessions use temporary worktree-local state and remove real provider credentials from the test environment. Tests requiring full public datasets may skip when those datasets are absent; the synthetic correctness and failure tests still run. The live scope requires explicitly exported credentials. The previous ./scripts/verify.sh --no-live spelling remains an alias for the full offline scope. See AGENTS.md for the module map and invariants.