LocalArena

Run the models you choose against the tasks you choose, using live provider responses from your machine or an explicitly configured API endpoint.

LocalArena executes the complete model × task grid concurrently, applies deterministic scorers or a live judge model, records failures as data, derives order-invariant arena ratings with uncertainty, and writes a standalone HTML report. It never substitutes recorded answers or copied leaderboard results for a live call.

• llama.cpp, Ollama, and LM Studio on loopback or any reachable host
• OpenRouter and OpenAI with separate runtime credentials
• Any compatible Chat Completions endpoint through a custom base URL
• Python CLI and library, plus matching JavaScript and TypeScript APIs
• Versioned, content-addressed native task packs and JSONL interoperability
• Multi-reference, choice, extraction, token-F1, JSON, numeric, and judge scoring
• Bradley–Terry ratings, task-clustered confidence intervals, and honest inconclusive results
• Privacy-safe result files by default: prompts and answers require opt-in
• No runtime dependencies in either package

First live score in one minute

The one-minute command-line path uses Python 3.10+:

pip install localarena

Once a provider is running and its model is available, one command performs a real generation, scores it, saves JSON, and writes a standalone HTML report:

localarena quickstart ollama qwen3:0.6b

The default files are localarena-results.json and localarena-report.html. The same command works with llama.cpp, LM Studio, OpenRouter, OpenAI, and any compatible custom endpoint.

Model downloads are not part of the one-minute path and can take much longer. LocalArena never downloads or starts models. The complete getting-started guide gives copy-paste setup and quickstart commands for every provider, macOS/Linux and PowerShell credential setup, privacy boundaries, two-model comparisons, live judges, and troubleshooting.

The Node.js 18+ package exposes the matching JavaScript library, TypeScript types, examples, documentation, and schemas; it does not install the Python CLI:

npm install localarena

Run a reusable evaluation

Create a JSON configuration with any number of model targets and scored tasks, then run the complete Cartesian product:

localarena run evaluation.json \
  --output results.json \
  --report report.html

The command prints progress as calls complete and returns status 2 if any generation or scoring process errored. An ordinary scorer mismatch remains a valid row with score zero. Failures never abort or silently shrink the matrix.

Start from the complete multi-provider reference. It shows all six profiles and configuration fields, but it is not expected to run unchanged because it assumes all endpoints and credentials are ready at once. For one target at a time, use the genuinely runnable provider configurations. Every committed example is executed against a compatible live fixture in CI and before a release.

Providers

All built-in profiles use bounded, non-streaming Chat Completions requests and model discovery through /v1/models.

| Profile | Default base URL | Default credential | | --- | --- | --- | | llamacpp | http://127.0.0.1:8080/v1 | None | | ollama | http://127.0.0.1:11434/v1 | None | | lmstudio | http://127.0.0.1:1234/v1 | None | | openrouter | https://openrouter.ai/api/v1 | OPENROUTER_API_KEY | | openai | https://api.openai.com/v1 | OPENAI_API_KEY | | custom | Required in config | Optional api_key_env |

A model entry may override base_url, select any model ID, set generation parameters, and read credentials or sensitive headers from environment variables. Literal API keys are rejected by the CLI configuration loader. Hosted-provider keys are isolated: a standard cloud key is never forwarded automatically when that profile is pointed at a different base URL.

Useful commands:

localarena providers
localarena models ollama
localarena quickstart ollama qwen3:0.6b
localarena models custom --base-url http://127.0.0.1:9000/v1
localarena report results.json --output report.html

For an authenticated custom endpoint:

printf 'Custom endpoint API key: '
read -r -s LOCALARENA_CUSTOM_API_KEY
printf '\n'
export LOCALARENA_CUSTOM_API_KEY
localarena models custom \
  --base-url https://inference.example.com/v1 \
  --api-key-env LOCALARENA_CUSTOM_API_KEY

The getting-started guide also includes the PowerShell credential flow.

Requests make one attempt by default because replaying an ambiguous generation can duplicate work or billing. Retries are explicit through policy.max_attempts.

Tasks and scoring

Every native task is provider-independent JSON-safe chat messages plus one optional evaluator:

| Evaluator | Use | | --- | --- | | exact | Complete answer equality with explicit whitespace/case rules | | match | Any accepted reference at the exact, beginning, end, or anywhere location | | choice | Boundary-safe leading multiple-choice or classification label | | extract | Regex capture followed by multi-reference normalized matching | | token_f1 | Partial credit from whitespace-token multiset F1 | | contains | Require all or any configured strings | | regex | Search or full-output format validation | | json | Strict JSON validation or type-sensitive JSON comparison | | numeric | Numeric answer with an absolute tolerance | | model_judge | Live rubric scoring from another configured model | | null | Generate and retain run metadata without a score |

Reusable task packs

Keep tasks separate from model and provider configuration with a native pack:

{
  "schema_version": 1,
  "name": "Release gate",
  "version": "1.0.0",
  "license": "internal",
  "tasks": [
    {
      "id": "final-answer",
      "prompt": "Show brief work, then end with #### 42.",
      "evaluator": {
        "type": "extract",
        "expected": ["42"],
        "pattern": "####\\s*([0-9]+)",
        "group": 1
      }
    }
  ]
}

Reference it from an evaluation config using a path relative to that config:

{
  "models": [
    {
      "name": "candidate",
      "provider": "ollama",
      "model": "qwen3:0.6b"
    }
  ],
  "task_files": ["release-gate.localarena"]
}

The loader validates every task, rejects duplicate IDs, records the pack version and cross-runtime SHA-256 digest, and never executes downloaded code. It also auto-detects JSONL rows with input and ideal, so simple existing Evals-style datasets can run live without conversion. See the executable task-pack example.

A content digest proves which pack was loaded; it does not make an untrusted pack safe. Review packs before running them. In particular, regex and extract patterns execute in the runtime's native regular-expression engine while scoring and can have pathological running time.

Set "judge_only": true on a model entry to use that model for model_judge tasks without evaluating or ranking it as a contestant.

concurrency bounds the number of in-flight rows. repetitions repeats the entire grid. Each model keeps its own base URL, credential, model ID, and generation settings, so local and hosted targets can run in the same evaluation.

Results and reports

results.json follows the versioned evaluation run schema. It contains:

• one record for every model, task, and repetition;
• normalized status, timing, finish reason, token use, and attempt count;
• deterministic or live-judge scores;
• raw mean and a decision score where failed scored rows count as zero;
• score coverage, pass rate, errors, latency, tokens, and judge aggregates;
• order-invariant Bradley–Terry ratings, task-clustered 95% confidence intervals, comparison components, and inconclusive flags;
• legacy sequential Elo for replay compatibility; and
• an arena snapshot whose match history can move between Python and JavaScript.

JSON Schema validates the portable document shape. Load untrusted snapshots with run_from_dict or runFromSnapshot as well; the SDK loaders enforce the cross-field guarantee that every configured model × task × repetition has exactly one row.

The HTML report is a single local file with no scripts, fonts, stylesheets, or network requests. Its default leaderboard is ordered by the reliability- adjusted decision score, not by a raw mean that silently ignores failures. It also shows the arena interval and whether the evidence actually separates a model.

Prompt text, evaluator criteria and reference answers, generated text, score reasons, task metadata, and detailed errors are excluded from saved results and reports by default. Non-secret evaluator type and live-judge target provenance remain visible so scoring failures are attributable. Opt in only when the content is safe to retain:

localarena run evaluation.json \
  --include-content \
  --output results-with-content.json \
  --report report-with-content.html

Credentials, authorization headers, base URLs, and configured headers are never serialized into an evaluation run.

Python API

from localarena import (
    EvaluationRunner,
    ExactMatch,
    ModelTarget,
    PromptTask,
    create_provider,
    write_html_report,
)

provider = create_provider("ollama")
target = ModelTarget(
    name="local-model",
    provider=provider,
    model="qwen3:0.6b",
    max_tokens=64,
    temperature=0,
)
task = PromptTask.from_text(
    "capital",
    "Reply with exactly Paris.",
    evaluator=ExactMatch("Paris"),
)

run = EvaluationRunner([target], [task], max_concurrency=2).run(
    name="Python API example"
)
print(run.to_json())
write_html_report(run, "report.html")

Provider construction does not perform network I/O. Calls occur only through list_models(), generate(), or an evaluation runner.

JavaScript API

import {
  EvaluationRunner,
  ExactMatch,
  ModelTarget,
  PromptTask,
  createProvider,
  writeHTMLReport,
} from "localarena";

const target = new ModelTarget({
  name: "local-model",
  provider: createProvider("ollama"),
  model: "qwen3:0.6b",
  requestOverrides: { maxTokens: 64, temperature: 0 },
});
const task = PromptTask.fromText(
  "capital",
  "Reply with exactly Paris.",
  { evaluator: new ExactMatch("Paris") },
);

const run = await new EvaluationRunner([target], [task], {
  maxConcurrency: 2,
}).run({ name: "JavaScript API example" });

console.log(run.toJSONString());
await writeHTMLReport(run, "report.html");

AbortSignal is supported for JavaScript provider calls and complete evaluation runs.

Python's arun() stops scheduling new rows when its coroutine is cancelled. A synchronous provider request already running in a worker thread continues only until that provider call completes or reaches its configured timeout.

Terminal and container benchmarks

A prompt-only completion is not an official terminal benchmark run. Tasks that depend on a repository, shell, tools, container image, mutable state, time limits, or a benchmark-specific grader must stay in their authoritative harness. An adapter can import that harness result and reproducibility metadata into the rating layer; LocalArena does not pretend that sending the task prompt to a chat endpoint reproduces the environment.

See provider and task support for the protocol rationale and external-adapter boundary.

Pairwise ranking core

The dependency-free arena remains available for human or external verdicts. Sequential Elo describes a changing history; bradley_terry() fits a static comparison set without depending on replay order:

from localarena import Arena, Result

arena = Arena(["model-a", "model-b"])
arena.record("model-a", "model-b", Result.LEFT)
print(arena.standings())
print(arena.bradley_terry(bootstrap_samples=1000, seed=7))

The JavaScript package exposes the same bradleyTerry(), Arena, Result, roundRobin, and schema-v1 snapshot contract. Read evaluation methodology and ecosystem for the statistical choices, current limitations, integration boundary, and the product moat.

Development and release

python3 -m pip install -e python
python3 -m unittest discover -s python/tests
python3 scripts/check_runtime_parity.py
python3 scripts/check_examples.py

npm ci
npm run check
npm test
npm pack --dry-run

CI covers Python 3.10–3.13 and Node.js 18, 20, 22, and 24. A matching immutable v* tag builds and verifies both distributions, then publishes the exact artifacts to PyPI and npm.

License

Apache-2.0. See LICENSE.