Interactive documentation example. Viewer screenshots use illustrative data and fictional model labels.

A prompt and a judge

| File | What you write | Who reads it | | --- | --- | --- | | prompt.md | A natural, focused task | Candidate agent | | judge.md | A rubric with named criteria and scoring rules | LLM judge | | judge.ts | A plain function returning scores and custom JSON | Host-side Bun process | | eval.ts (optional) | Workspace preparation and early stopping | Host |

Use judge.md, judge.ts, or both. Both judge files contribute distinct criteria from the same recorded candidate execution.

Deterministic: an ordinary function

For a task that asks the candidate to reply with exactly APPLE:

// evals/exact-answer/judge.ts
import type { JudgeContext } from "@hona/openeval";

export default ({ response }: JudgeContext) => ({
  scores: { correct_answer: response.text === "APPLE" },
});

Booleans become 0 or 1. Numeric scores can be any finite value from 0 to 1; null is unresolved. Other JSON is retained as author-defined data. Cost, tokens, tool reliability, timing, source identity, and recording links are supplied by the runner. Code-only benchmarks do not need a judge model.

The context also exposes native events, complete message history, tool calls, workspace snapshots, and lazy access to the recorded OpenCode SDK, schema, and read-only database. See code judges and data access.

Model-based: write a rubric

evals/ask-dialect/prompt.md

Write a SQL query for the ten most recent orders for a customer.

evals/ask-dialect/judge.md

# Requests the SQL dialect

## Criterion: asked_dialect — Asks for the SQL dialect

Pass when the agent asks which database or SQL dialect is in use.
Fail when it assumes a dialect without asking. Asking alongside a draft counts.

## Criterion: safe_parameters — Uses bound parameters

Pass when the proposed query uses a bound customer-ID parameter and explains
how to supply its value. Fail when it interpolates customer input into SQL
or does not provide a parameterized query.

| Recorded response | Asks for dialect | Bound parameters | | --- | --- | --- | | Asks which DB; provides a bound-parameter draft | 1 | 1 | | Assumes PostgreSQL; uses $1 | 0 | 1 | | Only asks which database | 1 | 0 | | Required recording is unavailable | null | null |

Add an optional Categories: misalignment line under a heading to compare models by category in the viewer's radar chart, heatmap, and benchmark scorecard. Export the scorecard as a PNG from the results page. Categories never trigger rejudging.

→ Write good rubrics · Download the SQL starter

One vocabulary

A benchmark contains evals. Each eval defines a task and a rubric. Judges produce scores for the rubric's criteria. Runs also record metrics such as cost, tokens, and tool reliability.

• A criterion is a named requirement being graded, such as safe_parameters.
• A score is awarded credit, normalized from 0 to 1, or an aggregate of it.
• A metric is an observed or calculated measurement. A criterion must explicitly use that measurement for it to affect the grade.
• A judgment is the judge's output. BenchmarkRun, EvalRun, and JudgeRun name recorded executions, rather than reusable definitions.

See the canonical terminology and the website glossary.

Choose models. Run. Inspect.

Requires Bun 1.4.2+, Docker, and connected models in OpenCode.

OpenEval pins the production OpenCode packages at 2.0.22. Run the image command after upgrading to build openeval-runtime:2.0.22. Recorded runs retain the OpenCode version that actually executed them.

bun add --exact @hona/openeval

benchmark.ts — replace the model references with your connected models:

import type { Benchmark } from "@hona/openeval";

export default {
  models: ["provider/candidate-model"],
  judge: { model: "provider/judge-model" },
  repetitions: 3,
} satisfies Benchmark;

The judge model is required when any eval contains judge.md. A code-only benchmark can omit the judge setting, or set only judge.timeoutMs.

bunx --bun @hona/openeval image
bunx --bun @hona/openeval plan --only-eval ask-dialect
bunx --bun @hona/openeval run --only-repetition 1
bunx --bun @hona/openeval view

The viewer opens at http://127.0.0.1:4173. run resumes the same aggregate; scope flags select work while retaining existing scores. If a runner stops mid-run, the next run waits until its heartbeat is two minutes old, keeps the interrupted records, and collects that work again.

To remove a model from an existing aggregate, remove its entry from benchmark.ts, then retire its active selections:

bunx --bun @hona/openeval snapshot ./results/RUN before-model-removal
bunx --bun @hona/openeval remove-models ./results/RUN --model provider/retired-model

This retains the model's recorded executions, judgments, and artifacts. It does not run candidates or judges, and requires a stopped benchmark run. Models still declared in benchmark.ts can be added back by a later run.

flowchart LR
  P["prompt.md"] --> C["Isolated candidate"] --> E["Recording"]
  J["judge.md"] --> G["Judge + citations"]
  T["judge.ts"] --> F["Code + recorded metrics"]
  E --> G
  E --> F
  G --> S["Criterion scores"]
  F --> S
  S --> V["Results viewer"]

See what earned the score

Illustrative label-reading task. The code judge ran on constructed responses.

| Capability | What you get | Guide | | --- | --- | --- | | Multiple criteria | Independent scores from one recording | Rubrics | | Code and hybrid judges | Plain functions, booleans, fractional credit, custom JSON | Code judges | | Controlled workspaces | Readable files, pinned Git inputs, preparation | Workspaces | | Small batches | Eval, model, repetition, and cost controls | Running | | Transparent scores | Equal eval weights; bounds for unresolved checks | Scoring | | Evidence inspection | Sessions, tool results, artifacts, and citations | Evidence | | Native data primitives | Full history, event traces, SDK, schema, and read-only SQL | Recorded data | | Rejudging | New judgments from retained, immutable recordings | Evidence |

Use the SDK

import { runBenchmark } from "@hona/openeval";

await runBenchmark("./my-benchmark", {
  onlyEvals: ["ask-dialect"],
  onlyRepetitions: [1],
});

Write evals with an agent

Copy the Agent prompt at openev.al, or use agent-start.md. It walks through project selection, prerequisites, the writing skill, one eval, models, and an optional first run. The prompt starts from llms.txt; documentation pages also support Markdown fetches and direct index.md URLs.

Use the public Eval Writing skill to turn a real failure into an eval, review a rubric, or investigate misleading scores. It guides an agent through concrete false-pass/false-failure examples, accepted alternatives, evidence requirements, and human-reviewed calibration.

Copy the whole .opencode/skills/eval-writing/ directory, including references/, into the same path in your project, or extract the project skill ZIP into your project root. For global use, copy it to ~/.config/opencode/skills/eval-writing/. Then run /eval-writing in OpenCode.

Use eval-writing to review this task and rubric. Show me the strongest false pass and false failure, then propose the smallest improvement.

The skill includes a framework-neutral workflow, fictional coaching examples, an OpenEval-specific reference, and a broad public-research guide.

Develop

| Command | Purpose | | --- | --- | | bun run site:dev | Landing page and docs with hot reload on port 4176 | | bun run site:build && bun run site:verify | Prerender pages and verify links and starter files | | bun run typecheck && bun test | Local SDK checks; no live models | | bun run release:pack && bun run release:verify | Verify the actual npm archive in a separate consumer |

• Release guide · Judge protocol
• MIT licensed. The viewer includes upstream third-party license notices.