lastlight-evals

Which model should run your agent? Find out — with receipts.

▶ Explore the live scorecard — interactive, with per-instance detail. (Above: code-fix tier across 9 models.)

lastlight-evals takes Last Light's real production workflows — the actual prompts, skills, and agent loop that ship — and runs them end to end against a fully mocked GitHub, for whatever models you throw at it. No toy benchmarks: grading is deterministic by default (did the agent apply the right labels? did the held-out tests turn green?) — with one scoped LLM-judge for the pr-review tier's precision/recall/F-beta (F1 by default) — then ranked side by side on pass/score rate, cost, and latency.

The payoff is one scorecard that tells you, for your workflows, exactly what each model delivers — and what it costs you per run. Swap a model, re-run, see the difference. Drop in your own issues and repos and it evaluates your agent.

🛰️ Part of Last Light — the AI agent that triages, reviews, and fixes your GitHub repos. lastlight.dev · Core repo · Eval repo

It's SWE-bench-compatible, and nothing here touches real GitHub: the agent's github_* tool calls are served by an in-process fake (seeded + recording) and git push goes to a local bare repo. The only deviations from production are the two we can't do unattended — approval gates are disabled and outward side-effects are mocked. Everything else is exactly what ships.

instance (SWE-bench shape)
   │
   ├─ start fake GitHub (seeded with the issue, records every mutation)
   ├─ (code-fix) seed workspace: fixture repo @ base_commit + local bare origin
   ├─ load the REAL workflow YAML (issue-triage / build / …) from lastlight core
   ├─ runWorkflow(sandbox:"none", githubApiBaseUrl→fake, approvalConfig:{})
   └─ grade:
        • execution  — apply held-out tests, run them → FAIL_TO_PASS / PASS_TO_PASS
        • behavioral — recorded GitHub calls vs the instance's expectations

Working on the harness itself? See CLAUDE.md for the seams and invariants (the base-URL mock, static-token mode, the no-clone seeding trick, the asset-bootstrap footgun, the metrics drain).

Get started

Needs Node 24+ and a provider API key.

Easiest: let the Last Light agent skill set up your own workspace

Want to eval your own deployment — your workflows, your agent persona, your config — not just the shipped samples? If you drive Last Light from an agent (e.g. Claude Code), install its skills once and then just ask — no flags to remember:

lastlight skills install            # installs the Last Light agent skills

Then, in a new empty folder, tell your agent (point it at your instance overlay repo):

Let's set up an evals workspace here, using my existing Last Light instance config in cliftonc/lastlight-instance.

The lastlight-evals skill scaffolds the workspace, clones your overlay into instance/, seeds the sample datasets, and wires it all up — under the hood it runs lastlight-evals init . --clone cliftonc/lastlight-instance, after which a bare lastlight-evals run "just works" (it auto-detects ./instance as the overlay and ./evals/datasets). Now you're evaluating your agent against the models you care about. Prefer to drive it by hand? Keep reading.

The skill itself lives in a separate repo — it's bundled in the lastlight plugin (nearform/lastlight, under plugins/lastlight/skills/lastlight-evals/) and tracks this CLI's init / run surface, so the two are kept in sync.

Manual: scaffold with init

The fastest CLI path is init — it scaffolds your own evals workspace (your workflows + your datasets, seeded from the built-in samples) and optionally creates a private GitHub repo for it:

npm install -g lastlight-evals
export OPENAI_API_KEY=...                # or ANTHROPIC_ / FIREWORKS_ / OPENROUTER_

# 1. Scaffold your workspace (offers to `git init` + `gh repo create`).
lastlight-evals init my-evals
cd my-evals

# 2. Run it — drives the real workflows against your datasets, prints a scorecard.
lastlight-evals run --overlay .

That's the loop: edit evals/datasets/ with your own issues/repos (and workflows/ with your own workflows), then re-run. init gives you a self-contained, version-controllable repo that shadows the built-in workflows/skills and datasets by name — see overlays and the configuration docs.

Just kicking the tires? Skip init and run the shipped samples directly:

npm install -g lastlight-evals
lastlight-evals run triage               # or: npx lastlight-evals run triage

Installing pulls in lastlight (and agentic-pi). lastlight-evals is a thin CLI on the lastlight package — it runs core's published workflows/, skills/, and agent-context/, so the evals exercise the exact same assets production does.

Configuration (.env)

The only thing you must provide is a model provider key. Set it in the environment, or drop a .env file in the directory you run from (the runner loads it automatically — KEY=VALUE lines, no quotes needed):

# .env — at least ONE of these. Set keys only for the providers you want to eval.
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
FIREWORKS_API_KEY=fw-...          # GLM / DeepSeek / GPT-OSS (open models)
OPENROUTER_API_KEY=sk-or-...

• The default run uses one model (default in models.json); --compare fans out across the compare set, running only the models whose key is present — so set the keys for the providers you care about and the rest are skipped.
• A compare entry may carry an optional cost (USD per million tokens: input/output/cacheRead/cacheWrite) — a fallback price used only when the run transcript reports $0, e.g. flat-rate/subscription plans (like the kimi.com/code console) whose provider registry has no pay-as-you-go price. Real reported costs always win, so it's safe to leave on. Ships in the overlay's evals/models.json, so a deployment prices its own models without patching pi-ai.
• No GitHub credentials are needed — GitHub is mocked end to end. The harness sets a dummy GITHUB_TOKEN internally; don't put a real one in .env.
• An init-scaffolded repo already gitignores .env, so your keys never get committed.

What a run does

Each eval instance (an issue fixture, optionally with a code fixture + held-out tests) is taken through the real production workflow end to end:

1. An in-process fake GitHub starts, seeded with the issue and recording every mutating call the agent makes.
2. For code-fix, the workspace is seeded with the fixture repo at its base commit plus a local bare origin, so git push works fully offline.
3. The real workflow YAML (issue-triage, build, …) is loaded from lastlight and run with sandbox:"none" by default (in-process), the agent's github_* tools pointed at the fake, and approval gates disabled (so it never pauses). Pass --sandbox gondolin to isolate the agent's bash/file tools in a QEMU micro-VM (see Isolation below).
4. The result is graded deterministically (with one scoped judge for pr-review):
   • behavioral — the recorded GitHub calls (labels, comments, PRs) vs the instance's expect_github / triage_gold.
   • execution (code-fix) — the held-out tests are applied and run; the case is resolved only if every FAIL_TO_PASS passes and every PASS_TO_PASS stays green (SWE-bench's criterion).
   • review (pr-review) — the posted review is matched to a human-verified gold set by an LLM judge → precision / recall / F-beta (F1 by default, Martian's leaderboard metric; EVAL_F_BETA reweights). The one, deliberately-scoped exception; triage/code-fix stay judge-free.
5. Token usage, cost, and latency are collected per run.

Run multiple models and you get a side-by-side scorecard (HTML + JSON) ranking them on pass rate, cost, and latency.

Isolation (--sandbox)

By default the agent runs in-process (sandbox:"none") — fast and CI-friendly, but with no filesystem restriction: the agent process can read any absolute path, including this repo's held-out gold data (datasets/<tier>/tests/, instances.json, .eval-cache/). A capable model that explores the disk could find and spoil the answer key.

--sandbox gondolin (or EVAL_SANDBOX=gondolin) closes that gap: the agent's bash/file tools execute inside a QEMU micro-VM that only sees its own workspace, so host gold paths are invisible. The agent runtime and github_* tools stay in-process, so the fake-GitHub mock still works unchanged — this is why gondolin, and not docker, is the supported isolation backend (docker/smol run the whole agent in the container/VM, where the in-process fake GitHub isn't reachable).

Gondolin needs QEMU with hardware acceleration and runs natively (macOS via Apple's Hypervisor.framework, Linux via KVM) — install it with brew install qemu (macOS) or your distro's qemu-system package. It does not work inside a container on macOS (no /dev/kvm, and the failure is a silent hang), so the harness runs a fail-fast preflight and aborts with guidance rather than wedging. Expect a one-time ~13s VM cold start plus per-tool-call overhead, so keep --sandbox gondolin for trustworthy/anti-spoil runs and leave the default none for quick iteration.

Run it

# no tier args → interactively pick which tiers to run (one or all).
# Non-interactive (CI / piped) falls back to the cheapest default.
lastlight-evals run

# name tiers explicitly to skip the prompt
lastlight-evals run triage
lastlight-evals run code-fix            # the full build cycle (heavy)
lastlight-evals run triage code-fix     # both → combined tabbed report

# cross-vendor comparison (OpenAI + Anthropic + open source) — see models.json.
# Families run in PARALLEL; serial within a family. Force serial with --serial.
lastlight-evals run --compare

# pick ONE model (fuzzy-matched against models.json id/label)
lastlight-evals run triage --model haiku
lastlight-evals run triage --model glm,deepseek   # a comma-list also works

# repeat each case N times; verdicts WORST-case, cost/tokens/latency MEAN
lastlight-evals run triage --runs 3

# isolate the agent's tools in a QEMU micro-VM so it can't read host gold data
# (anti-spoil; needs QEMU natively — see "Isolation" above). Default is none.
lastlight-evals run pr-review --sandbox gondolin

# run against an overlay repo's OWN workflows + datasets (see below)
lastlight-evals run --overlay ~/work/lastlight-instance

# add your own datasets dir without an overlay
lastlight-evals run --datasets ~/my-evals/datasets

# CONFIG run type — eval a deployment's REAL per-step model config (different
# models per workflow phase, from the overlay's config.yaml) instead of forcing
# one model. This is the setup you actually ship. Try the bundled sample overlay:
lastlight-evals run code-fix --mode config --overlay examples/overlay
lastlight-evals run code-fix --mode config --overlay A --overlay B   # 2 configs side-by-side

# ad-hoc model set / focus one instance / no browser
EVAL_MODELS="openai/gpt-5.5,anthropic/claude-sonnet-4-6" lastlight-evals run
EVAL_INSTANCE=off-by-one lastlight-evals run code-fix
lastlight-evals run triage --no-open

The report is a JSON-driven dashboard, not generated HTML — the harness only ever writes scorecard.json, updating it (atomically) as the run proceeds. The runner starts a tiny local server and opens http://localhost:PORT deep-linked at the run, so you watch the scorecard fill in live (the SPA polls the JSON). When the run finishes the server stays up so the dashboard keeps working — press Ctrl-C to stop it. Each run lands in its own timestamped folder, so runs accumulate instead of overwriting — ./eval-results/<tiers>/<runId>/ (override the root with LASTLIGHT_EVALS_OUT), where runId is <timestamp>-<git-sha>:

• scorecard.json — structured roll-up per model + per-instance results, carrying run meta.
• predictions.jsonl — SWE-bench predictions shape.

The dashboard's overview lists every run newest-first with a per-model trend sparkline and links into each run's full scorecard; the run view is the model-comparison table plus per-instance rows. To browse past runs anytime without running models, start the server on its own:

lastlight-evals serve            # opens the dashboard over ./eval-results
lastlight-evals serve --port 4319

Needs a provider key (OPENAI_API_KEY / ANTHROPIC_API_KEY / FIREWORKS_API_KEY / OPENROUTER_API_KEY) in the environment or a cwd .env. The runner exits non-zero only if the harness itself errors — a weak model scoring poorly is the measurement, not a build failure.

Two run types

A run compares N arms along one of two axes — pick with --mode (or, in a TTY with no model flags, you're asked):

• models (default) — compare models, each forced across every workflow step. --model/--compare select the set. Lands in eval-results/<tier>/ (or <tier>-compare/).
• config (--mode config) — run a deployment's real per-step model config: the models/variants maps from an overlay's config.yaml, merged over core's config/default.yaml exactly as production does, so each phase can run on a different model. The arm is the config/overlay; pass --overlay more than once to compare configs side-by-side, or re-run over time to compare as you tweak prompts/skills/workflow/model-config. --model overrides a config's default for quick what-ifs. Lands in eval-results/<tier>-config/, on its own trend line. The run view shows a Per-step models panel with each phase's resolved model. See examples/overlay for a ready-to-run sample.

Your own workflows + datasets (overlays)

An overlay is a directory (often its own repo, like lastlight-instance) that carries its own workflows/ / skills/ / agent-context/ (which shadow the core built-ins by name) and its own evals/datasets/. It's the same deployment-overlay mechanism the production harness uses — see the Last Light configuration docs for the full story. One flag wires both:

lastlight-evals run --overlay ~/work/lastlight-instance     # or LASTLIGHT_OVERLAY_DIR

• Overlay workflows/skills are layered over core via core's asset overlay (same mechanism the production harness uses).
• Overlay datasets are discovered at <overlay>/evals/datasets/<tier>/, and shadow built-in tiers of the same name.
• An overlay evals/models.json is picked up automatically (or pass --models-file).

lastlight-evals init [dir] — scaffold an evals workspace

Two shapes, depending on whether you already have a deployment overlay repo:

Plain — a self-contained overlay+evals repo (its own workflows/ skills/ agent-context/ + evals/):

lastlight-evals init my-evals
cd my-evals && lastlight-evals run --overlay .

Scaffolds workflows/ skills/ agent-context/ (empty, to fill in), evals/datasets/ + evals/models.json (seeded from the shipped samples), config.yaml, and a .gitignore/README, then offers to git init + create a private GitHub repo via gh (reusing core's lastlight server setup flow).

Separate (--clone) — the recommended shape when you already have a deployment overlay (e.g. lastlight-instance) and want to eval its config. The overlay is cloned into <dir>/instance/ (its own git checkout, git-ignored) with the evals at the workspace root; a bare run auto-detects both, no flags:

lastlight-evals init my-evals --clone cliftonc/lastlight-instance
cd my-evals && lastlight-evals run        # auto: overlay ./instance + ./evals/datasets

This is exactly what the lastlight-evals agent skill does for you. Update the overlay later with cd instance && git pull; your evals stay out of the deployment repo. Run lastlight-evals init --help for all flags (--yes, --no-git, …).

Datasets & tiers

A tier is a directory containing instances.json (+ an optional tier.json declaring its defaultWorkflow). Tiers are discovered from three roots, merged by name with overlay > user (--datasets) > built-in precedence:

• built-in (shipped here): triage → issue-triage, code-fix → build, pr-review → pr-review (ships empty — populate with scripts/import-martian.ts; see PR-review tier below and datasets/pr-review/README.md), repo-config → answer (see Per-repo config layer below).
• user: --datasets <dir> / LASTLIGHT_EVALS_DATASETS.
• overlay: <overlay>/evals/datasets/*.

Add a case

Triage — append to a tier's instances.json:

{
  "instance_id": "triage__my-case",
  "repo": "lastlight-evals/widget",
  "workflow": "issue-triage",
  "problem_statement": "short title",
  "issue": { "number": 110, "title": "…", "body": "…", "labels": [] },
  "triage_gold": { "category": "bug", "state": "ready-for-agent" },
  "expect_github": { "labels_added": ["bug"] }
}

Or scaffold one from a real, resolved issue — its content, the labels that were applied (with who applied them), and reviewer comments become the gold case:

lastlight-evals add-case --issue https://github.com/owner/repo/issues/42 --dry-run

It seeds the issue without its triage labels (so the agent triages fresh), sets expect_github.labels_added to the applied labels (+ issue_closed if it was closed), and prints the labels/comments as evidence; you then assign triage_gold (category/state) per your deployment's taxonomy.

Code-fix (vendored fixture) — three things keyed by instance_id, all under the tier dir:

<tier>/instances.json     # the SweBenchInstance (FAIL_TO_PASS / PASS_TO_PASS)
<tier>/repos/<id>/        # fixture repo at base_commit (NO held-out tests)
<tier>/tests/<id>/        # held-out test files, copied in at grade time

Code-fix from a real PR (git-source) — point the CLI at a merged PR instead of hand-building a fixture:

lastlight-evals add-case --pr https://github.com/owner/repo/pull/123 --dry-run

It reads the PR with gh, computes base_commit (the merge-base of the base branch and the PR head) + head_commit, captures the PR's test diff as the held-out test_patch, and — unless --no-validate — runs the tests at base (red) vs head (green) to fill FAIL_TO_PASS / PASS_TO_PASS. Drop --dry-run to write it (to --datasets <dir> / --overlay <dir>, else ./datasets). No repos/<id>/ is vendored: at run time the harness clones the repo into the gitignored ./.eval-cache/ and checks out base_commit. Non-node --test runners work via --test-cmd "<cmd>" (+ --setup-cmd "<cmd>"), graded on the test command's exit code (suite mode) when it emits no TAP names. The repo's tests run real code — only use trusted repos.

A new tier just needs a directory with an instances.json and a tier.json ({ "name", "defaultWorkflow", "description" }); per-instance workflow wins when present.

Per-repo config layer (.lastlight/)

A managed repo can commit a .lastlight/ directory that overrides a bounded subset of config for runs against that repo — models, variants, cron participation, disabled.workflows, approval gates, plus prompt / skill / agent-context overrides. A case declares one by dropping the tree at <tier>/lastlight/<instance_id>/, laid out exactly as the repo commits it:

<tier>/lastlight/<id>/lastlight.yml               # the config override
<tier>/lastlight/<id>/workflows/prompts/*.md      # prompt overrides
<tier>/lastlight/<id>/skills/<name>/SKILL.md      # skill overrides
<tier>/lastlight/<id>/agent-context/*.md          # persona/rules additions (add-only)

No instance field and no flag: the directory's presence is the declaration, and a case without one runs on the plain operator config (which is every other tier). The mock serves the tree through the same fetchRepoConfigTree seam the real GitHub client exposes, and the harness then runs the production resolver over it unmodified — so a run measures the real feature, not a re-implementation. What the layer contributed lands on the result as repoLayer (applied, assets, warnings, and refused when the repo opted the workflow out).

The shipped repo-config tier is the end-to-end proof: two cases, same question, on the answer workflow — one repo whose .lastlight/ prompt override tells the agent to add a marker label, one with no .lastlight/ at all. It's cheap (one agent call each, no clone, no judge) and opt-in:

lastlight-evals run repo-config --model haiku

A fixture that sets models: overrides the run's arm for that case. That's faithful to production and occasionally the point, but it makes the case's model column stop describing the arm — so only do it deliberately.

PR-review tier (Code Review Bench)

The pr-review tier measures review quality against Martian's Code Review Bench: the review the real pr-review workflow posts is scored against a human-verified gold set of the issues a reviewer should have caught. It's the one tier graded by an LLM judge — matching free-text findings to semantic gold comments can't be done deterministically — so triage and code-fix stay judge-free.

Cases come from Martian's offline set — 50 real merged PRs across Sentry, Grafana, Cal.com, Discourse, and Keycloak, each carrying inlined golden_comments. They ship empty (datasets/pr-review/instances.json is []) because they're large real-repo PRs — generated, not vendored:

npx tsx scripts/import-martian.ts            # resolve all 50 via gh (pins base/head SHAs)
npx tsx scripts/import-martian.ts --limit 3  # a quick subset first

Seeding clones the real repo into the gitignored ./.eval-cache/ and checks out the PR head (mirroring production's pre-clone contract), so the skill's git diff origin/<base>...HEAD works fully offline — no fixture is vendored.

Grading (gradeReview, src/grade.ts) is a two-step LLM judge:

1. Extract the review's distinct, concrete findings (drop praise/summaries).
2. Match each finding to a gold comment ("same underlying issue?").

From the matches: precision = matched ÷ posted, recall = matched ÷ gold, combined as F-beta. The headline is F1 (β=1 — precision and recall weighted equally, Martian's leaderboard metric). Pass --f-beta 0.5 (or EVAL_F_BETA=0.5) to weight precision 2× (F0.5), mirroring Martian's adjustable F-beta; the dashboard relabels itself F{β} to match.

Gold-set caveat. Martian's own methodology documents the gold set as incomplete — it caps at human performance, so a real issue the annotators missed is scored as a false positive. That understates precision, which is why the default is F1, not the precision-weighted F0.5. Treat the score as a relative signal and inspect each match with the dashboard's judge button.

The judge model is independent of the models under test — a strong default per your provider key (EVAL_JUDGE_MODEL overrides). A judge failure marks the case errored (ungraded), never a silent zero. Alongside the judge score, a cheap deterministic review_submitted proxy checks a review was actually posted.

Diff-blind by default. The judge sees only the posted review (body + inline comments) matched against the gold set — not the PR diff — mirroring Martian's offline judge. This can penalize terse, location-anchored comments (off-by-one here on a line the judge can't see). Pass --judge-with-diff to feed the PR diff into the judge for higher-fidelity matching (the judge is instructed never to invent findings from the diff); this trades away leaderboard parity, and the dashboard marks such grades diff-aware.

Run it (heavy — clones real repos + calls the judge):

lastlight-evals run pr-review --model <model>            # full tier
lastlight-evals run pr-review --model <model> --limit 3  # first 3 cases (controlled)
lastlight-evals run pr-review --model <model> --f-beta 0.5        # weight precision 2×
lastlight-evals run pr-review --model <model> --judge-with-diff   # give the judge the diff

In the dashboard, each row's judge button opens the judge's working — the findings it extracted, the gold set, the finding↔gold pairing (matched / false positive / missed), and its raw replies — so the F1 score is inspectable, not a black box.

Improving an eval — the loop (lastlight-evals-loop)

Running an eval gives you a score; the improvement loop raises it without gaming it. It's driven by the sibling lastlight-evals-loop skill (say "raise the pr-review F1") and two read-only helpers in scripts/. The method — mine the failures → propose a few minimal candidate fixes → keep the one that survives a blind held-out gate — follows Self-Harness: Harnesses That Improve Themselves, adapted to keep the anti-gaming discipline below.

One round:

1. Diagnose (mine). scripts/mine-failures.ts reads the TRAIN split of a scorecard and clusters the judge's falseNegatives (recall loss, weighted by severity) and falsePositives (precision loss) into a ranked signature bundle — the systematic patterns, ordered by F1 headroom, instead of reading traces by hand.

   npx tsx scripts/mine-failures.ts <train-scorecard>.json --train <train-ids> --keywords

2. Propose. Draft a few (K=2–4) minimal, diverse candidate edits for the top pattern — lowest lever first: a generic overlay prompt/skill/persona edit, or a synthetic AGENTS.md injected into the checkout. Never a core change.

3. Select on TRAIN, confirm on HELD-OUT once. Rank candidates on the train split, then give the single winner one blind held-out confirmation (gating every candidate on held-out would inflate it). scripts/diff-runs.ts computes the keep/revert verdict:

   npx tsx scripts/diff-runs.ts <baseline>.json <winner>.json \
       --train <train-ids> --heldout <heldout-ids>
   # VERDICT: KEEP — train ↑ and held-out held  (or REVERT — OVERFIT: held-out regressed)
   # --symmetric swaps in the paper's non-regressive gate (neither split may regress).

4. Keep one, journal, repeat until a target F1 or a plateau.

What keeps it honest: a fixed train / blind held-out split (the empirical gate), one change kept per round (attribution), an adversarial generality + leak auditor that rejects any edit naming a specific repo/file or encoding the gold answer, and generic-first levers — core is never touched. The loop produces two durable outputs: workflow improvements (better prompts/skills for every repo) and per-repo recommendations (context a maintainer can commit), each backed by a measured held-out lift.

Models (models.json)

• default — the single model run uses.
• compare — the cross-vendor set --compare fans out over. Each entry has an id (the agentic-pi/pi-ai provider/model spec), a label, and an envKey. An entry only runs if its envKey is present, so the compare set auto-trims to whatever keys you have.

Roadmap

• lastlight-evals extract <owner>/<repo>#<n> — generate eval cases from GitHub historical issues/PRs (issue → fixture, merged PR → held-out tests).
• Docker-backed sandboxed runs (needs the fake GitHub reachable from inside the container — --sandbox gondolin already gives native isolation today); real SWE-bench Lite ingestion; per-fixture test runners.
• LLM-as-judge stays out by design — grading is deterministic.