eve-eval-trend-reporter

An eve eval reporter. It keeps every run and renders the trend as one standalone HTML page.

Braintrust and JUnit send one run to a destination. This one adds runs up instead. Each run appends a line to a committed evals/history.jsonl, and the page it writes shows what changed across all of them. A single run tells you whether the evals passed today, which is worth knowing but not much on its own. The things I want to catch only show up over a series: a judge score that has been sliding for a week, or an eval that flips its verdict on the same commit.

Look at the page before installing anything. Those are real renders of made-up histories, one per shape a project can be in, light and dark.

Install

npm install --save-dev eve-eval-trend-reporter

Needs Node 24 or newer. The package is written in TypeScript and ships the JavaScript tsc emits, with the declarations beside it.

Use

Name it in evals.config.ts and eve eval runs it:

import { defineEvalConfig } from "eve/evals";
import { EveEvalTrend } from "eve-eval-trend-reporter";

export default defineEvalConfig({
  reporters: [EveEvalTrend()],
});

Every run now writes .eve/eve-eval-trend.html and appends one line to evals/history.jsonl, pass or fail. Keep the failures. Those are the runs you come back to.

Commit evals/history.jsonl. .eve/ is gitignored and gets cleaned, so without the archive the trend reaches back only as far as your last clone.

Options

EveEvalTrend({ archive: "reports/history.jsonl", open: true });

| Option | Default | Does | | --------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | | archive | evals/history.jsonl, or none where there is no evals/ directory | Where to keep the history. false reports on .eve/evals alone and writes nothing back. | | open | false | Hand the finished page to a browser. | | root | the working directory eve eval ran in | The project to report on. |

The archive directory is never created for you. A project without an evals/ directory is not using this convention, and I did not want a reporter making directories in your repo to hold a file you never asked for. Name a path and you get one.

The command

The same report, without running any evals:

npx eve-eval-trend              # rebuild from what is already on disk
npx eve-eval-trend --open       # and open it
npx eve-eval-trend --archive reports/history.jsonl
npx eve-eval-trend --no-archive

Use it after eve eval --skip-report, which turns off config reporters and so turns off this one. Also on a fresh clone where the archive is all that is left, or while working on the report itself.

It reads EVAL_COMMIT and EVAL_BRANCH for the commit under test, and stamps only the runs that started after EVAL_STARTED_AT. Without that timestamp it stamps nothing. The command can run long after the run it is reading, by which time HEAD has moved, and stamping from HEAD would relabel every earlier record with today's commit.

The reporter needs none of that. It reads git itself at onRunStart, before a judge-heavy run has had time to move the commit under it, and limits the stamp to the run's own start time. Set EVAL_COMMIT or EVAL_BRANCH to override it in CI, where a detached checkout reports HEAD as its branch.

What the page shows

Shorter to look at than to read: the rendered previews, seven histories from a suite that has never gone wrong to one holding a single run.

First, whether the suite is getting better or worse, in those words. The evidence beside it is the mean judge score across every run, with the runs where the model or the judge changed marked on the line. A score drop next to a model swap is the likeliest false alarm this page can raise. The line covers all of history. Every claim made about it stops at the last 20 runs.

Then what moved over those 20 runs, one line per eval, worst first:

• a verdict that differs at the two ends of the window
• a judge score that moved by 0.10, or crossed the eval's own minimum
• an eval that changed its mind twice or more inside the window

Improvements get a line too, ranked below the declines. The list stops at eight rows, so a suite that moved all at once cannot push everything else off the screen. Anything without a row is counted rather than listed. The exception is an eval that has been failing the whole time, which is named: it has not moved, but it is still wrong.

Below that, one card per eval covering the last 20 runs, with its verdict per run and its judge trend on a shared axis. Then every run in the history, collapsed, and expandable down to each assertion, the reply that produced it, and any images the run generated.

There is no latest-run panel. The newest run is the last point on the line and the top row of the Runs list, and its conditions are inside that row.

The page is one self-contained file with no external requests, so it survives being moved or sent to someone.

What it reads

The artifacts eve eval already writes under .eve/evals/<run>/, rather than the results the reporter gets handed. Eve has written them by the time the reporter runs, and the page covers every run the project has ever done, only one of which is in that call. Reading from disk is also how the runs you made before installing this show up at all.

Two things come out of the captured event streams rather than the run summary. The model id and eve version are read from session.started. Images are picked up by an image/* media type on a tool result, not by the name of the tool that returned it.

Judge conditions are scraped from .closedQA("…") in your eval sources, because the runner records the score and not the condition it graded against. The source file doubles as the liveness check. An eval whose file is gone keeps its history under Retired, where nothing reports on it: it cannot be fixed, so it is not news.

Images are never archived. One is a few hundred KB of base64 that no trend is computed from, so they are read back from the run directories for as long as those survive, newest first, up to about 6 MB of base64 per page. Past that, older runs render without their pictures and the console summary says how many went without. The logs written with t.log are archived, because they are the only account of why a score landed where it did.

Ingest is keyed on the run directory name, so running it twice changes nothing.

Development

See DEVELOPMENT.md for the checks, the preview and how a release goes out. Issues and pull requests are welcome.

MIT licensed.