@spendgraph/evals

Score LLM output instead of asserting equality. Deterministic metrics, an optional LLM judge, and the bill for both. Zero dependencies.

npm install @spendgraph/evals

import { evaluate, formatReport } from "@spendgraph/evals";
import { jsonCorrectness, wordLimit } from "@spendgraph/evals/metrics";

const report = await evaluate(
  [{ input: "price as JSON", actualOutput: '{"input": 3, "output": 15}' }],
  [jsonCorrectness({ requiredKeys: ["input", "output"] }), wordLimit({ limit: 60 })]
);

console.log(formatReport(report));

A score, not a pass

Every metric returns a number between 0 and 1 and a reason for it. The threshold turns that into a verdict, and it is per metric and overridable per use:

const metric = report.results[0].metrics[0];

metric.name;       // "json_correctness"
metric.score;      // 1
metric.reason;     // "valid JSON with all required keys"
metric.threshold;  // 1
metric.passed;     // true
metric.kind;       // "deterministic" | "judge"

The report totals that across every case, and says what the judging cost:

report.passed;            // cases where every metric cleared its bar
report.failed;
report.meanScore;
report.byMetric;          // mean, passed and failed per metric name
report.skipped;           // metrics that never ran, and why
report.judgeCostMicros;   // what the judge charged, in micro-USD
report.judgeSavedMicros;  // what the response cache avoided

Equality assertions do not survive contact with a model: the same answer, reworded, is still the answer. A score says how far off it was, which is the thing you can set a bar against and watch move between versions.

Two fields exist so a report cannot lie about what happened:

• error is set when the metric threw. A broken metric must never read as a pass.
• skipped is set when it never ran — no judge configured, or the case is missing a field the metric needs.

A case

input and actualOutput are all most metrics read. The rest are there for the metrics that check retrieval or tool use.

{
  input: "Why was I charged twice?",
  actualOutput: "A proration for the mid-cycle upgrade.",
  expectedOutput: "You upgraded mid-cycle.",  // judged for meaning, never string equality
  retrievalContext: ["MSA 2.4 …"],            // for RAG groundedness
  toolsCalled: ["lookup_contract"],
  expectedTools: ["lookup_contract"],
}

The nine

Import a metric from /metrics and it costs nothing to have around. Only geval makes a model call.

| | | | | --- | --- | --- | | jsonCorrectness | output parses, and carries the keys you named | free | | containsAll | every required phrase is present | free | | wordLimit | the answer stayed inside its budget | free | | absence | a phrase that must not appear did not | free | | noLeakedPlaceholders | no {placeholder} survived rendering | free | | toolCorrectness | the tools called are the tools expected | free | | toolOrder | and in the order that made sense | free | | stepValidity | each step in a rollout was a legal move | free | | geval | a model grades it against criteria you write | paid |

kind is the field that answers "did this cost money". The runner uses it to skip judge metrics when no key is configured and to say so in the report, rather than quietly reporting a run where half the metrics never executed.

That split is the point of having both. The free eight run on every commit; the judge runs where a rubric is genuinely the only way to score the thing, and you can see what it charged.

Docs

| | | | --- | --- | | Getting started | Your first eval, and how to read the report | | Metrics | All nine, with what each one is for | | The judge | Wiring a model, sampling, logprobs, caching, cost | | Testing and CI | assertTest, regression gates, free-vs-paid runs | | Datasets and reliability | Repeat runs, pass^k, feedback and held-out splits | | Custom metrics | defineMetric, the registry, evals defined as data | | Assay | The prompt optimizer |

Where this is going: PLAN.md — the five phases from scorer to prompt optimizer, and which schema decisions cannot be backfilled.

Layout

src/
  core/       types, scoring maths, defineMetric, the registry
  judge/      Judge contract, logprob weighting, response cache
  metrics/    one file per metric, grouped by family
  runner/     evaluate, assertTest, reporters

Nothing in core/ or runner/ knows which metrics exist. Adding one touches two files: the metric, and the registration list.

Adding a metric

export const yamlCorrectness = defineMetric<YamlOptions>({
  id: "yaml_correctness",          // registry key — stable, snake_case
  kind: "deterministic",           // or "judge" if it costs money
  defaultThreshold: 1,
  requires: ["actualOutput"],      // fields the case must carry
  description: "Output parses as YAML.",
  build: (opts) => (testCase) => ({ score: 1, reason: "…" }),
});

Then registerMetric(yamlCorrectness) in src/metrics/index.ts and export it. That is the whole change: every metric gets threshold and name overrides for free, appears in listMetrics(), and becomes constructible from config. The full walkthrough is in docs/custom-metrics.md.

requires is what produces a skipped rather than a crash when a case does not carry the field — a tool metric run over a dataset with no tool calls says so instead of scoring zero.

What is exported where

The main entry is twelve names — enough to run an evaluation and read the result. Importing it registers the built-in metrics, which is what lets createMetric({ metric: "word_limit" }) resolve one by name.

import { evaluate, formatReport, createMetric, assay } from "@spendgraph/evals";

The rest is grouped by what it is for:

| | | | --- | --- | | /metrics | the built-in metric functions — wordLimit · geval · jsonCorrectness · … | | /core | scoring primitives, the metric registry, newSpend | | /dataset | loading and splitting cases | | /judge | judge plumbing — caching, the default judge, logprobs | | /report | score matrices and candidate comparison | | /rollout | repeat runs and reliability | | /runner | assertTest, measureOne, mapLimit | | /taxonomy | MAST failure codes |

Importing a metric from /metrics directly does not register it, which is the point: take the scoring function without the registry if that is all you want.

License

MIT