@spendgraph/evals

Score LLM output instead of asserting equality. Deterministic metrics, an optional LLM judge, and the bill for both.

npm install @spendgraph/evals

import { evaluate, formatReport } from "@spendgraph/evals";
import { jsonCorrectness, wordLimit } from "@spendgraph/evals/metrics";

const report = await evaluate(cases, [jsonCorrectness(), wordLimit({ limit: 60 })]);
console.log(formatReport(report));

What you get

• Seventeen metrics, ten of them free and instant — and an eighteenth is one object away.
• A judge you can check before you trust it — cohensKappa against labels a person wrote, because raw agreement flatters a suite that mostly passes, and confusion for which way it errs when it does — two judges can share an agreement rate and a kappa while failing in opposite directions.
• An optional LLM judge, with the bill for it in front of you rather than behind.
• Paired statistics, so you can tell a real improvement from noise on a suite you can afford to run.
• Skipped is not zero. A metric that could not run says so instead of averaging a guess into your score.

Docs

| | | | --- | --- | | Getting started | One number in five minutes | | Datasets | Where cases come from, and splits | | Choosing one | What do you want to check? | | Deterministic | Free, instant, no judge | | You write the criteria | checklist, rubric, geval | | Retrieval | RAG: grounded, and complete | | Writing a metric | defineMetric, the registry, and skipping | | The judge | Wiring a model, cost, trusting it | | Comparing runs | Did the change help? CI gates |

License

Apache-2.0