@demystify/evals

Pick what works, and notice when it stops working.

Deterministic, offline eval harness for LLM-shaped work. It answers exactly three questions:

1. Selection — is candidate A better than candidate B for this use case?
2. Protection — did quality regress against a recorded baseline?
3. Cost — what did that quality cost?

It reports all three together, and it refuses to collapse them into one number.

Zero dependencies. ESM. Node >= 22. No keys, no network, no clock, no database.

It never calls a model

This is the architectural rule the whole package is built around, and it is enforced by a test, not a promise (test/no-transport.test.ts).

A candidate is a plain async function the host supplies:

type Candidate<I, O> = (input: I) => O | Promise<O>;

There is no provider name, no base URL, no API key and no HTTP client anywhere in this source tree. The suite calls what you give it.

That is not purity for its own sake. It is what makes the numbers comparable: the same eval code runs in CI against a deterministic stub and in staging against a live provider. If the harness owned the transport, the CI number and the live number would come from two different programs, and comparing them would be a guess.

// CI: deterministic, free, offline.
const stub = (doc: string) => FIXTURES[doc];

// Staging: your client, your keys, your retries.
const live = async (doc: string) => {
  const { text, usage } = await myOcrClient.read(doc);
  return measured(text, { costMicroUsd: usage.microUsd });
};

// Identical from here down.
const report = await runSuite(cases, live, { scorer: fieldF1() });

Quickstart

npm install @demystify/evals

import { runSuite, fieldF1, compareCandidates, measured } from "@demystify/evals";

const cases = [
  {
    id: "inv-001",
    input: "invoice-001.png",
    expected: { invoice_number: "INV-2024-0158", total_amount: 125000 },
    tags: ["invoice", "printed", "english"],
  },
  {
    id: "inv-014",
    input: "invoice-014.png",
    expected: { invoice_number: "INV-2024-0171", total_amount: 98000 },
    tags: ["invoice", "handwritten", "hindi"],
  },
];

const sarvam = await runSuite(cases, readWithSarvam, {
  scorer: fieldF1(),
  name: "sarvam-ocr",
  suite: "invoice-extraction",
});
const docling = await runSuite(cases, readWithDocling, {
  scorer: fieldF1(),
  name: "docling",
  suite: "invoice-extraction",
});

const comparison = compareCandidates([
  { name: "sarvam-ocr", report: sarvam },
  { name: "docling", report: docling },
]);

console.log(comparison.tradeoff);
// [ { name: "docling",    n: 2, meanScore: 0.83, microF1: 0.83,
//     meanCostMicroUsd: 400, costStatus: "known", meanLatencyMs: 210 },
//   { name: "sarvam-ocr", n: 2, meanScore: 0.91, microF1: 0.91,
//     meanCostMicroUsd: 2100, costStatus: "known", meanLatencyMs: 640 } ]

console.log(comparison.byTag.find((t) => t.tag === "hindi")?.leaders);
// [ "sarvam-ocr" ]   <- the answer to "which OCR", per document type

Why there is no single overall winner

compareCandidates returns a trade-off table and one ranking per axis. It does not return a winner, and it will not.

Collapsing quality, cost and latency into one score requires exchange rates — how many µUSD is a point of F1 worth, how many milliseconds — and those rates belong to the business, not to a library. A model 2% better at 5x the cost usually loses. Sometimes it is the only one that clears a compliance bar and it wins at any price. Both are correct, and no weighting shipped in this package would be right for both.

So you get the numbers, side by side, with n on every one of them, and you make the call. Per-tag quality leaders are reported — "best on Hindi invoices" is a single-axis fact, not a verdict.

Determinism is the product

An eval that answers differently on the same input cannot detect a regression: every diff is ambiguous between "the model changed" and "the harness wobbled", and a gate nobody trusts is a gate nobody enforces.

So the core has:

• no clock, no Math.random, no process.env, no ambient I/O;
• a fixed, explicit ordering: cases run sequentially in the order given, results come back in that order, and everything aggregated (tags, fields) is sorted by name so a hand-reordered case list cannot change the report;
• stableStringify for byte-stable JSON (sorted keys).

Two runs of the same suite produce a byte-identical report. That is asserted in test/determinism.test.ts, over a seeded corpus.

Latency is the exception, and it is quarantined. It varies with the machine and the network, so:

• it is excluded from the determinism guarantee — use withoutLatency(report) when byte-comparing timed runs;
• it never flips a regression verdict unless you explicitly opt in with compareToBaseline(current, baseline, { includeLatency: true, latencyToleranceMs: 250 }), and that opt-in requires the tolerance. A gate that fires when CI is busy is a gate that gets switched off.

Latency is only measured if something measures it: inject a clock, or have your candidate report it through measured(output, { latencyMs }).

Cases and tags

interface Case<I, E> {
  id: string;
  input: I;
  expected: E;
  tags?: readonly string[];
}

tags is how a business slices the answer. "Which OCR is best?" almost always resolves to "depends on the document type" — so every aggregate carries a per-tag breakdown, and every tag slice carries its own n.

Every mean states its n

A 0.9 over 3 cases and a 0.9 over 300 are not the same claim. There is no shape in this package that can carry a mean without its count: aggregate.n, byTag[].n, cost.n and cost.knownCount, latency.n, fieldF1.n, and support on every per-field row.

compareToBaseline also reports comparable: false when the two runs scored a different number of cases, because the per-case findings still hold but the means do not compare.

Scorers

Pure (actual, expected) => { score, reason }. Every scorer returns a reason — a 0.6 with no explanation is not actionable.

| Scorer | For | Score | |---|---|---| | exact | labels, ids, enums | 1 or 0 | | numericTolerance({ toleranceMinor }) | money | 1 or 0 | | fieldF1({ kinds, envelopeKinds, matchers }) | extracted records | micro-F1 | | containsAll({ caseSensitive }) | required phrases/disclosures | graded | | jsonSchemaValid(schema) | structural conformance | 1 or 0 |

numericTolerance — money in integer minor units

Money is integer minor units with the currency stated: paise for INR, cents for USD, the same rule @demystify/indic-numerals parses to and @demystify/agent-kernel records spend in.

A tolerance expressed in rupees is how you accept a wrong ledger: 0.01 looks like "a paisa" and is in fact a float. So toleranceMinor is an integer count of minor units, and it defaults to 0 — "close enough" on money has to be asked for out loud.

numericTolerance({ toleranceMinor: 100 }); // INR: within one rupee

A float in the expected value throws (your suite is broken). A float from the candidate scores 0 with a reason (the model is broken) — that distinction is the whole point of an eval.

fieldF1 — ported, not reinvented

Precision/recall/F1 over extracted key-value fields, ported from modules/dmstfy-extract/evals/run_evals.py, which has been gating extraction releases in production. The counting rule is the part worth reading twice:

match           -> tp
present, wrong  -> fp AND fn      <-- both
absent          -> fn

A wrong value counts as both a spurious extraction and a missed one. This is not the textbook definition, and it is deliberate: a garbled GSTIN is not a half-answer, it is a fabricated one plus a missing one. A consequence worth knowing: when nothing is missing, fp == fn, so precision, recall and F1 are all equal. That is the rule working.

Micro-F1 is not the mean of per-case F1s. Counts are pooled across cases first, so a 40-field document weighs more than a 2-field one. The two numbers differ by several points on any real corpus; aggregate.fieldF1.microF1 is the pooled one, and n sits beside it.

Fields the candidate invented that gold does not mention are not counted — same as the Python. This metric measures whether the fields you asked for came back right, not whether the model added extras.

jsonSchemaValid — small, and honest about it

Implements type, properties, required, items, enum, additionalProperties. Any other keyword throws at construction. Silently ignoring pattern or $ref would report a green that means nothing; if you need full JSON Schema, use ajv in your own code and wrap it as a scorer.

Baselines and the CI gate

import { runSuite, fieldF1, compareToBaseline } from "@demystify/evals";
import { readBaselineIfExists, writeBaseline } from "@demystify/evals/node";

const report = await runSuite(cases, candidate, { scorer: fieldF1() });
const baseline = readBaselineIfExists("evals/baseline.json");

if (baseline === null) {
  writeBaseline("evals/baseline.json", report, { meta: { commit: process.env.SHA! } });
  process.exit(0);
}

const verdict = compareToBaseline(report, baseline.report);
console.log(verdict.reasons.join("\n"));
if (verdict.verdict === "REGRESSED") process.exit(1);

compareToBaseline returns improved | unchanged | REGRESSED and names which cases, tags and fields moved and by how much. A verdict with no names is an alert people learn to ignore.

It reports REGRESSED when any of these drops by more than the tolerance:

• the overall mean score,
• any per-case score,
• any per-tag mean,
• any per-field F1, or the pooled micro-F1,

or when a case present in the baseline is missing from this run.

The tolerance

DEFAULT_REGRESSION_TOLERANCE = 0.01. It is exported and documented, never buried in a comparison. It is inherited from run_evals.py's F1_REGRESSION_TOLERANCE, where it has been blocking releases in production.

Pass your own if your suite is small: over 20 cases a single case is worth 0.05, so 0.01 will flag noise you cannot act on.

Cost is reported in the comparison and never flips the verdict. A candidate that got 3% cheaper and 1% worse is a decision, not a build failure.

Cost is integer µUSD, and unknown is not zero

1 USD = 1,000,000 µUSD, matching what gateway-moat meters. Always an integer; means are divided half-up in exact integer arithmetic, never through a float. (Book money is minor units and is a separate thing — see numericTolerance.)

A candidate reports cost by wrapping its output:

return measured(text, { costMicroUsd: usage.microUsd });

If it reports nothing, cost is unknown — null totals, status: "unknown", and unranked in compareCandidates. It never becomes 0, because a free-looking candidate wins every trade-off table it appears in. CostSummary.status is known / partial / unknown, with knownCount beside the total so a partial mean cannot be read as a full one.

A candidate that throws also leaves cost null: a crashed call may well have been billed.

Failure is data

A candidate that throws is recorded, not propagated: that case scores 0 with ok: false and the thrown message as its reason, and the suite keeps going. "Crashes on 3% of inputs" is a finding, and an eval that aborts on the first crash cannot report it. aggregate.errors counts them.

A scorer that throws is propagated — that is a bug in your suite, and a report built on a broken metric is worse than no report.

The Node subpath

The core is pure. @demystify/evals/node is the only thing that touches a disk:

import { readBaseline, readBaselineIfExists, writeBaseline } from "@demystify/evals/node";

Baselines are written with sorted keys and a trailing newline, so re-recording an unchanged baseline produces a zero-line diff — a file that churns every run teaches reviewers to skim past baseline diffs, which is exactly when a real drop slips through. No timestamp is written; pass a commit sha through meta if you want provenance. Parsing and validation live in the pure core (parseBaseline), so JSON from an artifact store gets the same checks as JSON from a file.

What this package does NOT do

• It does not call models. No providers, no retries, no rate limiting, no batching. That is the host's client.
• It does not pick a winner. See above.
• It does not do LLM-as-judge. Every scorer here is deterministic. A judge is a model call, which belongs on the other side of the Candidate boundary — and a non-deterministic scorer would destroy the regression guarantee.
• It does not ship a fuzzy name matcher. The Python matches names with difflib.SequenceMatcher at >= 0.9, whose ratio (including its autojunk heuristic) is a Python-stdlib implementation detail. A hand-rolled approximation would agree on the easy pairs and disagree on the ones that matter. Supply your own: fieldF1({ kinds: { vendor_name: "name" }, matchers: { name: (p, g) => sim(p, g) >= 0.9 } }).
• It does not implement full JSON Schema. It refuses schemas it cannot fully check rather than half-checking them.
• It does not do statistical significance. It reports n honestly and leaves the inference to you.
• It does not sample, shuffle or split. Train/test splits are the caller's — flywheel_eval.py does its own, deliberately, and so should you.
• It does not run cases concurrently. Sequential and boring, so a report is reproducible.

Known limits

• Rounding is to 4 decimals, half-up. Python's round() is half-even, so the two can differ only on an exact tie at the 5th decimal — impossible from a ratio of small integers, but stated rather than hidden.
• flatten treats inputs as JSON-shaped. Date, Map, Set and class instances are not special-cased.
• Per-case scores in a report follow input order; everything aggregated is sorted. If you rely on report ordering, rely on that.

Relationship to dmstfy-extract

This package generalises modules/dmstfy-extract/evals/ — its F1 definition, per-field breakdown, baseline file and 0.01 regression tolerance — into something a stranger's codebase can adopt. What stayed behind is domain policy: the OCR confusion error model, the train/test split, the extraction pipeline's {raw, value} envelope (available here, but off by default via envelopeKinds), and the degraded-mode fault injection. Mechanisms travel; policy does not.

Testing

pnpm install
pnpm test          # 236 tests
pnpm run lint
pnpm run build

Tier 1 only: pure, offline, no infrastructure. The worked example in test/field-f1.test.ts is checked against numbers produced by running the actual Python functions from run_evals.py, not against what this implementation happened to output.

Licence

MIT