ragbench

Score a RAG system against a golden dataset, and fail the build when it gets worse.

No API key. No service. No per-run cost.

npx ragbench gate --baseline main --threshold recall@k=0.8

  2 cases, baseline "main"

  recall@k       0.5000  -0.5000
  ndcg@k         0.5000  -0.5000
  exact_match    0.5000  -0.5000

  x recall@k fell 0.5000, from 1.0000 to 0.5000

  3 check(s) failed

Exit code 1. Your pull request is red, and you know which metric moved before anyone reviews the diff.

Why this instead of the alternatives

The eval platforms are priced for teams. Confident AI runs Free, Starter at $200 a month and Team at $2,000 with Enterprise above that; Braintrust Pro is $249; Galileo Pro is $100. Each has a free tier, and each meters it: Confident AI's is two seats, one project and five test runs a week, which a build gate exhausts by Tuesday.

The open source libraries, RAGAS and DeepEval, compute good metrics but leave you to build the storage, the comparison and the CI gate yourself. So a solo developer stitches three or four tools together and usually ends up with no gate at all.

Pricing checked on the vendors' own pages on 8 September 2026, and it moves, so check it again before quoting it back at anyone.

The other problem is that almost every metric on offer needs an LLM judge. That costs money per run and returns a slightly different number each time, and neither is acceptable in a build gate. A check that costs a dollar gets run less, and a check that drifts cannot tell a regression from noise.

Everything here is deterministic and free. Same dataset, same predictions, same numbers, on any machine.

Metrics

Retrieval, from labelled relevant documents:

| | | | :--- | :--- | | precision@k | How much noise is in the context window. Predicts hallucination. | | recall@k | Whether the model could possibly have got it right. It cannot cite what retrieval never fetched. | | mrr | Reciprocal rank of the first relevant document. | | ndcg@k | Notices a relevant document sliding from position one to position eight, which precision and recall both consider identical. | | hit_rate | The floor below which nothing else matters. |

Answers, from expected text:

| | | | :--- | :--- | | exact_match | After SQuAD style normalisation, so "The answer." matches "answer". | | token_f1 | Partial credit. Counts multiplicity, so padding the answer cannot inflate it. | | contains_all | Required phrases present, for when an amount or a date must appear. | | groundedness | Share of answer content words supported by the retrieved context. A cheap proxy: it will not catch a fluent misreading, but it catches an answer invented wholesale, and it costs nothing. |

A metric whose inputs are absent is omitted, not scored zero, and aggregation skips missing values rather than averaging them in. A partially labelled dataset should report what it can measure, not a column of zeroes that looks like failure.

The gate

Two checks that answer different questions.

A threshold asks "is this good enough". You set it once.

A regression asks "is this worse than last time". That is the one that catches the change nobody meant to make, and most tools do not have it. A suite sitting comfortably above its thresholds will absorb a ten point drop without a word.

ragbench gate --baseline main --tolerance 0.01 --threshold recall@k=0.8

The tolerance exists because retrieval scores move slightly for reasons that are not your change, and a gate that fires on noise gets disabled within a week. Drops inside it are reported as warnings rather than failures.

A threshold on a metric the run never produced is a failure, not a pass. Treating it as passing hides that the check never ran.

When the labels stop describing the corpus

Every retrieval metric above is computed against relevant_docs, which somebody wrote by hand against the corpus as it stood that day. The ranker is re-run on every commit. The labels are not. Documents get re-chunked, re-ided, merged and dropped, and the labels quietly stop pointing at anything, at which point nDCG applies its discount curve to a stale notion of relevant and every number under it measures the wrong thing with great precision.

Three signals, all deterministic, all reported on every run:

Orphaned labels. A labelled document that came back for no question in the whole run. It may have left the corpus or it may now rank below k everywhere, and from the outside those look the same. Either way recall for the cases that need it cannot reach 1, whatever you do to the ranker.

Which ones are new. The count on its own decays as a signal exactly when the dataset gets big enough to be worth having: orphans accumulate for benign reasons, the number only ever goes up, and a line reading 3 of 47 every morning stops being read long before it says 4. So the orphan ids are stored per run and the warning leads with the change. Two more than at the baseline, and here they are is an event. Three are unreachable is weather. Labels that come back are reported too, on their own line, because somebody repairing a dataset should be able to watch the repair land.

Corpus turnover. How much the retrieved set of document ids moved against the baseline run. Past 25 percent, configurable with --turnover, the labels are worth re-auditing on a sample before the comparison means anything.

Here is the case that makes it worth having. A re-chunk changes every document id, nothing about retrieval quality changes at all, and this is what the gate says:

  recall@k       0.0000  -1.0000
  ndcg@k         0.0000  -1.0000

  x recall@k fell 1.0000, from 1.0000 to 0.0000
  x ndcg@k fell 1.0000, from 1.0000 to 0.0000

  Labels
  ~ 3 labelled document(s) went unreachable since the baseline:
    handbook/exceptions, handbook/refunds, handbook/retention. 3 of 3 labelled
    documents (100 percent) are retrieved for no question in this run, so
    recall cannot reach 1 for the 3 case(s) that need them, whatever the
    ranker does.
  ~ the retrieved corpus turned over 100 percent since the baseline (4 new
    document(s), 4 gone). Labels written against the old shape are worth
    re-auditing on a sample before trusting this comparison.

Without the last two lines that is a day spent bisecting a ranker that never changed.

None of it gates. A corpus that changed shape is usually somebody doing their job, and a check that failed the build for it would be switched off within a week. The warnings sit under their own heading, in the terminal and in the pull request comment, and never touch the verdict. They are about whether the measuring instrument still fits the thing it is measuring, which is a different question from whether this change made it worse.

Getting started

npx ragbench init

Writes an example cases.jsonl:

{"id": "settlement-timing", "question": "When does the Benin gateway settle?", "expected_answer": "Overnight", "relevant_docs": ["payments/settlement.md"], "must_contain": ["overnight"]}

Your system reads cases on stdin and writes one prediction per line. Any language, as long as it speaks JSON lines:

{"id": "settlement-timing", "answer": "It settles overnight", "retrieved": ["payments/settlement.md"]}

ragbench run --dataset cases.jsonl --exec "python my_rag.py"

Or produce the file however you like and score it:

ragbench run --dataset cases.jsonl --predictions out.jsonl

In CI

- uses: catidegla/[email protected]
  with:
    dataset: cases.jsonl
    exec: python my_rag.py
    baseline: main
    thresholds: recall@k=0.8

The action scores once and writes both the machine readable report and the table a reviewer reads, because with exec a second pass means running your whole pipeline again. Pin it to a tag: the action runs the source at that ref, so the version you pinned is the code that runs.

Or call it directly, which is the same thing without the wrapper:

- run: npx ragbench gate --dataset cases.jsonl --exec "python my_rag.py"
                         --baseline main --threshold recall@k=0.8

- run: npx ragbench gate --dataset cases.jsonl --exec "python my_rag.py"
                         --baseline main --threshold recall@k=0.8

For a pull request comment, --markdown emits a table:

| Metric | Baseline | Now | Change | | :--- | ---: | ---: | ---: | | recall@k | 1.0000 | 0.5000 | -0.5000 down | | ndcg@k | 1.0000 | 0.5000 | -0.5000 down |

Record the baseline on merge:

- run: npx ragbench run --dataset cases.jsonl --exec "python my_rag.py" --label main

History lives in a local SQLite file. Nothing is sent anywhere, which is the point.

Things it refuses to do quietly

A case with nothing to check against is refused. It passes every metric vacuously and lifts your average.

A duplicate id is refused. Predictions join by id, so a duplicate makes half the suite vanish without a word.

A case with no prediction is scored as empty, not skipped. Skipping means a system that answered nothing scores the same as one that answered perfectly on what it attempted.

A first run with no baseline says so, instead of reporting a clean comparison that never happened.

Every dataset problem is reported in one pass, with line numbers, so fixing a file is not a game of whack-a-mole.

What this is not

Not a tracing or observability platform. Langfuse and Phoenix do that well and this does not try.

There is no LLM judge. One would be useful for answer quality and the hook is there, but it will not be on the path that gates your build.

Testing

npm test    # 84 tests, nothing to install

The metric tests check the arithmetic against hand computed values, not just asserting the code runs. The ndcg test in particular verifies that a document found first scores higher than the same document found last, while precision and recall report both as identical.

Requirements

Node 22.13 or newer, which is where the built-in node:sqlite stopped needing a flag. 22.5 shipped the module but only behind --experimental-sqlite, so anything between the two crashes on import rather than running. Nothing else.

License

MIT