laserbrain-check

Deterministic checks on things that stop being true without anything breaking.

No model, no API key, nothing sent anywhere. The same inputs give the same answer forever, and every finding shows the words it was based on, so you can disagree with it.

npx laserbrain-check try

That runs all three checks on built-in examples and needs no files. Then point it at your own — the filenames below are placeholders, not files that ship with the package:

npx laserbrain-check docs <your-openapi.json> --base https://your-api.example.com
npx laserbrain-check gate --context <chunk...> --answer <answer.txt>

Why

A README is a set of claims about a running system, and a generated answer is a set of claims about the context it was given. Both stop being true quietly. The docs still render, the tests still pass, the endpoint still answers — it just answers something the page no longer describes. Nothing in a normal pipeline fails when that happens.

docs — does your documentation still describe your API?

Reads markdown or an OpenAPI / Swagger JSON spec. With --base, probes the live API and reports which documented endpoints, response fields and status codes still hold.

laserbrain-check docs <your-README.md> --base https://your-api.example.com
laserbrain-check docs https://your-api.example.com/openapi.json

It never sends a write. GET, HEAD and OPTIONS only. A documented POST or DELETE is reported unchecked, permanently, and there is no flag to change that. A documentation checker that fires writes at a live system to see whether they are documented correctly would, on a good day, create junk records — and on a bad day delete a customer.

A claim it cannot resolve is unchecked, never a pass. If the prober is unauthenticated, or the spec declares a required query parameter, or the server returns 5xx, the documented path was never exercised. Reporting that as drift would flag nearly every authenticated endpoint on earth, and a checker that flags everything is read as flagging nothing.

YAML specs are refused with a reason rather than half-read — most APIs that publish a spec publish openapi.json alongside it.

gate — is a generated answer grounded in its retrieved context?

laserbrain-check gate --context <chunk...> --answer <answer.txt>

| verdict | meaning | exit | |---|---|---| | block | a quotation in the answer is not in any chunk | 1 | | flag | a figure or name is not in any chunk — a worklist | 0 | | pass | every checkable claim is grounded | 0 | | no_signal | nothing in the answer was checkable — not a pass | 0 |

Only quotations block, and that is measured rather than chosen. Figures blocked in the first version, on the argument that a number cannot be paraphrased. True, and the wrong argument: the question is whether a claim could be grounded in a source the checker was not handed — an infobox, a citation, a chunk that was not retrieved. Against real human-written grounded summaries:

figures blocking     74.3% of correct answers BLOCKED
figures flagging      8.6%   same corpus, one policy line
excluding titles      6.7%   "The Devolution Project" is a title, not a quotation

A flag does not fail your build. 91.7% of real grounded summaries flag, because a correct answer routinely carries a figure or a name from a source the checker never saw. Treat flags as a list of claims to spot-check. --strict fails on them if you want that.

no_signal is not a pass. It means the answer contained no figure, quotation or name, so nothing was checkable. It exits 0 — failing a build because an answer had no numbers punishes the wrong thing — and prints NOT CHECKED so it can never be read as a clean result.

anchor — which sentences came from the source?

laserbrain-check anchor --source <doc.txt> --answer <answer.txt>

Reports sentence-level lexical support, plus the three readings that survive paraphrase: figures, names and quotations not present in the source.

What it is not

• Not semantic. A faithful paraphrase scores badly on the support number. That is why figures, names and quotations are reported separately — none of them paraphrase.
• Not an entity model. Names come from capitalisation, which returns Everything and Questions as entities on real text. That is why names only flag.
• Not attribution-aware. A real name attached to the wrong claim passes. A verbatim quote attributed to the wrong speaker passes. It asks whether the words are in the document, not whether that person said them.

Every one of these is pinned as a passing test, so none can be quietly "fixed" without the claim here changing too.

In CI

- uses: actions/setup-node@v4
  with: { node-version: 20 }
- run: npx laserbrain-check docs ./openapi.json --base ${{ secrets.STAGING_URL }}
#      ^ your spec's real path in the repo

Exit 0 pass/flag/no_signal · 1 block or drift · 2 usage or input error.

Licence

MIT · phronesis.world/laserbrain/docs