jsonshim

Score any JavaScript JSON-recovery parser against the MALFORMED-300 conformance suite, using exactly the grading spec behind the published leaderboard. Zero dependencies, no network call at runtime, no telemetry. CC0-1.0 — public domain.

It is the scorer and adapter protocol, not a parser. It measures whichever parser you point it at, including your own.

Install

$ npm i jsonshim
added 1 package, and audited 2 packages in 606ms

found 0 vulnerabilities

That is the real output of the command run on 2026-08-20 against the public registry. Or, without the registry, straight from a plain HTTPS tarball on the project's own domain:

npm i https://toolkitlabs.org/pkg/jsonshim-1.0.0.tgz

Every artefact's exact byte count and sha256 is served as JSON at https://toolkitlabs.org/pkg/index.json, so you can check what you downloaded before you run it. Nothing is minified and there is nothing to install globally.

Use it

import { scoreCases, loadCorpus, canon } from 'jsonshim';

const cases = loadCorpus('malformed-open12.jsonl');
const summary = await scoreCases(cases, (text) => JSON.parse(text));
console.log(summary.exact, summary.invented_values, summary.false_refusals);

Or from the shell, against 12 open cases served live with no signup:

curl -o malformed-open12.jsonl https://toolkitlabs.org/malformed300/open12.jsonl

jsonshim-score-js --corpus malformed-open12.jsonl --parser json          # stdlib control
jsonshim-score-js --corpus malformed-open12.jsonl --parser ./my-parser.mjs#recover
jsonshim-score-js --corpus malformed-open12.jsonl --parser-cmd "./myparser --stdin"

Add --json for the summary object only, so it wires into CI. npm test runs the harness selftest; every expected value in it was derived by hand from the spec below before the file was run once.

See it in a browser first

A free page runs strict JSON.parse, jsonrepair 3.15.0 and JSON5 2.2.3 over text you paste and compares their answers with the same canon() this package uses -- no signup, no card, nothing installed: https://toolkitlabs.org/try/?s=npm-jsonshim-107

The grading spec

| expected_kind | passes only by | |---|---| | value | returning exactly that value | | unrecoverable | refusing — throwing, or returning undefined / null |

Any other return on an unrecoverable case — including {}, [] or "" — is counted as an invented value. That is the number this suite exists to measure: an invented value silently corrupts state downstream, while a refusal is loud and can be handled.

Comparison is by a canonical form computed by this harness for both sides, so key order and 1 vs 1.0 can never decide a case. One run each, nothing tuned afterwards.

Results already published

Ten JavaScript parsers have been scored on all 300 cases; the table is free and CC0 at https://toolkitlabs.org/leaderboard/?s=npm-jsonshim-107, machine-readable at https://toolkitlabs.org/api/. Nobody pays to be listed, ranked or removed. The Python library this harness is named after scores 282/300 exact and invents 5 values on unrecoverable cases — a real defect, documented and not fixed; on the 275 recoverable cases json-repair beats it, 264 to 262.

The full corpora are paid

The open 12 cases per suite are a sample. The full labelled sets — every case with the rationale for its label, and the sealed answers — are sold, EUR 29 each:

• MALFORMED-300, all 300 cases — https://toolkitlabs.org/malformed300/?s=npm-jsonshim-107
• TOOLCALL-300, all 300 cases — https://toolkitlabs.org/toolcall300/?s=npm-jsonshim-107

Direct checkout, one click, no page in between: single developer EUR 29 — https://toolkitlabs.org/go/m300?s=npm-jsonshim-readme; team/CI licence EUR 99 — https://toolkitlabs.org/go/m300-team?s=npm-jsonshim-readme. Those two links are 302 redirects on our own domain to Stripe-hosted checkout; the only thing they add is the source tag, so we can tell an npm reader from a search visitor. Nothing is tracked here.

Nothing in this package is gated behind them; it scores any corpus you hand it.

What funds this

This package is not gated, metered or upsold, and it does not phone home. The corpora above cover part of the bill; the rest is paid by an unrelated product on the same domain: Companion, an AI companion that is openly software and says so — EUR 9.00 a month, ten messages free without a card, 500 messages a month with a hard cap and no overage. Cancel from the Stripe billing portal on your own conversation page; cancelling stops the next payment, the page stays readable to the end of the paid month and then becomes read-only, and a one-click delete erases the stored record immediately. It is for people 18 and over, and it is not a developer tool.

https://toolkitlabs.org/companion/?s=npm-jsonshim-107

Buying it changes nothing about this package. The harness, the leaderboard and the API stay free and CC0 for ever.

Structured-output conformance matrix — edition 1 (free)

How reliably four LLM endpoints return output that actually validates against a given JSON Schema. Free, no signup, no email — the matrix, the 216-row raw log with the exact error strings, and the harness: https://toolkitlabs.org/conformance/?s=npm-jsonshim-107 Raw data and a stdlib-only script that reprints every table from it, public domain, on GitHub: https://github.com/YtinuMoc/llm-structured-output-conformance (data CC BY 4.0, code CC0 1.0).

Read the limitations first: this is 4 endpoints x 6 schemas x 3 modes x 3 trials = 216 live calls in one run, and 63 of 72 cells returned at least one HTTP 200. At three trials a cell's pass rate can only take four values (0, 1/3, 2/3, 1), so per-cell rates are a coarse signal, not a benchmark score. That is why it costs nothing.

What does survive three trials, because it is categorical rather than a rate:

• The dominant failure is the provider rejecting the schema before the model ever runs. In native mode only 44 of 72 calls got an HTTP 200. gemini-3.1-flash-lite rejected 18 of 18 with its own words: Unknown name "additionalProperties" at 'generation_config.response_schema': Cannot find field.
• Mechanically sanitising the schema per provider takes acceptance from 44/72 to 71/72 — and validation still runs against the original schema, so nothing is smuggled past the check.
• Valid is not the same as parseable. Plain prompting produced schema-valid output in 70 of 72 calls but only 52 of 72 parsed with a bare json.loads. A consumer without an extraction step sees 0.722, not 0.972.

Every number above is recomputed from raw-runs.jsonl by the shipped harness; run harness.py --score . yourself.

Two things a recount of the same log on 2026-08-27 added, both of which cut against the headline:

• The 20 plain-mode strict-parse failures are not spread across the fleet. 18 of them are one model, groq/qwen3.6-27b, which parsed 0 of 18 strictly and yet was schema-valid 18 of 18 once a fenced or embedded object was extracted; the other 2 are gpt-oss-20b returning no JSON at all. Pooled, that reads as a general parsing problem. It is one model and an extraction step.
• The sanitiser that lifts native acceptance to 71/72 bought part of its win by deleting constraints. It altered 54 of 72 schemas: minimum dropped x36, maximum x24, pattern x24, additionalProperties x18, required widened x9. Acceptance gained by removing the checks is not conformance, so both columns are published.