diff-conformance

A correctness harness for JavaScript structural diff libraries. It measures what a diff actually reports, in both directions, against whatever supported packages you have installed. Nothing is bundled and nothing is pinned.

npx diff-conformance all

The premise

A diff makes two promises, and only one of them ever gets tested.

Everybody checks that a change is reported. Almost nobody checks the other direction: that a change which did not happen is not reported. That second direction is where this field quietly fails, and its failure mode is the worse of the two. A phantom patch in a state-sync loop is an infinite loop, because every poll produces a patch that produces a write that produces a patch. In an audit trail it is a permanent record of edits nobody made.

Three of the seven most-used packages emit a patch for two values that are structurally identical. Including the most downloaded one in the field.

What it found

Measured on 2026-07-30 against the versions in docs/REPORT.md, which is generated by the command above rather than typed by hand.

| subject | missed a real change | reported a phantom one | HTER | |---|---|---|---| | just-diff | 8 (34.8%) | 0 (0.0%) | 17.4% | | microdiff | 9 (39.1%) | 0 (0.0%) | 19.6% | | json-diff-ts | 2 (8.7%) | 6 (42.9%) | 25.8% | | rfc6902 | 13 (56.5%) | 0 (0.0%) | 28.3% | | deep-object-diff | 14 (60.9%) | 0 (0.0%) | 30.4% | | fast-json-patch | 11 (47.8%) | 2 (14.3%) | 31.1% | | jsondiffpatch | 11 (47.8%) | 2 (14.3%) | 31.1% |

Out of 23 detection probes and 14 identity probes, so the two errors are reported as rates and summarized by their mean. The name comes from biometrics, where these two are the false non-match and false match rates and the mean is the half total error rate. That field usually quotes an equal error rate instead, the point where the two cross as a decision threshold sweeps, and it does not apply here: none of these libraries exposes a threshold, so each is a single operating point rather than a curve. HTER is what biometrics uses when the operating point is fixed.

Read the first column alone and json-diff-ts is far and away the best in the field: 2 misses where everyone else has 7 to 13. It is the only subject that sees a change inside a Map or a Set.

It does not see them. Handed two identical Maps it reports a change too:

diff({ v: new Map([["k", 1]]) }, { v: new Map([["k", 1]]) })
// [{ "type": "UPDATE", "key": "v", "value": {}, "oldValue": {} }]

The {} gives it away. It collapses the Map exactly like everyone else and then reports UPDATE unconditionally, so it cannot be wrong about a real change and cannot be right about an unchanged one. A detector that always fires has a perfect detection rate and no value, and only running both directions can tell the difference. That is the whole reason this harness exists.

Two more results worth pulling out:

fast-json-patch, handed two identical Dates, emits 24 operations.

compare({ v: new Date("2020-01-01") }, { v: new Date("2020-01-01") })
// [{op:"add",path:"/v/0",value:"2"}, {op:"add",path:"/v/1",value:"0"}, ...]

It serializes the Date to its ISO string and diffs it character by character. Passing the same instance twice returns [], so this only appears with distinct-but-equal Dates, which is exactly what deserializing the same document twice gives you.

Nobody handles Map or Set. All seven report nothing for a changed Map value, an added Set member, or a swapped one. In a JSON-model walker a Map presents no enumerable properties, so it looks like {} and a real change is silently dropped. Typed arrays are fine, since they are indexable; it is specifically the keyed collections that vanish.

The suites

| suite | question | failure | |---|---|---| | detection | is a real change reported? | reporting nothing | | false-positives | are two equal values left alone? | reporting anything | | moves | how big is the patch? | more operations than the minimum | | round-trip | does applying the patch give the right value? | a wrong or unpatchable result | | scaling | how does cost grow with depth and width? | superlinear growth, or dying |

The depth axis reaches 8,192. It used to stop at 512, and that is not deep enough to catch anything: an engine whose memory is quadratic in depth looks perfectly linear up to a few thousand levels, so the suite certified a 1.00 depth exponent for a subject that exhausts a multi-gigabyte heap at twenty thousand. A range that excludes the failure cannot report the failure. At the current depths every recursive implementation in the field throws RangeError before the last two sizes, which the table records rather than hides.

Default is the first three. all runs everything.

Throwing is never a failure. A library that refuses a cyclic input is telling the truth about its scope; one that returns a confident wrong patch is not. Every suite records throws as its own outcome instead of folding it into a pass or a fail, and every table keeps that column.

Every probe states why it is fair. The reason each pair genuinely differs ships with the data rather than living in prose, because that sentence is the first thing to attack when reading someone else's benchmark. If a probe is unfair, it is unfair in public.

Probes get added when something slips past. Two of them exist because a diff engine measured by this harness scored clean and was still wrong: a shared reference (the same object reachable by two paths) reported under only one of them, and a Map entry inserted ahead of an existing one coming back in the wrong order, because set can only append. Both produce a patch that applies without complaint and leaves a document that is not the target, which is the worst failure mode here and the one a detection-only suite cannot see. Of the seven subjects, json-diff-ts gets the second one wrong.

Two things the harness had to get right first

Operations are counted per format, not uniformly. These libraries emit five incompatible shapes for the same idea: a flat operation list, an object mirroring the input whose leaves are the changes, a delta with sigil keys and array tags, and a recursive changes tree. Counting top-level entries scored a four-change patch as one operation in the audit that led to this tool. Each format has its own extractor, checked against real output.

Subjects run in a capped child process. jsondiffpatch exhausts the heap on a cyclic input, and rfc6902 exhausts a 256 MB heap on the scaling suite. An in-process harness dies with them and loses every unrelated result, and a try/catch cannot help: by the time the allocator gives up there is nobody left to write the answer down. So "exhausted the heap" and "never returned" become recorded outcomes like any other.

The isolation is two-tier, because both obvious strategies are bad. One fork per subject is fast and loses everything for a subject that dies; one fork per probe never loses anything and pays a process launch per cell. This forks per suite, and only when that child dies does it retry that suite one probe at a time to find out which probe killed it. The common case costs one fork.

Bring your own

import { defineSubject, runDetection, runFalsePositives } from "diff-conformance";

const mine = defineSubject({
  name: "mine",
  format: "rfc6902",
  run: (a, b) => myDiff(a, b),
  count: (patch) => (patch as unknown[]).length,
  apply: (patch, a) => myApply(patch, a), // optional, enables round-trip
});

console.log(runFalsePositives(mine).falsePositives);

count is not boilerplate. It is how the harness knows an empty patch from a non-empty one, which is the question both correctness suites are built on, so it has to be exact rather than approximate.

A subject you define is a closure, and a closure cannot cross a process boundary, so it runs in this process rather than an isolated one. The CLI says so at the end of a run instead of leaving you to assume every row was equally protected.

Install

npm install --save-dev diff-conformance

Node 18+. Adapters load dynamically and are skipped if the package is not installed, so you can measure one library without installing seven, and the report names what it did not measure. A report that hides its own gaps is worse than no report.

License

MIT