schema-parity

schema-parity checks whether a JSON Schema accepts exactly the same set of values as the runtime validator it was generated from, and reports every value where the two disagree.

Here is a gap you can reproduce in a clean directory:

mkdir -p /tmp/schema-parity-repro && cd /tmp/schema-parity-repro
npm init -y && npm i -s [email protected] [email protected]
node -e '
const {z}=require("zod"), Ajv=require("ajv/dist/2020");
const t=z.tuple([z.string(), z.number()]);
const js=z.toJSONSchema(t);
const ok=new Ajv({strict:false}).compile(js);
console.log(JSON.stringify(js));
for (const v of [[], ["a"], ["a",1], ["a",1,2], ["a","b"]])
  console.log(JSON.stringify(v).padEnd(12), "zod:", String(t.safeParse(v).success).padEnd(5), "json-schema:", ok(v));
'

{"$schema":"https://json-schema.org/draft/2020-12/schema","type":"array","prefixItems":[{"type":"string"},{"type":"number"}]}
[]           zod: false json-schema: true
["a"]        zod: false json-schema: true
["a",1]      zod: true  json-schema: true
["a",1,2]    zod: false json-schema: true
["a","b"]    zod: false json-schema: false

Three of the four major TypeScript schema libraries, ArkType, Valibot and TypeBox, emit a tuple schema that constrains length exactly as their validator does; zod 4.4.3 emits prefixItems with no minItems and no items: false, so the JSON Schema accepts [], ["a"] and ["a",1,2] that zod itself rejects.

This is a library, not a command-line tool. Install it and point it at your own schemas, the gap above is one instance of a class of gap, and the class is what you actually want to find.

npm install --save-dev schema-parity

import { assertParity } from "schema-parity";
import { z } from "zod";
import Ajv from "ajv/dist/2020";

const schema = z.object({ pair: z.tuple([z.string(), z.number()]) });
const json = z.toJSONSchema(schema);
const validate = new Ajv({ strict: false }).compile(json);

assertParity(
  (v) => schema.safeParse(v).success,
  (v) => validate(v) === true,
  { seeds: [{ pair: ["a", 1] }] }
); // throws ParityError listing the values the JSON Schema wrongly accepts

Why this matters

You define a validator, convert it to JSON Schema, and hand the JSON Schema to a language model so it knows what a tool's arguments must look like. The model reads the JSON Schema, produces arguments, and your code re-checks those arguments with the original validator.

If the JSON Schema is wider than the validator, the model produces arguments the provider approves and your own validator then refuses at runtime. The tool call fails and nothing in the toolchain warned you, because the conversion reported success. If it is narrower, you have silently shrunk what the model is allowed to produce, and you will never see the requests it did not make.

What the two kinds mean

| kind | meaning | consequence | |---|---|---| | wider | the JSON Schema accepts a value the validator rejects | the model emits arguments your own code then refuses | | narrower | the validator accepts a value the JSON Schema rejects | the model is stopped from producing arguments that would have worked |

assertParity treats both as failures by default. Pass allowWider or allowNarrower when a gap is understood and accepted, a z.string().refine(...) has no JSON Schema representation, and a widening there is expected lossiness rather than a defect.

How it works

You supply two predicates. schema-parity supplies the values.

1. A fixed, hand-written adversarial corpus, ordered simplest first.
2. Your seed values.
3. Every deterministic structural mutation of those seeds, every prefix, every dropped key, every element replaced by every corpus value.
4. Seeded pseudo-random second-order mutations, using a fixed mulberry32 stream.

Both predicates run over every value. Because generation is simplest-first and duplicates keep their earliest position, the first counterexample you see is the smallest one, and no shrinking step is needed.

The package has zero runtime dependencies. It never parses a JSON Schema, never converts anything, and never imports your validator, which is why it works with any library, including ones that did not exist when it was written.

Adapters

Each pairing is the same three lines: build the schema, compile the emitted document, hand assertParity the two predicates.

Valibot

import * as v from "valibot";
import { toJsonSchema } from "@valibot/to-json-schema";
import Ajv from "ajv";

const schema = v.tuple([v.string(), v.number()]);
const validate = new Ajv({ strict: false }).compile(toJsonSchema(schema));
assertParity((x) => v.safeParse(schema, x).success, (x) => validate(x) === true, { seeds: [["a", 1]] });

ArkType

import { type } from "arktype";
import Ajv from "ajv/dist/2020";

const schema = type(["string", "number"]);
const validate = new Ajv({ strict: false }).compile(schema.toJsonSchema());
assertParity((x) => schema.allows(x), (x) => validate(x) === true, { seeds: [["a", 1]] });

TypeBox

import { Type } from "@sinclair/typebox";
import { Value } from "@sinclair/typebox/value";
import Ajv from "ajv";

const schema = Type.Tuple([Type.String(), Type.Number()]);
const validate = new Ajv({ strict: false }).compile(schema);
assertParity((x) => Value.Check(schema, x), (x) => validate(x) === true, { seeds: [["a", 1]] });

TypeBox emits draft-07 keywords, so it needs the default ajv export rather than ajv/dist/2020. Compiling a TypeBox document with the 2020-12 export fails with schema is invalid: data/items must be object,boolean: a dialect mismatch, not a parity defect.

API

checkParity(source: Predicate, schema: Predicate, options?: ParityOptions): ParityReport
assertParity(source: Predicate, schema: Predicate, options?: AssertOptions): void
generateValues(options?: ParityOptions): readonly unknown[]
CORPUS: readonly unknown[]

ParityOptions:

| option | default | meaning | |---|---|---| | seeds | [] | values the source predicate is expected to accept; mutations of these find most gaps | | seed | 1 | seed for the pseudo-random phase | | count | 2000 | total values to check, before de-duplication | | maxCounterexamples | 5 | counterexamples recorded per kind | | expectVacuous | false | set only when the source predicate genuinely accepts nothing |

AssertOptions adds allowWider and allowNarrower, both false.

A ParityReport carries checked, agreed, wider, narrower, sourceAccepted, schemaAccepted and counterexamples. Every counterexample value is a deep copy; nothing you receive points back into CORPUS or into your seeds.

When it refuses to answer

A harness that cannot measure has not found parity, it has found nothing. checkParity throws HarnessError rather than returning a clean report when:

| reason | trigger | |---|---| | too-few-values | fewer than 32 distinct values were generated, or count is not a positive integer | | predicate-threw | either predicate threw; a throw is a broken adapter, never a rejection | | predicate-not-boolean | either predicate returned a promise, a generator, or any other non-boolean | | no-accepted-values | the source predicate accepted nothing and expectVacuous was not set |

A HarnessError is never converted into a ParityError. Predicate is synchronous by design: an async predicate returns a promise, which trips predicate-not-boolean.

Findings

FINDINGS.md is regenerated from the pinned libraries by npm run scoreboard and committed. It contains no timestamp, so it cannot go quietly stale, VERIFY.md has the one command that proves it is current.

License

MIT