agentic-test-contracts

JSON Schemas for the documents that agents in a test pipeline hand to each other, plus a blank profile template describing the system those agents are pointed at.

Nothing in here runs a test or talks to a tool. It is the boring layer underneath: the shapes the documents have to be, and the questions you have to answer about your own system before any of it is worth automating.

npx agentic-test-contracts init

Why this exists separately

If you build a pipeline where one agent reads requirements, another plans checks, another writes them, another runs them and another triages what failed, the interesting failure is not any one of those steps. It is the handoff. An agent emits an empty array; the next one cannot tell whether that means "nothing applies here" or "the thing I query was down". An agent invents an identifier scheme; the next run invents a different one and appends instead of updating. A plan says a check covers a requirement, and nobody recorded what the check actually compared against, so the claim cannot be audited later.

Those are contract problems, and a contract is only useful if it is written down somewhere neither end owns. That is all this package is. The schemas are opinionated about a handful of things that turn out to matter, and studiously unopinionated about everything else — there is no built-in list of test layers, no enumeration of surfaces, no assumption about which tracker or runner you use.

The eight schemas

common.schema.json holds the shared definitions; the other seven are documents.

| Schema | The document it describes | | --- | --- | | requirement | one requirement, its source, and an explicit readiness verdict | | test-plan | planned cases with their intent, risk tier, chosen layer and oracle | | test-artifact | the files a generator wrote, and whether they build | | run-result | what executed, where, and what each case did | | triage | a classification per failure, with evidence and a recommended action | | finding | something worth someone's attention, deduplicable | | approval | a pipeline parked at a human gate, in a form it can resume from |

Every document carries the same envelope: which profile and version it was produced under, which agent produced it and when, and whether the document is complete, partial or failed — with a reason when it is not complete. That last field is the one that stops downstream agents from guessing.

Using them

const { getSchema, getAllSchemas, listSchemas } = require('agentic-test-contracts');

const { Ajv2020 } = require('ajv/dist/2020.js');
const addFormats = require('ajv-formats');

const ajv = new Ajv2020({ strict: true });
addFormats(ajv);
for (const schema of getAllSchemas()) ajv.addSchema(schema); // common comes first

const validate = ajv.getSchema(getSchema('test-plan').$id);
if (!validate(myPlan)) console.error(validate.errors);

getSchema also takes 'test-plan.json' or 'test-plan.schema.json', and returns a fresh object each call so one consumer mutating a schema cannot affect another. If you would rather have the file directly, each one has a subpath export:

const commonSchema = require('agentic-test-contracts/contracts/v1/common.schema.json');

getSchemaPath(name) gives you an absolute path for tooling that wants a file, and contractsDir gives you the directory.

Ajv is a dev dependency here, not a runtime one. This package has no runtime dependencies at all; bring your own validator.

The profile template

npx agentic-test-contracts init [dir] copies fourteen files into dir (default ./profile). They are almost entirely comments and empty collections. That is deliberate — the value is in the questions, and a template pre-filled with plausible answers is a template people accept without reading.

| File | What it settles | | --- | --- | | meta.yaml | the loader entry point: profile version, trust state, what invalidates what | | product.yaml | what the software is for, which parts matter, who owns them | | bindings.yaml | what a "module" actually is — paths, tickets, endpoints, suites, owners | | stack.yaml | how code and tests are built | | layers.yaml | your layer ladder, with what each layer can and cannot decide | | conventions.yaml | the house style, mined from the corpus rather than wished for | | environments.yaml | where checks may run and what may be done there | | policy.yaml | what agents may do, keyed on blast radius rather than environment | | gates.yaml | every threshold, in one tunable place | | risk.yaml | weighting, and obligations that are not merely advisory | | integrations.yaml | which adapter serves each port, and its field vocabulary | | domain.json | glossary, entities, and the invariants worth checking | | indexes/ | derived indexes, regenerated and never hand-edited | | learned/ | corrections that have to survive the next rebuild |

init will not write into a directory that already has files in it. Pass --force if you mean it.

You can also do it from Node:

const { initProfile, templateDir } = require('agentic-test-contracts');
const { files } = initProfile('./config/test-profile');

What this deliberately gets right

Offline resolution. Every $ref between the schemas is a relative filename. The $ids are absolute URIs because JSON Schema wants globally unique identifiers, but nothing here fetches them, and a test fails if any $ref becomes an absolute URI. You can validate on a machine with no network.

No hardcoded vocabulary. Layer names, surfaces and environments are strings you declare in your own profile, not enumerations baked into the schemas. An earlier version shipped a unit | contract | api | ui | e2e layer list, which was itself a stack-specific vocabulary that several real systems could not be described in.

A representable "I could not do that". The status block distinguishes complete, partial and failed, with a reason drawn from a fixed list. An empty array with no reason is indistinguishable from a successful finding of nothing, which is the shape most agent handoffs fail in.

Provenance on everything. Every document records the agent, the prompt identity, the inputs it derived from with hashes, and the principal it acted under. A regression traces to a prompt change instead of being guessed at.

Nothing environment-specific. No hostnames, no credentials, no tenants, no defaults for your staging box. A test greps the whole working tree for that and fails if any of it creeps back in.

Tests

npm install
npm test

node --test, no test framework. The suite compiles all eight schemas under ajv 2020 in strict mode, asserts every $ref resolves, asserts each document schema rejects {} while naming every missing required field, and — this is the part that matters — validates a minimal valid document against each schema and then deletes each required field in turn to confirm the rejection is real. A schema that rejects everything would pass a {}-only suite.

It also checks the things that break quietly: that moving one schema's $id without moving the others breaks resolution (so the relative refs are load-bearing rather than decorative), that no subschema is unsatisfiable through requiring a property it does not declare, and that npm pack ships exactly the file allowlist and nothing more.

Provenance

These schemas and the profile template were extracted from a larger unpublished architecture. That source tree had no license and no commit history, so there is no attributable history behind the first commit here, in either direction: nothing in that tree carries a third-party copyright, author tag or proprietary marker, and equally nothing in it records who wrote what. Everything else from that tree — the agent specifications, the port adapters, the runtime — is not part of this package and is not planned for it.

Extraction changed three things: the $id base moved from a hostname with no TLD to this repository's raw URL, a description pointing at a document that is not shipped was rewritten, and one genuine defect was fixed — test-plan.schema.json required a property named layer on a closed object whose property was actually layerId, making that subschema impossible to satisfy.

License

MIT