@dogfood-lab/verify

Central verifier for testing-os. Validates submissions against schema and policy, produces persisted records.

Part of the testing-os monorepo — the operating system for testing in the AI era.

The verifier sits between dispatch and persist: every dogfood submission passes through here before it's written to records/. Returns a structured verdict (ok / rejection_reasons[]) so callers — including the @dogfood-lab/ingest pipeline — can decide whether to persist the record or surface the rejection to the operator.

Install

npm install @dogfood-lab/verify

Usage

import { verify } from '@dogfood-lab/verify';

const result = verify(submission, {
  policy,
  schemas,        // from @dogfood-lab/schemas
  provenance: 'github',
});

if (!result.ok) {
  // rejection_reasons is an array of STRINGS with stable prefixes (see
  // "Error shape" below). Operators discriminate failure class via the
  // prefix; the rest of the string carries the human-readable detail.
  for (const reason of result.rejection_reasons) {
    console.error(reason);
  }
  process.exit(1);
}

// result.record is the persistable artifact

Validators

@dogfood-lab/verify/validators/* ships discrete validators that can be composed or called directly:

| Validator | Purpose | |---|---| | validators/schema.js | JSON Schema check against @dogfood-lab/schemas (SHAPE gate) | | validators/schema-version.js | schema_version VALUE gate — refuses an incompatible MAJOR against SUPPORTED_SCHEMA_VERSIONS | | validators/policy.js | Per-repo policy compliance (prototype-pollution-safe deep merge) | | validators/provenance.js | GitHub Actions run-ID confirmation via API (with timeout guard) | | validators/steps.js | Step-by-step contract checks (gate accumulation, ordering) | | validators/verdict.js | Final verdict synthesis from upstream validator results |

Import a single validator:

import { validateSchema } from '@dogfood-lab/verify/validators/schema.js';
import { validateProvenance } from '@dogfood-lab/verify/validators/provenance.js';

CLI (dogfood-verify)

The package ships a dogfood-verify bin with two verbs.

Verify a submission (local dry-run / explain — never writes):

dogfood-verify --file submission.json --explain   # human verdict, reasons classified by who-fixes-it
dogfood-verify --file submission.json --json       # machine-readable
# exit: 0 accepted · 1 rejected · 2 operator error

Lint a policy file (VERIFY-F3, author-time — no submission needed):

dogfood-verify lint policies/repos/<org>/<repo>.yaml        # or global-policy.yaml
dogfood-verify lint policies/global-policy.yaml --json      # for CI
# exit: 0 clean or warnings-only · 1 errors · 2 operator error

lint runs the structural schema gate plus the data-independent predicate checks (unknown_field, max_depth, node_budget) over every when predicate, and emits an advisory warning on the [] footgun (a negative operator over a [] path fails open) with the fail-closed not(any(...)) rewrite as a suggestion — never auto-applied, never a hard error. It is the opa check analogue. It cannot statically catch a type_mismatch or a fan-out overrun (both are data-dependent) and says so. Full contract + coverage boundary: docs/policy-lint.md.

Submission envelope

The full envelope shape is defined by @dogfood-lab/schemas (dogfood-record-submission.schema.json). Minimum required fields:

{
  "repo": "org/repo",
  "commit": "<git-sha>",
  "submitted_at": "2026-05-14T15:00:00Z",
  "records": [/* one or more dogfood-record envelopes */]
}

Provenance fields (github_run_id, github_workflow_ref) are required when provenance: 'github' is set. The verifier confirms the run ID against the GitHub Actions API before accepting.

Error shape

rejection_reasons[] is an array of strings — the persisted-record schema (dogfood-record.schema.json → verification.rejection_reasons) enforces items: { type: 'string' }. Machine-readable discrimination happens via stable string prefixes.

Prefix taxonomy

The verifier emits rejection-reason strings under stable prefixes, each mapping to one of four routing classes:

Discrimination happens by class, surfaced by parseRejectionReason (below). Every prefix maps to one of four classes: submission-bad (the submitter fixes the payload), operational (the verifier/tooling faulted), ingest (an ingest-side load fault), or unknown (unrecognized prefix).

Retryable (F-f8952a50, wave 10) is a SEPARATE, narrower per-prefix flag, orthogonal to class: may a same-run_id resubmission whose ONLY prior rejection carries this prefix still reach an acceptance, once corrected? packages/ingest/persist.js's duplicate guard (isDuplicate → isRetryableRejection) reads this flag — never class or prefix directly — to decide whether a stale _rejected record blocks a same-run_id retry that is now headed to acceptance. Every operational / ingest / unknown prefix is retryable: false (an ops fault or an unrecognized signal is never the submitter's to retry past). Within submission-bad, the taxonomy splits further — shape/addressing prefixes ("we could not even read/place/shape your submission") are retryable; content-verdict prefixes ("we read your submission and rendered a verdict against its own reported content") are not, so a submitter cannot launder a genuinely-bad run into an accepted one by resubmitting different self-reported content under the same run_id. The table below marks each submission-bad prefix's retryable value explicitly.

Submission-bad — class: 'submission-bad' (the submitter's payload failed a validator gate; fix the submission and resubmit):

| Prefix | Retryable | Source | Meaning | |---|---|---|---| | schema: | Yes — shape | validators/schema.js | JSON Schema check on the submission/record envelope failed. The rest of the string carries the AJV path + message. | | policy: | No — content verdict | validators/policy.js | Per-repo policy gate failed (forbidden tags, missing required fields, surface evidence/CI requirements, or a declarative when/custom_rules predicate matched — see the policy DSL). Judges the run's OWN reported content (tags, evidence, scenario results); consuming the run_id is the deliberate anti-gaming behavior. | | policy-config: | Yes — shape | validators/policy.js | VERIFY-F1. A REPO custom-rule predicate hit an eval-time semantic fault the schema could not catch — an unknown leading field, a numeric operator against a non-number, or a depth/width/fan-out budget. The repo authored the bad RULE (config), not the run's content, so the fix belongs to the submitter and is retryable. (A malformed GLOBAL predicate is VALIDATOR_FAULT_POLICY: operational instead — see below.) | | steps[<id>]: | Yes — shape | validators/steps.js | Step-level contract check failed on a specific step id (gate accumulation, ordering, evidence shape — a completeness/structure mismatch, not a verdict on whether the steps passed). | | provenance: | No — content verdict | validators/provenance.js | The run was genuinely absent / not confirmable — a 404 from the provider API, or the run head did not match the submitted commit/repo. The provider could not confirm THIS specific run happened as claimed; resubmitting different self-reported content under the same run_id to "become confirmable" is exactly the laundering the anti-gaming doctrine blocks. (Operational provider faults — 429/5xx/401/403 — are NOT this class; see provenance-fault: below.) | | repo: | Yes — shape | index.js cross-field guard | submission.repo does not match the owner/repo encoded in source.run_url (anti-forgery guard). Emitted as repo:mismatch: …. Pure identity/addressing — the run happened, only the repo/run_url pairing was mis-stated. | | submission-contains-verifier-field: | Yes — shape | index.js | The submission carried a verifier-owned field (policy_version, verification, or an object overall_verdict) it must not author. | | CONTRACT_SCHEMA_TOO_OLD: | Yes — shape | validators/schema-version.js | The submission's schema_version declares a MAJOR below the supported floor. The submitter must re-emit against the current contract. A patch/minor delta inside the supported major range is NOT rejected. (NOT symmetric with CONTRACT_SCHEMA_TOO_NEW: below — see that row for why.) | | unsafe-record-path: | Yes — shape | packages/ingest/run.js, writeRecord() catch | The record passed schema validation but computeRecordPath()'s traversal guard (isUnsafeSegment, stricter than the schema's repo pattern — e.g. ../etc) still refused to place it on disk. The submitter's own repo string is the problem; nothing is persisted (there is no safe path to write to). |

Operational — class: 'operational', always retryable: false (the validator itself threw an internal error; investigate the verifier, do NOT bounce to the submitter):

| Prefix | Source | Meaning | |---|---|---| | VALIDATOR_FAULT_SCHEMA: | runValidator('schema', …) catch | Internal exception inside the schema validator. The rest of the string carries the thrown .message. | | VALIDATOR_FAULT_POLICY: | runValidator('policy', …) catch | Internal exception inside the policy validator — including a GLOBAL declarative-rule predicate fault (VERIFY-F1): a broken policies/global-policy.yaml is an ops incident (the studio's own config), so its predicate fault throws here rather than bouncing to the submitter. The repo-authored counterpart is policy-config: submission-bad. | | VALIDATOR_FAULT_STEPS: | runValidator('steps', …) catch | Internal exception inside the steps validator. | | VALIDATOR_FAULT_CONTRACT_SCHEMA_VERSION: | runValidator('contract_schema_version', …) catch | The version gate was called with an unknown contract key (a programmer error at the call site, not a submission fault). | | CONTRACT_SCHEMA_TOO_NEW: | validators/schema-version.js | F-be0deacd (wave 20). The submission's schema_version declares a MAJOR above what this build supports (see SUPPORTED_SCHEMA_VERSIONS in @dogfood-lab/schemas) — THIS BUILD is behind a schema major its own submitters have already adopted. No resubmission, corrected or not, can ever satisfy a major > maxMajor comparison until testing-os itself ships an upgrade — the operator must upgrade testing-os. Page ops; do NOT bounce it back to the submitter. Unlike every other row in this table, validators/schema-version.js RETURNS this as an ordinary rejection string rather than throwing, so (unlike the VALIDATOR_FAULT_*/provenance-fault:/scenario-fetch-fault: rows) it genuinely persists to records/_rejected/ and is reachable by packages/ingest/persist.js's isRetryableRejection() — F-51780da9 (wave 22): rather than trusting the frozen retryable: false this prefix's own classification carries, isRetryableRejection() re-derives retryability for this ONE prefix against the CURRENT build's SUPPORTED_SCHEMA_VERSIONS ceiling, so a stale TOO_NEW rejection unblocks once the operator upgrades testing-os past the declared major. Distinct from the submission-bad CONTRACT_SCHEMA_TOO_OLD: above — the two prefixes are asymmetric despite the shared emitter. | | submission-malformed: | index.js null/non-object early-return | The submission itself was null or not an object — a malfunctioning dispatcher sent garbage, not a submitter who authored a bad-but-shaped payload. Page ops / inspect the dispatch pipeline; do NOT bounce it to a submitter. | | provenance-fault: | index.js provenance catch | The provenance adapter THREW an operational error confirming the run — a provider 429 rate-limit, 5xx outage, or 401/403 token fault (validators/provenance.js throws these on purpose for non-404 responses). The submitter's payload is fine; the verifier could not reach a verdict. Page ops / retry; do NOT bounce it to a submitter. Distinct from the submission-bad provenance: (genuine absence/404). | | scenario-fetch-fault: | packages/ingest/load-context.js | The scenario fetcher THREW after exhausting its retry budget (5xx/429 outage, transport reject) or hit a 401/403 credential fault loading a scenario definition. The submission may be perfectly good — the fetch infrastructure faulted. The ingest CLI lets this propagate (exit 2, nothing persisted); a true missing file is the ingest-class scenario-load: … (reason: not_found) instead. |

Any future VALIDATOR_FAULT_<NEW>: prefix is classified operational by family — parseRejectionReason matches the VALIDATOR_FAULT_ head, so a new validator class needs no parser edit. The submission-malformed: prefix is matched literally (it is not part of the VALIDATOR_FAULT_ family).

Ingest — class: 'ingest', always retryable: false (an ingest-side load fault, not a verifier gate):

| Prefix | Source | Meaning | |---|---|---| | scenario-load: | packages/ingest/run.js | A scenario referenced by scenario_results could not be loaded from the source repo (typed-reason: timeout / not_found / parse_error / invalid_id / too_large / schema_invalid). Outages and credential faults are NOT this class — they throw scenario-fetch-fault: (operational, above) instead of rejecting the submission. |

Operator hygiene

Discriminate by class, not by hand-rolled .startsWith() chains. parseRejectionReason(reason) returns { class, prefix, detail, retryable }:

import { parseRejectionReason } from '@dogfood-lab/verify';

for (const r of result.rejection_reasons) {
  const { class: cls, prefix, detail, retryable } =
    parseRejectionReason(r);
  switch (cls) {
    case 'operational':
      // Verifier-side fault. Page ops; do NOT bounce
      // back to the submitter as "fix your payload".
      notifyOps(prefix, detail);
      break;
    case 'submission-bad':
      // The payload failed a gate — surface to the
      // submitter so they fix it and resubmit.
      surfaceToSubmitter(prefix, detail);
      break;
    case 'ingest':
      // Ingest-side scenario fetch. The typed reason in
      // `detail` (timeout vs not_found/…) decides triage.
      triageScenarioLoad(detail);
      break;
    default: // 'unknown'
      // Unrecognized prefix — log + surface raw text.
      log.warn('unknown rejection_reason', r);
  }
}

Persistence note: every entry above is round-tripped verbatim through verification.rejection_reasons in the persisted-record JSON; the schema enforces array of string so any consumer of the audit-DB ground truth sees the same prefix vocabulary.

Warnings channel (accepted-with-warning)

Not every policy signal is a rejection. A policy rule declared severity: warn produces a policy: <id>: <message> entry on verification.warnings (an optional array of string on the persisted record) without flipping the verdict to rejected — the submission is accepted and recorded, the warning rides alongside it. (severity: info rules are logged only and never persisted; severity: reject rules go to rejection_reasons as above.) Consumers that want advisory signals read verification.warnings; the routing decision (parseRejectionReason) only concerns rejection_reasons. A clean accepted submission carries no warnings key at all.

Docs

📖 Full handbook: https://dogfood-lab.github.io/testing-os/handbook/

License

MIT © 2026 mcp-tool-shop