@systemproof/core

Deterministic tooling for the portable QA/e2e harness — the reference implementation of the file contracts in docs/skills/qa-harness (QADATA dashboard data, flow graphs, environment contract, scenario/verdict frontmatter, campaign manifest) plus the qa CLI that executes them.

Use

CLI (the supported invocation contract):

pnpm --filter @systemproof/core run qa -- <command> [args]
pnpm --filter @systemproof/core run qa -- env check --file eval/catalog/environments.yaml

Library:

import {
  assembleQadata,
  stableStringify,
  qadataSchema,
  checkEnvironments,
} from '@systemproof/core';

const qadata = assembleQadata('qa/runs/camp-20260701-0900');

CLI capabilities

| Command | What it does | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | qa dashboard | QADATA → static self-contained dashboard.html for one campaign; --demo writes synthetic sample data | | qa flow lint | validate flows/*.flow.yaml against schema + lint rules | | qa flow materialize | named flow paths → canonical scenario files | | qa book lint | validate an Action Book (books/*.book.yaml): structured grounding, state resolution, provenance, --flow reachability | | qa book materialize | cross-check a book's action coverage of a flow's edges; reports GAPs | | qa codify | compile runs/<campaign>/<flow>/trace.json × a scenario oracle into a book-grounded Playwright spec (zero literal selectors, byte-identical on re-run); refuses a no-oracle journey, --strict fails on a codify gap | | qa env check | validate catalog/environments.yaml (access tiers + capabilities), list grants per env | | qa findings lint | validate findings.md (F-NNN id, HARNESS\|ENV\|APP attribution, non-empty status); out-of-order ids warn, duplicates fail | | qa task validate | validate a task.json agent-lane work order against the task contract | | qa report audit | re-grade a runs tree strong\|partial\|weak\|failed\|unknown; exit 1 if any run grades failed/weak (--allow-weak drops the weak gate), --json for the payload | | qa campaign plan | capability-gated, ordered scenario selection for a target environment | | qa campaign report | verdicts → manifest.json → QADATA dashboard |

Exit codes are the contract: 0 ok, 1 contract or harness violation, 2 usage error. qa report audit reads its floor from the grade counts: 0 when nothing grades failed/weak (or only weak under --allow-weak), 1 otherwise.

Exports

• Zod schemas + inferred types for every file contract: qadataSchema (runs/bursts/suites/step keyframes), traceFileSchema, flowSchema, environmentsFileSchema, scenarioFrontmatterSchema, verdictFrontmatterSchema, manifestSchema
• assembleQadata(campaignDir) — runs/<campaignId>/ tree → QADATA object
• stableStringify(value) — sorted-key, 2-space-indent, trailing-newline JSON (golden-file/byte-determinism format)
• checkEnvironments(file) — environment-contract check as a library call
• extractFrontmatter, parseScenarioFrontmatter, parseVerdictFrontmatter
• findingSchema + parseFindings(markdown) — the F-NNN findings register contract + heading parser; formatFinding / nextFindingId emission helpers
• ContractViolationError / parseContract — typed violations naming file + field

Campaign manifest (runs/<campaignId>/manifest.json)

The assembler's input, written by the campaign runner:

{
  "campaignId": "camp-20260701-0900",
  "project": "shop-demo",
  "generatedAt": "2026-07-01T09:00:00Z", // copied into QADATA verbatim
  "flows": [
    {
      "flowKey": "checkout-happy", // = the flow's directory name
      "title": "…",
      "suite": "shop",
      "kind": "e2e", // e2e | actionbook | burst | eval
      "status": "passed", // passed | failed | flaky | error
      "retries": 2,
      "durationMs": 8400,
      "surface": "browser",
      "burst": 3, // burst flows fold into QADATA bursts[]
      "instances": [{ "id": "i1", "status": "passed", "failAt": 4200 }],
      "findings": ["human-readable line"],
      "artifacts": {
        "video": "…",
        "screenshots": ["…"],
        "trace": "checkout-happy/trace.json",
      },
    },
  ],
}

Assembler rules:

• e2e/actionbook flows must have <flowKey>/trace.json; a missing or unreadable trace forces the row to status: "error" with a human-readable findings line — never a silent omission, even if the manifest claimed passed.
• eval rows carry no trace (their truth is the verdict); status/findings come from the manifest verbatim.
• Burst instances are manifest-declared; per-instance steps are read from <flowKey>/<instanceId>/trace.json when present.
• Suites rollup is computed from runs[]; error rows count as failed (an error row never lets a campaign look green).
• Artifact paths are relative to the campaign directory and preserved as data, but dashboard media links allow only relative or http(s) URLs.
• Step at offsets come from trace at or cumulative durationMs — never the wall clock.

Queue layout (qa/runs/queue/)

The agent-lane bus is the filesystem. One directory per task:

qa/runs/queue/
  <taskId>/
    task.json       # work order (task contract), written by qad or a human — in
    report.json     # runner result (report contract), written by the runner — out
    <artifacts>     # trace.json, shots/*, book-draft.yaml, findings.patch.md, …

A runner (qa-agent or a fallback session) claims a task, does the work, and writes report.json + artifacts only under its own task dir (plus qa-root drafts). Same report.json schema regardless of runner, so the daemon validates results identically. Path resolvers (queueDir, taskDir, taskPath, reportPath) are exported for tooling; validate a work order with qa task validate <file>.

No executor lives here — these are the contracts + layout only. The qad Go daemon (crash-safe claim/heartbeat/reclaim, cron/watch, agent-lane supervision) lands in Phase 3 and validates queue files against the generated JSON Schemas.

Report audit (qa report audit [runsDir])

The meta-grader (ported from fheswagshop's report-audit, adapted to our contracts) re-grades a runs tree strong | partial | weak | failed | unknown, one row per campaign-manifest flow and per agent-lane report.json. It keeps fheswagshop's grading heuristics and the hollow-pass rule: a green UI verdict whose logs/oracles show trouble (browserSignalIssues, backend 5xx/level=error) — or which captured no evidence (no trace/screenshots/video) — grades weak, not strong. Discovery is by our artifacts (manifest.json via manifestSchema, report.json via reportSchema) instead of fheswagshop's result.json/ summary.json. Output (the table and the --json payload) is byte-deterministic — no clock, no LLM, golden-tested — so it can gate CI. auditRuns(runsDir) is exported for tooling.

Findings register (qa/findings.md)

The QA findings log is heading-based (matched against the real susume-poc artifact, not the front-matter blocks the impl sketch guessed at). One finding per level-2 heading:

## F-001 — Burst video recording destabilizes the consistency gate — HARNESS, FIXED

- **Surface:** …
- **Disposition:** video is off by default for the burst
- **Status:** ✅ Fixed

## F-NNN — <title> — <ATTRIBUTION>, <STATUS>, where ATTRIBUTION ∈ {HARNESS, ENV, APP} and STATUS is free text (FIXED, OPEN (low), MITIGATION PROPOSED, …). parseFindings extracts id/title/attribution/status plus the body Disposition field and any backticked run/scenario links; qa findings lint enforces a valid unique F-NNN id, a valid attribution, and a non-empty status (out-of-order ids warn but don't fail). HTML-comment templates are ignored. formatFinding/nextFindingId emit new contract-valid findings for the explore/judge lanes.

Action Books (books/*.book.yaml)

The durable cache between exploration and replay (design doc §3): per-surface, versioned YAML that names a surface's states (identified by a11y anchors) and the intent-grounded actions between them. Codified specs never contain literal selectors — they call book.action('<book>/<action>'), so healing the book once fixes every test. The anti-brittleness rule is structural, not conventional:

• Grounding is a structured locator — {role, name} | {testId} | {label}, a strict union with no css/xpath/nth variant, so ordinal/DOM-ancestry grounding is unrepresentable (fails schema validation by construction).
• Anchors are strict — an a11y fact ({role, name?, level?}) that identifies a state; a smuggled css/xpath key is rejected.
• invariants[].expect is kept loose for now (a bare record); the precise oracle-expect contract lands in Phase 2 (P2.2).

qa book lint <file...> [--qa-root <dir>] [--flow <flow.yaml>] is deterministic (no browser/clock/LLM) and enforces: schema, every action from/to resolving to a declared state (dangling-state), no positional/ordinal grounding names (ordinal-grounding), book-level provenance present (no-provenance), and — with --flow — every state reachable from the flow's entry (unreachable-state, reusing the flow linter's reachability). Orphan states, an unverifiable auth fixture, and a missing --flow warn (never gate). Exit 0/1/2 like the other linters. loadBookFile(file) and lintBookFile(file, opts) are exported.

qa book materialize --book <id> --flow <flow.yaml> [--qa-root <dir>] is an advisory, byte-deterministic coverage cross-check: each flow edge (user action) maps to the book action bridging that state pair, and edges with no book action are reported as GAPs. Exit 0 always unless an input fails to parse.

Environment superset (P0.5)

environments.yaml entries may carry optional superset fields (old bare-scalar catalogs still parse unchanged):

• kind: live | owned | sim
• taps: { <name>: <tap> } — log-oracle taps, discriminated on type (file-ndjson, command, kubectl, http-json, http-ndjson); string fields may hold a literal {{runId}} template (interpolated later by qa-oracles).
• correlate: { header?, env? } — the runId correlation carrier.
• load: { maxVus?, maxDurationS? } — fail-closed load caps.
• artifacts: { burstHeavyOptIn? } — default true; read via resolveBurstHeavyOptIn(env) (a getter, never a mutation).
• Per-value env-var overrides (fheswagshop pattern): a field <name> may carry a sibling <name>Env key naming an env var; applyEnvOverrides(env) returns a copy with process.env[<name>Env] substituted when set and non-empty.

Conventions

• Deterministic core, LLM edges: no LLM calls anywhere in this package; explore/judging/dispatch stay with the qa-* skills and sessions.
• Byte-identical generation: no Date.now()/randomness on any path that lands in generated output; all timestamps come from inputs; goldens enforce it.
• No app-specific logic: contracts and generation only; each app keeps its own runners (qa-run execution half stays with the app).
• Contracts here mirror docs/skills/qa-*; if implementation forces a contract change, update the skill in the same PR.