@vanmarkic/agent-verifier

Deterministic verification for code an AI agent wrote.

Conventional CI answers did the tests pass? When the same agent that writes the code can also edit the tests, that question stops being sufficient:

- expect(result).toEqual(expected);
+ expect(result).toBeDefined();

- it('adds', () => { ... });
+ it.skip('adds', () => { ... });

Both make CI greener. Neither makes the software better. This package answers a different question:

Given the change currently in this repository, is there enough deterministic evidence to accept it?

No model is involved in that verdict. Same input, same answer, every time.

Install

npm i -D @vanmarkic/agent-verifier
npx agent-verify init
npx agent-verify doctor

Use

npm run verify:fast     # integrity, format, lint, typecheck, unit
npm run verify:agent    # + coverage + production build — the bar before "done"
npm run verify:full     # + sub-threshold findings promoted to failures

agent-verify policy     # the integrity gate alone
agent-verify explain    # the single failure to fix next
agent-verify report --format=json

Exit codes: 0 pass · 1 verification failure · 3 verifier or config error · 4 infrastructure failure · 5 terminated by the supervisor.

A run that does not finish — the profile deadline expired, or you interrupted it — is never reported as a pass. It fails with VERIFICATION_INCOMPLETE and exit 5, because a partial run establishes nothing.

What it actually does

Verification integrity. Diffs the working tree — uncommitted and untracked work included — against the baseline branch and reports changes that weaken the evidence rather than satisfy it:

| Rule | Default | | --- | --- | | TEST_SKIPPED, TEST_FOCUSED, TEST_EXCLUDED | BLOCK | | COVERAGE_POLICY_WEAKENED, VERIFIER_POLICY_MODIFIED | BLOCK | | ASSERTION_DELETED, ASSERTION_WEAKENING_SUSPECTED | REVIEW | | SUPPRESSION_ADDED, TIMEOUT_POLICY_WEAKENED, BUDGET_POLICY_WEAKENED | REVIEW | | TEST_DELETED, SNAPSHOT_REPLACED, CI_MODIFIED | REVIEW | | DEPENDENCY_CHANGED | INFO |

The engine detects; your policy config decides. Findings below the failure threshold stay in the report as warnings instead of disappearing.

Process supervision. Every external command runs in its own process group under a wall-clock deadline and an idle-output deadline. On expiry the whole tree gets SIGTERM, then SIGKILL — workers included, not just the direct child. On Linux, CPU accounting separates a busy loop (CPU_RUNAWAY_SUSPECTED) from a stall (PROCESS_HANG); elsewhere the process still dies, the cause is just named less precisely. A timeout is never reported as a proven infinite loop.

Local Angular 21 gates. Format, lint, tsc --noEmit, Vitest, and ng build --configuration production, each supervised, each normalized into the same failure shape. Set any command to null to disable its gate.

Coverage, twice. A global floor and a floor on the lines this change added — the second is what notices a hundred new untested lines dropped into a repository already sitting at 85%.

One structured report. .verification/latest/report.json, schema version 1.0, kept out of git (init adds the entry, and the directory carries its own .gitignore) and never read back as part of a change, with a primaryFailure chosen so an agent fixes the compile error before it starts arguing about a skipped test. Every failure carries a stable fingerprint, so a repair loop can tell it is going in circles.

Configure

.agent-verify/config.mjs (also .js, .json, or .ts under a TS loader):

import { defineConfig } from '@vanmarkic/agent-verifier';

export default defineConfig({
  schemaVersion: 1,
  preset: 'angular-21',            // or 'generic'
  baseline: { branch: 'main' },
  coverage: { lines: 80, changedLines: 90 },
  policy: { ASSERTION_WEAKENING_SUSPECTED: 'BLOCK' },
  commands: { format: null },      // null disables the gate
});

Precedence is preset → repository config → CLI flags. A CLI flag that would loosen acceptance — a lower coverage floor, a laxer threshold — is refused and reported unless the repository sets allowCliPolicyOverride: true. Otherwise --coverage-lines=0 is all it takes for the thing under test to excuse itself.

Programmatic

import { verify } from '@vanmarkic/agent-verifier';

const { report } = await verify({ cwd: process.cwd(), profile: 'agent' });
if (report.status !== 'PASS') {
  console.error(report.primaryFailure?.summary);
  process.exitCode = 1;
}

The CLI is a renderer over this. Agents should read report.json, not scrape terminal output.

Trust model — read this before relying on it

A local run is tamper-evident, not tamper-proof. An agent with shell access as your user can edit .agent-verify/, swap dependencies, or replace the verifier itself. The integrity gate makes such edits visible; it cannot make them impossible. Neither can an agent-runner permission table, since a denied edit is defeated by a shell one-liner.

Only two things are real boundaries: a separate uid, or a container where the verifier image is fixed and the repository is the only writable mount. Until you have one of those, treat local runs as fast feedback and re-run the same profile in CI from a pinned version for an independent verdict.

Known limitations

Stated plainly, because a verifier that oversells itself is worse than none.

• Single-line scanning. String literals and line comments are excluded from pattern matching, but a directive spanning a multi-line template literal can still be misread. Regex literals naming the directives may produce a SUPPRESSION_ADDED warning.
• A config the branch itself introduced has no baseline to be lower than, so a brand-new file setting a 5% coverage floor is not reported as weakened. Changing an existing one is.
• Assertion weakening is a heuristic, defaulting to REVIEW. The same diff shape occurs in legitimate refactors. Raise it to BLOCK only once you have seen its false-positive rate on your own repository.
• Changed-line coverage needs istanbul JSON output (coverage-final.json). Without it that gate is skipped, and says so. The --coverage flags are only appended to a direct vitest invocation; behind an npm run wrapper they would not reach the runner, so configure the reporters in vitest.config.ts.
• CPU classification is Linux-only. Elsewhere, hangs are still terminated; only the reason is coarser.
• Not in this release: browser/runtime verification, memory and leak analysis, risk-based gate selection, affected-test selection, and the repair orchestrator. Commands for them are absent rather than stubbed.

Releasing

Releases run from .github/workflows/agent-verifier-release.yml, either by pushing an agent-verifier-v* tag or by dispatching the workflow from master. package.json is the single source of truth for the version.

Publishing authenticates with npm trusted publishing (OIDC): the workflow proves its own identity to the registry and no NPM_TOKEN is involved. npm only lets you register a trusted publisher for a package that already exists, so the very first version has to be published by hand. That is a one-time cost.

One-time setup, in order:

1. Publish 0.1.0 from a local checkout, with 2FA:

   npm login
   cd packages/agent-verifier
   npm install --no-workspaces --legacy-peer-deps
   npm publish --access public   # prompts for your 2FA code

2. On npmjs.com, go to Packages → @vanmarkic/agent-verifier → Settings → Trusted publishing, and add a GitHub Actions publisher:

   | Field | Value | | ----------------- | --------------------------------- | | Organization/user | vanmarkic | | Repository | lagendwa | | Workflow filename | agent-verifier-release.yml | | Environment | (leave empty) |

3. Delete the NPM_TOKEN repository secret. Once the trusted publisher exists the workflow no longer reads it, and a token that is not there cannot leak.

Every release after that is just: bump package.json, merge, dispatch the workflow (or push the tag).

If you cannot do step 1 right now, the workflow still falls back to NPM_TOKEN — but it must be a granular access token with both read/write on the @vanmarkic scope and "Bypass 2FA" enabled. A token missing the scope fails with 404 Not Found - PUT; a token missing the 2FA bypass fails with EOTP. Both of those were hit before trusted publishing was set up, and the publish step now prints the corresponding fix. npm removes direct publishing for these tokens in January 2027, so this path is a stopgap.

Versioning

Failure codes and the report schema are public API. Any release that can change an existing repository's PASS/FAIL outcome says VERDICT_BEHAVIOR_CHANGE in its notes, whatever the semver bump.

License

MIT