VerdictCI

Fail pull requests when AI behavior regresses.

VerdictCI runs prompt, agent, and RAG evals in CI, writes a stable result JSON artifact, renders a readable Markdown summary, and exits non-zero when configured quality gates fail.

Install

Requirements: Node.js 20 or newer.

npm install -D @syntaxname/verdictci
npx verdictci --help

Run without adding a dependency:

npm exec --package @syntaxname/verdictci -- verdictci --help

CLI quick start

Create verdictci.yaml:

suites:
  - id: support-bot
    fixture: examples/support-bot/fixtures/pass.json
    thresholds:
      passRate: 1
      maxFailures: 0

Run VerdictCI:

npx verdictci run --config verdictci.yaml --output verdictci-result.json

Useful options:

npx verdictci run \
  --config verdictci.yaml \
  --output verdictci-result.json \
  --summary verdictci-summary.md \
  --fixture-mode

Exit codes:

| Code | Meaning | | ---: | --- | | 0 | Required eval checks passed. | | 1 | Required eval checks ran but failed thresholds. | | 2 | Usage or configuration error. | | 3 | Provider, adapter, or runtime execution error. | | 4 | Unexpected internal error. |

GitHub Action

name: VerdictCI

on:
  pull_request:

jobs:
  verdictci:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v6
      - uses: syntax-dot/[email protected]
        with:
          config: verdictci.yaml
          output: verdictci-result.json
          summary: verdictci-summary.md
          upload-artifact: "true"

The Action publishes the Markdown summary to the GitHub Actions job summary and uploads the result JSON when upload-artifact is enabled.

Demo

Clone the repo and install dependencies:

pnpm install

Run the passing fixture:

pnpm verdictci run --config examples/support-bot/verdictci-pass.yaml --output .tmp/pass-result.json --fixture-mode

Run the failing fixture:

pnpm verdictci run --config examples/support-bot/verdictci-fail.yaml --output .tmp/fail-result.json --summary .tmp/fail-summary.md --fixture-mode

The failing fixture exits 1, writes .tmp/fail-result.json, and creates a Markdown summary that starts with # VerdictCI: failed.

See the full walkthrough in docs/demo/public-demo.md.

Public docs

• CLI spec
• Config YAML spec
• Result JSON spec
• Exit codes
• GitHub Action spec
• PR summary spec
• Eval templates

Privacy

VerdictCI runs locally or in your GitHub Actions environment by default. The MVP does not upload prompts, eval cases, model outputs, traces, provider keys, or result artifacts to a hosted VerdictCI service.

Contributing and security

• Contributing
• Security policy
• Changelog
• License