@verica-app/cli

Run a Verica eval from CI and gate the merge on the result.

The eval — criteria, golden set, graders, and the pass condition — lives in Verica. The CLI just triggers a run, waits for the verdict, writes a JUnit report, and exits 0/1 so your pipeline can block the merge. It's a thin HTTP client: the only secret it needs is a workspace token (provider keys stay in Verica via BYOK).

Install

No install needed — run it with npx:

npx @verica-app/cli run --eval <eval-id> --prompt prompts/agent.txt --junit report.xml

Or add it as a dev dependency:

npm i -D @verica-app/cli

Usage

verica run \
  --eval eval_8x2k9d \
  --prompt prompts/support-agent.txt \
  --system-prompt prompts/support-agent.system.txt \  # optional
  --tools prompts/support-agent.tools.json \          # optional
  --model gpt-4.1-mini \                              # optional — defaults from the eval's last run
  --junit verica-results.xml \
  --json

The run waits for the verdict and gates the exit code by default — add --no-wait to fire-and-forget instead. Multi-prompt via a manifest:

verica run --manifest .verica.yml --junit report.xml

# .verica.yml
evals:
  - id: eval_8x2k9d
    prompt: prompts/support-agent.txt
    systemPrompt: prompts/support-agent.system.txt
    tools: prompts/support-agent.tools.json # a path to a JSON file…
    sampling: { temperature: 0.2, maxTokens: 512 }
    model: gpt-4.1-mini
  - id: eval_3p1m7q
    prompt: prompts/triage.txt
    tools: # …or an inline array
      - name: get_order
        description: Look up an order by id
        parameters: { type: object, properties: { id: { type: string } }, required: [id] }
    model: claude-sonnet-4-6

Environment

| Var | Required | Notes | | -------------- | -------- | -------------------------------------------- | | VERICA_TOKEN | yes | Workspace API token (Settings → API tokens). |

VERICA_TOKEN is the only thing you set. The CLI talks to the hosted Verica API by default — you don't configure a URL.

Key flags

• --eval <id> / --manifest <file> — what to run.
• --prompt <file> / --system-prompt <file> / --tools <file> — prompt content to push (versioned by content). See Prompt content.
• --model <model> · --sampling <file.json> — execution config. --model is optional: omit it and the server defaults to the eval's most recent run's model.
• --no-wait — fire-and-forget: trigger the run and exit 0 without waiting. By default the run waits and gates the exit code; --wait is accepted but inert (it's already the default).
• --junit <file> · --junit-mode rows|gate — JUnit report (default rows).
• --json — machine-readable results on stdout.
• --threshold <0..1> · --baseline-ref <ref> · --baseline-run <id> — override the gate per branch.
• --reuse-if-unchanged · --reuse-max-age <hrs> · --reuse-same-ref — reuse a recent completed run instead of re-executing an unchanged config. See Reuse.
• --git-sha / --git-ref / --git-repo-url — provenance + the repo web base that links the SHA in the run UI (all auto-detected from CI env otherwise).

Local dev / self-hosting only: point the CLI at another instance with --base-url (or the VERICA_BASE_URL env var). Clients never need this.

Prompt content (what you push)

The repo owns the prompt: the message chain (--prompt), the system prompt (--system-prompt), and the tool definitions (--tools). The dataset, graders, and gate stay in Verica — they're the test scenario, managed by whoever owns the eval.

Each of the three prompt fields is independent and optional: push the ones you changed and every omitted field is inherited from the current version. A push creates a new prompt version only if the merged content actually differs.

--prompt reads a file whose contents become the initial user turn: a single-turn chain that replaces the version's chain as a unit. Multi-turn / simulated-tool chains are authored in Verica; omit --prompt to inherit them unchanged (e.g. push only --system-prompt).

verica run --eval eval_8x2k9d --system-prompt prompts/agent.system.txt --model gpt-4.1-mini
# message chain + tools inherited; only the system prompt re-versions

The prompt references dataset columns by name — e.g. What is the capital of {{ pais }}? (the column is pais). Grader/judge prompts can also reference the model output via {{ output.text }} / {{ output.tool_calls }}.

Tools are pushed as JSON — a --tools <file>, or a path / inline array under tools: in the manifest. Each entry may be Verica's flat shape or the OpenAI wrapper (auto-unwrapped), so you can paste your real schemas as-is:

[
  {
    "name": "get_order",
    "description": "Look up an order by id",
    "parameters": {
      "type": "object",
      "properties": { "id": { "type": "string" } },
      "required": ["id"]
    }
  },
  {
    "type": "function",
    "function": { "name": "cancel_order", "description": "…", "parameters": { "type": "object" } }
  }
]

Tools are never executed — the model's decision to call one (and with which arguments) is what the eval grades.

Reuse (skip re-running an unchanged config)

By default every verica run executes — re-running an unchanged eval is often the point in CI (it catches silent model drift and run-to-run variance, since an eval's output isn't a pure function of its inputs). When you'd rather save the tokens, opt in with --reuse-if-unchanged:

verica run --eval eval_8x2k9d --model gpt-4.1-mini --reuse-if-unchanged
# if the same config ran & completed in the last 24h, returns that verdict — no new run

"Unchanged" means the prompt version + model + sampling + dataset snapshot + graders all match a prior run (the gate is not part of it — it decides the verdict, not the output). On a hit the CLI exits on the prior run's frozen verdict and the --json element carries "reused": true plus a reusedFrom block (the API also answers 200 instead of 202).

• --reuse-max-age <hrs> — how stale a reusable run may be (default 24, max 720). There is no "forever": reuse can't see provider-side drift behind a stable model id, so it's always bounded — that bound is your staleness budget.
• --reuse-same-ref — only reuse a run on the same git ref. Off by default: an identical config produces the same output distribution regardless of branch.
• Only completed runs are reused (never a partial/failed one).
• Incompatible with --threshold / --baseline-ref / --baseline-run. Reuse hands back a prior run's verdict, frozen under the gate that applied when it ran, so a new --threshold can't be recomputed against it. --baseline-ref is worse than stale: no-regression compares against the last run on the ref — a moving target — so a cached verdict can never be a fresh no-regression check. Gate on either → run fresh (omit reuse).

Omit --reuse-if-unchanged (the default) any time you want a guaranteed fresh run.

Exit codes

0 passed · 1 gate failed · 2 validation/transport error.

Stability

This CLI is pre-1.0 (0.x). The command surface, the --json payload, the JUnit output, and the prompt-push behavior are still settling and may change. Exit codes (0/1/2) are stable.

During 0.x the minor version is the breaking lever, so pin accordingly:

// package.json
"@verica-app/cli": "~0.1"   // >=0.1.0 <0.2.0 — gets patches, not breaking minors

We bump the minor for any breaking change (flags, output shapes, push behavior) and the patch for additive features and fixes. 1.0 will freeze the commands, flags, exit codes, and output shapes under standard semver. See the bundled CHANGELOG.md for what changed in each release.

MIT licensed. There's no IP in the client — the engine, graders, gate, and crypto all run server-side behind the token API.