VerifyFlow

Evidence-backed delivery verification for Linear-driven pull requests.

VerifyFlow answers one question:

Did this PR actually deliver what the linked ticket asked for?

It is not code review. Crosscheck reviews code and merge readiness; VerifyFlow runs after that, executes the ticket acceptance criteria, captures evidence, and reports the delivery verdict.

Linear issue + GitHub PR -> execution evidence -> delivery verdict

The CLI is verifyflow, with the short alias vf.

Quickstart

The fastest way to see a full run — no credentials, no checkout, fully offline:

npx github:humanbased-ai/verifyflow demo

vf demo runs bundled fixtures through the whole pipeline and writes a report you can read.

Then check your environment and (optionally) get a guided setup:

vf doctor      # are gh / claude / LINEAR_API_KEY / Playwright / a sandbox runtime ready?
vf onboard     # guided first-run setup; prints the exact fix command for anything missing

vf onboard can save your Linear key to ~/.verifyflow/credentials.json (mode 0600). At runtime LINEAR_API_KEY is resolved from the environment first, then that credentials file — so you never have to export it again.

Install

After the first npm release:

npm install -g @humanbased/verifyflow
vf doctor

Before npm publish, run straight from GitHub:

npx github:humanbased-ai/verifyflow doctor
npx github:humanbased-ai/verifyflow run \
  --fixtures fixtures/example-cli \
  --linear EX-1 \
  --pr example/greet#7 \
  --level functional

Verify a PR — vf run

vf run \
  --linear IN-123 \
  --pr humanbased-ai/monorepo#456 \
  --level auto \
  --checkout \
  --comment

• --linear is optional: if omitted, VerifyFlow derives the issue from the PR body's Linear link or the branch name.
• --checkout clones the repo and checks out the PR head for real execution. Or point at an existing checkout with --workdir <dir>, or run offline with --fixtures <dir>.
• --comment posts the markdown report as a PR comment (idempotent — updates in place).
• --linear-writeback also posts the delivery verdict back to the linked Linear issue.

Preview what VerifyFlow would do without checking out or executing anything (exits 0):

vf run --linear IN-123 --pr humanbased-ai/monorepo#456 --level auto --dry-run

Verify a PR that has no resolvable ticket, against its own description (verdict capped at manual_review_required):

vf run --pr humanbased-ai/monorepo#456 --allow-no-ticket

Levels — --level

| Level | What it does | Needs | | --- | --- | --- | | functional | Command/test probes against a checkout | checkout/workdir | | ui | AI-driven browser checks via Playwright | Playwright + a running app | | journey | Multi-step end-to-end (backend + browser) | checkout + Playwright | | auto | Picks the level from the ticket; downgrades to functional if no browser is available, and says why | — |

For ui / journey, point at the app with --base-url <url> (otherwise VerifyFlow tries to find a deployment preview from the PR's checks), and supply an authenticated session with --ui-auth <storageState.json> (create it with playwright codegen --save-storage=auth.json).

Merge policy — --policy

| Policy | Behavior | | --- | --- | | advisory | Default. Reports only, never blocks. | | merge_gate | Exits non-zero on needs_fix, so a failing verdict blocks merge. | | strict | Also blocks on manual_review_required / accept_with_risks. |

Other commands

| Command | What it does | | --- | --- | | vf run | Verify one PR against its Linear ticket (see above). | | vf step | Orchestrator-facing step (Symphony/Jazzband): advisory-only, auto-resolves the issue, checks out + executes + comments, and prints one machine-readable JSON line to stdout. Never blocks. | | vf watch | Independent daemon: watch a repo's Crosscheck-approved PRs, verify delivery, and (with --auto-merge) squash-merge on a clean accept. | | vf report | Aggregate accumulated runs into quality metrics; --trend, --since, --repo, --level, --json filters. | | vf replay <runId> | Re-run the verdict engine against a past run's stored evidence — no probes/tests re-execute. | | vf show <runId> | Re-render a past run's report.md (or report.json). | | vf signal <runId> | Pretty-print a past run's improvement-signal. | | vf memory | Inspect reusable test-point memory: vf memory ls, vf memory show <key>, vf memory clear [--repo <o/r>] [--yes]. | | vf init | Scaffold a verifyflow.config.json in the target repo (auto-detects npm / uv / go / cargo / make). | | vf doctor | Check that the tools/env VerifyFlow relies on are ready. | | vf onboard | Guided first-run setup; --non-interactive to skip prompts. | | vf demo | Offline demo with bundled fixtures; --open to open the report. |

Full flag-by-flag reference: docs/commands.html. Run vf --help for the same usage in the terminal.

Watch a repo's Crosscheck-approved PRs:

vf watch --repo humanbased-ai/monorepo --interval 120              # monitor + verify + comment
vf watch --repo humanbased-ai/monorepo --auto-merge --interval 120 # also squash-merge on accept

What works today

• One Linear issue against one GitHub PR.
• Acceptance-criteria extraction from the Linear issue (or the PR description with --allow-no-ticket).
• functional, ui, journey, and auto levels.
• Real command/test execution against a checkout or workdir.
• Browser-backed UI and journey checks through Playwright.
• Markdown and JSON reports, screenshots, traces, command logs, and reusable test memory.
• Idempotent PR comments and optional Linear writeback.
• advisory / merge_gate / strict merge policies.
• Standalone CLI, orchestration step, and Crosscheck-approved watcher modes.
• Colorized terminal output in an interactive shell (auto-disabled when piped or under NO_COLOR).

VerifyFlow is conservative: uncertainty becomes blocked or not_evaluable, not a fake product failure.

Requirements

VerifyFlow stores no secrets. It reuses local tools and environment variables.

| Tool | Needed for | Required? | | --- | --- | --- | | Node.js >= 20 | CLI runtime | required | | gh authenticated | GitHub PR context and comments | required | | LINEAR_API_KEY (env or ~/.verifyflow/credentials.json) | Linear issue reads | required (or use --fixtures) | | claude authenticated | LLM planning and judging; otherwise rules-only fallback | optional | | docker or podman | Sandbox isolation for executing PR code (IN-555); without it, probes run on the host | optional | | Playwright | ui and browser-backed journey runs | optional | | npm, uv, etc. | Target repo setup and tests | as the target needs |

Roadmap

Next major layer: project-level verification.

Linear project -> ticket/PR matrix -> leveled runs -> evidence bundle -> project report

Planned work:

• Read a Linear project and map tickets to PRs.
• Generate a coverage matrix across tickets, criteria, PRs, SHAs, and evidence.
• Run per-ticket functional/UI/journey checks.
• Produce one project-level implementation gap report.
• Keep stronger sandbox isolation for untrusted PR execution.
• Add opt-in Linear status transitions and follow-up issue filing.

Boundaries

VerifyFlow does not:

• review code or decide merge readiness; that is Crosscheck
• decompose tickets or dispatch coding agents; that is Symphony today and Jazzband next
• move money, send production email, or perform irreversible side effects
• replace CI
• provide native mobile evaluation yet

Toolchain fit

Symphony / Jazzband -> Crosscheck -> VerifyFlow
orchestration          code review    delivery verification

Symphony is the current Python orchestration layer. Jazzband is the planned open TypeScript/npm successor. VerifyFlow works with both, and also runs alone, by using public artifacts: GitHub PR metadata, Linear issue links, SHA-bound comments, CLI JSON, and evidence files.

More detail:

• Command reference
• End-to-end auto evaluation
• Coordination contract
• Publishing
• UI level

Development

npm install
npm run typecheck
npm test
npm run build
npm pack --dry-run