qa-runner

AI-powered visual QA automation. Write tests in plain English, let Claude drive the browser, catch the bugs, and explain what went wrong.

qa-runner reads .qa.md test files (Markdown + a little YAML), drives a real browser via Playwright, and uses Claude to evaluate assertions against the live page — no CSS selectors, no XPath, no test code. It produces a rich HTML report with per-step pass/fail, anomaly screenshots, accessibility violations, and visual-regression diffs.

---
suite: Login Flow
base_url: https://app.example.com
---

## Test: Valid Login
- Navigate to /login
- Fill email: [email protected]
- Fill password: secret
- Click: Sign In
- Assert: Dashboard is visible
- Assert: User name shown in header
- Screenshot: post-login

That's the entire test. No selectors. No code.

Table of contents

• Features
• How it works
  • Architecture
  • Execution flow
  • The AI evaluation pipeline
• AI provider: current state & future direction
• Getting started on a fresh machine (end-to-end)
• Writing tests — the .qa.md format
• CLI reference
• Using qa-runner with a coding agent (MCP)
• Device profiles
• Reports
• Optional: web dashboard
• Project structure
• Development
• Environment variables
• Roadmap / next steps
• Known limitations
• License

Features

• Plain-English tests — natural-language steps in Markdown, a YAML config header, zero programming.
• AI page evaluation — Claude interprets the page's accessibility tree + a distilled DOM element layer to judge assertions, instead of brittle selectors.
• Anomaly detection — Claude flags visual/functional issues independently of pass/fail.
• Self-healing — on a failed step, Claude diagnoses the page and proposes a corrected step (auto-applied or confirm-first).
• Multi-device matrix — run one test across Desktop, iPhone, Pixel, and iPad in a single command.
• Accessibility checks — axe-core WCAG 2 A/AA scan; violations surface in the report.
• Visual regression — pixel-level diffs (pixelmatch) against stored baselines.
• Network emulation — throttle a suite to slow-3g or offline.
• Deterministic login + auth caching — declare login in auth:; handles single-form and email-first flows, learns the landing URL, and caches the session across suites/runs. qa-runner login captures SSO/OTP sessions by hand.
• Test generation & recording — generate a .qa.md from a URL, or record browser actions into one.
• Watch mode — re-run on file change with a debounce for tight authoring loops.
• Slack / Teams alerts — post pass/fail summaries to a webhook.
• Rich reports — HTML (default), JSON, or Markdown.

How it works

Architecture

The orchestrator only ever talks to the IBrowserEngine contract and the AI client — it never imports Playwright or the AI SDK directly. This is what makes the browser engine and (in future) the AI provider swappable.

flowchart TD
    CLI["cli.ts<br/>(commander)"] --> Runner["runner.ts<br/>suite + device orchestration"]
    Runner --> Parser["parser/<br/>.qa.md → TestSuite"]
    Runner --> Engine["IBrowserEngine"]
    Runner --> Orch["orchestrator/<br/>ClaudeRunner"]

    Engine --> PW["PlaywrightEngine<br/>(Chromium)"]
    Engine -.future.-> Chrome["ChromeEngine<br/>(stub)"]

    Orch --> AIClient["anthropic-client.ts<br/>(AI abstraction)"]
    AIClient -->|"ANTHROPIC_API_KEY set"| SDK["@anthropic-ai/sdk<br/>(hosted API)"]
    AIClient -->|"no key (default)"| CC["claude CLI subprocess<br/>(local Claude Code)"]

    Orch --> Healer["self-healer.ts"]
    Runner --> Reporter["reporter/<br/>HTML · JSON · Markdown"]
    Reporter --> Report["qa-report.html"]

Execution flow

flowchart LR
    A["📄 Parse .qa.md<br/>YAML + steps"] --> B["🖥️ Resolve devices<br/>desktop / mobile / tablet"]
    B --> C["🌐 Launch browser<br/>Playwright Chromium"]
    C --> D["▶ Execute step<br/>navigate · click · fill · scroll"]
    D --> E{"Assert<br/>step?"}
    E -->|no| D
    E -->|yes| F["📸 Capture page snapshot<br/>ARIA + DOM elements"]
    F --> G["🤖 Claude evaluates<br/>pass / fail / anomaly"]
    G --> H{"Failed?"}
    H -->|yes| I["🔧 Self-heal<br/>diagnose + retry"]
    H -->|no| D
    I --> D
    D --> J["📊 Report<br/>HTML + JSON + anomalies"]

Key performance behaviours:

• Pipelined evals — the browser action runs immediately; for Assert steps the page snapshot is captured and sent to Claude in the background, so the next browser step starts without waiting for the API.
• Batch asserts — consecutive Assert steps share one snapshot and one Claude call.
• Model tiering — Haiku for fast ARIA/DOM-logic asserts; Sonnet for visual anomaly detection and failure diagnosis.
• Prompt caching — the stable system prompt is cached (5-min TTL) so repeat calls are cheaper.

The AI evaluation pipeline

For each assertion, qa-runner captures an enriched page snapshot and feeds it to Claude. The snapshot is the ARIA accessibility tree plus a compact, curated list of interactive DOM elements (selectors like id / class / data-testid, form values, visibility, and disabled/checked state) — richer than ARIA alone, which matters because some elements are hard to identify from the accessibility tree. The snapshot is cached and reused until a DOM-mutating action invalidates it.

sequenceDiagram
    participant O as ClaudeRunner
    participant E as PlaywrightEngine
    participant Cache as Snapshot cache
    participant AI as AI client

    O->>E: getPageSnapshot()
    alt cache clean
        E->>Cache: return cached snapshot
    else cache dirty
        E->>E: ariaSnapshot() + single page.evaluate() DOM walk
        E->>Cache: store (mark clean)
    end
    E-->>O: { url, title, ariaTree, elements[] }
    O->>AI: system prompt (cached) + snapshot + assertion(s)
    AI-->>O: JSON { status, explanation, anomaly, ... }
    Note over O,E: next click/fill/navigate/etc. → invalidate cache

Assert: page passes WCAG AA is special-cased: instead of Claude, it injects axe-core and runs a WCAG 2 A/AA audit, failing on serious/critical violations.

AI provider: pluggable

qa-runner talks to the model through a small pluggable interface (AiProvider in src/ai/types.ts) — the same way IBrowserEngine makes the browser swappable. Every call site funnels through one facade (callAi / callClaudeApi in src/ai/provider.js), so the vendor is configurable without touching the orchestrator. Pick a provider with --provider, the QA_RUNNER_PROVIDER env var, or the companion UI Settings page (persists to ~/.qa-runner/config.json). The default is copilot — the no-key path most teams already have.

| Provider | id | Auth | Notes | | --- | --- | --- | --- | | GitHub Copilot CLI | copilot (default) | none — uses your local Copilot login | Drives the official copilot CLI non-interactively (prompt piped to copilot -s --model …). No API key; pick any model your Copilot plan exposes (copilot help). | | Anthropic API | anthropic | ANTHROPIC_API_KEY | Hosted Claude SDK. Fastest; reports token usage. Falls back to the local claude CLI when no key is set. | | OpenAI-compatible | openai | OPENAI_API_KEY (+ optional OPENAI_BASE_URL) | Any OpenAI-compatible endpoint — OpenAI, Azure OpenAI, GitHub Models, or a local model. | | Claude Code CLI | claude-code | none — uses your local Claude Code login | Drives the local claude CLI subprocess. |

qa-runner run --provider openai .          # OpenAI key path
qa-runner run --provider copilot .         # no key — your local Copilot CLI

Notes & caveats. CLI providers (copilot, claude-code) report no token counts (the CLIs don't expose them) and spawn one process per call. Screenshot-based checks (--visual, failure diagnosis, self-heal, generation) need a vision-capable model; text/snapshot checks work on any model. The local claude path runs with --tools "" and the copilot path with --deny-tool 'shell,write' --no-ask-user, so neither local agent can run commands or touch files even when fed untrusted page content — they only return a text/vision answer.

⭐ Using qa-runner with GitHub Copilot (no API key)

This is the smoothest path if your team already has Copilot: no API key, no billing setup — qa-runner reuses the GitHub Copilot CLI you're already signed in to. Here's the end-to-end in a consuming app (e.g. your web project):

1. Install & sign in to the GitHub Copilot CLI (one-time, per machine)

npm install -g @github/copilot      # installs the `copilot` binary
copilot                             # launch once and complete the GitHub login
copilot help                        # (optional) lists the models your plan can use

Verify it's on your PATH and non-interactive mode works (pipe the prompt on stdin — don't use -p, which tokenizes multi-word prompts on Windows):

echo ping | copilot -s --model claude-sonnet-4.5

2. Add qa-runner to your app

npm install --save-dev @og1o/qa-runner
npx qa-runner install-browsers      # one-time Chromium download

3. Tell qa-runner to use Copilot — pick any one:

# a) per-run flag
npx qa-runner run --provider copilot

# b) environment variable (good for CI)
QA_RUNNER_PROVIDER=copilot npx qa-runner run

# c) persist it once for the project/machine
#    ~/.qa-runner/config.json
#    { "provider": "copilot", "copilotModel": "claude-sonnet-4.5" }

Or open the dashboard (npx qa-runner ui) → Settings → pick GitHub Copilot CLI (no key) and optionally set a model. The choice persists to ~/.qa-runner/config.json, and every run, the dashboard, and the MCP server then use Copilot.

4. Run a test

echo '---
suite: Smoke
base_url: http://localhost:3000
---

## Test: Home loads
- Navigate to /
- Assert: page title contains "Home"
- Screenshot: home
' > smoke.qa.md

npx qa-runner run smoke.qa.md --provider copilot

Using Copilot through an AI coding agent (MCP). If your users drive qa-runner from an agent (Claude Code, Cursor, etc.) via the bundled MCP server, register it once:

// .mcp.json (or your agent's MCP config)
{ "mcpServers": { "qa-runner": { "command": "npx", "args": ["qa-runner", "mcp"] } } }

The agent can then call the run_qa_tests tool with {"provider": "copilot"} (or rely on the configured default), and read the qa-runner://guide resource for the full syntax. Because the MCP server shells out to the same CLI, it honors the Copilot provider with no extra setup.

Notes for Copilot users. Use a vision-capable model (claude-sonnet-4.5, gpt-4o) if your tests use screenshots/visual assertions; text and snapshot assertions work on any model. copilot must be installed and signed in on the machine that runs the tests (your dev box or CI runner). This uses the official Copilot CLI as intended — no token scraping, no proxy.

Credentials & login

Logging into the site under test is uniform: secrets are referenced from .qa.md only via ${VAR} (never inlined), and login is declared once in auth: frontmatter.

1. Put secret values in a gitignored .qa/.env at your project root (or set them in the shell / CI secret store). qa-runner auto-loads .qa/.env every run. Precedence: shell env > --env file > .qa/.env. The Companion UI can edit it under Settings → Test credentials.

# .qa/.env  (gitignored — never commit real secrets)
[email protected]
QA_PASSWORD=your-password

2. Declare login in the suite frontmatter:

---
suite: Account
base_url: https://app.example.com
auth:
  loginUrl: https://app.example.com/login
  username: ${QA_USERNAME}
  password: ${QA_PASSWORD}
---

qa-runner navigates to loginUrl, detects and fills the username/password fields, submits, and caches the authenticated session (Playwright storageState, keyed by loginUrl+username) under .qa-output/sessions/. Login runs once per credential per run — every suite sharing that credential reuses the cached session, and it's auto-refreshed when it goes stale, so you don't pay the login cost per test or per suite.

Login is deterministic (no AI) and works on most apps with zero tuning:

• Field detection uses an ordered cascade (autocomplete attributes → input types → name/id/placeholder/aria → labels), so it finds the right inputs without CSS selectors.
• Email-first / two-step forms (fill email → Continue → fill password) are handled automatically.
• Completion is detected by racing several signals (URL change, a successSelector, the form disappearing, cookie/storage mutation) — usually resolving in seconds.
• Zero-config validation: after the first login qa-runner learns the landing URL and reuses it to validate the cached session in a single navigation on later runs — no need to set validationUrl by hand (though an explicit value still wins). For SPAs that auth in-place, set successSelector to an element only present when logged in.
• On failure, the suite is skipped and the HTML report shows exactly why: final URL, page title, the on-page error text, which signals timed out, and a screenshot.

SSO / OTP / forms the heuristics can't drive: run qa-runner login once to log in by hand — it opens a headed browser, and closing the window saves the session where the runner looks for it:

# Save a session keyed to a suite's auth: block — the next run reuses it, no scripted login:
qa-runner login --suite src/checkout/.qa/cart.qa.md

# Or save an explicit storageState file to reference via auth.sessionFile:
qa-runner login https://app.example.com/login --output .qa/sessions/admin.json

auth.sessionFile also accepts any pre-captured Playwright storageState file directly.

Recording a login? qa-runner record detects the login sequence and emits an auth: block (with ${VAR} placeholders) instead of raw steps, writing the real secrets to .qa/.env — never into the .qa.md. If a suite logs in via raw fill/click steps without an auth: block, the parser warns you to convert it (raw-step logins re-login in every fresh browser context; auth: caches and shares it).

Login is Playwright-engine only. Secrets in .qa/.env are plaintext on your machine and gitignored; in CI, provide them as real environment variables.

Getting started on a fresh machine (end-to-end)

These steps take a brand-new machine (Windows, macOS, or Linux) from nothing to a passing test run.

1. Prerequisites

| Tool | Version | Check | |------|---------|-------| | Node.js | 18+ (20 LTS recommended) | node --version | | npm | bundled with Node | npm --version | | git | any recent | git --version | | An AI path | one of the two below | — | | → Anthropic API key | — | recommended — reliable, no local agent | | → Claude Code CLI | latest, logged in | for the keyless local path; claude --version |

2. Clone and install

git clone <your-repo-url> qa-runner
cd qa-runner
npm install

3. Install the Playwright browser

Playwright needs its Chromium binary downloaded once:

npx playwright install chromium

4. Choose how Claude runs

Option A — Anthropic API key (recommended):

# macOS / Linux
export ANTHROPIC_API_KEY="sk-ant-..."

# Windows (PowerShell)
$env:ANTHROPIC_API_KEY = "sk-ant-..."

…or put it in a file and pass it with --env:

# .env
ANTHROPIC_API_KEY=sk-ant-...
QA_USERNAME=...      # optional — referenced as ${QA_USERNAME} in test files
QA_PASSWORD=...

npm run qa -- tests/fixtures/example.qa.md --env .env

Option B — local Claude Code CLI (no API key):

Install Claude Code and sign in so the claude command works in your shell. qa-runner detects the missing key and drives claude locally, with all tools disabled (--tools "") so it can only read the prompt/screenshot and answer — it cannot run commands or modify files. See the AI provider note.

5. Run your first test

# dev mode (TypeScript via tsx — no build needed)
npm run qa -- tests/fixtures/example.qa.md --device "Desktop Chrome"

When it finishes, open the report (the path is printed at the end; default qa-report.html):

open qa-report.html        # macOS
start qa-report.html       # Windows
xdg-open qa-report.html    # Linux

6. (Optional) Build and install the global qa-runner command

npm run build      # compiles TypeScript to dist/
npm link           # makes `qa-runner` available on your PATH
qa-runner tests/fixtures/example.qa.md

Writing tests — the .qa.md format

A .qa.md file is an optional YAML frontmatter block followed by one or more ## Test: sections.

---
suite: Checkout Flow                  # display name (defaults to filename)
base_url: https://app.example.com     # base for relative Navigate steps
network: slow-3g                      # fast | slow-3g | offline
tags: [smoke, regression]
devices: [desktop, mobile]            # default device set for the suite
auth:
  username: ${QA_USERNAME}            # ${VARS} expand from the environment
  password: ${QA_PASSWORD}
  loginUrl: /login
  # optional: successSelector for SPA logins; sessionFile for SSO/OTP; accessToken for token auth.
  # validationUrl is auto-learned after the first login — see "Credentials & login".
---

## Test: Place Order
priority: critical                    # low | medium | high | critical
tags: [smoke]
devices: [desktop]                    # override the suite default for this test
depends_on: Login                     # skip this test if "Login" did not pass
sla: 5000ms

- Navigate to /shop
- Click: Add to Cart
- Assert: Cart shows 1 item
- Screenshot: cart

Frontmatter fields

| Field | Description | |-------|-------------| | suite | Suite display name (defaults to the filename) | | base_url | Base URL for relative Navigate steps | | tags | Suite-level tags for filtering | | devices | Default device set for the suite | | network | fast | slow-3g | offline | | auth.username / password | Login credentials (${VARS} expand from env) | | auth.loginUrl | Login page URL; relative paths resolve against base_url | | auth.validationUrl | URL only reachable when logged in; auto-learned after the first login | | auth.successSelector | Element present only when authenticated (for SPA in-place logins) | | auth.sessionFile | Pre-captured Playwright storageState (SSO/OTP; see qa-runner login) | | auth.accessToken | Bearer/JWT injected directly — skips UI login (tokenKey, tokenDomain tune it) |

See Credentials & login for the full auth model.

Per-test metadata

| Field | Description | |-------|-------------| | priority | low | medium | high | critical | | tags | Test-level tags (combined with suite tags) | | devices | Override the suite device list for this test | | depends_on | Skip this test if the named test did not pass | | sla | Expected max duration, e.g. 5000ms | | network | Override the suite network profile |

Supported steps

| Step | Syntax examples | Notes | |------|-----------------|-------| | Navigate | Navigate to /path, Navigate to https://full.url | relative paths use base_url | | Click | Click: Sign In, Click Submit button | colon or space form | | Fill | Fill email: [email protected], Fill password with hunter2, Fill search with: locator | colon, with, or with: separators | | Assert | Assert: Dashboard is visible (Check: / Verify: aliases) | evaluated by Claude; Assert: page passes WCAG AA runs axe-core instead | | Screenshot | Screenshot: name | capture-only (use --visual for anomaly detection) | | Hover | Hover: dropdown menu | | | Scroll | Scroll: footer, Scroll to bottom, Scroll down, Scroll: 500 | element, keyword, or pixels | | Wait | Wait: 2000 | milliseconds | | Select | Select: USD from currency dropdown | requires from | | Press | Press: Enter | keyboard key |

${VAR} references in step values, base_url, and auth fields are expanded from the environment at parse time (unset variables are left intact and a warning is printed).

Where tests live — discovery

A test is any *.qa.md file in the project; qa-runner walks the whole tree to find them (ADR-0006). You can:

• Co-locate a .qa/ folder next to the code it tests — and have many: src/checkout/.qa/cart.qa.md, src/auth/.qa/login.qa.md.
• Keep a root .qa/ folder, optionally with feature subfolders (.qa/checkout/…).
• Or drop a loose .qa.md anywhere.

Discovery skips node_modules, .git, build/output dirs (dist, web-dist, .next, build, coverage, .qa-output), other dot-directories, and _-prefixed folders (use _shared/ for snippets that shouldn't run on their own).

Each suite gets a group — its folder location relative to the project root, with any .qa segment stripped (src/checkout/.qa/cart.qa.md → group src/checkout). The group is an organizational label for --group selection and report sectioning; it is not a URL. Run a subset by pointing at a folder: qa-runner run src/checkout. Run everything with qa-runner run. List what would run with qa-runner list.

A top-level config.yml (in .qa/ or the project root) is merged into all suites:

# .qa/config.yml — defaults for all suites
base_url: https://app.example.com
tags: [smoke]
network: fast
auth:
  username: ${QA_USERNAME}
  password: ${QA_PASSWORD}
  loginUrl: /login

Precedence: test metadata overrides suite frontmatter overrides global config.yml.

CLI reference

qa-runner [run] [input]      # no input → every .qa.md in the project; a file/folder → just that
qa-runner list [input]       # list discovered .qa.md files (add --json)
qa-runner generate <url>     # generate a .qa.md from a URL using Claude
qa-runner record <url>       # record browser actions into a .qa.md (login → auth: block)
qa-runner login [url]        # log in by hand once; save the session for future runs
                             #   --suite <path>  key it to a suite's auth: block
                             #   --output <path> save a storageState for auth.sessionFile

Exit code 0 = all passed, 1 = one or more failed.

run options

| Option | Default | Description | |--------|---------|-------------| | --engine <engine> | playwright | playwright | chrome (chrome is a stub) | | --devices <profile> | desktop | desktop | mobile | tablet | all | | --device <name> | — | a single named device, e.g. "iPhone 14 Pro" | | --output <path> | qa-report.html | report output path | | --format <format> | html | html | json | markdown | | --watch | off | re-run on file change | | --no-headless | off | open a visible (headed) browser | | --disable-cors | off | launch with CORS disabled (local dev) | | --baseline <dir> | — | baseline screenshots dir for regression diffs | | --network <profile> | — | fast | slow-3g | offline | | --notify <target> | — | slack | teams (uses QA_WEBHOOK_URL) | | --group <pattern> | — | run only tests in a group/folder (e.g. src/checkout) | | --tag <tag> | — | filter tests by tag | | --ab-url <url> | — | second URL for A/B comparison | | --env <path> | — | load environment variables from a file | | --visual | off | use screenshots + Claude vision for asserts (slower) | | --no-heal | off | disable self-healing | | --heal-confirm | off | require approval before applying a heal |

Examples

# Single file on desktop
npm run qa -- tests/fixtures/example.qa.md

# Full device matrix, JSON report
npm run qa -- .qa --devices all --format json --output report.json

# Watch mode with a visible browser while authoring
npm run qa -- tests/fixtures/example.qa.md --watch --no-headless

# Throttle network and post results to Slack
QA_WEBHOOK_URL=https://hooks.slack.com/... npm run qa -- .qa --network slow-3g --notify slack

# Generate a test from a live URL
npx tsx src/cli.ts generate https://playwright.dev --output playwright.qa.md

# Record a session into a test file
npx tsx src/cli.ts record https://example.com/checkout --output checkout.qa.md

npm run qa -- forwards everything after -- to the CLI. After npm run build you can use the qa-runner binary directly.

Using qa-runner with a coding agent (MCP)

qa-runner ships a built-in MCP server so a coding agent (Claude Code, etc.) inside your repo can learn the .qa.md format and drive the tool without guessing. Point your agent host at the bundled server:

// e.g. Claude Code: .mcp.json  ->  "mcpServers"
{
  "qa-runner": {
    "command": "qa-runner-mcp"
    // equivalently: { "command": "npx", "args": ["qa-runner", "mcp"] }
  }
}

Both qa-runner-mcp and qa-runner mcp start the same stdio server.

Resources

| URI | Contents | |----------------------|----------| | qa-runner://guide | The full agent guide — writing + running tests, with best practices. | | qa-runner://syntax | A compact syntax cheat-sheet (verbs, frontmatter, assertion classes). |

Tools

| Tool | Purpose | |-----------------------|---------| | qa_runner_guide | Fetch the full guide (read this first). | | qa_syntax_reference | Fetch the step/frontmatter/assertion cheat-sheet. | | list_examples | List bundled example .qa.md files. | | get_example | Fetch one bundled example by name. | | validate_qa_test | Parse a .qa.md file and report structure + issues (unknown verbs, missing base_url, empty tests) so the agent can self-correct before running. | | run_qa_tests | Run qa-runner on a file/folder (as a child process) and return a summary with the exit code. |

The canonical guide the MCP serves lives at docs/AGENT_GUIDE.md. If the MCP server is unavailable, that guide and this README are the documented fallback — an agent can read them directly to author and run tests. (The server itself falls back to this README if the guide file is missing on disk.)

A typical agent loop: call qa_runner_guide → author .qa/*.qa.md → validate_qa_test → run_qa_tests → read the summary and fix any failing steps.

Device profiles

Groups (--devices):

| Group | Devices included | |-------|------------------| | desktop | Desktop (1280×800) | | mobile | iPhone 14 Pro, Pixel 7 | | tablet | iPad Air | | all | all of the above |

Individual devices (--device): desktop, desktop-hd, iphone-14-pro, iphone-se, pixel-7, galaxy-s23, ipad-air, surface-pro. See devices/ for the full set and viewports.

Reports

HTML (default) — a self-contained file with summary cards, a device matrix (pass/fail per test per device), step-by-step results with Claude's explanation and screenshot thumbnails, an anomaly gallery (severity + suggested fix), an accessibility panel (axe-core violations), regression diffs, and a screenshot lightbox.

JSON (--format json) — flat structured output for CI pipelines and dashboards: all test results, anomalies, regression diffs, and run metadata.

Markdown (--format markdown) — a concise text summary.

Optional: web dashboard

A Next.js dashboard lives in web/ (live run log, recorder UI, file picker).

npm run web:install   # one-time
npm run web           # starts the dashboard dev server

Project structure

src/
  cli.ts                 — command-line entry (commander)
  runner.ts              — suite + device orchestration, report assembly
  types.ts               — all shared interfaces and types
  anthropic-client.ts    — AI client: Anthropic SDK or local CLI fallback
  claude-code-client.ts  — local `claude` CLI subprocess driver
  parser/                — .qa.md file + .qa/ folder parsing
  engine/                — IBrowserEngine contract + Playwright/Chrome impls
  orchestrator/          — per-step interpretation, AI eval, self-healing
  features/              — generator, recorder, recorded-login, login-capture,
                           regression-differ, notifier, watch-mode
  engine/auth.ts         — deterministic scripted login + session cache/metadata
  reporter/              — HTML / JSON / Markdown report generation
devices/                 — device profile definitions
templates/               — Handlebars HTML report template
tests/                   — vitest unit tests + .qa.md fixtures
web/                     — optional Next.js dashboard

Development

npm run typecheck      # tsc --noEmit
npm run lint           # eslint (Airbnb TypeScript)
npm run format         # prettier --write
npm run check          # typecheck + lint + format:check
npm test               # vitest run
npm run test:watch     # vitest watch

Conventions are documented in .claude/CLAUDE.md — notably: no as casts, no any, explicit return types on exports, .js extensions on relative imports (ESM), and direct imports (no barrel index.ts). Run npm run check before considering a change complete.

Environment variables

| Variable | Description | |----------|-------------| | ANTHROPIC_API_KEY | Use the hosted Anthropic API instead of the local claude CLI. Recommended. | | QA_WEBHOOK_URL | Slack or Teams incoming webhook URL for --notify. | | Any custom var | Available in .qa.md step values, base_url, and auth fields as ${VAR_NAME}. |

Roadmap / next steps

Near-term (engineering)

• Deterministic-first asserts — classify the ~70% of assertions that are mechanical (URL contains X, element visible, count = N) at parse time and resolve them without a Claude call. Targets a large cut in AI spend and latency.
• Browser pooling — keep Chromium instances alive across suites to remove the 1–2 s cold-start per device.
• Parallel suite execution — a worker pool to run multiple suites concurrently and cut wall-clock time on large CI jobs.
• AI provider abstraction — replace the dual SDK/CLI paths with a single configurable, pluggable AIProvider (see AI provider note).

Medium-term (product)

• Web dashboard v1 — test history, trend charts, team workspace.
• GitHub Actions integration — a first-class action with automatic PR-comment reports.
• VS Code extension — syntax highlighting and inline run for .qa.md files.
• Cloud runner beta — hosted infrastructure so teams can run without a local Playwright setup.

Known limitations

• AI latency adds roughly 300–800 ms per assertion (mitigated by pipelining, batching, and caching; the deterministic-first path above removes it for mechanical asserts).
• AI dependency — every assertion currently flows through Claude. No API key (and no local claude CLI) means no evaluations.
• Non-determinism on vague assertions — "looks correct" can vary run to run. Prefer precise, checkable wording.
• Local CLI path — the keyless fallback drives the local claude with all tools disabled (--tools ""), so it cannot run commands or write files; it only returns a text/vision answer. An API key remains the cleanest, fully-isolated path.
• API cost scales with volume — large parallel suites against big pages accumulate spend; deterministic-first + caching are the mitigations.

License

Apache-2.0