Prose-QA

Beta — Prose-QA is under active development. Expect rough edges and breaking changes between releases.

Write what you want to test in plain text, and let Prose-QA do the rest. This autonomous, LLM-driven testing engine executes complex web workflows and validation checkpoints without the overhead of heavy browser wrappers, bringing frictionless QA to modern development.

Requires Node.js 24+, PQA_LLM_API_KEY, and llm.provider / llm.model in config.

Quick start

npm install prose-qa

pqa install-browser chrome   # or lightpanda (headless, lighter for CI)

pqa config llm.provider [ollama|fireworks|openai|anthropic|google]
pqa config llm.model [model-string]

export PQA_LLM_API_KEY=...
pqa run scenarios/**/*.md

New project checklist

1. Install the package in your app repo (or globally with npm install -g prose-qa).
2. Install a browser: pqa install-browser chrome or pqa install-browser lightpanda (headless only but lighter — good for CI).
3. Create pqa.config.json — use pqa config <key> <value> or copy the minimal example.
4. Add scenarios under scenarios/ (see 0_hello-world.md; advanced patterns in create-pqa-scenario skill).
5. Run pqa run or pqa debug.

What you get

• Natural language scenarios — # Goal, # Steps, and # Then checkpoints (format guide)
• CI + local debug modes with HTML/JSON reports
• MCP Server to help your usual agent create scenario tailored to your codebase
• Cache, healing, recording, and analysis — see HOWTO
• Extensions / hooks — lifecycle callbacks for auth, browser health, and custom logic (extensions guide)

Documentation

| Doc | Purpose | | ------------------------------------------------------------ | ------------------------------------------------------------------------------------------ | | docs/HOWTO.md | Step-by-step guide: scenarios → run → CI → MCP → record → cache → healing → analyze | | docs/CONFIG.md | Full configuration reference | | docs/extensions.md | Hooks system, defaultExtensionHooks, parallel workers | | CONTEXT.md | Glossary (run unit, provisioning, profiles) | | CONTRIBUTING.md | Pull request guidelines | | SECURITY.md | Vulnerability reporting, secrets, and run artifacts | | recorder-extension/README.md | Chrome extension recorder (WIP) |

Run pqa help or pqa help <command> for the full CLI reference (src/cli/help.ts).

CLI

| Command | Description | | --------------------------------------------------- | -------------------------------------------------- | | pqa config <key> <value> | Set a value in pqa.config.json | | pqa run [globs] | Run scenarios (headless by default) | | pqa debug [globs] | Verbose debug run (headed by default) | | pqa clear-cache [scenario] | Clear scenario replay cache | | pqa analyze [run...] | Post-run analysis and flaky detection (--last N) | | pqa auth list / clear [profile] | List or clear cached auth profiles | | pqa install-browser chrome / lightpanda | Install browser binaries for agent-browser | | pqa record start / note / checkpoint / stop | Record browser actions → scenario markdown | | pqa record generate <dir> | Regenerate scenario markdown from a saved recording | | pqa skills list / show / sync | Discover and inspect agent skills | | pqa mcp | Start MCP server (Cursor, Claude Desktop, …) | | pqa help [command...] | Show CLI help for one or all commands |

Tag filters, retries, and cache flags: see HOWTO §3–§4 and HOWTO §10.

Exit codes: 0 pass · 1 failure · 2 config/harness error

Configuration

Supported filenames (first match wins): pqa.config.json, pqa.config.mjs, pqa.config.js, pqa.config.ts.

{
  "envVars": ["PQA_TEST_EMAIL", "PQA_TEST_PASSWORD"],
  "llm": {
    "provider": "anthropic",
    "model": "claude-sonnet-4-20250514"
  }
}

| Variable | Required when | | ------------------ | ---------------------------------------- | | PQA_LLM_API_KEY | Any cloud llm.provider (not ollama) | | PQA_LLM_PROVIDER | Optional env shortcut for llm.provider | | PQA_LLM_MODEL | Optional env shortcut for llm.model |

All options, env vars, and a full example: docs/CONFIG.md.

MCP

Add the following to your mcp.json:

{
  "mcpServers": {
    "prose-qa": {
      "command": "npx",
      "args": ["-y", "prose-qa", "mcp"],
      "cwd": "/path/to/your-app-with-pqa.config"
    }
  }
}

Tools: validate_scenario, run_scenario, get_create_pqa_scenario_skill. Details: HOWTO §7.

Development (this repo)

git clone https://github.com/FreakDev/Prose-QA.git
cd Prose-QA
npm ci && npm run install-chrome

npm run build   # required — bundled pqa.config.ts imports dist/hooks/defaults.js

export PQA_LLM_API_KEY=...

npm run demo:server   # terminal 1 — http://127.0.0.1:8080/
npm run dev -- debug scenarios/0_hello-world.md
npm run dev -- run scenarios/**/*.md --tags example   # CI smoke subset
npm run dev -- run scenarios/**/*.md --tags demo       # full demo suite

Bundled demo scenarios cover smoke tests, calculator widgets, auth profiles, multi-field forms, validation errors, navigation, and partials — see scenarios/ and demo-site/. Add scenarios/auth/login-admin.md locally to exercise example-authenticated (see create-pqa-scenario skill).

See CONTRIBUTING.md and docs/HOWTO.md for the full walkthrough.

License

MIT — see LICENSE.