qabot

qabot is an agentic QA framework that serves primarily as my personal assistant and a playground for testing new approaches to software testing and QA in general. Because of that it is constantly under development, which means there may be rough edges or missing parts.

Important: qabot is used to spearhead a non-traditional approach to software testing — one not built on bloated test cases and test plans, but on clear intents that group sets of verifiable requirements. Those requirements are then tested either by automation scripts or in semi-automated human + AI-agent live sessions.

The baseline: it is safe to use, but rely on your own judgment and understand it may not suit your use case.

It covers the software testing stages from analysis and intent drafting through test generation, automation scripting, and a runner — one config file, with a human approval gate between every phase.

At the moment it requires Rust Token Killer — a CLI proxy that compresses tool output for 60–90% token savings. Install from https://github.com/rtk-ai/rtk before using qabot. /qa will hard-fail without it. (This may change later.)

Install

mkdir qa && cd qa
npx qabot-cli init

qabot lives in a folder you own — call it qa. That folder is the qabot root: a standalone repo, or a folder beside mobile/, backend/ in a monorepo. Run npx qabot-cli init inside it. Installs skills into .claude/skills/, hooks into .claude/hooks/, wires .claude/settings.json, and scaffolds the layout (intents/, tests/, reports/, ...) directly into the folder. The init also offers an optional Tolaria vault scaffold (see Tolaria).

Skills and hooks are project-local and git-ignored — each developer runs npx qabot-cli init once per clone. Hook wiring (.claude/settings.json) is committed so teammates get it automatically.

How to Use

The qabot root folder holds everything qa-related. Open Claude Code in that folder and run /qa to invoke the master orchestrator skill — it checks prerequisites, shows pipeline status, and routes you to the right phase automatically. If qa-config.yml is missing, /qa auto-routes to /qa-init.

First Run

After /qa-init scaffolds your project, fill in qa-config.yml:

project:
  name: "My App"
  github_repo: "org/repo"
  jira:
    url: "https://org.atlassian.net"
    project_key: "PROJ"

gen:
  playwright:
    enabled: true
    base_url: "http://localhost:3000"

Run /qa — it auto-routes to /qa-intent if no acceptance intents exist yet.

Phase Flow

/qa-explore  →  /qa-intent  →  /qa-codegen  →  /qa-run  →  /qa-heal  →  /qa-adversarial
   0.5              1               2              3          3            2.5
(optional)                                          (on failures)  (optional, post-run)

/qa-coverage (intent + run coverage report)
/qa-sync (after each release cycle)
/qa-triage (before each QA cycle)
/qa-ci (one-time setup)

Each phase shows a gate before proceeding. Use [f] full run from the menu to chain Intent → Codegen → Run with a single confirmation per step.

Skills Reference

| Skill | Phase | What it does | |-------|-------|--------------| | /qa | — | Orchestrator — prereq checks, status, routing | | /qa-init | — | Re-scaffold qa/ dirs, config, .gitignore (skills + hooks installed by npx qabot-cli init) | | /qa-explore | 0.5 | Browser-based live app discovery before intent drafting | | /qa-intent | 1 | Draft acceptance intents via grill loop + Draft + Validator agents | | /qa-live | 1.5 | Buddy for live manual debug session | | /qa-codegen | 2 | Generate Playwright / Maestro / XCUI automation | | /qa-run | 3 | Execute tests (local or CI artifacts) + analyse failures | | /qa-heal | 3 | Repair failing specs — locator/timing heal, Stagehand, flake gate, quarantine | | /qa-adversarial | 2.5 | Edge-case battery against isolated sandbox - TBD!! | | /qa-coverage | — | Coverage report from intent frontmatter + runs.jsonl | | /qa-sync | 4 | PR sync — new intents for changed features | | /qa-sync --daily | 4 | Auto-approve covered PRs, open review PR | | /qa-triage | — | Score Jira tickets against release signals | | /qa-ci | — | Write GitHub Actions workflow files - TBD!! | | /qa-bug | post-run | File failures from run-analysis / adversarial draft as Jira or GitHub issues | | /qa-retire | — | Mark intents deprecated: true for removed features (PR scan or manual) |

Architecture

See docs/ARCHITECTURE.md for full details. Key points:

• qa-config.yml is read once by /qa. All resolved values passed inline to sub-skills.
• Session ID (QABOT_SESSION) generated at startup — all reports from a session share the same ID.
• RTK wraps doc reads, test execution, and PR fetching for 60–90% token savings.
• Coverage tally computed inline by orchestrator — no subagent.
• /qa-run has two modes: local (execute via RTK) and CI-ingest (download JUnit/JSON artifacts from a GitHub Actions run and analyse them).
• Token-conscious runner: a green run is script-only plus one tiny summary subagent — no analysis or heal subagents spawn unless there are failures.

Intent Schema (canonical)

Framework is set so that the intent ID is immutable and ideally carries through the whole process. This way it should be free of friction and conflict.

templates/intent.md + docs/ACCEPTANCE-INTENT-FORMAT.md (the full schema is copied into docs/ on init). Key fields:

• schema_version: 1
• id: INT-{FEATURE}-{NNN} — uppercase feature slug, zero-padded sequence, immutable after write
• Immutable frontmatter: schema_version, id, title, feature, priority, platform
• Agent-written frontmatter: status, last_run, last_code_change, automation, jira_keys, deprecated
• Body: ## Intent, ## Deterministic acceptance (codegen blueprint), ## Exploratory acceptance (qa-live), ## Out of scope / known limits
• runs.jsonl — append-only run timeline; one row per run {intent_id, ts, mode, verdict, evidence, bug?}

Agent Patterns

Grill → Draft → Validator loop (/qa-intent): An upfront grill loop resolves planning decisions with the user, then the Draft agent writes intent files from RTK-injected docs. Validator checks quality + schema. Max 2 iterations.

Codegen (/qa-codegen): The codegen agent reads each intent's ## Deterministic acceptance bullets as the automation blueprint and emits runnable specs with inline assertions, then backfills the automation frontmatter list.

Heal (/qa-heal): Repairs failing specs — broken locators, timing, navigation, preconditions. Tags changes with a confidence score (HEAL_REVIEW for <0.70), runs a flake gate, quarantines flaky specs. Invoked on demand after /qa-run reports failures — a green run pays zero heal cost.

Output Structure

The qabot root folder (you own it — qa/ in a monorepo, or a standalone repo) holds everything. init scaffolds directly into it — no nested qa/.

<qabot-root>/                # the qa folder — its own repo, or a monorepo folder
├── .claude/
│   ├── settings.json    # hook wiring — committed
│   ├── skills/qa*/      # qabot skills — git-ignored, reinstall via npx qabot-cli init
│   └── hooks/           # qabot hooks — git-ignored, reinstall via npx qabot-cli init
├── tests/               # automation scripts — committed
│   ├── web/             # Playwright
│   ├── mobile/          # Maestro
│   └── ios/             # XCUI
├── intents/             # acceptance-intent .md files — committed (the test base)
├── qa-config.yml        # workflow config — committed
├── templates/           # reference templates — committed
├── docs/                # intent schema + source docs — committed
├── runs.jsonl           # append-only run timeline — git-ignored
├── reports/             # intent/codegen/run-analysis reports — git-ignored
├── sync-log.md          # sync state — git-ignored
├── .context/            # explore + adversarial artifacts — git-ignored
├── .env                 # creds, never committed
│
│   # optional — Tolaria vault scaffold:
├── intent.md            # Intent Tolaria type definition
└── views/               # saved views (by-status, work-queue, by-feature)

tests/ and intents/ are committed by default — intents/ is the acceptance-intent test base, versioned via git and browsable in Tolaria. qabot working artifacts (runs.jsonl, reports/, .context/, sync-log.md, .env) are git-ignored. Adjust by editing .gitignore.

Hook Configuration

The bundled pre_tool_use.py hook enforces intent-frontmatter immutability and blocks destructive patterns. It reads optional env vars:

| Var | Purpose | Example | |-----|---------|---------| | QABOT_BLOCKED_URLS | Comma-separated regex; block WebFetch/curl to these URLs so you avoid prod env. | https?://api\.acme\.com,https?://.*\.prod\. | | QABOT_BLOCKED_BASH | Comma-separated regex; override default destructive-bash blocklist | rm\s+-rf\s+/,DROP\s+TABLE | | QABOT_WORKSPACE | Absolute path; warn on writes outside this root | /Users/me/proj/qa | | QABOT_TC_IMMUTABLE | 1 enables intent-frontmatter immutability (set by the orchestrator) | 1 |

Observability Layer (off by default)

hooks/send_event.py exists for opt-in observability but is not wired by default. To enable:

1. Run a local obs server (script in folder /obs/start-obs.sh - it defaults to http://localhost:4000); override with OBS_SERVER_URL.
2. Add a PostToolUse entry calling python3 .claude/hooks/send_event.py --event-type tool_use in .claude/settings.json.

Adding a New Automation Framework

1. Add gen.<name>: block to templates/qa-config.yml
2. Add skills/qa-codegen/<name>.md framework rules file + a row to the Framework Rules Loading table
3. Add a run command block to skills/qa-run/SKILL.md (Step 1a). Heal is framework-agnostic — no /qa-heal change needed unless the framework needs bespoke locator repair
4. automation backfill, RTK wrapping, immutability all apply unchanged

Tolaria vault (optional)

Tolaria is a local markdown-vault UI — YAML frontmatter, first-H1 display titles, type: fields, saved views. Acceptance intents already are Tolaria notes, so the qabot root folder can double as a Tolaria vault: a human-readable, filterable view over the test base — the TestRail replacement. Tolaria only renders; the qa folder's own git versions it.

npx qabot-cli init (and /qa-init) prompts:

Set up Tolaria vault scaffold? [y/N]

On y it writes, into the qabot root:

• intent.md — the Intent Tolaria type definition (type: Type, icon, color, sort, list-property display).
• views/ — saved views: intents-by-status, intent-work-queue, intents-by-feature, all filtering type: Intent and deprecated != true.
• AGENTS.md — vault guidance. Skipped if AGENTS.md already exists (a monorepo may have its own — never overwritten).

Intents carry type: Intent in frontmatter (set by templates/intent.md), so Tolaria renders them under the Intent type. Reports and docs are scanned as generic notes but the type: Intent filter keeps the intent browse clean.

Open the qabot root folder as a Tolaria vault to browse and filter the test base. Tolaria's in-app Claude Code (Power User mode) resolves qabot skills from .claude/skills/ at the vault root — /qa runs the full pipeline from inside the vault.