@qatonic_innovations/qaios

v0.5.2

Published

2 days ago

AI QA engineer in your terminal — designs, writes, runs, heals, and explores tests for web UI and APIs with audit-grade traceability.

0High
0Medium
0Low

qatonic_innovations

qa testing playwright ai agent anthropic claude mcp cli test-generation test-automation visual-regression self-healing

QAIOS

AI QA engineer in your terminal. Designs, writes, runs, heals, and explores tests for web UI and APIs — with audit-grade traceability.

QAIOS acts like a QA engineer on your team. Point it at a feature description or an OpenAPI spec; it produces a test design, generates Playwright code, runs the tests, classifies failures, heals broken locators, files defects, scans for accessibility issues, and keeps a hash-chained audit log of every decision.

Mental model: Claude Code, but for QA work instead of feature coding.

Status: early alpha (v0.1). Core flows — init, doctor, test design & generation, Playwright execution, self-healing, accessibility, and the audit log — work today. Expect rough edges and please report issues. After installing, a quick smoke test confirms everything resolves:
qaios --version
qaios --help
qaios doctor          # checks Node, API key, Playwright, config

Install

npm install -g @qatonic_innovations/qaios
# or:  pnpm add -g @qatonic_innovations/qaios

The published package is @qatonic_innovations/qaios; the installed command is qaios.

Requirements

Node.js 22 LTS or newer. QAIOS uses Node's built-in SQLite (node:sqlite, stable in Node 22) for its local audit log — no native module to compile, so npm install is just a download. (On Node < 22 the built-in isn't available; upgrade Node.)
An LLM provider API key. QAIOS uses Anthropic by default — get a key at console.anthropic.com and export it:
```
export ANTHROPIC_API_KEY=sk-ant-...      # macOS/Linux
setx ANTHROPIC_API_KEY "sk-ant-..."      # Windows (open a NEW shell after)
```
Prefer OpenAI? Set llm.provider: openai and export OPENAI_API_KEY instead — see Choose your LLM provider below. Keys are read from the environment (process.env) and never written to disk by QAIOS. QAIOS does not auto-load a .env file — export the key in your shell (or use a .env runner like dotenv-cli) before running.
Playwright in your project, for qaios run / snapshot / explore / a11y:
```
npm i -D @playwright/test && npx playwright install
```
(@playwright/test is for run; explore/a11y use the playwright package, which it pulls in.)
For qaios a11y, also: npm i -D @axe-core/playwright

Install troubleshooting

QAIOS has no native dependencies — it uses Node's built-in SQLite — so install is friction-free. If qaios init reports a SQLite/node:sqlite error, your Node is too old: run node -v and upgrade to Node 22 LTS or newer.

qaios doctor will tell you exactly which check failed and what to run.

60-second quick start

cd my-app
qaios init                       # creates .qaios/, QAIOS.md; detects Playwright
qaios doctor                     # verify your environment is ready

# Generate tests from plain English, then run them:
qaios test "user can log in with email and password" --run

qaios test produces a DesignSpec, a Playwright spec, and a Page Object — then (--run) executes them. That's the core loop.

workflow: 01JTZ1MV7Q8X3R2WFP9D5K8YRH

▸ requirements.intake   ✓  (2 reqs,  confidence 0.85)
▸ risk.score            ✓  (overall HIGH — touches PII)
▸ design.web            ✓  (4 scenarios, confidence 0.84)
▸ write.web             ✓  (1 spec, 4 tests, 1 page object)

Generated:
  tests/auth/login.spec.ts          (4 tests)
  tests/auth/LoginPage.po.ts
  .qaios/specs/01JTZ1…/DesignSpec.json

Cost: 11,403 tokens · $0.06 (3 of 15 LLM calls used)

Gates: on HIGH/CRITICAL-risk features QAIOS pauses for review. Approve interactively with qaios review, or pass --yes to auto-approve for a non-interactive run.

What each command does

| Command | What it does | | ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | qaios init [path] | Bootstrap a project: create .qaios/, QAIOS.md, run migrations. | | qaios doctor | Health-check your environment (Node, API key, Playwright, config). --json for CI. | | qaios test <description> | Design + generate tests from natural language. --type api for OpenAPI. --run to execute. --yes to auto-approve gates. | | qaios run [pattern] | Execute Playwright tests, classify failures (real / flaky / locator drift), auto-heal locator breaks, file defects. | | qaios fix [target] | Self-heal a broken locator. --last-failure to target the most recent failure, --auto to apply, --dry-run to preview. | | qaios explore <url> | Time-boxed exploratory session against a live URL. --duration <seconds>, --focus "<hint>". | | qaios a11y <url> | Deterministic axe-core accessibility scan + AI triage. --wcag-level A\|AA\|AAA. Exits 1 on real violations (CI-gateable). | | qaios snapshot capture\|check\|review | Visual regression: capture baselines, diff, and triage AI-classified diffs. | | qaios review | Walk pending human-review gates and approve/reject; resumes the workflow. | | qaios history [--verify] | List workflows, inspect the audit log, and verify the hash chain. | | qaios config get\|set\|show\|edit | Read/write .qaios/config.yaml (schema-validated). | | qaios mcp list\|add\|remove\|enable\|disable\|test | Manage MCP servers (GitHub, Jira, Slack, custom). |

Run qaios <command> --help for the full option list of any command.

Common workflows

# Generate API tests from an OpenAPI spec
qaios test --type api --api-spec ./openapi.yaml "exercise the /orders endpoints"

# Run a suite; QAIOS classifies failures and self-heals locator drift
qaios run

# Heal a specific broken test, previewing the patch first
qaios fix tests/auth/login.spec.ts --last-failure --dry-run

# Accessibility scan, gate CI on real WCAG violations
qaios a11y https://staging.myapp.com --wcag-level AA

# Exploratory testing with a focus hint
qaios explore https://staging.myapp.com --duration 900 --focus "checkout flow"

# Visual regression
qaios snapshot capture        # baseline
# ...change CSS...
qaios snapshot check          # diff
qaios snapshot review         # triage interactively

Configuration

Config lives in .qaios/config.yaml (created by qaios init):

version: 1
mode: LITE # LITE | FULL | TRUST
app:
  baseUrl: https://staging.myapp.com
llm:
  provider: anthropic # anthropic | openai
  maxLlmCallsPerWorkflow: 15
  costAlertThresholdUsdCents: 50
testing:
  framework: playwright
  testDir: tests
  testIdAttribute: data-testid # e.g. set to 'data-test' for some apps
  locatorPreference: [data-testid, role, text]
gates:
  autoExpireOnTimeout: abort # abort | auto_approve | auto_reject
defects:
  target: stdout # stdout | github | jira

qaios config show prints the resolved config; qaios config set <key> <value> validates against the schema before writing. API keys are never written to config — they come from environment variables.

Choose your LLM provider

QAIOS works with Anthropic (default) or OpenAI. Set the provider in .qaios/config.yaml and export that provider's key:

# Anthropic (default)
llm:
  provider: anthropic # reads ANTHROPIC_API_KEY

# OpenAI
llm:
  provider: openai # reads OPENAI_API_KEY
  model: gpt-4o # optional; default gpt-4o (try gpt-4o-mini for lower cost)

export OPENAI_API_KEY=sk-...
qaios doctor   # confirms the configured provider's key is reachable

The rest of QAIOS is provider-blind — every command (test, run, fix, explore, a11y, …) works identically on either. Structured output uses each provider's native guaranteed-schema mode (Anthropic forced tool-use / OpenAI strict function calling), so generated artifacts stay schema-valid. You can override the key's env-var name with llm.apiKeyEnv if needed.

Picking a model

llm.model is a free-form string passed straight to the provider — any model that provider's API accepts works, not just the defaults. When llm.model is omitted, QAIOS uses a sensible default per provider:

| Provider | Default | Other examples you can set | | ----------- | ------------------- | ------------------------------------------------- | | anthropic | claude-sonnet-4-6 | claude-opus-4-7, claude-haiku-4-5-20251001, … | | openai | gpt-4o | gpt-4o-mini (cheaper), gpt-4.1, … |

llm:
  provider: openai
  model: gpt-4o-mini # any OpenAI model id

Two things to know about non-default models:

Cost tracking. QAIOS knows exact pricing for a built-in set of models (Sonnet/Opus/Haiku and gpt-4o/4o-mini/4.1/4.1-mini). A model outside that set still runs fine, but the USD figure in the audit log is approximate — it bills at a default rate and prints a one-time no pricing for model … warning. The per-workflow call/cost cap still applies regardless.
Structured output. QAIOS relies on the provider's strict tool/function calling. The defaults are chosen for strong schema adherence; a much older or unusual model may fail validation more often (the built-in retry loop recovers from occasional misses).

The defaults are the live-validated starting points — every skill is exercised end-to-end against them. Other models share the same code path but aren't individually certified.

Operating modes

LITE (default) — HIGH/CRITICAL risk pauses for review; routine work flows through.
FULL — every gate active, every AI decision reviewable.
TRUST — auto-approve when confidence > 0.85 and risk ≤ MEDIUM; CRITICAL still notifies.

qaios config set mode TRUST

Cost & privacy

Skills run on Claude Sonnet (Anthropic) or gpt-4o (OpenAI) per your llm.provider. A typical qaios test costs ~$0.04–0.10. Each workflow is capped (default min(15 calls, $0.50), configurable) and aborts if exceeded.
Local-first. No telemetry, no phone-home. The only outbound traffic is (a) LLM calls to your configured provider (Anthropic or OpenAI) with the prompts QAIOS builds, and (b) any MCP servers you configure. You bring your own API key; QAIOS does not proxy your requests. See SECURITY.md for exactly what's sent.

Exit codes (for CI)

Every command exits with a stable, documented code so you can gate pipelines:

| Code | Meaning | Example | | ----- | ------------------------------- | ----------------------------------------------------------------------- | | 0 | Success | tests generated; doctor all-green; a11y clean | | 1 | User error / actionable failure | bad flag; a11y found real violations; run had failing tests | | 2 | Gate blocked (informational) | a workflow paused for human review — run qaios review or pass --yes | | 3 | Tool/dependency error | Playwright or axe not installed; SQLite binding missing | | 4 | LLM error | rate limit, timeout, or invalid API key | | 5 | Internal error | unexpected crash (re-run with --debug for a stack trace) | | 130 | Cancelled | you pressed Ctrl+C |

qaios doctor --json and --json on most commands emit machine-readable output for CI consumption.

Documentation

Getting Started — first 15 minutes
Commands Reference — every CLI command
Skills Reference — what each skill does
MCP Servers — adding integrations
A runnable end-to-end example lives in examples/live-saucedemo.

Status & roadmap

v0.1 (now): Web UI + API + visual regression + self-healing + exploratory + accessibility, as an open-source CLI. Early — feedback wanted.

v0.2 — optional cloud sync + web dashboard
v0.5 — mobile testing
v0.7 — performance + security testing
v1.0 — database testing, stable public API

The roadmap is gated on real-world validation, not a calendar.

License

MIT. The CLI will always remain MIT; cloud features (v0.2+) ship under a separate license.