@qualitywatcherai/cli

qwai — the QualityWatcher AI command line. Trigger automation runs from CI, re-run failures, wait for results, and export JUnit — from any shell.

Authoring is not a CLI surface: automations are authored by the Playwright-agent pipeline in the web app (Automate a plan), and agents outside QualityWatcher will author through the MCP tool surface (in design).

npm install -g @qualitywatcherai/cli
qwai auth login --api-key qwai_live_…
qwai plans run <planId> --env staging --wait --junit reports/qwai.xml

Requires Node.js 20.19+.

Authenticate

Create an API key in the web app under Settings → API Keys. Scopes: read for status/exports, write for posting results and creating cases, and execute for anything that runs automation — qwai plans run, qwai runs rerun, qwai runs trigger, and the MCP run/generation tools. A CI key typically needs read + execute. Then either:

qwai auth login --api-key qwai_live_…        # stores it in ~/.config/qwai/config.json
export QWAI_API_KEY=qwai_live_…              # or: environment variable (recommended for CI)

Configuration precedence (last wins): ~/.config/qwai/config.json → ./qwai.config.json → QWAI_API_KEY / QWAI_BASE_URL / QWAI_DEFAULT_PROJECT / QWAI_DEFAULT_ENVIRONMENT → flags. qwai init writes a local qwai.config.json with a default project and environment so plans list / plans run need fewer flags.

Run a plan from CI

qwai plans list --project <projectId>            # find the plan id
qwai plans run <planId> --env staging --wait     # create run, execute, block, exit 0/1

plans run creates an automation run from the plan and queues every case that has an active automation. The server completes the run itself when the last case finishes (a 10-minute finalizer covers dead workers), so you can fire and forget. With --wait the CLI polls until the run is complete and exits:

| exit | meaning | |-----:|---------| | 0 | run PASSED | | 1 | run FAILED / ABORTED | | 2 | authentication problem (missing/invalid key, wrong scope) | | 3 | usage / validation error (bad id, no automated cases, unknown environment…) | | 4 | --timeout elapsed — the run keeps going on the server and completes on its own; re-attach with qwai runs wait <runId> |

Useful flags (shared by plans run, runs rerun, runs wait):

--env <id|name>      environment (defaults to the org's default environment)
--wait               block until the run settles
--timeout 30m        give up waiting (45s, 20m, 1h…)
--poll 10s           polling interval
--junit <path>       write a JUnit XML report when the run settles (implies --wait)
--json               NDJSON events: run-created, run-progress, run-finished, junit-written, log
--title, --tags a,b  run metadata
--max-cases N        cap how many automated cases are queued

Trigger-time flags (plans run, runs rerun):

--no-auto-complete   leave the run IN_PROGRESS until someone completes it (pre-0.3 behaviour)
--report             generate a Quality Report on the server when the run completes
--report-name <s>    name for that report (implies --report)
--send-to a@x,b@y    email the report to these organization members when the run completes
--public-link        put the public share link in the email instead of the in-app link
--message <s>        a short note included in the email
--no-finalize        (with --no-auto-complete) do not complete a settled run yourself

# Nightly regression: no CI job babysitting, the report lands in inboxes when it is done.
qwai plans run <planId> --env UAT --report --send-to [email protected],[email protected]

Recipients must be members of your organization; anyone else is rejected by name before the run starts (UNKNOWN_RECIPIENTS).

Cases without an automation are marked SKIPPED at trigger time so the run can settle without manual input (pass markUnautomatedSkipped: false via the REST API if you want them left UNTESTED).

Re-run an existing run

qwai runs rerun <runId> --wait                 # every case again, as a NEW run
qwai runs rerun <runId> --failed-only --wait   # only FAILED / BLOCKED / INVALID cases

A re-run is always a new run linked to the same plan, environment, tags and configuration (each overridable). The source run is untouched, so history is preserved and terminal runs can be re-run.

Create a plan

qwai plans create "Release Smoke — UAT" --project <projectId> \
  --cases MSC-003,SC-001,SC-002 --description "Cross-channel smoke pack"
qwai plans create "Prod Read-Only Sanity" --cases-file cases.txt

--cases / --cases-file take case db ids or human labels (MSC-003); labels resolve within the project. Order is preserved as run order, so put the highest-signal cases first — plans run --max-cases N queues the first N.

Run a saved API-test suite

qwai api suites --project <projectId>            # find the suite id
qwai api run "Full Channel Lifecycle" --wait     # by name (needs --project or default)
qwai api run <suiteId> --env "UAT Environment" --wait --junit reports/api.xml

Runs the suite's endpoints/flows server-side against its saved workspace environment and records an ordinary run (type API) — the same --wait, --timeout, --junit, and exit codes as plans run. A suite that contains write requests (POST/PUT/PATCH/DELETE) aimed at an environment flagged as production is refused unless you pass --allow-writes. API runs finalize themselves; no complete step is needed.

Inspect and export

qwai runs status <runId>                        # one poll: counts + settled flag
qwai runs wait <runId> --junit out.xml          # re-attach to an in-flight run
qwai results export <runId> --format junit -o reports/qwai.xml
qwai results export <runId> --format json

Quality Reports

qwai reports create --run <runId> --send-to [email protected],[email protected]   # generate + email
qwai reports create --run <runId>,<runId> --name "Release 4.2"  # joint report
qwai reports send <reportId> --to [email protected] --public-link        # email an existing report
qwai reports list --project <projectId>

Emails link to the report inside QualityWatcher (sign-in required) unless you pass --public-link, which embeds the report's share link (anyone with the link can view). Every send is recorded on the report.

GitHub Actions example

name: QualityWatcher regression
on:
  deployment_status:
jobs:
  regression:
    if: github.event.deployment_status.state == 'success'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm install -g @qualitywatcherai/cli
      - name: Run plan
        env:
          QWAI_API_KEY: ${{ secrets.QWAI_API_KEY }}
        run: |
          qwai plans run ${{ vars.QWAI_PLAN_ID }} \
            --env "${{ github.event.deployment.environment }}" \
            --tags "ci,${{ github.sha }}" \
            --wait --timeout 45m \
            --junit reports/qwai.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: qwai-junit, path: reports/qwai.xml }

Any CI that can run a shell works the same way — the CLI only needs QWAI_API_KEY (and QWAI_BASE_URL for non-production deployments).

Connect an agent (MCP)

QualityWatcher exposes its test-case, Knowledge Base, automation and run operations over the Model Context Protocol (revision 2026-07-28) at /api/mcp, so Claude Code, Cursor, Codex or a CI agent can work in your projects without scraping the UI — 29 tools, all scoped to the organization that owns the API key:

| Area | Tools | |---|---| | Orientation | qw_status, list_projects, get_project_context, list_environments, list_suites, list_plans | | Test cases | search_test_cases, get_test_case, create_test_cases, update_test_case (optimistic expectedUpdatedAt), lint_test_cases, generate_test_cases (QualityWatcher's engine, optional save) | | Knowledge & memory | search_knowledge_base, list_knowledge_sources, search_team_memory, propose_memory (proposal only — a human approves) | | Automation code | list_automation_versions, get_automation_version, lint_automation_code, submit_automation_code, validate_automation_version, watch_execution, approve_automation_version | | Runs | trigger_run, rerun_failed, watch_run, complete_run, list_runs, get_run |

Your agent can write the Playwright spec itself: submit_automation_code stores it as a DRAFT version and queues a validation run in QualityWatcher's runner; specs with no expect() or with hard-coded ids are refused up front; approve_automation_version only activates a version that has a PASSED validation. Errors come back in-band as { error: { code, message, retriable } }.

qwai mcp setup                      # prints ready-to-paste client config
# remote (recommended):
claude mcp add --transport http qualitywatcher https://www.qualitywatcher.ai/api/mcp \
  --header "Authorization: Bearer $QWAI_API_KEY"
# stdio, for clients that can't send a bearer header:
claude mcp add qualitywatcher -- qwai mcp serve

qwai mcp serve is a stdio MCP server that proxies every tool call to your deployment with the key from qwai auth login / QWAI_API_KEY. It writes only protocol frames to stdout; diagnostics go to stderr. Requires the deployment to have MCP enabled for your organization (ask an admin — it is gated while the security workstream completes).

Everything else

qwai auth login|status|logout       API key management
qwai init                           write qwai.config.json (default project/env)
qwai projects list                  projects in the active organization
qwai envs list                      test environments
qwai suites list                    test suites
qwai plans list|create|run          plans: list, create from case ids/labels, trigger runs
qwai runs trigger|rerun|status|wait saved-automation replay, re-runs, polling
qwai results export                 JUnit / JSON export
qwai reports create|send|list       Quality Reports: generate, email to teammates, list
qwai api suites|run                 saved API-test suites: list and execute
qwai api import|export|frameworks   generate API tests from OpenAPI / collections

qwai <command> --help documents every flag. Add --json to any command for machine-readable output; spinners are suppressed automatically when stdout is not a TTY.

Under the hood

The CI commands use the public REST API (/api/v1, described at GET /api/v1/openapi.json):

| CLI | REST | |-----|------| | plans run | POST /api/v1/plans/{id}/run | | runs rerun | POST /api/v1/runs/{id}/rerun | | runs status / runs wait | GET /api/v1/runs/{id}/status → POST /api/v1/runs/{id}/complete | | results export | GET /api/v1/runs/{id} |

Development

cd cli
npm install
npm run dev          # tsup --watch
npm test             # node:test via tsx
npm run typecheck
npm run build        # dist/index.js (single ESM bundle, shebang included)
node dist/index.js --help

Releasing: bump version in package.json, commit, tag cli-v<version>, push the tag. .github/workflows/publish-cli.yml typechecks, tests, builds, verifies the tag matches the version, asserts the tarball is compiled output only (npm run check:tarball — no source, no source maps), and publishes to npm.

License

Proprietary — © QualityWorks Consulting Group. The package ships as a compiled bundle only; see LICENSE. Use is governed by your QualityWatcher terms of service.