runnerqa (mqa)

The RunnerQA CLI runs your Maestro mobile tests locally on your own simulator or device and reports the results back to RunnerQA. RunnerQA is the mobile QA control plane for AI-built apps: an AI agent prepares, reads, and analyzes tests over the hosted MCP endpoint, while this CLI does the actual local execution.

npm install -g runnerqa

Commands

Connect and run

mqa connect <server-url> <runner-token>              # link the CLI to your account
mqa status                                           # show connection status
mqa pull [project-id]                                # download test YAML into ./.mqa-tests
mqa run <test-case-id> [--output json]               # run one test locally, report result
mqa run-all [--project <id>] [--output json]         # run all pulled tests, sequentially

Running tests — deterministic exit codes + machine-readable results

mqa run and mqa run-all execute your Maestro tests locally, one at a time (sequential, single-device) and report each result to RunnerQA. They are built for agents and CI: the process exit code is authoritative and, with --output json, stdout carries exactly one JSON document (no progress text, no Maestro logs, no ANSI, no stack traces).

Exit codes:

| Code | Meaning | |------|---------| | 0 | At least one test was requested, every requested test executed, every result was reported to RunnerQA, and every test passed. | | 1 | Execution completed reliably, but one or more Maestro tests failed. | | 2 | No trustworthy complete suite verdict — invalid usage, missing/invalid config, missing Maestro, missing local YAML, unretrievable suite, zero selected test cases, a result-reporting failure, or an unexpected internal error. |

If both a test failure and an operational/reporting error occur, exit 2 wins. A SIGINT (Ctrl-C) keeps conventional interruption behavior (exit 130) and is never reported as a pass or fail.

A generated test whose local YAML is missing is an incomplete suite (missing_yaml, exit 2), not a passing skip. A locally executed result that cannot be reported to RunnerQA is not a complete verification (reported: false, exit 2); the CLI never auto-retries a report (so it never creates duplicate run records) and never re-runs a test because reporting failed.

--output json contract (stdout only, one document):

{
  "schema_version": 1,
  "command": "run-all",
  "execution_mode": "sequential",
  "project_id": null,
  "status": "passed",
  "exit_code": 0,
  "message": null,
  "totals": {
    "requested": 2, "executed": 2, "passed": 2, "failed": 0,
    "errors": 0, "missing_yaml": 0, "reported": 2
  },
  "runs": [
    {
      "test_case_id": "…",
      "status": "passed",
      "duration_ms": 8213,
      "reported": true,
      "run_id": "…",
      "error_kind": null,
      "message": null
    }
  ]
}

schema_version is pinned at 1. Per-run status is passed | failed | error | missing_yaml; error_kind is assertion | timeout | maestro_unavailable | missing_yaml | reporting | internal | null. Totals are internally consistent (requested = passed + failed + errors + missing_yaml, executed = passed + failed, reported ≤ executed). The durable run_id lets an agent retrieve full logs and the UI hierarchy over hosted MCP — full Maestro logs are never placed in this summary. Per-run identity uses test_case_id and run_id; test_case_name is intentionally excluded from machine JSON because names are user-controlled and may contain PII, secrets, or local paths. Output never contains a runner token, environment value, raw config, email address, full home path, test case name, or a raw provider/server error; a top-level message (and each run's message) is a bounded, safe string. --project <id> is forwarded to the runner API, so another project's tests are never downloaded and filtered locally. Human mode stays readable and prints a concise per-test + summary footer.

mqa doctor — diagnose local execution readiness

Read-only. It never modifies config, installs software, creates/revokes tokens, changes permissions, pulls or runs tests, controls a simulator, or prints your token.

mqa doctor            # human-readable report
mqa doctor --json     # machine-readable report (see contract below)

It checks: CLI + Node version (Node >= 20), RunnerQA config (existence, valid JSON, required fields, a valid server URL, and — on Unix — a warning if permissions are broader than 0600), API connectivity (validates the runner token against the status endpoint, distinguishing invalid/revoked vs unreachable vs timeout vs malformed, with a bounded timeout), Maestro (PATH then ~/.maestro/bin fallback, maestro --version), iOS readiness on macOS (xcode-select, xcrun, booted simulator count — never boots or modifies anything), Android readiness (adb devices; absence is a warning, not a failure, when iOS is available), and local pulled tests (.mqa-tests presence + a lightweight structural check).

The server URL is validated and normalized: HTTPS origins are accepted (HTTP only for loopback — localhost, 127.0.0.1, [::1]), while embedded credentials, query strings, fragments, and non-root paths are rejected. The runner token is only ever sent to a normalized origin, and an invalid URL is a blocking failure that skips connectivity entirely — no request is made and the rejected value is never printed.

Exit codes: 0 when there are no blocking failures (warnings are fine); 1 when at least one blocking requirement fails (old Node, missing/invalid config, invalid server URL, invalid/revoked token, or missing Maestro).

--json contract (stdout only, no decoration):

{
  "schema_version": 1,
  "status": "pass",
  "checks": [
    {
      "id": "maestro",
      "status": "pass",
      "summary": "Maestro 1.39.0 (PATH)",
      "detail": "/usr/local/bin/maestro"
    }
  ],
  "totals": { "passed": 9, "warnings": 0, "failed": 0 }
}

schema_version is pinned at 1. Output never contains a token, Authorization header, cookie, or credential. Filesystem paths inside your home directory are shown ~-relative (e.g. ~/.mobile-qa-agent/config.json, ~/.maestro/bin/…), so the raw home path and your local username never appear, and only normalized server origins are shown. Text and JSON modes are produced from the same check engine through one redaction boundary, so they never disagree.

RunnerQA Verify skill for Claude Code

Install a skill that teaches Claude Code to verify a mobile feature with a real local run before ever claiming it works — confirm the connected account and scopes via runnerqa_connection_status first, inspect existing tests over hosted MCP, run mqa doctor --json, execute locally, and require your explicit approval before any fix is applied.

mqa agent install claude   # install .claude/skills/runnerqa-verify/SKILL.md
mqa agent status           # report install state (not installed / current / outdated / conflict)

The installer is safe: project-relative paths only, path-traversal and symlink-escape rejection, atomic writes, a RunnerQA-managed marker + version, and it refuses to overwrite a file it does not manage (no --force). It does not touch global Claude config, MCP config, OAuth, or Agent API tokens.

mqa setup --agent claude — guided setup

mqa setup --agent claude          # interactive: prompts before writing
mqa setup --agent claude --yes    # non-interactive: required to write without a TTY

Runs the same doctor engine, shows any blocking local-execution problems, installs/updates the Verify skill only after confirmation (or --yes), then explains the two separate connections and prints the next safe commands. It never installs Maestro, never creates credentials, and never runs a test.

Exit codes: setup exits 0 only when local execution is actually ready — doctor found no blocking failure and the skill is current or was successfully installed/updated. It exits 1 when doctor has a blocking failure (even if --yes still installed the skill), when a non-interactive run omits --yes, when you decline the install, or when a conflict/unsafe path prevents the write.

Two connections:

• Hosted MCP (OAuth): your agent's access to RunnerQA — https://www.runnerqa.dev/api/mcp/mcp. Prepares, reads, analyzes. It does not drive the simulator.
• Runner token / CLI config: local run reporting. The CLI runs tests on your own simulator or device and reports results back.

Docs

https://www.runnerqa.dev/docs