qa-agent

Write Playwright tests in plain English. AI models run them, cache what works, fix what breaks, and cross-check every assertion.

await runSteps({
  page, test, expect,
  userFlow: "Add product to cart",
  steps: [
    { description: "Navigate to https://www.saucedemo.com" },
    { description: "Type username", data: { value: "standard_user" } },
    { description: "Type password", data: { value: "secret_sauce" } },
    { description: "Click Login", waitUntil: "Products page is visible" },
    { description: "Click Add to cart on Sauce Labs Backpack" },
    { description: "Open the cart", waitUntil: "Your Cart page is visible" },
  ],
  assertions: [{ assertion: "The cart contains Sauce Labs Backpack" }],
});

Contents

1. Why
2. Install
3. Your first test
4. Writing tests in YAML
5. Choosing a model provider
6. Running on Lightpanda
7. API
8. Assertions
9. Placeholders and email
10. Caching
11. Reference
12. Project

Why

| Problem with selector-based tests | What qa-agent does instead | |---|---| | A renamed class or moved button breaks the suite | Steps are natural language. The AI finds the element on the current page | | AI on every run is slow and costly | Steps that worked are cached in Redis and replayed without any AI call | | A cached action stops matching the page | It falls back to AI, re-executes, and re-caches (auto-heal) | | One model's judgement can be wrong | Assertions run on two models; a third arbitrates if they disagree | | Toasts vanish before the screenshot | video: true assertions judge the whole recorded run |

It also suits checking code written by coding agents: the suite keeps passing while the DOM is reshaped underneath it, so it works as a pre-review check on every change.

How a step runs

flowchart LR
    A[Step] --> B{Cached?}
    B -- yes --> C[Replay cached action]
    C -- ok --> F[Done]
    C -- fails --> D[AI executes step]
    B -- no --> D
    D --> E[Cache result] --> F
    F --> G[Assertions: 2 models + arbiter]

Install

1. Create a Playwright project (TypeScript) and add the package:

npm init playwright@latest my-tests
cd my-tests
npm install @ahmadw/qa-agent dotenv

2. Load .env in playwright.config.ts:

import dotenv from "dotenv";
import path from "path";
dotenv.config({ path: path.resolve(__dirname, ".env") });

3. Add keys to .env. The defaults use Anthropic and Google models:

ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_GENERATIVE_AI_API_KEY=AIza...

Prefer a single key? See Choosing a model provider.

Your first test

tests/cart.spec.ts:

import { test, expect } from "@playwright/test";
import { runSteps } from "@ahmadw/qa-agent";

test.use({ headless: !!process.env.CI });

test("cart", async ({ page }) => {
  test.setTimeout(120_000); // AI steps need more than the default
  await runSteps({
    page, test, expect,
    userFlow: "Log in and add a product to the cart",
    steps: [
      { description: "Navigate to https://www.saucedemo.com" },
      { description: "Type username", data: { value: "standard_user" } },
      { description: "Type password", data: { value: "secret_sauce" } },
      { description: "Click Login", waitUntil: "Products page is visible" },
      { description: "Click Add to cart on Sauce Labs Backpack" },
      { description: "Open the cart", waitUntil: "Your Cart page is visible" },
    ],
    assertions: [{ assertion: "The cart contains Sauce Labs Backpack" }],
  });
});

npx playwright test cart.spec.ts --project chromium
npx playwright show-report   # includes an AI-written summary of the run

Writing tests in YAML

Write tests as plain English in YAML files, with no TypeScript per test. Each file is a suite with a list of tests:

# qa/cart.yaml
suite: Shopping cart
tests:
  - name: Add backpack to cart
    steps:
      - Navigate to https://www.saucedemo.com
      - Type username: standard_user
      - Type password: secret_sauce
      - step: Click Login
        waitUntil: Products page is visible
      - Click Add to cart on Sauce Labs Backpack
      - step: Open the cart
        waitUntil: Your Cart page is visible
    expect:
      - The cart contains Sauce Labs Backpack

Add one runner spec, once. It loads every .yaml and .yml file under the folder, including subfolders:

// tests/yaml.spec.ts
import path from "path";
import { test } from "@ahmadw/qa-agent/lightpanda";
import { loadYamlTests } from "@ahmadw/qa-agent/yaml";

for (const t of loadYamlTests(path.join(__dirname, "../qa"))) test(t.title, t.run);

Call configure() in the runner if you need non-default models. Tests appear in Playwright as Shopping cart: Add backpack to cart, so --grep, retries, reports, and the Chromium and Lightpanda projects all work as usual.

Step forms

| Form | Example | Becomes | |---|---|---| | Sentence | - Click Add to cart | { description } | | Sentence with a value | - Type username: standard_user | { description, data: { value } } | | Full form | - step: Click Login with value: and/or waitUntil: | { description, data, waitUntil } |

A sentence that contains ": " but has no value must be quoted, for example - "Note: the banner closes". Otherwise YAML reads it as a sentence with a value.

Test fields: name and steps are required. expect is a list of assertions. timeout is in seconds and defaults to 300. Placeholders such as {{run.email}} work in steps and assertions.

An invalid file stops the run before any test starts, with an error that names the file and the field:

ValidationError: Invalid YAML test file qa/broken.yaml:
  - tests[0].steps[0]: a step must be a sentence, "sentence: value", or an object with "step" (and optional "value", "waitUntil")

Choosing a model provider

Pick one row. Model IDs are always written as provider/model, for example google/gemini-3-flash.

| Route | configure({ ai: { gateway } }) | Keys in .env | |---|---|---| | Direct (default) | "none" | ANTHROPIC_API_KEY, GOOGLE_GENERATIVE_AI_API_KEY (+ OPENAI_API_KEY for openai/*) | | Z.AI GLM Coding Plan | "none", models zai/* | ZAI_API_KEY | | Vercel AI Gateway | "vercel" | AI_GATEWAY_API_KEY | | OpenRouter | "openrouter" | OPENROUTER_API_KEY | | OpenCode Zen | "opencodezen" | OPENCODEZEN_API_KEY | | Cloudflare AI Gateway (proxy) | "cloudflare" | CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_AI_GATEWAY, the provider keys, and CLOUDFLARE_AI_GATEWAY_API_KEY if the gateway is authenticated |

import { configure } from "@ahmadw/qa-agent";
configure({ ai: { gateway: "openrouter" } });

Z.AI (GLM-5.3 / GLM-5.3-Flash)

configure({
  ai: {
    models: {
      stepExecution: "zai/glm-5.3-flash",
      userFlowLow: "zai/glm-5.3-flash",
      userFlowHigh: "zai/glm-5.3-flash",
      assertionPrimary: "zai/glm-5.3-flash",
      assertionSecondary: "zai/glm-5.3-flash",
      assertionArbiter: "zai/glm-5.3-flash",
      utility: "zai/glm-5.3-flash",
    },
  },
});

• glm-5.3 is text-only. Steps, assertions, and wait conditions send screenshots, so use the multimodal glm-5.3-flash for them.
• GLM always reasons. Set ZAI_REASONING_EFFORT=low|high|max (the API defaults to max, which is slow) and raise test timeouts.
• zai/* models can be mixed with other providers in the same config.

CUA mode (OpenAI computer use)

The default mode reads ARIA accessibility snapshots. mode: "cua" drives the page visually with screenshots and coordinates instead, using OpenAI's built-in computer tool.

configure({ ai: { mode: "cua", gateway: "none" } }); // needs OPENAI_API_KEY

• Model is locked to gpt-5.5; models.cua cannot be set.
• Requires gateway: "none". The computer tool is only available on direct OpenAI access.
• Redis caching is skipped: coordinates don't carry across viewport sizes.
• Your OpenAI key needs access to the CUA model and the Responses-API computer tool.

Mixing modes per step

ai can be set on configure(), on the runSteps / runUserFlow call, and on each step. The most specific wins: step → call → global.

configure({ ai: { gateway: "openrouter" } });

await runSteps({
  page, test, expect,
  userFlow: "Buy product on sale",
  steps: [
    { description: "Navigate to /products" },
    { description: "Drag the price slider to $40", ai: { mode: "cua", gateway: "none" } },
    { description: "Click Add to cart" },
  ],
});

Running on Lightpanda

Lightpanda is a headless browser with no rendering engine. It starts instantly and runs page actions in milliseconds, so you can run the same specs on it next to Chromium.

1. Start Lightpanda and leave it running:

brew install lightpanda-io/browser/lightpanda
lightpanda serve --host 127.0.0.1 --port 9222

2. Add a lightpanda project. Projects without the option keep Playwright's normal browser:

// playwright.config.ts
import { defineConfig, devices } from "@playwright/test";
import type { LightpandaWorkerOptions } from "@ahmadw/qa-agent/lightpanda";

export default defineConfig<object, LightpandaWorkerOptions>({
  projects: [
    { name: "chromium", use: { ...devices["Desktop Chrome"] } },
    { name: "lightpanda", use: { lightpanda: {} } }, // or { lightpanda: { endpoint: "ws://host:9222" } }
  ],
});

3. Import test and expect from the Lightpanda entry point instead of @playwright/test:

import { test, expect } from "@ahmadw/qa-agent/lightpanda";
import { runSteps } from "@ahmadw/qa-agent";

The endpoint is resolved from the project option, then LIGHTPANDA_WS_ENDPOINT, then ws://127.0.0.1:9222. If nothing is listening there, the run fails with a ConfigurationError that names the endpoint.

What changes on Lightpanda

| Feature | On Lightpanda | |---|---| | Snapshot mode (default): steps, waitUntil, assertions | Works. Screenshots are a text-only rendering of the page, so assertions about layout, colors or images can't be checked. | | CUA mode | Rejected with ConfigurationError, since element coordinates aren't real. | | video: true assertions | Recording is skipped and the assertion uses the screenshot and snapshot instead. | | Hidden elements | The accessibility snapshot can include elements hidden with CSS, such as closed menus. |

Lightpanda is AGPL-3.0. qa-agent only connects to it over CDP and doesn't ship it.

To use it without the fixture, call connectLightpanda({ endpoint }), which returns a Playwright Browser. isLightpandaPage(page) tells you whether a page came from it.

API

| Function | Use it for | |---|---| | runSteps(options) | An ordered list of natural-language steps, with caching, auto-heal, and assertions | | runUserFlow(options) | One agent call that completes a loosely described goal (exploratory) | | assert(options) | A single multi-model assertion against the current page | | configure(config) | Global settings: models, gateway, mode, Redis, email, telemetry, upload path |

// runUserFlow: effort "low" (default) or "high" selects userFlowLow / userFlowHigh
const result = await runUserFlow({
  page,
  userFlow: "Complete a purchase",
  steps: "Navigate to store, add an item, checkout with test card",
  effort: "high",
});

// assert: standalone check
await assert({ page, expect, assertion: "The dashboard shows 3 active projects" });

Also included: AI data extraction from pages and URLs, AI-evaluated wait conditions (waitUntil) with backoff, an AST-validated Playwright script runner, and a Pino logger.

Assertions

Each assertion is checked by a primary and a secondary model in parallel. When they agree, that's the result. When they disagree, consensusPolicy decides:

| consensusPolicy | On disagreement | |---|---| | "consult-arbiter-on-disagreement" (default) | The arbiter model makes the call | | "fail-on-disagreement" | Fail immediately; the reasoning shows both models' views |

configure({ assertions: { consensusPolicy: "fail-on-disagreement" } });

Video assertions

For UI that only shows for a moment, record the whole run:

assertions: [{ assertion: "An 'Added to cart' toast appears", video: true }]

• One recording (via page.screencast) spans every step in the runSteps call and is shared by all video: true assertions.
• Judged by Gemini only (gemini-3-flash-preview), uploaded straight to Gemini's Files API whatever the gateway, so GOOGLE_GENERATIVE_AI_API_KEY (or GEMINI_API_KEY) is always required.
• Files go to /tmp/qa-agent-recordings/ and are deleted afterwards. Change with configure({ videoDir }).
• If recording can't start, the assertion falls back to the screenshot path.

Placeholders and email

Put placeholders in data.value to inject values at runtime:

| Placeholder | Lives for | Value | |---|---|---| | {{run.email}} · {{run.fullName}} · {{run.phoneNumber}} · {{run.shortid}} | one test | random (faker) | | {{run.dynamicEmail}} | one test | address on the configured email domain | | {{global.email}} · {{global.dynamicEmail}} | every test sharing an executionId | stored in Redis | | {{data.key}} | the project | stored in Redis | | {{email.type:prompt}} | resolved when used | extracted from a received email |

{
  description: "Enter the verification code",
  data: { value: "{{email.otp:get the 6 digit verification code:{{run.dynamicEmail}}}}" },
}

Email extraction needs a provider. Use the bundled emailsink one (works without a key; EMAILSINK_API_KEY raises limits) or write your own:

import { emailsinkProvider } from "@ahmadw/qa-agent/providers/emailsink";
configure({ email: emailsinkProvider({ apiKey: process.env.EMAILSINK_API_KEY }) });

// or
configure({
  email: {
    domain: "my-test-mail.com",
    extractContent: async ({ email, prompt }) => fetchAndExtract(email, prompt),
  },
});

Caching

Set REDIS_URL or configure({ redis: { url } }) (the configure value wins). Without Redis, caching, {{global.*}}, and {{data.*}} are off.

• Key: userFlow + step.description.
• Only runSteps, and only steps that resolved in a single AI action, are cached.
• Skip the cache with bypassCache: true on a step or the whole run. Playwright retries skip it automatically.

Reference

| Slot | Default | Used for | |---|---|---| | stepExecution | google/gemini-3-flash | Each runSteps step | | userFlowLow | google/gemini-3-flash | runUserFlow, effort low | | userFlowHigh | google/gemini-3.1-pro-preview | runUserFlow, effort high | | assertionPrimary | anthropic/claude-haiku-4.5 | First assertion model | | assertionSecondary | google/gemini-3-flash | Second assertion model | | assertionArbiter | google/gemini-3.1-pro-preview | Tie-breaker | | utility | google/gemini-2.5-flash | Extraction, wait conditions, JSON conversion | | cua | gpt-5.5 (locked) | CUA mode |

| Variable | When needed | Notes | |---|---|---| | ANTHROPIC_API_KEY | anthropic/* models (default) | | | GOOGLE_GENERATIVE_AI_API_KEY | google/* models (default), video assertions | | | OPENAI_API_KEY | openai/* models, CUA mode | | | ZAI_API_KEY | zai/* models | | | ZAI_REASONING_EFFORT | optional | low · high · max | | ZAI_BASE_URL | optional | default https://api.z.ai/api/coding/paas/v4; use https://api.z.ai/api/paas/v4 for pay-as-you-go | | AI_GATEWAY_API_KEY | gateway vercel | | | OPENROUTER_API_KEY | gateway openrouter | | | OPENCODEZEN_API_KEY | gateway opencodezen | | | CLOUDFLARE_ACCOUNT_ID · CLOUDFLARE_AI_GATEWAY | gateway cloudflare | | | CLOUDFLARE_AI_GATEWAY_API_KEY | authenticated Cloudflare gateway | sent as cf-aig-authorization | | REDIS_URL | caching, global/project data | configure({ redis }) wins | | AXIOM_TOKEN · AXIOM_DATASET | telemetry | configure({ telemetry }) wins | | QA_AGENT_LOG_LEVEL | optional | debug · info (default) · warn · error · silent |

Off unless both token and dataset are set, via env vars or:

configure({ telemetry: { axiomToken: process.env.MY_AXIOM_TOKEN, axiomDataset: "qa-agent-traces" } });

Every AI call is wrapped in a span.

Project

• Contributing: CONTRIBUTING.md · Troubleshooting: TROUBLESHOOTING.md
• Known gap: test coverage is thin, especially around failure modes. PRs welcome.
• License: FSL-1.1-ALv2, converting to Apache 2.0 two years after each release. qa-agent is a fork of Passmark by LinearBytes Inc.