QA Wolf AI

AI-assisted Playwright actions and semantic assertions. Use your existing browser, pages, locators, and test runner with your own TypeSafe API key. A QA Wolf account or runner is not required for standalone web usage.

Install and run

Install alongside your existing Playwright project:

Requires Node.js >=24.14.1 <25 and Playwright >=1.55.0 <2. Both ESM and CommonJS Playwright projects are supported.

npm install --save-dev @qawolf/ai

Set TYPESAFE_API_KEY in your shell or CI secrets. Get a key from TypeSafe. Keep the key in the Node.js test process rather than browser code.

import { test, expect } from "@playwright/test";
import { createAi } from "@qawolf/ai/web";
import { typesafe } from "@qawolf/ai/providers/typesafe";

const ai = createAi({ provider: typesafe() });

test("a customer requests a refund", async ({ page }) => {
  await page.goto("/support");
  await ai.act(page, "Open the latest customer message", { timeout: 15_000 });
  const message = page.getByTestId("message");
  await expect(message).toBeVisible();
  await ai.expect(message).toSatisfy("The customer is requesting a refund");
});

Use your existing baseURL and authentication setup. ai.act chooses actions on the supplied page; it does not launch a browser or navigate to a generated URL. Keep ordinary Playwright commands and assertions for deterministic steps.

Playwright Test fixture

Add an ai fixture to your existing test definition. This works with custom fixtures and gives each test an independent client:

// fixtures.ts
import { test as base, expect } from "@playwright/test";
import { extendAi } from "@qawolf/ai/playwright";
import { typesafe } from "@qawolf/ai/providers/typesafe";

export const test = extendAi(base, { provider: typesafe() });
export { expect };

// support.spec.ts
import { test, expect } from "./fixtures";

test("a customer requests a refund", async ({ page, ai }) => {
  await page.goto("/support");
  await ai.act(page, "Open the latest customer message");
  const message = page.getByTestId("message");
  await expect(message).toBeVisible();
  await ai.expect(message).toSatisfy("The customer is requesting a refund");
});

Operations appear as named Playwright test steps. The fixture caps each operation's timeout to the configured test timeout, and Playwright's enclosing test timeout still applies. When the fixture ends, it cancels pending AI work. A qawolf-ai JSON attachment records goals, selected control labels, model names, probabilities, and assertion judgments, including decisions made before a failure. It does not include page text, accessibility snapshots, context, or fill values; goals and labels can still contain application data. Provider failures appear in the failed step's error.

Provider and client configuration

typesafe() reads TYPESAFE_API_KEY and the SDK's TYPESAFE_DEFAULT_MODEL environment variable, defaulting to jev-latest. You can configure the key and model explicitly:

const ai = createAi({
  provider: typesafe({ apiKey, defaultModel: "jev-latest" }),
  act: { maxSteps: 5, timeout: 20_000 },
  expect: { threshold: 0.9, timeout: 10_000 },
});

The provider also accepts baseURL, fetch, timeout (per HTTP attempt), and retry options. Operation deadlines cover all provider attempts. SDK payload logging is disabled. Browser observations go directly to TypeSafe; no QA Wolf service is contacted by this provider.

createAi accepts any provider with decideAction(state, signal) and evaluate(request) methods. createAct(decide) and createExpect({ evaluate }) remain available for lower-level integrations. Each client owns its configuration; creating one never changes another or the runner-configured API. Pass an AbortSignal as createAi({ provider, signal }) to cancel the client's operations, or pass signal to an individual ai.act call. Completed actions are not rolled back.

The namespace functions act and expect exported directly from @qawolf/ai/web require host configuration. Standalone users should call createAi; host integrations can continue using @qawolf/ai/_runner.

Web behavior

The package separates platform implementations under src/web and src/mobile; Android and iOS APIs are described in Android and iOS.

The loop observes visible, labelled controls in the current page, asks for one action from that closed set, executes it with Playwright, and observes again. The decision maker can select done or stuck, or wait when the page reports loading through visible aria-busy or progressbar elements. Waiting, transient page-read retries, and stale-decision retries share the original timeout. Retrying a page read never repeats an already executed action. Completion requires a separate judgment of the current page; a chosen click alone does not count as success. Return values contain { steps, history, url, model }. steps contains successfully executed actions; history also includes failed attempts, their choice and goal probabilities, and whether the next observation changed. A changed observation is evidence for the next decision, not proof of goal completion. Keep explicit assertions for the behavior the test is meant to verify.

Prefer one observable outcome per call. Completion confidence still needs calibration: live evaluation can reject completion even after the requested controls have been updated, including for focused goals. Keep explicit assertions for each result; validate completion behavior against your application before relying on it.

Options:

• maxSteps: number of action attempts (including failed Playwright actions), default 5, integer 1–20. A final completion check is allowed after the last action.
• timeout: total budget in milliseconds, default 60000, maximum 300000. Each Playwright action gets at most 5 seconds within that budget. Expiry cancels model evaluation and never schedules another action; actions already executed are not rolled back.
• signal: an optional AbortSignal that cancels the operation, including a pending provider request.
• values: up to 10 named strings used verbatim for filling. The model sees the names and whether each field equals a supplied value. After act fills a field with one supplied value, it does not fill that field with another in the same call; use a separate ai.act call to change it. The transport does not include the raw values, although page text or a goal that contains them is still sent as evidence. Use value names in the goal.

Supported actions are click, fill, hover, selectOption for native single-select controls, setChecked for checkboxes and radios, and press (Enter, Escape, Tab) on a focused control. Hover can reveal controls for the next observation, including CSS-only menus and tooltips on disabled controls. Filling never implicitly submits. Custom dropdowns use click/keyboard actions on their visible controls.

await ai.act(page, "Open Analytics from the Products menu");
await ai.act(page, "Select Pro in Plan");
await ai.act(page, "Check Email updates");

Controls are resolved through unique locators using observed IDs, names, labels, text, placeholders, or titles. The loop validates the selected target's description immediately before execution; a changed target triggers a fresh observation. Equivalent rerenders can use the same locator. Ambiguous controls without a unique locator are omitted, rather than resolved by DOM position. Playwright performs its normal action-specific waiting, without forced clicks or coordinate fallbacks.

The API operates in the main frame, including open shadow roots. The model does not generate selectors, URLs, input text, or JavaScript. Frames, popup navigation, uploads, drag-and-drop, and multi-select controls are outside its current scope. Use Playwright directly for those operations and for deterministic steps with stable locators. The caller chooses the page and goal and remains responsible for actions requested by that goal.

The loop rejects uncertain choices (selected probability below 0.8, unless it is at least 0.5 and at least 0.3 above the most likely other option), unconfirmed completion (goal probability below 0.9, or a done choice below 0.5; while the page reports loading, a done choice below 0.8 waits and observes again), missing choices, repeated actions with unchanged page text, headings, and control state, and budget exhaustion. These initial policy thresholds are not an accuracy guarantee. Failed Playwright actions are recorded before reobserving; attempts still consume the action budget. Error messages for unconfirmed completion include both probabilities and the attempt count without including fill values.

Observation caps page text at 32000 characters and rejects more than 500 action candidates. Changes to unrelated page text do not invalidate a selected action. URL/title changes or changes to the selected control trigger reobservation. Before accepting completion, the loop rechecks all observed evidence, including text and control states. Continuously changing completion evidence can still exhaust the timeout.

The application controls transport, authentication, and provider selection through its injected DecideAction implementation.

Semantic assertions

ai.expect(locator).toSatisfy(statement, { threshold, timeout, context }) judges whether the statement is true of one element. ai.expect offers only toSatisfy and .not.toSatisfy; use Playwright's expect for every other matcher. The evaluator returns { probability, model } for a snapshot of the element; provider access is supplied by the client configuration.

import { createExpect } from "@qawolf/ai/web";

const semanticExpect = createExpect({ evaluate });
await semanticExpect(page.locator("#message")).toSatisfy(
  "The customer is requesting a refund",
);

The default probability threshold is 0.9. Positive assertions require P(true) >= threshold; negated assertions require P(false) >= threshold. Uncertain probabilities fail in both directions. This threshold is an initial policy choice; validate it on representative examples for your application.

The assertion reads the selected element's innerText, with textContent as a fallback for non-HTML elements, and its Playwright accessibility snapshot, which carries roles, states such as checked and selected, and table structure. Values typed into text, search, number, and combobox fields, and link URLs, are removed from the snapshot. If a value typed into any field on the page would still appear in it, for example in another element's name, or the snapshot is longer than 64000 characters, the snapshot is left out and only the text is sent. It does not assert visibility or inspect screenshots. It waits for missing or empty text, rejects ambiguous locators, and evaluates each distinct snapshot once per assertion. The text and accessibility snapshot are rechecked before accepting a result. Evaluator failures never pass under negation. Wait for the application's completion signal before asserting streamed content.

The timeout defaults to 30000 milliseconds and covers reads, evaluation, and diagnostics; createExpect({ timeout }) changes the default and each assertion can override it. Evaluators receive an AbortSignal. onEvaluation can record the text, accessibility snapshot, context, probability, and model; diagnostic retention is the host's responsibility. Durations must be positive and finite.

Assertion tests use deterministic evaluators and cover retries, cancellation, stale text, and negation.

Android and iOS

@qawolf/ai/android and @qawolf/ai/ios provide vision-based taps and screenshot assertions:

import * as ai from "@qawolf/ai/ios";

await ai.tap(driver, "the checkout button");
await ai
  .expect(driver)
  .toMatchScreenshot("cart-page", "the cart with a Checkout button");

Both entries export tap and expect directly. Import them as a namespace so expect does not shadow the flow's assertion expect.

AI runs once per device model and OS version, and later runs replay the result without AI:

• ai.tap asks AI where the element is, stores the coordinates with a crop of the screen under them, and on later runs taps the stored coordinates once the crop matches again.
• toMatchScreenshot takes a name and a prompt. The name is a slug like cart-page (letters, digits, ., _ or -) that identifies the baseline; an invalid name fails with a suggested slug. The prompt describes what the screen should show. AI checks the first baseline for a name on each platform against the prompt before storing it. On a new device, AI judges the screenshot against the prompt and baselines from other devices of the same platform, and an accepted screenshot becomes that device's baseline. Later runs compare pixels.

The screenshot prompt can also name content that varies between runs, such as dates or balances, so AI ignores it when judging a new device or a heal.

Pass verify to ai.tap to have AI check what the screen shows after tapping an element it just found. If the check fails, the step fails and nothing is stored. Replays never run the check, and without verify no extra AI call is made:

await ai.tap(driver, "the Pay button", {
  verify: "the order confirmation is shown",
});
await ai
  .expect(driver)
  .toMatchScreenshot("cart-full", "the cart has 2 items; prices may differ");

A stored tap belongs to its exact prompt and verify, and a baseline to its name; changing a screenshot's prompt applies from the next new device or heal. If a stored value is wrong, reword the tap's prompt or verify, or rename the screenshot, and AI starts over on the next run.

When a replay fails, the step fails. Pass { autoHeal: true } to let AI re-assess the failure and update the stored value when it is not a real change. Results are stored in team storage under _ai_/. Each call logs its outcome to the run log, and the runner also writes it to the pod log, with heals under "@qawolf/ai healed a stored value". Calls that need a model require the flows_ai_assertions team feature.

The runner configures these APIs with configureAi from @qawolf/ai/_runner, providing the request channel to tester-agent, the storage directory, the run's target, logging, and artifact reporting. Everything under src/core is internal. Run npx jest in this package for the mobile unit tests.