autopilot-web

The web backend for AutoPilot — runs the same declarative JSON test plans against a browser, via Playwright. It is a first-class test backend (headless, for CI parity, like the macOS/iOS/Android backends) and a demo/screencast driver (headed, with on-page highlight/caption overlays and paced typing).

AutoPilot plans are plain JSON, designed to be authored by hand or by an AI agent. The same plan the native runners execute drives the browser here — the JSON plan schema is the single cross-platform contract. This runner reimplements the plan-execution loop over Playwright, exactly as the Android runner reimplements it over UiAutomator and the iOS runner over XCUITest.

AutoPilot is a web testing tool. (Earlier AutoPilot docs said it was not — that is no longer true; web is a supported platform.)

What runs on web, and what is skipped

Everything with a browser equivalent is exercised and asserted:

• Actions: launch (open the URL), click, doubleClick, rightClick, press, type (with clear/commit/paced typing), setValue, keyPress, scroll, drag, waitFor, screenshot, wait, assert, and the demo actions highlight / caption / pace.
• Assert properties: value, title, enabled, focused, marked (checkbox/ radio), position, size, count — with ops equals / notEquals / contains / matches / exists / notExists / greaterThan / lessThan.

Steps with no web equivalent are skipped (reported as skipped, never failed — the same "skip-don't-branch" contract the mobile backends use):

• menu (native menu bar), assertPixel / assertRegion / snapshot (native pixel capture), exec (host command), and the assert properties clipboard / stdout / stderr / exitCode.

Prerequisites — install Node.js (18+)

You need Node.js 18 or newer. Install it for your OS:

• Debian: sudo apt-get update && sudo apt-get install -y nodejs npm (for a current version, use NodeSource: curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash - && sudo apt-get install -y nodejs)
• Ubuntu: same as Debian (NodeSource recommended for a current version).
• RHEL / Fedora: sudo dnf install -y nodejs npm (or NodeSource: curl -fsSL https://rpm.nodesource.com/setup_20.x | sudo bash - && sudo dnf install -y nodejs)
• Windows: download the LTS installer from https://nodejs.org/ and run it, or winget install OpenJS.NodeJS.LTS.
• macOS: brew install node, or download the installer from https://nodejs.org/.

Verify: node --version (should print v18… or higher) and npm --version.

Install

From the repo directory:

npm install                      # install dependencies (Playwright + TypeScript)
npx playwright install chromium  # download the Chromium browser Playwright drives
npm run build                    # compile the TypeScript to dist/

Run a plan

# Headless (the normal test path) against the bundled TestHost page:
node dist/src/index.js run plans/test-web-capabilities.json --url testhost/index.html

# Against a live site:
node dist/src/index.js run plans/my-plan.json --url https://example.com

# Print the report as JSON (for tooling / CI):
node dist/src/index.js run plans/my-plan.json --url https://example.com --json

--url accepts an http(s):// URL or a local file path (a bare path is opened as a file:// URL). A plan may also set target.url; the --url flag overrides it.

The process exits non-zero if the plan does not pass, so CI can gate on it.

Demo / screencast mode

# Headed browser, with on-page highlight + caption overlays and paced typing,
# recording a video of the run:
node dist/src/index.js run plans/my-plan.json --url https://example.com \
  --headed --demo --record ./recordings

• --headed shows the browser window (omit for headless).
• --demo renders highlight/caption overlays and applies pace cadence (per-char typing + post-step delays). Without --demo, the demo actions are passing no-ops — a plan runs as a fast, clean test.
• --record <dir> writes a .webm video of the run into <dir>.

Test (the CI parity gate)

npm test

This builds, then runs the web-capabilities plan against the bundled testhost/index.html and asserts the expected pass/skip counts — the gate that makes web a real test backend. It uses Node's built-in test runner (node:test); no extra test framework is required.

Layout

• src/planModel.ts — the Plan/Step/Selector/Assert types, mirroring the core schema.
• src/runner.ts — the plan-execution loop over Playwright (selector mapping, actions, assertions, demo overlays, skip-don't-branch).
• src/index.ts — the CLI.
• testhost/index.html — the web TestHost (mirrors the native TestHostApp surface).
• plans/test-web-capabilities.json — the plan run by the parity gate.
• test/runner.test.ts — the parity-gate tests.

Selector mapping

AutoPilot selectors map onto Playwright locators:

| AutoPilot selector | Playwright locator | | --- | --- | | identifier: "foo" | page.locator('#foo') | | role: "button", title: "Save" | page.getByRole('button', { name: 'Save' }) | | role: "list" | page.getByRole('list') | | title: "Save" (no role) | page.getByText('Save') | | index: 2 | .nth(2) | | within: {...} | resolve the parent, then locate the child inside it |

Native AX roles (e.g. AXButton, AXTextField) are mapped to their ARIA equivalents (button, textbox); plain ARIA roles pass through unchanged.