Playwright Pilot AI

Turn plain English into production-ready Playwright tests — no coding required

What is Playwright Pilot AI?

Playwright Pilot AI is a plugin for Claude Desktop and Cursor that automatically writes, runs, and heals Playwright browser tests — just by describing what you want to test in plain English.

You do not need to know how to write Playwright code. You do not need to inspect HTML. You just describe the test in your own words, and Playwright Pilot AI generates everything for you.

The Problem It Solves

Writing Playwright tests manually is slow and repetitive:

• You open the browser, inspect elements, copy selectors
• You write Page Object Models by hand
• When the UI changes, selectors break and tests fail
• You spend hours figuring out why — not testing features

Playwright Pilot AI eliminates all of that.

How It Works — Simple Explanation

You type a message in Claude or Cursor describing what to test
              ↓
Playwright Pilot AI opens a real browser and looks at the page
              ↓
It reads every button, input, link, and form on the page automatically
              ↓
It generates a complete Page Object file (zero AI tokens — done locally)
              ↓
It asks Claude to write just the test file (~600 tokens — very cheap)
              ↓
It saves ready-to-run .js test files into your project folder
              ↓
You run: npx playwright test ✅

Everything from "looking at the page" to "building the Page Object" happens locally inside the plugin — no AI tokens used. Claude is only called once, to write the test logic. This makes it ~95% cheaper than using AI for every step.

Key Features

1. 🗣️ Plain English Test Generation

Just describe what you want to test. No selectors, no code, no HTML inspection needed.

"Generate a test for the login page at http://localhost:3000/login
— enter email and password, click Login, verify the dashboard appears"

2. 🌐 Live Browser Analysis

Playwright Pilot AI opens a real Chromium browser, navigates to your page, and automatically discovers every interactive element — buttons, inputs, dropdowns, checkboxes, links, tables, modals, and more.

3. 📄 Auto-Generated Page Object Models

The complete Page Object Model (POM) is generated locally with zero AI tokens. It includes:

• One property per element on the page
• Smart locators using getByRole, getByLabel, getByText (Playwright best practices)
• Rich metadata stored for future self-healing
• Ready-to-use action methods (fillEmailField(), clickLoginButton(), etc.)

4. 🎬 Video Recording to Test

Record yourself using your application and Playwright Pilot AI converts the recording into a Playwright test automatically. It extracts frames, reads the screen, identifies each action, and writes the test.

5. 📋 Steps-Only Mode (No Browser Needed)

If you don't have a URL ready, just list the steps manually. Playwright Pilot AI will write the test from your description alone — no browser required.

6. 🏗️ Auto-Scaffolds New Projects

Point it at an empty folder. Playwright Pilot AI will automatically create a complete, production-standard project setup including:

• package.json with all dependencies
• playwright.config.js configured for 3 browsers
• GitHub Actions CI workflow
• Folder structure (pages/, tests/, ai-helpers/, ai-artifacts/)

Then it writes your test into that fresh project.

7. 🔄 Self-Healing Tests

When the UI of your application changes (buttons renamed, elements moved), tests don't just break — they heal themselves:

• Stores rich metadata about every element (role, label, text, placeholder, testId)
• When a locator fails, automatically tries 5+ alternative ways to find the same element
• Never stops mid-test — collects all failures and reports at the end
• Zero AI tokens needed for healing in most cases

8. 📦 Portable YAML Test Definitions

Every generated test also creates a YAML file that describes the test in a portable format. You can re-run the test directly from the YAML without needing the spec file — useful for quick re-runs, CI pipelines, or sharing tests across teams.

9. 💰 Token-Efficient (Up to 95% Cheaper)

Unlike tools that send the entire page to AI for every step, Playwright Pilot AI does almost everything locally:

| Task | Who does it | Cost | |------|-------------|------| | Open browser + read DOM | Local | Free | | Find all elements | Local | Free | | Build Page Object Model | Local | Free | | Choose best locators | Local | Free | | Write test spec | Claude (1 call) | ~600 tokens |

Compare that to tools that use thousands of tokens per page interaction.

10. 🎯 Works With Your Existing Project

If you already have a Playwright project, Playwright Pilot AI detects your:

• File naming conventions
• Import style (CommonJS or ES modules)
• Test folder structure
• Existing base classes

Generated code matches your project's style exactly.

What You Get After Running It

For every test you generate, these files are saved to your project:

your-project/
├── pages/
│   └── login-page.js              ← Page Object with all elements and methods
│
├── tests/
│   └── login.spec.js              ← The Playwright test spec
│
├── ai-helpers/
│   └── step-tracker.js            ← Self-healing runtime helper
│
└── ai-artifacts/
    ├── yaml/login.yaml            ← Portable test definition (run without spec)
    └── intents/login.json         ← Stored intent for healing and re-execution

Run the test immediately:

npx playwright test tests/login.spec.js

Real Example

You type:

Generate a test for http://localhost:3000/login
Steps:
1. Navigate to the login page
2. Leave email and password empty
3. Click the Login button
4. Verify the error message "Please enter your credentials" appears

Save to /Users/me/projects/my-app

Playwright Pilot AI shows you a plan:

## Playwright Pilot AI — Generation Plan

Test: empty-login-validation
URL:  http://localhost:3000/login
Codebase: New project — will be scaffolded automatically

Files that will be created:
  pages/empty-login-validation-page.js
  tests/empty-login-validation.spec.js
  ai-helpers/step-tracker.js
  ai-artifacts/yaml/empty-login-validation.yaml

Tools that will run:
  1. Open browser (local, 0 tokens)
  2. Analyze page (local, 0 tokens)
  3. Generate test (Claude writes spec, ~600 tokens)
  4. Save files (local, 0 tokens)

Shall I proceed?

You say "yes" — and it generates:

// pages/empty-login-validation-page.js (generated locally — 0 tokens)
const { expect } = require('@playwright/test');

const ELEMENT_CONTEXTS = {
  emailField:   { key:'emailField',   label:'Email',    actionType:'fill',  originalLocator:"page.getByLabel(/email/i)" },
  passwordField:{ key:'passwordField',label:'Password', actionType:'fill',  originalLocator:"page.getByLabel(/password/i)" },
  loginButton:  { key:'loginButton',  text:'Login',     actionType:'click', originalLocator:"page.getByRole('button',{name:/login/i})" },
  errorMessage: { key:'errorMessage', actionType:'assert',                  originalLocator:"page.getByText(/please enter your credentials/i)" },
};

class EmptyLoginValidationPage {
  constructor(page) {
    this.page = page;
    this.url = 'http://localhost:3000/login';
    this.emailField    = page.getByLabel(/email/i);
    this.passwordField = page.getByLabel(/password/i);
    this.loginButton   = page.getByRole('button', { name: /login/i });
    this.errorMessage  = page.getByText(/please enter your credentials/i);
  }
  async goto() { await this.page.goto(this.url, { waitUntil: 'domcontentloaded' }); }
  async fillEmailField(value) { await this.emailField.fill(value); }
  async fillPasswordField(value) { await this.passwordField.fill(value); }
  async clickLoginButton() { await this.loginButton.click(); }
}
module.exports = { EmptyLoginValidationPage, ELEMENT_CONTEXTS };

// tests/empty-login-validation.spec.js (written by Claude — ~600 tokens)
const { test, expect } = require('@playwright/test');
const { StepTracker } = require('../ai-helpers/step-tracker');
const { EmptyLoginValidationPage, ELEMENT_CONTEXTS } = require('../pages/empty-login-validation-page');

test.describe('empty login validation', () => {
  test('shows error when login is attempted without credentials', async ({ page }) => {
    const tracker = new StepTracker(page, 'empty-login-validation', ELEMENT_CONTEXTS);
    const po = new EmptyLoginValidationPage(page);

    await tracker.step(0, 'Navigate to login page', () => po.goto());
    await tracker.step(1, 'Click Login without filling fields', () => po.clickLoginButton());
    await tracker.step(2, 'Verify error message appears', async () => {
      await expect(po.errorMessage).toBeVisible();
      await expect(po.errorMessage).toHaveText(/please enter your credentials/i);
    });

    await tracker.done();
  });
});

Then run it:

npx playwright test   # ✅ passes

Installation

Step 1 — Install the package

npm install -g playwright-pilot-ai

Step 2 — Set up Claude Desktop

Open: ~/Library/Application Support/Claude/claude_desktop_config.json

Add this entry:

{
  "mcpServers": {
    "playwright-pilot-ai": {
      "command": "node",
      "args": ["/usr/local/lib/node_modules/playwright-pilot-ai/dist/mcp/server.js"]
    }
  }
}

Find your exact path: run npm root -g in terminal, then append /playwright-pilot-ai/dist/mcp/server.js

Step 3 — Restart Claude Desktop

Close and reopen Claude Desktop. You should see playwright-pilot-ai in the MCP tools list.

Step 4 — Install Playwright browsers (first time only)

npx playwright install chromium

Setup for Cursor

Add to .cursor/mcp.json in your project:

{
  "mcpServers": {
    "playwright-pilot-ai": {
      "command": "playwright-pilot-mcp"
    }
  }
}

Setup From Source (Developers)

cd playwright-pilot-ai
npm install
npm run build

Then point your Claude Desktop config at: /path/to/playwright-pilot-ai/dist/mcp/server.js

How to Use — Just Type Naturally

Once set up, open Claude Desktop or Cursor and describe what you want to test. Playwright Pilot AI activates automatically.

Generate a test from a live page

Generate a test for the login page at http://localhost:3000/login
Save the tests to /Users/me/projects/myapp

Generate a test with specific steps

Create a Playwright test for https://myapp.com/register with these steps:
1. Fill in first name, last name, email and password
2. Check the Accept Terms checkbox
3. Click the Register button
4. Verify a success message appears
Save to /Users/me/projects/myapp

Generate from a screen recording

I recorded the checkout flow at /Users/me/recordings/checkout.mp4
Convert it to a Playwright test for https://shop.example.com
Save to /Users/me/projects/shop-tests

Run a test from its YAML definition (no spec file needed)

Run the yaml test for "login-flow"

Re-run a saved test with healing

Run the "empty-login-validation" test

Fix a broken test step

The "checkout-flow" test step 2 is failing — the promo code input wasn't found.
Please heal it.

The 9 Tools (Reference)

You never call these directly — Claude calls them automatically. This section explains what each tool does so you understand what's happening.

pwpilot_preview_generation — Always Called First

What it does: Before opening any browser or spending any tokens, this tool shows you a complete plan:

• Exactly which files will be created and where
• Which tools will run in what order
• Whether it will scaffold a new project
• The detected style of your existing codebase (if any)

It waits for you to say "yes" or "go ahead" before doing anything.

Cost: Zero — runs entirely locally.

pwpilot_launch_browser — Opens the Browser

What it does: Opens a visible Chromium browser window and navigates to your URL. You can watch it happen.

Cost: Zero — runs locally.

pwpilot_analyze_page — Reads the Page

What it does: Scans the entire page DOM and extracts every interactive element. Runs 9 local analysis skills:

| Skill | What it finds | |-------|--------------| | Element detection | Every button, input, link, dropdown, checkbox | | Smart locators | Best Playwright locator for each element | | Form detection | Labels, required fields, submit buttons | | Navigation detection | Menus, tabs, sidebars, breadcrumbs | | Login flow detection | Email/password fields, OTP, MFA | | Assertion builder | Ready-to-use expect() lines | | Table detection | Data grids and tables | | Popup detection | Modals, alerts, cookie banners | | Healing metadata | Stores rich info for future self-healing |

Cost: Zero — runs entirely locally, no AI tokens.

pwpilot_generate_test — Builds the POM, Asks Claude for Spec

What it does:

1. Generates the complete Page Object Model locally (zero tokens)
2. Sends a compact ~600-token summary to Claude
3. Claude writes only the test spec — it never sees the full DOM

Cost: ~600 tokens (Claude writes the spec only).

pwpilot_save_test — Writes Files to Disk

What it does: Saves all generated files to your project folder:

• Page Object Model (.js)
• Test spec (.js)
• StepTracker self-healing helper
• Intent JSON (for future healing and re-execution)
• YAML test definition (for running without the spec)

If the folder is empty, it also scaffolds a complete Playwright project first.

Cost: Zero — runs locally.

pwpilot_video_to_test — Converts Video or Steps to Test

What it does:

• From video: Uses ffmpeg to extract frames, reads the screen (OCR), identifies each action, writes the test
• From steps: Takes your list of steps and writes the complete test (POM + spec)

No live browser needed for either mode.

Cost: ~800 tokens (Claude writes both POM and spec).

Requires ffmpeg for video mode. Install: https://ffmpeg.org/download.html

pwpilot_run_yaml — Runs Test Directly from YAML

What it does: Every saved test has a ai-artifacts/yaml/<name>.yaml file. This tool runs that YAML file directly in a Playwright browser — no spec file needed.

Each step in the YAML has:

• A primary locator — tried first
• Fallback locators — tried automatically if the primary fails (self-healing)

Cost: Zero — runs entirely locally.

Use it when:

• You want to quickly re-run a test without the spec file
• The spec file was deleted or not committed
• You want to share tests as YAML across teams

pwpilot_execute_test — Runs a Saved Test

What it does: Executes a saved test intent in a headed browser. Uses StepTracker to:

• Run each step
• Auto-heal any failing steps using stored element metadata
• Never stop on first failure — collect all failures and report at the end

Cost: Zero for local healing. Small token cost if healing needs to escalate to Claude.

pwpilot_heal_test — Repairs a Broken Step

What it does: When a test step fails because a locator no longer works (e.g. a button's text changed), this tool:

1. Looks up the stored metadata for that element (role, label, text, placeholder, testId)
2. Tries 5+ alternative locators automatically
3. If one works — reports the healed locator back (you can update your POM)
4. If none work — generates a detailed prompt for Claude to find a new locator

Cost: Zero for local healing. Small token cost only if escalated to Claude.

All Supported Flows

Flow 1 — Test From Live URL (most common)

You describe the test + give a folder path
         ↓
Preview plan shown → you confirm
         ↓
Browser opens → page analyzed → POM built (0 tokens)
         ↓
Claude writes spec (~600 tokens)
         ↓
Files saved to your project ✅

Flow 2 — Test From Screen Recording

You provide a video file + URL + folder path
         ↓
Preview plan shown → you confirm
         ↓
ffmpeg extracts frames → OCR reads screen → steps detected (local)
         ↓
Claude writes POM + spec (~800 tokens)
         ↓
Files saved to your project ✅

Flow 3 — Test From Manual Steps

You list numbered steps + URL + folder path
         ↓
Preview plan shown → you confirm
         ↓
Claude writes POM + spec from your steps (~800 tokens)
         ↓
Files saved to your project ✅

Flow 4 — Run From YAML (no spec needed)

"Run the yaml test for X"
         ↓
Loads ai-artifacts/yaml/X.yaml
         ↓
Playwright executes each step directly
Primary locator → fallbacks if needed (self-healing)
         ↓
Pass/fail report shown ✅

Flow 5 — Execute Saved Test + Auto-Heal

"Run the X test"
         ↓
Loads saved intent → opens browser
         ↓
Runs each step with StepTracker
         ↓
On failure: tries 5+ alternative locators (0 tokens)
         ↓
Still failing: escalates to Claude for new locator
         ↓
Full report: passed / healed / failed per step ✅

New Project Scaffolding

If you give Playwright Pilot AI an empty folder, it automatically sets up a complete project before writing any tests.

What gets created:

| File | Purpose | |------|---------| | package.json | All Playwright dependencies, npm scripts | | playwright.config.js | Chromium + Firefox + WebKit, HTML reports, CI settings | | .eslintrc.json | JavaScript linting rules | | .env.example | Template for your app's base URL | | .gitignore | Excludes node_modules, test results, AI cache | | .github/workflows/playwright.yml | GitHub Actions CI — runs on 3 browsers in parallel | | pages/ | Folder for Page Object Models | | tests/ | Folder for test specs | | ai-helpers/ | Folder for StepTracker helper | | ai-artifacts/ | Folder for YAML definitions and healing data | | README.md | Project documentation |

After scaffolding, run:

cd /your/project/folder
npm install
npx playwright install --with-deps
cp .env.example .env    # Edit .env and set BASE_URL
npx playwright test     # Run your first test ✅

Self-Healing — How It Works

Every element in a generated Page Object has rich metadata stored alongside it:

loginButton: {
  key: 'loginButton',
  role: 'button',
  text: 'Login',
  label: 'Login',
  originalLocator: "page.getByRole('button', {name:/login/i})"
}

When a test step fails (e.g. the button text changed to "Sign In"), the StepTracker:

1. Tries getByRole('button', {name:/sign in/i}) — from stored role
2. Tries getByLabel(/login/i) — from stored label
3. Tries getByText(/login/i) — from stored text
4. Tries getByPlaceholder(...) — from stored placeholder
5. Tries getByTestId(...) — from stored testId
6. Scans the live DOM semantically for anything matching the description
7. If healed → logs the new locator so you can update your POM
8. If not healed → escalates to Claude with full context

Tests never abort on first failure. All steps run, all failures are collected, and a complete report is shown at the end.

YAML Test Definitions

Every test you generate automatically creates a portable YAML file:

# ai-artifacts/yaml/empty-login-validation.yaml
version: "1.0"
name: "empty-login-validation"
url: "http://localhost:3000/login"

steps:
  - index: 0
    action: navigate
    value: "http://localhost:3000/login"

  - index: 1
    action: click
    description: "Click Login button"
    locator:
      strategy: getByRole
      value: button
      options: { name: "Login" }
    fallbackLocators:
      - { strategy: getByText, value: "Login" }
      - { strategy: getByTestId, value: "login-btn" }

  - index: 2
    action: assert
    description: "Error message is visible"
    assertion:
      type: visible
      locator: { strategy: getByText, value: "Please enter your credentials" }
      expected: true

You can run this at any time:

Run the yaml test for "empty-login-validation"

Token Cost Comparison

| Approach | Tokens per test | |----------|-----------------| | Regular AI browser tools | 20,000 – 50,000+ | | Playwright Pilot AI | 500 – 800 | | Savings | ~95% |

Requirements

| Requirement | Version | |-------------|---------| | Node.js | >= 18 | | Claude Desktop or Cursor | Latest | | Playwright | >= 1.44 (installed as peer dependency) | | ffmpeg | Any recent version (video mode only) |

Troubleshooting

"I don't see Playwright Pilot AI tools in Claude" → Check the path in your claude_desktop_config.json points to the correct dist/mcp/server.js file, then restart Claude Desktop.

"Which folder should I save the tests to?" → Playwright Pilot AI always asks for a folder path if you don't include one. Provide the full absolute path, e.g. /Users/me/projects/myapp.

"Browser failed to open" → Run npx playwright install chromium to install the browser.

"ffmpeg not found" → Only needed for video-to-test. Install from ffmpeg.org. Steps-only mode works without it.

"Tests saved to wrong location" → Make sure your prompt includes a folder path, or provide it when Claude asks.

"A test step keeps failing" → Use pwpilot_heal_test to repair it. Check ai-artifacts/contexts/ for stored element metadata.

"Stale results / wrong elements detected" → Delete ai-artifacts/.ai-cache/ to force a fresh page scan. Cache expires automatically after 24 hours.

Node.js version error → Run node --version. You need v18 or higher. Download from nodejs.org.

License

MIT — see LICENSE