playwright-codegen-ai

Playwright's code recorder, enhanced with AI-powered assertion generation and smart locator picking.

Run it instead of playwright codegen and you get the same recorder UI plus enhanced capabilities:

• AI Assertions — click a button and Ollama suggests expect() statements based on the current page state
• Smart Locators (automatic) — recorded actions are automatically upgraded from generic getByRole/getByText to the most stable locator available (data-testid, #id, href, [name], etc.), with no manual tab or toggle required

Requirements

| Dependency | Version | |---|---| | Node.js | >= 18 | | playwright or @playwright/test | >= 1.40.0 | | Ollama | any (for AI assertions only) |

Ollama is optional. Smart locators work without it. You only need Ollama running if you want the ✨ AI Assert button to generate assertions.

Installation

npm install --save-dev playwright-codegen-ai

Usage

# Open the recorder pointed at a URL
npx playwright-codegen-ai https://example.com

# Save output to a file
npx playwright-codegen-ai -o tests/my-test.spec.ts https://example.com

# Generate Python instead of TypeScript
npx playwright-codegen-ai --target=python https://example.com

# Use a specific Ollama model
npx playwright-codegen-ai --model=llama3 https://example.com

# Use a remote Ollama instance
npx playwright-codegen-ai --base-url=http://my-server:11434 https://example.com

The recorder opens like playwright codegen with the AI assertion enhancements in the Inspector UI. Smart locator upgrades run automatically in the background while you record.

UI Overview

Toolbar buttons

| Button | What it does | |---|---| | ✨ AI Assert | Sends the current ARIA snapshot to Ollama and shows suggested expect() calls. Accept or dismiss each one individually. Accepted assertions are inserted into the generated code after the last recorded action. |

Smart locator generation runs automatically in the background as you record actions.

Panel tabs

| Tab | Description | |---|---| | ✨ AI Assertions | Pending AI assertion suggestions. Accept inserts into code, reject dismisses. |

The panel presents AI assertion suggestions, while smart locator enhancement runs automatically for recorded actions.

Locator confidence and decision flow

Smart locator confidence is computed during enhancement and used to rank locator strategies before replacement.

• Confidence is computed in cli.js per strategy and used for ranking
• In single-process mode, replacement is applied only when the top strategy is >= 60
• In two-process mode (PLAYWRIGHT_TWO_PROCESS_VALIDATION=1), uniqueness and target-identity validation gates decide replacement

CLI options

All standard playwright codegen options are supported. The following are added or relevant:

| Option | Default | Description | |---|---|---| | --model <model> | qwen3-coder:480b-cloud | Ollama model for AI assertions | | --base-url <url> | http://localhost:11434 | Ollama server URL | | --target <language> | playwright-test | Output language: javascript, playwright-test, python, python-pytest, python-async, csharp, csharp-mstest, csharp-nunit, java, java-junit | | -o, --output <file> | — | Save generated script to a file | | --test-id-attribute <attr> | — | Attribute to use for test ID selectors | | -b, --browser <type> | chromium | cr/chromium, ff/firefox, wk/webkit | | --device <name> | — | Emulate a device, e.g. "iPhone 15" | | --viewport-size <WxH> | — | e.g. "1280, 720" | | --color-scheme <scheme> | — | light or dark | | --geolocation <lat,lon> | — | e.g. "51.5,-0.1" | | --lang <locale> | — | e.g. "en-GB" | | --timezone <tz> | — | e.g. "Europe/London" | | --proxy-server <proxy> | — | e.g. "http://myproxy:3128" | | --save-storage <file> | — | Save auth state for reuse | | --load-storage <file> | — | Load previously saved auth state | | --save-har <file> | — | Save a HAR of all network activity | | --ignore-https-errors | — | Ignore TLS certificate errors | | --timeout <ms> | — | Timeout for Playwright actions | | --two-process-validation | — | Enable shadow validator for live DOM uniqueness checks |

Environment variables

These can also be set in a .env, playwright.env, or .env.local file in the project root and will be picked up automatically on startup.

| Variable | Default | Description | |---|---|---| | PLAYWRIGHT_AI_MODEL | qwen3-coder:480b-cloud | Ollama model name | | PLAYWRIGHT_AI_BASE_URL | http://localhost:11434 | Ollama server base URL | | PLAYWRIGHT_AI_LOG_FILE | ./playwright-ai-assertions.log | Path for the AI interaction log | | PLAYWRIGHT_TWO_PROCESS_VALIDATION | 0 | Enables the shadow validator pipeline. Set to 1 to require live DOM uniqueness proof before replacing a locator |

AI assertions in detail

When you click ✨ AI Assert, the tool:

1. Takes an ARIA snapshot of the current page
2. Diffs it against the snapshot before the last recorded action to identify what changed
3. Sends the snapshot, the diff, and the recorded test steps so far to Ollama
4. Parses the model's response into individual expect() statements
5. Displays each suggestion in the panel — accept or dismiss per assertion

Accepted assertions are inserted into the generated script immediately after the action that triggered them.

Supported output languages: TypeScript, JavaScript, Python, C#, Java (same set as --target).

Ollama setup

# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh   # Linux / macOS
# Windows: https://ollama.com/download

# Start the server (runs on port 11434 by default)
ollama serve

# Pull a model
ollama pull qwen3-coder       # recommended
# or
ollama pull llama3
ollama pull codellama

Smart locator generation in detail

The enhancer runs after each recorded action and targets only generic recorder locators:

• Included: getByRole(...), getByText(...), getByLabel(...)
• Skipped: locators already containing getByTestId, #id, .nth(...), or .first()

It extracts action context (click, fill, check, press), analyzes the live DOM, and ranks generated strategies by confidence.

Strategies generated in current code

| Strategy family | Typical confidence in code | Example output | |---|---|---| | href-exact for links | 98 | page.locator('a[href*="/products"]').click() | | role-semantic-input for textbox / searchbox | 97 | page.locator('input[aria-label="Search"]').fill('foo') | | unique-id | 95 | page.locator('#email').fill('[email protected]') | | test-id | 90 | page.getByTestId('submit-btn').click() | | name-attribute / type-name-combo | 85–90 | page.locator('input[type="email"][name="email"]').fill('[email protected]') | | Non-dynamic data-* attributes | 80–85 | page.locator('[data-qa="nav-home"]').click() | | Container-scoped chains (id/form/nav/class) | 65–95 | page.locator('#checkout').locator('[name="card"]').fill('...') | | Positional fallbacks | 40–75 | page.locator('li').nth(2).locator('a').click() |

Notes from implementation:

• Link strategies intentionally prioritize href-based selectors over text when text is dynamic.
• Dynamic state attributes such as data-current, data-active, data-selected, and data-state are excluded.
• IDs used in chained selectors are CSS-escaped before generation.

Replacement decision by mode

| Mode | Replacement rule | |---|---| | Single-process (default) | Best ranked strategy is applied when confidence is >= 60 | | Two-process (PLAYWRIGHT_TWO_PROCESS_VALIDATION=1) | Replacement only if shadow validator proves uniqueness and target identity; otherwise original locator is kept |

Shadow validator: live DOM uniqueness validation (two-process)

Optionally, enable the shadow validator for safer locator replacement:

npx playwright-codegen-ai --two-process-validation https://example.com
# or set environment variable:
PLAYWRIGHT_TWO_PROCESS_VALIDATION=1 npx playwright-codegen-ai https://example.com

PLAYWRIGHT_TWO_PROCESS_VALIDATION controls whether locator replacement uses the shadow-validation pipeline in addition to normal enhancement:

• 0 (default): single-process enhancement only
• 1: two-process validation enabled (recommended for maximum safety)

In two-process mode, a shadow validator pipeline runs alongside normal enhancement and only allows replacement when it can prove the replacement targets the exact same element the user interacted with.

When enabled, every generated locator is validated in real-time against the live page DOM:

1. DOM analysis — runs a DOM script on the target page to find all elements matching the recorded selector
2. Target identification — determines which exact element was clicked using recorder's internal selector hints
3. Strategy generation — generates candidate locators using stable attributes (id, data-testid, href, name, etc.)
4. Uniqueness validation — tests each strategy in the live DOM; only accepts if it matches exactly 1 element
5. Safety gate — if the clicked element cannot be proven, keeps the original locator instead of guessing
6. Replacement — applies the best validated strategy (highest confidence, count=1) or keeps the original

What "two-process" means

1. Recorder process captures user actions and the recorder's native locator
2. Shadow validator stage verifies uniqueness and target identity on the live DOM
3. The final locator is replaced only when both agree the candidate is safe

If validation fails, the tool keeps Playwright's original recorded locator to avoid false replacements.

Validation features

| Feature | Details | |---|---| | DOM stability | Waits for DOM readyState and debounces mutations | | Element counting | Runs CSS selector queries on live DOM | | Target disambiguation | Extracts position hints (nth, id, role-name) to identify exact clicked element | | Parent chain walk | Walks up parents to find unique containers (form, nav, section, id, class) | | Actionability check | Ensures element is visible and not disabled | | Confidence scoring | Uses per-strategy confidence values defined in cli.js (for example href-exact 98, unique-id 95, test-id 90) | | Ambiguity detection | Rejects replacement if exact target element cannot be proven |

Example: enhancement outcome by mode

Input action recorded by Playwright:

await page.getByRole('searchbox', { name: 'Search' }).press('Enter');

Possible enhanced candidate from current strategy generator:

await page.locator('input[aria-label="Search"]').press('Enter');

What happens next:

1. Single-process mode: applies the top candidate when confidence threshold is met.
2. Two-process mode: applies only if live DOM validation confirms count = 1 and target identity; otherwise keeps the original getByRole(...) line.

Development (standalone project)

This is a self-contained project. playwright-core is installed as a dev dependency.

# Install dependencies
npm install

# Build TypeScript → lib/
npm run build

# Watch mode during development
npm run build:watch

# Build and create .tgz for local testing
npm run pack

Project structure

playwright-codegen-ai-standalone/
├── cli.js                    # Entry point — patches RecorderApp at runtime
├── postinstall.js            # Shown after npm install
├── src/
│   ├── index.ts              # Registers the codegen-ai CLI command
│   ├── recorderInjection.ts  # Browser-side UI injected into Playwright Inspector
│   └── aiAssertions/
│       ├── index.ts          # generateAssertions() — main entry point
│       ├── assertionAdvisor.ts   # Ollama API caller with chat history
│       ├── promptBuilder.ts      # Builds system/user prompts for the LLM
│       ├── snapshotDiff.ts       # ARIA snapshot line-set diffing
│       ├── languageTemplates.ts  # Per-language assertion examples
│       └── aiLogger.ts           # Logs LLM interactions to file
├── lib/                      # Compiled output (generated by npm run build)
├── scripts/
│   └── pack.js               # Build + npm pack helper
├── package.json
├── tsconfig.json
└── README.md

How it works

cli.js does three things at startup:

1. Injects aiAssertions — registers our generateAssertions() implementation into Node's require.cache under the path that playwright-core's recorderApp.js uses, so core picks it up without any modifications to core files.

2. Patches RecorderApp — intercepts Module._load to catch when recorderApp.js is loaded, then monkey-patches three methods: _init, _createDispatcher, and _updateActions.

3. Registers the CLI command — loads playwright-core's Commander program and registers a codegen-ai [url] subcommand with extra --model and --base-url options. When npx playwright-codegen-ai is called with flags or a URL but no explicit subcommand, codegen-ai is automatically injected into the argument list so that npx playwright-codegen-ai --target=playwright-test https://example.com works directly without typing the subcommand name.

Publishing to npm

# 1. Build and verify
npm run pack

# 2. Publish
npm publish
# or publish the .tgz directly:
npm publish playwright-codegen-ai-0.1.0.tgz

Troubleshooting

playwright-core not found

npm install --save-dev @playwright/test

AI assertions button does nothing / times out

Ollama must be running and have a model pulled:

ollama serve
ollama pull qwen3-coder

Check the log file (playwright-ai-assertions.log by default) for error details.

Locator enhancement not applying

Enhancement only triggers for getByRole, getByText, and getByLabel actions — the ones Playwright's recorder generates that are most likely to be non-unique. If the recorder already produced a getByTestId or #id locator, no replacement is needed.

License

Apache-2.0 — same as Playwright.