playwright-genie

Find and interact with Playwright elements using natural language — powered by any LLM

playwright-genie lets you write Playwright tests in plain English. No more hunting for selectors — just describe the element and the genie finds it.

Features

• Natural Language — describe elements in plain English, no selectors needed
• 40+ Playwright Actions — click, fill, check, hover, drag, wait, screenshot and more
• Any LLM Provider — OpenAI, Claude, Ollama, Azure, or any OpenAI-compatible API
• Adaptive Page Analysis — automatically switches between ARIA, hybrid, and DOM-only modes based on page accessibility quality
• DOM Tree Pipeline — generates XPath and CSS selectors in-browser for pages with poor accessibility
• Smart Caching — two-tier cache (memory + disk .locator-cache.json) to minimize LLM calls
• Fallback Locator Chains — LLM returns multiple locator strategies; if the primary fails, fallbacks are tried automatically
• Action-Aware — fill('username') targets the input, not the label
• Auto-Retry — stale cached locators are automatically invalidated and re-resolved
• Batch Resolution — prefetch() resolves multiple queries in a single LLM call
• iframe Support — automatically detects and resolves elements inside iframes
• TypeScript Support — full type definitions included
• Single Page Object — one createSmartLocator(page) works across all navigations

Installation

npm install playwright-genie

Prerequisites

• Node.js >= 18
• Playwright >= 1.40
• An LLM API key (OpenAI, Anthropic, or any OpenAI-compatible provider)

Setup

Create a .env file in your project root:

# Option 1: OpenAI
LLM_API_KEY=sk-your-openai-key
LLM_MODEL=gpt-4o-mini

# Option 2: Anthropic (via OpenAI-compatible proxy)
LLM_API_KEY=your-anthropic-key
LLM_BASE_URL=https://your-proxy.com/v1
LLM_MODEL=claude-sonnet-4-20250514

# Option 3: Ollama (local, free)
LLM_API_KEY=ollama
LLM_BASE_URL=http://localhost:11434/v1
LLM_MODEL=llama3

# Option 4: Azure OpenAI
LLM_API_KEY=your-azure-key
LLM_BASE_URL=https://your-resource.openai.azure.com/openai/deployments/your-deployment
LLM_MODEL=gpt-4o-mini

Also supports OPENAI_API_KEY, ANTHROPIC_API_KEY, or ROUTELLM_API_KEY as fallbacks.

Quick Start

With Playwright Test

import { test } from '@playwright/test';
import { createSmartLocator } from 'playwright-genie';

test('login flow', async ({ page }) => {
  const smart = createSmartLocator(page);

  await page.goto('https://myapp.com/login');
  await smart.fill('username', 'admin');
  await smart.fill('password', 'secret123');
  await smart.click('login button');
  await smart.waitForVisible('welcome heading');
});

Standalone Script

import { chromium } from 'playwright';
import { createSmartLocator } from 'playwright-genie';

const browser = await chromium.launch();
const page = await browser.newPage();
const smart = createSmartLocator(page);

await page.goto('https://myapp.com');
await smart.click('sign in link');
await smart.fill('email field', '[email protected]');
await smart.fill('password field', 'secret');
await smart.click('submit button');

await browser.close();

How It Works

When you call smart.click('login button'), the library:

1. Collects page structure — gathers the ARIA accessibility tree, interactive elements, special attributes (data-testid, placeholder, aria-label), and a full DOM tree with XPath/CSS selectors
2. Evaluates ARIA quality — scores the page as good, sparse, or none based on how many named interactive elements the ARIA tree contains
3. Builds an adaptive payload — selects the best strategy:
   • aria mode — rich ARIA tree with good accessibility; uses ARIA snapshot + special elements
   • hybrid mode — sparse ARIA; combines the ARIA tree with DOM tree nodes for better coverage
   • dom mode — no useful ARIA; sends DOM tree with XPaths and CSS selectors generated in-browser
4. Queries the LLM — sends the payload with your natural language query; the LLM returns a Playwright locator string (e.g., getByRole('button', { name: 'Login' })) along with fallback locators
5. Validates and caches — verifies the locator resolves to a real element, caches it to memory and disk, and returns a SmartAction for interaction
6. Auto-recovers — if a cached locator goes stale, it's invalidated and re-resolved; if the primary locator fails, fallback locators are tried automatically

API Reference

createSmartLocator(page, options?)

Creates a smart locator instance bound to a Playwright page. Works across navigations — no need to recreate it.

const smart = createSmartLocator(page, { verbose: true });

Options:

| Option | Type | Default | Description | |--------|------|---------|-------------| | verbose | boolean | false | Log resolved locators to console | | debug | boolean | false | Enable detailed debug logging | | model | string | env var | Override LLM model | | temperature | number | 0 | LLM temperature | | maxTokens | number | 1024 | Max response tokens | | actionTimeout | number | 10000 | Timeout for actions in ms |

Interaction Actions

await smart.click('login button');
await smart.click('submit', { force: true });

await smart.dblclick('editable cell');

await smart.fill('username', 'Admin');
await smart.fill('email field', '[email protected]', { timeout: 5000 });

await smart.type('search box', 'hello');

await smart.pressSequentially('otp input', '123456', { delay: 100 });

await smart.press('search box', 'Enter');

await smart.clear('email field');

await smart.hover('profile menu');

await smart.focus('first input');

await smart.tap('mobile menu icon');

await smart.select('country dropdown', 'India');

await smart.selectText('paragraph content');

Checkbox & Radio

await smart.check('remember me checkbox');

await smart.uncheck('newsletter opt-in');

await smart.setChecked('terms checkbox', true);

File Upload

await smart.setInputFiles('file upload', '/path/to/file.pdf');
await smart.setInputFiles('avatar input', ['/img1.png', '/img2.png']);

Drag & Drop

const { source, target } = await smart.dragTo('card item', 'drop zone');

Wait Actions

await smart.waitForVisible('success toast');
await smart.waitForVisible('modal', 10000);

await smart.waitForHidden('loading spinner');

await smart.waitForAttached('dynamic table');

await smart.waitForDetached('old modal');

await smart.waitFor('element', { state: 'visible', timeout: 5000 });

State Queries

const visible  = await smart.isVisible('error message');
const hidden   = await smart.isHidden('loading spinner');
const enabled  = await smart.isEnabled('submit button');
const disabled = await smart.isDisabled('locked field');
const checked  = await smart.isChecked('terms checkbox');
const editable = await smart.isEditable('readonly field');
const found    = await smart.exists('optional element');

Content Retrieval

const text  = await smart.getText('welcome heading');
const inner = await smart.getInnerText('article body');
const html  = await smart.getInnerHTML('rich content area');
const value = await smart.getInputValue('email field');
const attr  = await smart.getAttribute('link', 'href');
const box   = await smart.getBoundingBox('hero image');
const num   = await smart.count('list items');

Scroll & Visual

await smart.scrollIntoView('footer section');

const buffer = await smart.screenshot('chart area', { path: 'chart.png' });

await smart.highlight('target element');

smart.locate() — Resolve Once, Act Many Times

When you need multiple actions on the same element, use locate() to resolve the locator once:

const el = await smart.locate('username');
await el.clear();
await el.fill('NewAdmin');
await el.press('Tab');
console.log(await el.inputValue());
console.log(await el.isEnabled());

// Access the raw Playwright locator
const loc = el.rawLocator;

// SmartAction has 40+ methods matching Playwright's Locator API
await el.click();
await el.hover();
await el.screenshot({ path: 'element.png' });
await el.waitForVisible();
await el.evaluate((node) => node.style.border = '2px solid red');

smart.prefetch() — Batch Resolve

Pre-resolve multiple locators in a single LLM call to save time and cost:

await smart.prefetch('username', 'password', 'login button');

// These now hit the cache — no LLM calls
await smart.fill('username', 'Admin');
await smart.fill('password', 'secret');
await smart.click('login button');

Cache Management

smart.clearCache();       // clear in-memory cache only
smart.clearAllCaches();   // clear both memory + disk (.locator-cache.json)

Use clearCache() after SPA navigations where the DOM changes significantly (e.g., after login) to force fresh page analysis.

Low-Level API

For advanced use cases, you can use the lower-level exports directly.

findLocator(page, query, options?)

Resolves a single natural language query to a Playwright locator string without performing any action.

import { findLocator } from 'playwright-genie';

const result = await findLocator(page, 'submit button');
console.log(result);
// {
//   found: true,
//   strategy: 'role',
//   locatorString: "getByRole('button', { name: 'Submit' })",
//   confidence: 0.95,
//   fallbackLocators: ["getByTestId('submit-btn')", "locator('#submit')"],
//   isInFrame: false,
//   frameSelector: null
// }

findAllMatches(page, query, options?)

Returns an array of all matching locator results for a query.

import { findAllMatches } from 'playwright-genie';

const matches = await findAllMatches(page, 'navigation link');

getPageStructure(page, forceRefresh?)

Returns the full page structure used for LLM resolution. Useful for debugging or building custom pipelines.

import { getPageStructure } from 'playwright-genie';

const structure = await getPageStructure(page);
console.log(structure.mainFrame.ariaQuality); // 'good' | 'sparse' | 'none'
console.log(structure.mainFrame.ariaTree);    // ARIA snapshot (YAML string)
console.log(structure.mainFrame.domTree);     // Array of DOM nodes with XPath/CSS
console.log(structure.mainFrame.interactiveElements); // Interactive element metadata
console.log(structure.mainFrame.specialElements);     // Elements with data-testid, etc.
console.log(structure.frames);                // iframe structures

resolveLocator(page, query, options?)

Low-level resolver that checks memory cache → disk cache → LLM. Returns the raw result object without creating a Playwright locator.

import { resolveLocator } from 'playwright-genie';

const result = await resolveLocator(page, 'login button', { action: 'click' });
// result.source is 'memory', 'disk', or 'llm'

getLocator(page, query, options?)

Resolves a query and returns both the Playwright Locator object and the result metadata. Handles stale cache invalidation and fallback chains.

import { getLocator } from 'playwright-genie';

const { locator, result } = await getLocator(page, 'email input', { action: 'fill' });
await locator.fill('[email protected]');

clearCache() / clearAllCaches()

Module-level cache clearing functions.

import { clearCache, clearAllCaches } from 'playwright-genie';

clearCache();       // memory only
clearAllCaches();   // memory + disk

Adaptive Page Analysis

The library automatically adapts to the accessibility quality of each page:

| ARIA Quality | Criteria | Mode | What Gets Sent to LLM | |---|---|---|---| | good | ARIA tree has 3+ named interactive elements, 20+ lines | aria | Trimmed ARIA tree + special elements + interactive elements | | sparse | ARIA tree exists but fewer named elements than the page has | hybrid | ARIA tree + DOM tree nodes (XPath/CSS) + interactive elements | | none | ARIA tree has < 5 lines or is missing | dom | DOM tree with XPaths and CSS selectors + interactive elements |

DOM Tree Pipeline

For pages with poor or no accessibility markup, the library walks the DOM in-browser and:

• Traverses up to 300 visible nodes (headings, links, buttons, inputs, landmarks, etc.)
• Generates XPath for each node (e.g., //*[@id="login"], //form/div[2]/input[1])
• Generates unique CSS selectors (e.g., #login, [data-testid="submit"], button.primary)
• Extracts text content, ARIA labels, placeholders, data-testid attributes, and parent context
• Filters nodes by relevance scoring against your query before sending to the LLM

This means the library works on any page — not just accessible ones.

Caching

playwright-genie uses a two-level cache to minimize LLM calls:

1. Memory cache — instant lookups within the same test run
2. Disk cache (.locator-cache.json) — persists across runs

Cache keys are scoped by URL pathname + action + query, so fill('username') on /login won't collide with click('username') on /dashboard.

If a cached locator becomes stale (element no longer exists), the library:

1. Tries fallback locators returned by the LLM
2. If all fallbacks fail, invalidates the cache and re-queries the LLM with fresh page structure

Set LOCATOR_CACHE_FILE env var to customize the cache file path.

LLM Provider Configuration

| Provider | LLM_API_KEY | LLM_BASE_URL | LLM_MODEL | |----------|---------------|-----------------|-------------| | OpenAI | sk-... | (default) | gpt-4o-mini | | Anthropic | sk-ant-... | proxy URL | claude-sonnet-4-20250514 | | Ollama | ollama | http://localhost:11434/v1 | llama3 | | Azure OpenAI | Azure key | deployment URL | gpt-4o-mini | | RouteLLM | key | proxy URL | model name |

Complete Examples

Login Flow

import { test, expect } from '@playwright/test';
import { createSmartLocator } from 'playwright-genie';

test('complete login flow', async ({ page }) => {
  const smart = createSmartLocator(page);
  await page.goto('https://myapp.com/login');

  await smart.fill('username', 'admin');
  await smart.fill('password', 'secret123');

  if (await smart.exists('remember me checkbox')) {
    await smart.check('remember me checkbox');
  }

  await smart.click('sign in button');
  await smart.waitForVisible('dashboard heading');

  const welcome = await smart.getText('welcome message');
  expect(welcome).toContain('admin');
});

E-commerce Flow

test('shopping flow', async ({ page }) => {
  const smart = createSmartLocator(page);
  await page.goto('https://shop.example.com');

  await smart.fill('search bar', 'wireless headphones');
  await smart.press('search bar', 'Enter');
  await smart.waitForVisible('product list');

  await smart.click('first product card');
  await smart.select('size dropdown', 'Medium');
  await smart.click('add to cart button');

  await smart.waitForVisible('cart badge');
  const count = await smart.getText('cart badge');
  expect(count).toBe('1');
});

SPA Navigation with Cache Clearing

test('SPA login and navigate', async ({ page }) => {
  const smart = createSmartLocator(page);
  await page.goto('https://spa-app.com/login');

  await smart.fill('username', 'admin');
  await smart.fill('password', 'secret');
  await smart.click('login button');

  // After SPA navigation, clear cache to force fresh page analysis
  await page.waitForURL('**/dashboard');
  smart.clearCache();

  await smart.click('settings tab');
  await smart.waitForVisible('settings panel');
});

Batch Pre-fetch for Performance

test('prefetch for faster tests', async ({ page }) => {
  const smart = createSmartLocator(page);
  await page.goto('https://myapp.com/form');

  // Resolve all locators in one LLM call
  await smart.prefetch(
    'first name input',
    'last name input',
    'email field',
    'phone number',
    'submit button'
  );

  // All cached — zero LLM calls from here
  await smart.fill('first name input', 'John');
  await smart.fill('last name input', 'Doe');
  await smart.fill('email field', '[email protected]');
  await smart.fill('phone number', '555-0123');
  await smart.click('submit button');
});

Dynamic Content & Modals

test('handle dynamic content', async ({ page }) => {
  const smart = createSmartLocator(page);
  await page.goto('https://app.example.com');

  if (await smart.exists('cookie consent popup')) {
    await smart.click('accept cookies button');
    await smart.waitForHidden('cookie consent popup');
  }

  await smart.scrollIntoView('footer section');
  await smart.waitForVisible('load more button');
  await smart.click('load more button');
  await smart.waitForHidden('loading spinner');
});

Using Low-Level API for Debugging

import { getPageStructure, findLocator } from 'playwright-genie';

test('debug locator resolution', async ({ page }) => {
  await page.goto('https://myapp.com');

  // Inspect page analysis
  const structure = await getPageStructure(page);
  console.log('ARIA quality:', structure.mainFrame.ariaQuality);
  console.log('DOM nodes:', structure.mainFrame.domTree.length);
  console.log('Interactive elements:', structure.mainFrame.interactiveElements.length);

  // See what the LLM resolves without acting
  const result = await findLocator(page, 'submit button');
  console.log('Strategy:', result.strategy);
  console.log('Locator:', result.locatorString);
  console.log('Fallbacks:', result.fallbackLocators);
});

Exports

import {
  createSmartLocator,   // Main entry — creates smart locator with 40+ action methods
  findLocator,          // Resolve a query to a locator string (no action)
  findAllMatches,       // Get all matching locator results
  getPageStructure,     // Get the full page structure (ARIA + DOM + interactive elements)
  getLocator,           // Resolve + validate + create Playwright Locator object
  resolveLocator,       // Low-level: cache lookup → LLM resolution
  clearCache,           // Clear in-memory cache
  clearAllCaches,       // Clear memory + disk cache
  SmartAction,          // Class wrapping a Playwright Locator with 40+ methods
  chatCompletion,       // Direct LLM call (for custom pipelines)
  getConfig,            // Get current LLM configuration
  loadDiskCache,        // Load disk cache manually
  invalidateDiskEntry,  // Invalidate a specific disk cache entry
} from 'playwright-genie';

Best Practices

Be specific about element type:

await smart.click('login button');        // good
await smart.click('button');              // too vague

Include context when needed:

await smart.click('delete button in first row');
await smart.fill('search box in header', 'shoes');

Use action-appropriate descriptions:

await smart.fill('username', 'Admin');    // finds the input
await smart.click('username label');       // finds the label

Reuse the same instance across navigations:

const smart = createSmartLocator(page);
await page.goto('/login');
await smart.fill('username', 'Admin');
await smart.click('login button');
// navigated to /dashboard — same smart object works
await smart.click('settings tab');

Use locate() for multiple actions on same element:

const el = await smart.locate('search box');
await el.fill('query');
await el.press('Enter');
// 1 LLM call instead of 2

Use prefetch() for forms and multi-element pages:

await smart.prefetch('name', 'email', 'password', 'submit');
// 1 LLM call instead of 4

Clear cache after SPA navigation:

await page.waitForURL('**/dashboard');
smart.clearCache();

TypeScript

import { test } from '@playwright/test';
import { createSmartLocator, SmartAction, SmartLocator } from 'playwright-genie';

test('typed example', async ({ page }) => {
  const smart: SmartLocator = createSmartLocator(page);

  const el: SmartAction = await smart.locate('username');
  await el.fill('Admin');

  const visible: boolean = await smart.isVisible('dashboard');
  const text: string | null = await smart.getText('heading');
});

Debug Mode

LLM_LOCATOR_DEBUG=true npx playwright test

This logs:

• Payload mode selected (aria/hybrid/dom) and payload size
• LLM queries and responses
• Cache hits/misses (memory and disk)
• Stale cache invalidations and fallback attempts
• Page structure collection timing

Environment Variables

| Variable | Description | |---|---| | LLM_API_KEY | API key for LLM provider | | LLM_BASE_URL | Base URL for OpenAI-compatible API | | LLM_MODEL | Model name (e.g., gpt-4o-mini) | | OPENAI_API_KEY | Fallback API key | | ANTHROPIC_API_KEY | Fallback API key | | ROUTELLM_API_KEY | Fallback API key | | LOCATOR_CACHE_FILE | Custom path for disk cache file | | LLM_LOCATOR_DEBUG | Set to true to enable debug logging |

Security Notes

• The library sends the page's accessibility tree and/or DOM structure to your configured LLM API
• Sensitive data visible in the DOM may be sent to the API
• Use environment variables for API keys — never hardcode them
• For sensitive environments, use a local LLM (e.g., Ollama)

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License — see the LICENSE file for details.