LocatAI

Stop fixing broken selectors. Let AI do it.

LocatAI is a self-healing locator layer for Playwright. When a selector fails, it reads the page, figures out what you meant, and drives the correct element — automatically caching the fix so the next run is instant.

The 10-second pitch

// Classic Playwright — breaks the moment a class name changes
await page.click('[data-testid="submit-btn"]');

// LocatAI — heals itself when the selector breaks
await page.locatai.click(
  page.locator('[data-testid="submit-btn"]'),
  'Submit button on the checkout form'
);

// Or skip selectors entirely — describe it, LocatAI finds it
await page.locatai.click('', 'Place order button');

That's the whole idea. Give it a selector and a plain-English description. The selector is the fast path; the description is the safety net. When the UI refactors and your #submit-btn disappears, LocatAI falls back to the description, locates the element with AI, and caches the healed selector for next time — no re-runs, no manual patching.

Why it exists

UI tests rot. A button becomes a <div role="button">. A data-testid gets renamed. A framework upgrade reshuffles the DOM. Suddenly half your suite is red and your afternoon is gone.

Most teams deal with this two ways: they either over-invest in brittle selector hygiene, or they write loose selectors that miss real bugs. LocatAI gives you a third option — ship your normal selectors, and have AI repair them in place when reality drifts from your test code. Your tests become resilient to UI churn without becoming vague.

Feature highlights

| | | |---|---| | Self-healing | Failed locators are re-resolved by AI against a live DOM snapshot. | | Selector-free mode | Pass '' and let the description alone locate the element. | | Four AI providers | OpenAI, Anthropic, Google, or fully local via Ollama. | | Zero-cloud option | Run healing entirely offline with a local LLM. No API keys. | | Persistent cache | Healed selectors are saved to disk so successful heals happen once. | | Token-efficient | DOM is compacted, pre-ranked, and capped to keep bills low. | | Drop-in fixture | One Playwright fixture — your existing tests keep working. | | TypeScript-native | Full typing for LocatAIPage and all heal methods. |

Install

npm install @testnexus/locatai

Playwright >= 1.40 is a peer dependency. Node >= 18 is required.

60-second setup

1. Pick a provider and export the env vars.

# OpenAI (default)
export AI_PROVIDER=openai          # alias: gpt
export AI_API_KEY="sk-..."
export SELF_HEAL=1

# …or Anthropic / Google / Local — see the Providers section below.

2. Wire up the fixture.

// fixtures.ts
import { test as base } from '@playwright/test';
import { createLocatAIFixture, LocatAIPage } from '@testnexus/locatai';

export const test = base.extend<{ page: LocatAIPage }>(createLocatAIFixture());
export { expect } from '@playwright/test';

3. Write tests.

import { test, expect } from './fixtures';

test('login', async ({ page }) => {
  await page.goto('https://example.com/login');

  await page.locatai.fill(
    page.locator('#email'),
    'Email input on login form',
    '[email protected]'
  );

  await page.locatai.fill('', 'Password input', 'hunter2');
  await page.locatai.click('', 'Sign in button');
});

That's it. Healing is off by default — SELF_HEAL=1 turns it on. Leave it off in local dev if you just want plain Playwright.

Two ways to wrap a page

Most users want the fixture. The raw wrapper is there if you need it.

createLocatAIFixture() — recommended

import { test as base } from '@playwright/test';
import { createLocatAIFixture, LocatAIPage } from '@testnexus/locatai';

export const test = base.extend<{ page: LocatAIPage }>(createLocatAIFixture());

Every page in your tests is now a LocatAIPage — all regular Playwright methods plus the page.locatai.* namespace.

withLocatAI(page) — manual wrap

import { withLocatAI } from '@testnexus/locatai';

test('one-off test', async ({ page }) => {
  const smart = withLocatAI(page);
  await smart.locatai.click(smart.locator('#go'), 'Go button');
});

Useful when you need healing inside a utility function, hook, or a page object that can't rely on the fixture.

The API surface

All page.locatai.* methods take (locator, description, …rest). The locator can be a Locator, a CSS selector string, or an empty string '' (AI-only mode). The description is plain English — write what you'd tell a teammate.

| Method | Signature | Use for | |---|---|---| | click | (locator, desc, options?) | Clicking anything | | fill | (locator, desc, value) | Typing into inputs, textareas | | selectOption | (locator, desc, value) | Native <select> dropdowns | | check | (locator, desc) | Checkboxes, radio buttons | | uncheck | (locator, desc) | Unchecking checkboxes | | dblclick | (locator, desc) | Double-click | | hover | (locator, desc) | Hover (useful before revealing hidden elements) | | focus | (locator, desc) | Focus without clicking | | locator | (selector, desc) | Returns a chainable self-healing Locator |

Chainable self-healing locators

page.locatai.locator() returns something you can chain off like a normal Playwright Locator, but with the description bound:

const newTodo = page.locatai.locator('.new-todo', 'New-todo input field');
await newTodo.fill('Buy milk');
await newTodo.press('Enter');

If .new-todo breaks, the description kicks in on the first action that runs.

Hidden / hover-revealed elements

Trash icons that only appear on hover are a classic flake source. Hover first, then force:

await page.locatai.hover('', 'Todo row for "Buy milk"');
await page.locatai.click('', 'Delete button for that row', { force: true });

Mix freely with plain Playwright

The wrapped page is a Playwright page. page.click(), page.fill(), expect(page).toHaveURL() — all still work. Use healing only where it earns its keep:

await page.fill('#username', 'qa-bot');                             // stable, no need to heal
await page.locatai.click('', 'Third-party widget "Accept" button'); // flaky, let AI handle it

Providers

Set AI_PROVIDER to pick a backend. Aliases are accepted for convenience.

| Provider | AI_PROVIDER values | Default model | Needs AI_API_KEY | |---|---|---|---| | OpenAI | openai, gpt | gpt-5.2 | Yes | | Anthropic | anthropic, claude | claude-sonnet-4-5 | Yes | | Google | google, gemini | gemini-3-flash | Yes | | Local (Ollama) | local, ollama | gemma3:4b | No — fully offline |

Override the model per provider with AI_MODEL:

export AI_PROVIDER=anthropic
export AI_MODEL=claude-opus-4-5
export AI_API_KEY="sk-ant-..."

Going fully local with Ollama

Heal without sending a byte to the cloud.

# 1. Install Ollama from https://ollama.com, then pull a model
ollama pull llama3.2:3b

# 2. Point LocatAI at it
export AI_PROVIDER=local
export AI_MODEL=llama3.2:3b
export SELF_HEAL=1

Benchmark from our element-zoo test suite (20+ interactive element types):

| Model | Size | Context | Pass rate | Notes | |---|---|---|---|---| | llama3.2:3b ⭐ | 2.0 GB | 128K | 5/5 | Best default. Fast, accurate, tiny footprint. | | mistral | 4.1 GB | 128K | 5/5 | Most accurate on ambiguous pages, slowest (~1.4 min/run). | | gemma3:4b | 3.3 GB | 128K | 4/5 | Middle ground — ~53s/run, occasional miss on lookalike elements. |

Any model in the Ollama library works — pick your tradeoff. If Ollama runs on a different machine:

export OLLAMA_HOST=http://192.168.1.100:11434

Small models (1–4B) can wobble when a page has multiple near-identical elements. Reach for mistral or a 7B+ model if accuracy matters more than latency.

Configuration reference

Environment variables

| Variable | Required | Purpose | |---|---|---| | SELF_HEAL | Yes (=1 to enable) | Master switch. When unset, LocatAI passes through to Playwright with no healing. | | AI_PROVIDER | Default: openai | Which backend to use. See table above. | | AI_API_KEY | Yes for cloud providers | API key for the selected provider. | | AI_MODEL | Optional | Override the default model for the chosen provider. | | OLLAMA_HOST | Optional | Custom Ollama endpoint. Default: http://127.0.0.1:11434. |

Fixture options

Every knob lives on createLocatAIFixture():

createLocatAIFixture({
  maxCandidates: 30,   // DOM elements sent to the AI (default 30)
  maxAiTries: 4,       // Strategies validated per heal (default 4)
  timeout: 5000,       // Locator timeout in ms (default 5000)
  provider: 'openai',  // Overrides AI_PROVIDER
  model: 'gpt-5.2',    // Overrides AI_MODEL
});

Tuning guidance:

• maxCandidates is the main lever for cost and speed. Simple pages → drop to 15. Dashboards with dozens of similar controls → raise to 50.
• maxAiTries trades cost for resilience. 4 is enough for almost every page we've seen.
• timeout applies to the initial selector attempt before healing kicks in.

How healing actually works

Every healed call follows the same four-step path:

1. Try the selector. If it resolves in timeout ms, LocatAI gets out of the way — zero AI calls, zero cost.
2. Check the cache. .self-heal/healed_locators.json stores previous heals keyed by description + URL. Hit → use the cached selector.
3. Ask the AI. LocatAI snapshots a compact list of candidate DOM elements, pre-ranks them by keyword / role / tag / test-id relevance, and sends the top candidates to the AI. The AI returns up to 3 strategies.
4. Validate and cache. Each strategy is tried against the live page in order. The first one that works is cached to disk so step 2 hits next time.

Every heal is logged to .self-heal/heal_events.jsonl — one JSON line per healing event, with token counts, provider/model, strategy used, and test name. Parse it for cost tracking, dashboards, or CI analytics.

Token & cost optimization

Healing should be cheap and fast. LocatAI takes several passes at keeping the prompt small:

• Attribute compaction — short keys (tid, txt, al) instead of their verbose originals. Null/empty fields are stripped.
• Visibility filter — invisible elements (display: none, visibility: hidden, zero-size) never make it into the payload.
• Semantic pre-ranking — rankCandidates() scores every candidate on keyword match, tag inference, ARIA role fit, and test-id presence. The lowest-scoring 20–30% is dropped before sending.
• Capped output — AI is told to return max 3 strategies.
• Cache-first — a successful heal writes to disk, so repeated runs on the same DOM drift cost nothing.

You'll see a one-line token breakdown in the console on every heal:

↑ 1350 input · 180 output · 1530 total tokens

The .self-heal/ directory

LocatAI writes two files to your project root:

| File | Purpose | Commit? | |---|---|---| | .self-heal/healed_locators.json | The cache. Survives across runs. | Yes — lets CI reuse heals and avoid AI calls on repeat runs. | | .self-heal/heal_events.jsonl | Append-only event log for analytics. | No — add to .gitignore. |

Minimum .gitignore addition:

.self-heal/heal_events.jsonl

Privacy notes

When the AI path fires, LocatAI sends a structured list of candidate DOM elements — tag, role, aria attributes, visible text, test ids, bounding info — to the configured provider. It does not send screenshots, full page HTML, or cookies.

If your DOM legitimately contains sensitive data (PII, internal URLs, financial info), you have options:

• Go local. AI_PROVIDER=local + Ollama keeps everything on your machine.
• Review the log. .self-heal/heal_events.jsonl shows exactly what went over the wire.
• Gate by environment. Set SELF_HEAL=1 only in CI against staging fixtures, not production data.

Full example

import { test as base, expect } from '@playwright/test';
import { createLocatAIFixture, LocatAIPage } from '@testnexus/locatai';

const test = base.extend<{ page: LocatAIPage }>(createLocatAIFixture({
  maxCandidates: 40,
}));

test('checkout a single item', async ({ page }) => {
  await page.goto('/shop');

  await page.locatai.click(
    page.locator('[data-testid="product-card-1"]'),
    'First product card'
  );

  await page.locatai.click('', 'Add to cart button');
  await page.locatai.click('', 'Shopping cart icon in header');

  await page.locatai.fill(
    page.locator('#email'),
    'Email field in checkout form',
    '[email protected]'
  );

  await page.locatai.selectOption('', 'Country dropdown', 'US');
  await page.locatai.check('', 'Agree to terms checkbox');
  await page.locatai.click('', 'Place order button');

  await expect(page).toHaveURL(/\/order\/confirmation/);
});

Live demo

Don't want to wire it into your own project yet? The test-published branch is a tiny standalone repo that consumes the published npm package against TodoMVC with intentionally broken selectors.

git clone -b test-published https://github.com/Divyarajsinh-Dodia1617/LocatAi.git locatai-demo
cd locatai-demo
npm install && npx playwright install chromium
cp example.env .env    # drop in your API key
npm test

Two smoke tests run:

• Broken selector heal — test points at #wrong-todo-input; AI finds the real input and caches the fix.
• Selector-free mode — no selector, just a description. AI locates the element cold.

FAQ

Does it work without SELF_HEAL=1? Yes — LocatAI becomes a no-op and your tests run as plain Playwright. This is intentional so local dev stays free and deterministic.

What happens when the AI can't find the element? LocatAI throws a Playwright-style error pointing at the description. Your test fails loudly with context, not silently.

Does healing work across iframes? Not yet. Healing happens against the main frame. For iframe content, use normal Playwright frameLocator().

Is the cache safe to commit? Commit healed_locators.json, ignore heal_events.jsonl. The cache is a net positive for CI — it's the difference between one AI call per heal versus one per run.

Can I use this with Cypress / WebdriverIO? Not currently. The library is built on Playwright's locator model. Cypress support would be a separate package.

Development

git clone https://github.com/Divyarajsinh-Dodia1617/LocatAi.git
cd LocatAi
npm install
npm run build
npm run test:unit
SELF_HEAL=1 npm run test:heal

The element-zoo fixture (examples/fixtures/element-zoo.html) is the primary integration-test harness — 20+ interactive element types used to stress both healing and AI-only paths.

License

MIT © Divyarajsinh Dodia

If LocatAI saves your Friday afternoon, a ⭐ on GitHub is a very nice thank-you.