genkitx-playwright

A Genkit middleware that gives LLMs access to a real web browser via Playwright. The model can navigate, "read" pages through accessibility snapshots, click, type, fill forms, manage tabs, handle dialogs, and more — all scoped safely to a single generate call.

Installation

npm install genkitx-playwright playwright
# Install the browser binaries (one-time)
npx playwright install chromium

genkitx-playwright lists playwright as a dependency and genkit (>= 1.37) as a peer dependency.

Usage

Register the plugin once (this is where you configure the browser — engine, headless mode, viewport, etc.), then opt into the tools per generate call (this is where you configure policy — allowed domains, read-only, etc.).

import { genkit } from 'genkit';
import { googleAI } from '@genkit-ai/google-genai';
import { playwright } from 'genkitx-playwright';

const ai = genkit({
  plugins: [
    googleAI(),
    playwright.plugin({ browser: 'chromium', headless: true }),
  ],
});

const { text } = await ai.generate({
  model: googleAI.model('gemini-flash-latest'),
  prompt: 'Go to example.com and tell me the main heading.',
  use: [playwright()],
});

A fresh browser is launched lazily on the first browser tool call and is closed automatically when the generation finishes (success or failure), so there are no lingering browser processes.

How the model "sees" the page

Rather than guessing CSS selectors, the model calls browser_snapshot to get an accessibility tree where each interactive element is tagged with a stable ref (e.g. [ref=e12]). Interaction tools (browser_click, browser_type, …) take that ref. This is the same approach used by the official Playwright MCP server and is far more reliable for LLMs.

Options

Plugin (browser-level) — playwright.plugin(options)

| Option | Type | Default | Description | | ---------------- | ----------------------------------------- | ------------ | -------------------------------------------------------- | | browser | 'chromium' \| 'firefox' \| 'webkit' | 'chromium' | Browser engine to launch. | | headless | boolean | true | Run without a visible window. | | launchOptions | Playwright LaunchOptions | — | Passed through to browserType.launch(). | | contextOptions | Playwright BrowserContextOptions | — | Passed through to browser.newContext() (viewport, …). |

Per-generate (policy-level) — playwright(options)

| Option | Type | Default | Description | | ---------------- | ------------------- | ------- | ------------------------------------------------------------------ | | readOnly | boolean | false | Inject only observe-only tools (no click/type/etc.). | | allowEvaluate | boolean | false | Enable the browser_evaluate arbitrary-JS escape hatch. | | tools | PlaywrightToolName[] | all | Allow-list of tools to inject. | | allowedDomains | string[] | — | Restrict navigation to these hostnames (suffix match). | | blockedDomains | string[] | — | Block navigation to these hostnames (suffix match). | | toolNamePrefix | string | '' | Prefix prepended to every injected tool name. |

Tools

Navigation

• browser_navigate(url) — go to a URL
• browser_navigate_back() / browser_navigate_forward()

Observation

• browser_snapshot() — accessibility tree with ref ids

• browser_screenshot({ fullPage? }) — captures a PNG; the image is injected into the conversation as a real multimodal part (not a JSON blob), so the model actually "sees" it on its next turn

• browser_get_console_logs() — page console messages

• browser_get_network_requests() — resources loaded by the page

Interaction

• browser_click(ref, { doubleClick? })
• browser_type(ref, text, { submit? })
• browser_fill(ref, value)
• browser_press_key(key, { ref? })
• browser_hover(ref)
• browser_select_option(ref, values[])
• browser_scroll({ direction?, ref? })
• browser_drag(startRef, endRef)
• browser_file_upload(ref, paths[])

Synchronization

• browser_wait_for({ text?, textGone?, time? })

Tabs

• browser_tab_list() / browser_tab_new({ url? }) / browser_tab_select(index) / browser_tab_close({ index? })

Dialogs

• browser_handle_dialog({ accept, promptText? })

Escape hatch (opt-in via allowEvaluate)

• browser_evaluate(script) — run arbitrary JS in the page

Safety notes

• Use allowedDomains / blockedDomains to constrain where the model can go.
• readOnly: true removes every mutating tool — handy for research/scraping.
• browser_evaluate is off by default; only enable it when you trust the prompt and need the flexibility.
• Combine with the toolApproval middleware to require human confirmation before sensitive actions.

Running the sample

npm install
npx playwright install chromium
export GEMINI_API_KEY=...   # or GOOGLE_API_KEY
npm run run-sample

License

Apache-2.0