A taiko plugin that adds natural language visual targeting to browser automation using Vision Language Models (VLMs).

Instead of relying on rigid DOM selectors (CSS, XPath) or fragile text matching, describe UI elements visually and click or type into them:

await vision.click("the blue shopping bag icon in the header");
await vision.write("[email protected]", "the email field in the signup form");

How it works

Every vision.* call runs a 4-step pipeline:

1. Capture — Grabs a Base64 viewport screenshot via the Chrome DevTools Protocol and runs a quick DOM script to gather candidate bounding boxes (x, y, width, height, tag, text) for visible interactive elements.
2. Provider delegation — Bundles the screenshot, prompt, and candidate list into a standardized payload and hands it to the configured VLM adapter.
3. Coordinate resolution — The model returns the target pixel { x, y }, which is validated against the viewport (fail-safe).
4. Action — The coordinate is routed into Taiko's native input engine (click({ x, y })).

The plugin owns zero LLM SDKs. Adapters talk to vendors over plain fetch.

Installation

• npm install taiko-vision --save

Usage

const { openBrowser, goto, closeBrowser, vision } = require("taiko");
const { ProviderType } = require("taiko-vision");

(async () => {
  try {
    await openBrowser();
    await goto("https://github.com");

    vision.setProvider({
      type: ProviderType.OpenAI,
      apiKey: process.env.OPENAI_API_KEY,
    });

    await vision.click("the search box in the top navigation");
    await vision.write("taiko", "the search box in the top navigation");
    // more actions
    // ...
  } finally {
    await closeBrowser();
  }
})();

Configuring a provider

Call vision.setProvider(...) with either a preset config object or your own VisionProvider instance.

The plugin never reads credentials from the environment. You must pass apiKey explicitly for the cloud providers (source it however you like, e.g. from your own process.env, a secrets manager, or a config file).

Prefer the exported ProviderType constants for type so you don't hard-code provider name strings:

const { ProviderType } = require("taiko-vision");

| Provider | ProviderType | Default model | apiKey required | | --------- | ------------------------ | -------------------------- | ----------------- | | OpenAI | ProviderType.OpenAI | gpt-4o | yes | | Google | ProviderType.Gemini | gemini-3.5-flash-lite | yes | | Anthropic | ProviderType.Anthropic | claude-3-5-sonnet-latest | yes | | Ollama | ProviderType.Ollama | llava | no (local) |

const { ProviderType } = require("taiko-vision");

// Cloud (apiKey is required)
vision.setProvider({
  type: ProviderType.OpenAI,
  apiKey: process.env.OPENAI_API_KEY,
  model: "gpt-4o",
});
vision.setProvider({
  type: ProviderType.Gemini,
  apiKey: process.env.GEMINI_API_KEY,
});
vision.setProvider({
  type: ProviderType.Anthropic,
  apiKey: process.env.ANTHROPIC_API_KEY,
});

// Local / self-hosted (no key needed)
vision.setProvider({
  type: ProviderType.Ollama,
  model: "llava",
  baseUrl: "http://127.0.0.1:11434",
});

model and baseUrl are optional and fall back to the defaults above. baseUrl can point at OpenAI-compatible gateways or internal endpoints.

Plain string literals ("openai", "gemini", …) still work if you prefer them.

APIs

The plugin exposes a vision namespace with the following APIs.

setProvider(providerOrConfig)

Configure the active model adapter. Accepts a preset config or a VisionProvider instance. Returns the resolved provider.

const { ProviderType } = require("taiko-vision");
vision.setProvider({
  type: ProviderType.OpenAI,
  apiKey: process.env.OPENAI_API_KEY,
});

getProvider()

Return the currently active provider (or undefined if none is set).

const provider = vision.getProvider();

select(prompt)

Resolve a visual description to a validated { x, y } coordinate without acting on it.

const { x, y } = await vision.select("the login button");

click(prompt, options)

Find and click an element visually. The optional options are forwarded to Taiko's click.

await vision.click("the login button");

write(text, prompt)

Find an input visually, focus it, and type the given text.

await vision.write("my-username", "the username field");

Custom / vendor-agnostic providers

The core abstraction is a tiny interface:

interface VisionProvider {
  name: string;
  resolve(input: {
    screenshotBase64: string;
    prompt: string;
    candidates: {
      x: number;
      y: number;
      width: number;
      height: number;
      tag: string;
      text: string;
    }[];
    viewport: { width: number; height: number };
  }): Promise<{ x: number; y: number }>;
}

Implement it to route through internal API gateways or other frameworks, then pass the instance to setProvider. A convenience wrapper for supplying a bare async function directly is planned for a future release.

Fail-safe validation

If a model returns malformed data, non-numeric coordinates, or a target outside the viewport, the plugin aborts the action with a descriptive TaikoVisionError (with a .code, e.g. OUT_OF_VIEWPORT) instead of driving the browser with bad input.

Use in Taiko REPL

To launch the REPL type taiko --plugin taiko-vision in your favorite terminal application. This will launch the Taiko Prompt.

e.g Version: 1.4.0 (Chromium:126.0.6468.0) Type .api for help and .exit to quit

You should now have full access to the vision APIs in the Taiko REPL window.

npx taiko --plugin taiko-vision
> openBrowser()
> vision.setProvider({ type: 'ollama' })
> goto('https://example.com')
> vision.click('the more information link')

To load the plugin with a runner like Gauge, set the TAIKO_PLUGIN environment variable:

TAIKO_PLUGIN=vision gauge run specs

Development

npm install
npm run format        # or: npm run format:check
npm run build         # emit dist/ (also runs via prepare on install)
npm test              # unit tests via ts-jest (no separate build required)

Unit tests import src/ through ts-jest and do not need a prior npm run build. CI still runs npm run build explicitly as a typecheck/emit step. Integration tests install the plugin globally (npm install -g .); prepare builds dist/ so Taiko can load main (./dist/index.js).

Unit tests run without a browser or an LLM. The integration tests need both a browser and a configured provider. Like taiko-storage, they install the plugin globally and load it via TAIKO_PLUGIN.

The provider is selected with TAIKO_VISION_PROVIDER (default ollama). There is a script per provider:

npm run test:integration            # default: ollama (local)
npm run test:integration:ollama     # OLLAMA_MODEL=... to override the model
OPENAI_API_KEY=sk-...    npm run test:integration:openai
GEMINI_API_KEY=...       npm run test:integration:gemini
ANTHROPIC_API_KEY=...    npm run test:integration:anthropic

The Ollama runs require a vision-capable model. The default is llava, so pull it first (ollama pull llava), or point the tests at another multimodal model you already have with OLLAMA_MODEL=<model> npm run test:integration. Text-only models (e.g. llama3.1, gpt-oss) will fail with model '...' not found or an image support error. Note that llama3.2-vision (the mllama architecture) fails to load on several recent Ollama builds with unknown model architecture: 'mllama'.

Each integration script runs npm install -g . (which builds via prepare) and sets TAIKO_PLUGIN=vision so Taiko resolves the plugin from the global module path (no symlinking required). Cloud providers require their API key in the environment; a missing key makes beforeAll fail fast.

CI notes

GitHub Actions caches Ollama models under /home/runner/.ollama (pinned via OLLAMA_MODELS). The install script's systemd service is stopped so pulls do not land in the system ollama user home (which Actions cannot cache). The first successful integration run warms the cache; later runs should restore it and skip the multi-GB pull when the model is already present.

Releasing

Releases follow SemVer (MAJOR.MINOR.PATCH). Cut one from GitHub Actions → Actions → release → Run workflow on main, choosing patch, minor, or major.

The workflow will:

1. Run format check, build, and unit tests
2. Bump package.json via npm version <bump>
3. Push the release commit and vX.Y.Z tag
4. Create a GitHub Release (notes generated from commits)
5. Publish to npm with provenance (npm publish --access public)

Required repository secret: NPM_TOKEN — an npm automation / granular access token with permission to publish taiko-vision. Create it under Settings → Secrets and variables → Actions.

Current version on main is the source of truth; do not hand-edit package.json version for a release unless you know you need an explicit pin.

License

MIT