shiplightai

The best way to write UI tests — YAML with natural language steps. Easy for AI agents to create and maintain, yet clear for humans to understand and control.

AI-powered execution with self-healing locators means near-zero maintenance — no more broken tests every time the UI changes. Compatible with Playwright, running alongside your existing .test.ts files with npx playwright test — no separate tooling needed.

AI does the work. Humans have visibility and control.

Quick Start

1. Install

npm install -D shiplightai @playwright/test

2. Configure

In your playwright.config.ts:

import { defineConfig, shiplightConfig } from 'shiplightai';

export default defineConfig({
  ...shiplightConfig(),

  testDir: './tests',
  use: {
    headless: true,
    viewport: { width: 1280, height: 720 },
  },
});

3. Set up API keys

At least one AI API key is required. You can either export it directly or use a .env file.

Option A: Environment variable

export ANTHROPIC_API_KEY=sk-ant-...
# or export GOOGLE_API_KEY=...

Option B: .env file (in your test directory or project root)

ANTHROPIC_API_KEY=sk-ant-...
# GOOGLE_API_KEY=...

shiplightConfig() auto-discovers .env files by walking up the directory tree — no manual dotenv setup needed.

The AI model is auto-detected from your API key (ANTHROPIC_API_KEY → claude-haiku-4-5, GOOGLE_API_KEY → gemini-3.1-flash-lite-preview, OPENAI_API_KEY → gpt-5.4-mini). Set WEB_AGENT_MODEL to override.

4. Write a YAML test

Create tests/login.test.yaml:

goal: Verify user can log in
url: https://example.com/login

statements:
  - Click on the username field and type "testuser"
  - Click on the password field and type "secret123"
  - Click the Login button
  - "VERIFY: Dashboard page is visible"

5. Run

npx playwright test

Playwright discovers both *.test.ts and *.test.yaml files. YAML files are transparently transpiled to .yaml.spec.ts files next to the source.

6. Gitignore generated files

Add to your .gitignore:

*.yaml.spec.ts
auth.setup.ts
.auth/
.env

Project Structure

A typical Shiplight test project follows standard Playwright conventions — a playwright.config.ts at the root, test files in subdirectories, and a .gitignore. On top of that, Shiplight adds two config files: .env for API keys (shared across all projects) and shiplight.config.json for per-project login credentials (only needed if the app requires authentication).

my-tests/
├── playwright.config.ts
├── package.json
├── .env                            # API keys — shared by ALL projects (gitignored)
├── .env.example                    # Checked in, documents required keys
├── .gitignore
│
├── airbnb/                         # Project 1: public site, no login needed
│   ├── search.test.yaml
│   ├── filter.test.yaml
│   └── listing.test.yaml
│
├── my-saas-app/                    # Project 2: requires login
│   ├── shiplight.config.json       # {"url":"https://my-saas.com","username":"...","password":"..."}
│   ├── dashboard.test.yaml
│   ├── settings.test.yaml
│   └── billing.test.yaml
│
└── admin-portal/                   # Project 3: different app, different login
    ├── shiplight.config.json       # {"url":"https://admin.my-saas.com","username":"...","password":"..."}
    ├── users.test.yaml
    └── audit-log.test.yaml

Root files

playwright.config.ts — Playwright config with shiplightConfig():

import { defineConfig, shiplightConfig } from 'shiplightai';

export default defineConfig({
  ...shiplightConfig(),

  testDir: '.',
  timeout: 120_000,
  use: {
    headless: false,
    viewport: { width: 1280, height: 720 },
  },
});

shiplightConfig() runs during Playwright config loading and:

1. Walks up from scanDir to the project root looking for .env files and loads them (closer files take precedence)
2. Scans for **/*.test.yaml files
3. Transpiles each to a *.yaml.spec.ts file next to the source

Playwright then discovers the generated .yaml.spec.ts files through its default testMatch pattern. If you override testMatch, make sure it includes *.spec.ts.

.env — API keys, shared by all projects (gitignored):

# At least one required. Model is auto-detected from the key.
ANTHROPIC_API_KEY=sk-ant-...
# GOOGLE_API_KEY=...

# Optional: override auto-detected model
# WEB_AGENT_MODEL=claude-haiku-4-5

.gitignore:

node_modules/
test-results/
*.yaml.spec.ts
auth.setup.ts
.auth/
.env

Subdirectories

Each subdirectory is a separate project that can test a different application. Subdirectories that require login include a shiplight.config.json with credentials. Subdirectories that don't need login (e.g., public sites) simply omit it.

my-saas-app/shiplight.config.json — login + variables for one app:

{
  "url": "https://my-saas.com",
  "username": "[email protected]",
  "password": "test-password",
  "variables": {
    "BASE_URL": "https://my-saas.com",
    "API_TOKEN": { "value": "sk-test-...", "sensitive": true }
  }
}

admin-portal/shiplight.config.json — different app, different credentials:

{
  "url": "https://admin.my-saas.com",
  "username": "[email protected]",
  "password": "admin-password"
}

Running tests

Run all tests:

npx playwright test

Run one project:

npx playwright test my-saas-app/

Debug a single test:

shiplight debug my-saas-app/dashboard.test.yaml

The visual debugger uses the same config discovery — it finds my-saas-app/shiplight.config.json for login and .env at the root for API keys.

Environment Variables

| Variable | Description | Default | |---|---|---| | ANTHROPIC_API_KEY | Anthropic API key (for Claude models) | — | | GOOGLE_API_KEY | Google AI API key (for Gemini models) | — | | OPENAI_API_KEY | OpenAI API key (for GPT/o-series models) | — | | WEB_AGENT_MODEL | AI model override | Auto-detected from API key | | OPENAI_BASE_URL | Custom base URL for OpenAI-compatible APIs | — | | SHIPLIGHT_LOGIN_EMAIL | Login email (with _PASSWORD, overrides shiplight.config.json) | — | | SHIPLIGHT_LOGIN_PASSWORD | Login password (with _EMAIL, overrides shiplight.config.json) | — | | SHIPLIGHT_LOGIN_URL | Login page URL (overrides shiplight.config.json url) | — | | SHIPLIGHT_LOGIN_TOTP_SECRET | TOTP secret for 2FA (overrides shiplight.config.json totp_secret) | — | | PLAYWRIGHT_STARTING_URL | Override the starting URL for all tests | — |

At least one AI API key is required. The model is auto-detected: ANTHROPIC_API_KEY defaults to claude-haiku-4-5, GOOGLE_API_KEY defaults to gemini-3.1-flash-lite-preview, OPENAI_API_KEY defaults to gpt-5.4-mini. Set WEB_AGENT_MODEL to override.

Configuration Options

shiplightConfig({
  // Directory to scan for .test.yaml files (default: process.cwd())
  scanDir: './tests',

  // API key for cloud features (optional)
  apiKey: process.env.SHIPLIGHT_API_KEY,

  // Auto-discover .env files by walking up the directory tree (default: true)
  // Set to false for CI pipelines where env vars are injected externally
  dotenv: false,
})

Agent Fixture

The package exports a custom test that extends Playwright's test with an additional agent fixture. It works exactly like Playwright's test in every other way — same API, same hooks, same assertions. expect is re-exported from Playwright unchanged. Generated YAML tests use this automatically:

import { test, expect } from 'shiplightai/fixture';

test('my test', async ({ page, agent }) => {
  // `agent` is a pre-configured WebAgent instance
  // `page` is the standard Playwright page
});

You can also use the fixture in your hand-written .test.ts files to get the same agent instance:

// tests/custom.test.ts
import { test, expect } from 'shiplightai/fixture';

test('custom test with agent', async ({ page, agent }) => {
  await page.goto('https://example.com');
  await agent.run(page, 'Click the login button', 'step-1');
  await agent.assert(page, 'User is on the dashboard', 'step-2');
});

Authentication (Optional)

If your app requires login, add credentials to shiplight.config.json and wire up a Playwright setup project. shiplightConfig() auto-generates an auth.setup.ts file in each directory that has credentials. Skip this section if you're testing public pages.

1. Create shiplight.config.json

Place this in your test subdirectory (see Project Structure above):

{
  "url": "https://your-app.com",
  "username": "[email protected]",
  "password": "your-password",
  "totp_secret": "JBSWY3DPEHPK3PXP"
}

The totp_secret field is optional — only needed if your app uses 2FA.

2. Add setup project to playwright.config.ts

Use Playwright's standard project dependencies to run the auto-generated auth.setup.ts before authenticated tests:

export default defineConfig({
  ...shiplightConfig(),

  projects: [
    // Setup project — runs auto-generated auth.setup.ts
    { name: 'my-app-setup', testDir: './my-app', testMatch: 'auth.setup.ts' },
    {
      name: 'my-app',
      testDir: './my-app',
      dependencies: ['my-app-setup'],
      use: { storageState: './my-app/.auth/storage-state.json' },
    },
  ],
});

That's it. No global-setup.ts needed — shiplightConfig() generates auth.setup.ts automatically from your shiplight.config.json.

Credential resolution order

Login credentials are resolved in this order (first match wins):

1. Environment variables: SHIPLIGHT_LOGIN_EMAIL + SHIPLIGHT_LOGIN_PASSWORD (+ optional SHIPLIGHT_LOGIN_URL)
2. Config file: shiplight.config.json or login.config.json, discovered by walking up from the test directory

This means you can use shiplight.config.json for local development and override with env vars in CI.

How it works

1. shiplightConfig() scans for shiplight.config.json files with credentials and generates auth.setup.ts next to each
2. Playwright runs the setup project before dependent projects
3. The AI agent logs in using shiplightai's WebAgent.loginPage() — no fragile selectors
4. The resulting cookies/localStorage are saved to <dir>/.auth/storage-state.json
5. All tests in the dependent project load this storage state — every test starts already authenticated
6. Each directory is self-contained: its own credentials, its own auth state, its own setup

2FA / TOTP support

Set totp_secret in shiplight.config.json or SHIPLIGHT_LOGIN_TOTP_SECRET as an env var, and the agent will generate and enter the 2FA code automatically.

YAML Test Format

Basic Structure

goal: Description of what this test verifies
url: https://your-app.com/starting-page

statements:
  - Step described in natural language
  - Another step
  - "VERIFY: Expected outcome"

teardown:
  - Clean up step

| Field | Required | Description | |---|---|---| | goal | Yes | Test description (used as the Playwright test name) | | url | Yes | Starting URL to navigate to | | statements | Yes | List of test steps | | teardown | No | Steps that always run after the test (like finally) |

Statement Types

Draft (natural language)

Plain strings are AI-resolved steps. The agent figures out what to click, type, etc.

statements:
  - Navigate to the settings page
  - Click the "Delete Account" button
  - Type "confirm" in the confirmation dialog

VERIFY

Asserts a condition using AI. Must be a quoted string prefixed with VERIFY:.

statements:
  - "VERIFY: The success message is displayed"
  - "VERIFY: User is redirected to the dashboard"

ACTION (with action entity)

Deterministic actions with explicit locators. These replay fast (~1s) without AI.

statements:
  - description: Click the Submit button
    action_entity:
      action_description: Click the Submit button
      locator: "getByRole('button', { name: 'Submit' })"
      action_data:
        action_name: click
        kwargs: {}

  - description: Type email address
    action_entity:
      action_description: Type email address
      locator: "getByLabel('Email')"
      action_data:
        action_name: input_text
        kwargs:
          text: "[email protected]"

  - description: Press Enter
    action_entity:
      action_data:
        action_name: press
        kwargs:
          keys: Enter

STEP (grouping)

Groups related statements under a label.

statements:
  - STEP: Fill in the registration form
    statements:
      - Type "John" in the first name field
      - Type "Doe" in the last name field
      - Type "[email protected]" in the email field

IF / ELSE (conditional)

Conditional branching with AI or JavaScript conditions.

statements:
  # AI condition
  - IF: A cookie consent banner is visible
    THEN:
      - Click the Accept button
    ELSE:
      - "VERIFY: No banner is blocking the page"

  # JavaScript condition
  - IF: "js: page.url().includes('/login')"
    THEN:
      - Enter credentials and log in

WHILE loop

Repeating steps with a timeout.

statements:
  - WHILE: There are more items in the list
    DO:
      - Click the next item
      - "VERIFY: Item details are shown"
    timeout_ms: 30000

Supported Actions

Actions used in action_entity.action_data.action_name:

| Action | kwargs | Description | |---|---|---| | click | — | Click an element | | double_click | — | Double-click an element | | right_click | — | Right-click an element | | hover | — | Hover over an element | | input_text | text | Type text into an input | | clear_input | — | Clear an input field | | press | keys | Press a keyboard key (e.g., Enter, Tab) | | send_keys_on_element | keys | Press a key on a specific element | | select_dropdown_option | text | Select a dropdown option by text | | scroll | down, num_pages | Scroll the page | | scroll_to_text | text | Scroll to text on the page | | go_to_url | url, new_tab | Navigate to a URL | | go_back | — | Browser back | | reload_page | — | Reload the page | | wait | seconds | Wait for a duration | | wait_for_page_ready | — | Wait for page load | | verify | statement or code | Assert a condition (AI or JS) | | js_code | code | Run inline JavaScript | | function | functionName, parameterNames, parameterValues | Call a function | | switch_tab | tab_index | Switch browser tab | | close_tab | — | Close current tab | | upload_file | file_path | Upload a file | | save_variable | name, value | Save a variable for later use |

Locators

Action entities can specify element locators in two ways:

# Playwright locator (preferred)
locator: "getByRole('button', { name: 'Submit' })"

# XPath
xpath: "//button[@id='submit']"

If both are present, locator takes priority. If neither is present, the AI agent resolves the element from the action description.

Frames

For elements inside iframes:

action_entity:
  frame_path:
    - "iframe#main"
  locator: "getByText('Hello')"
  action_data:
    action_name: click
    kwargs: {}

Extensions

Custom Test Name

Override the Playwright test name (defaults to goal):

name: Login with valid credentials
goal: Verify login flow works
url: https://example.com
statements:
  - ...

Tags

Add Playwright tags for filtering with --grep:

tags:
  - smoke
  - auth
goal: Login test
url: https://example.com
statements:
  - ...

Run: npx playwright test --grep @smoke

Playwright Fixtures

Pass options to test.use():

use:
  viewport:
    width: 375
    height: 812
  locale: fr-FR
goal: Mobile French layout
url: https://example.com
statements:
  - ...

Variables

Use {{VAR_NAME}} syntax in YAML tests to reference variables. Variables are resolved at runtime.

statements:
  - description: Type username
    action_entity:
      locator: "getByLabel('Username')"
      action_data:
        action_name: input_text
        kwargs:
          text: "{{TEST_USER}}"

Defining variables in shiplight.config.json

Declare variable defaults in your project's shiplight.config.json. These are loaded before each test runs.

{
  "url": "https://my-app.com",
  "username": "[email protected]",
  "password": "test-password",
  "variables": {
    "TEST_USER": "standard_user",
    "TEST_PASS": { "value": "secret_sauce", "sensitive": true }
  }
}

Variables can be either:

• Plain string — "TEST_USER": "standard_user"
• Object with sensitive flag — "TEST_PASS": { "value": "secret_sauce", "sensitive": true } (masked in logs)

Templates

Extract reusable flows into template files and include them with template:.

Template file (templates/login.yaml):

params:
  - username
  - password

statements:
  - description: Enter username
    action_entity:
      locator: "getByLabel('Username')"
      action_data:
        action_name: input_text
        kwargs:
          text: "{{username}}"
  - description: Enter password
    action_entity:
      locator: "getByLabel('Password')"
      action_data:
        action_name: input_text
        kwargs:
          text: "{{password}}"
  - description: Click login
    action_entity:
      locator: "getByRole('button', { name: 'Log in' })"
      action_data:
        action_name: click
        kwargs: {}

Using the template:

goal: Purchase flow
url: https://example.com

statements:
  - template: ../templates/login.yaml
    params:
      username: "{{TEST_USER}}"
      password: "{{TEST_PASS}}"
  - Navigate to the checkout page
  - "VERIFY: Order summary is displayed"

Template params ({{username}}) are substituted at transpile time. Variables ({{TEST_USER}}) pass through and are resolved at runtime.

Templates can be nested (max depth: 5) and circular references are detected.

Custom Functions

Call TypeScript functions from YAML using the function action with file#export syntax:

statements:
  - description: Seed test data
    action_entity:
      action_data:
        action_name: function
        kwargs:
          functionName: "../helpers/seed.ts#createTestUser"
          parameterNames:
            - page
            - email
          parameterValues:
            - page
            - "[email protected]"

This generates:

import { createTestUser } from '../helpers/seed';
// ...
await createTestUser(page, "[email protected]");