AutoPlaywright - AI-Powered Playwright Test Automation with Azure OpenAI

A library that integrates with Playwright tests to execute commands in plain English using Azure OpenAI. The LLM analyzes DOM snapshots and translates natural language commands into Playwright actions.

Features

• 🤖 Natural language test commands
• 📸 Intelligent DOM snapshot analysis
• 🔧 Extensible command system
• 🔄 Retry mechanisms for flaky elements
• 🐛 Debug mode for troubleshooting
• 🔐 Secure credential handling with Azure OpenAI
• 🔷 Powered by Azure OpenAI Service
• 🖼️ Full iframe support for embedded content

Installation

npm install autoplaywright-ai

# Peer dependency (if not already installed)
npm install @playwright/test

⚠️ Security Notice

NEVER commit credentials to version control!

• Always use environment variables for API keys and credentials
• Add .env to your .gitignore file
• Use .env.example as a template without real values
• Review all files before committing to ensure no secrets are included

Quick Start

1. Set up your Azure OpenAI environment variables

Create a .env file in the project root (copy from .env.sample):

# Copy the sample file
cp .env.sample .env

# Edit .env and add your actual values:
AZURE_OPENAI_API_KEY=your-actual-api-key
AZURE_OPENAI_RESOURCE=your-resource-name
AZURE_OPENAI_DEPLOYMENT=your-deployment-name

The environment variables are automatically loaded when running Playwright tests.

2. Use in your Playwright tests

import { test } from '@playwright/test';
import { auto } from 'autoplaywright-ai';

test('navigate GitHub with natural language', async ({ page }) => {
  const config = {
    azureResource: process.env.AZURE_OPENAI_RESOURCE!,
    azureDeployment: process.env.AZURE_OPENAI_DEPLOYMENT!,
    model: 'gpt-4o'
  };

  await page.goto('https://github.com/microsoft/playwright');
  
  // Use natural language commands
  await auto('Click on the Issues tab', { page }, config);
  await auto('Click on the first issue', { page }, config);
  
  const issueTitle = await auto('Get the issue title', { page }, config);
  console.log('Issue title:', issueTitle);
});

API Reference

auto(command, context, config)

Simple function for executing single commands.

• command: Natural language command
• context: Object with either page or locator property
• config: LLM configuration

AutoPlaywright class

For more control and advanced features:

const autoPlaywright = new AutoPlaywright(context, config, options);

// Execute single command
await autoPlaywright.execute('Click the submit button');

// Execute multiple commands
await autoPlaywright.executeMultiple([
  'Fill in the email field with "[email protected]"',
  'Fill in the password field',
  'Click the login button'
]);

// Retry command
await autoPlaywright.retryExecute('Click the dynamic button', 3, 1000);

// Helper methods
await autoPlaywright.exists('login button');
await autoPlaywright.getText('page heading');
await autoPlaywright.count('product cards');

Available Commands

The LLM understands these Playwright commands:

• Navigation: navigate, goto, reload, goBack, goForward
• Interaction: click, type, fill, select, check, uncheck, hover, press
• Waiting: wait, waitForNavigation
• Information: getText, getAttribute, isVisible, count
• Other: screenshot, evaluate

Configuration Options

interface LLMConfig {
  apiKey?: string;              // Falls back to AZURE_OPENAI_API_KEY env var
  azureResource: string;        // Your Azure OpenAI resource name
  azureDeployment: string;      // Your deployment name
  model?: string;               // Default: 'gpt-4o'
  temperature?: number;         // Default: 0.3
  maxTokens?: number;          // Default: 500
}

Debug Mode

Enable debug mode to see detailed logs:

const autoPlaywright = new AutoPlaywright(
  { page, test }, 
  config, 
  { debug: true }
);

Working with iframes

AutoPlaywright fully supports iframe interactions by accepting locators:

// Get iframe locator
const iframe = page.frameLocator('iframe#payment-frame');
const iframeBody = iframe.locator('body');

// Create AI instance for iframe
const iframeAI = new AutoPlaywright({ locator: iframeBody }, config);

// Interact with iframe content
await iframeAI.execute('Fill in the credit card number');
await iframeAI.execute('Click the submit button');

// Or use the simple auto function
await auto('Click the accept button', { locator: iframeBody }, config);

Best Practices

1. Be specific in your commands: "Click the blue Submit button" vs just "Click submit"
2. Wait for navigation when needed: Use page.waitForURL() after navigation commands
3. Use retry for dynamic content: Elements that load asynchronously
4. Check existence before interaction: Use exists() helper for conditional flows
5. For iframes: Always wait for the iframe to load before creating the AI instance

How It Works

1. DOM Snapshot: Captures current page state including visible text and interactable elements
2. Azure OpenAI Analysis: Sends snapshot + command to Azure OpenAI for interpretation
3. Command Generation: Azure OpenAI returns structured command with parameters
4. Execution: Playwright executes the generated command
5. Result: Returns success/failure with any returned values

Development

# Install dependencies
npm install

# Run TypeScript compiler
npm run build

# Run tests
npm test

License

MIT# autotest