playwright-ai-insights

v1.3.4

Published

11 days ago

AI-powered Playwright test failure analysis and self-healing with locator suggestions

0High
0Medium
0Low

anilkedia87

playwright testing ai automation self-healing test-analysis openai locator-suggestions

playwright-ai-insights

AI-powered Playwright test failure analysis and self-healing with automatic locator suggestions.

Features

✨ Core Capabilities

🤖 AI Failure Analysis - Analyzes test failures with OpenAI API
🔧 Self-Healing Locators - AI-suggested improved selectors
🟣 Flakiness Detection - Identify and quarantine flaky tests
📊 Smart Reports - HTML reports with AI insights and statistics
🎯 Minimal Setup - Just add API key, works with any Playwright framework

Installation

npm install playwright-ai-insights

Requires Node.js 18+ and an OpenAI API key.

Quick Start

Automatic Setup (Recommended)

The package automatically sets up your project when installed:

npm install playwright-ai-insights
npx playwright-ai setup

This creates:

.env - Configuration file (add your OpenAI API key)
.env.example - Template for team members
Enhanced test fixtures with AI context
CLI commands ready to use
PLAYWRIGHT_AI_SETUP.md - Integration guide

Manual Setup (Alternative)

If you prefer manual setup:

# Skip automatic setup
PLAYWRIGHT_AI_SKIP_SETUP=true npm install playwright-ai-insights

# Or configure manually later
npx playwright-ai setup --force

1. Add Your API Key

Create a .env file in your project root:

OPENAI_API_KEY=sk-your-openai-api-key-here
OPENAI_MODEL=gpt-4o-mini  # Optional, uses this by default

2. Update Your Tests

Import AI fixtures in your test files:

// Instead of:
import { test, expect } from '@playwright/test';

// Use:
import { test, expect } from './fixtures/aiHooks.js';

3. Analyze Test Failures & Generate Reports

# After tests run, generate an AI-powered HTML report
npx playwright-ai report

# Report will be created at: reports/ai-insights-report.html
# Open it in your browser to see AI insights and suggestions

API Usage Examples

Initialize Configuration

import PlaywrightAI from 'playwright-ai-insights';

PlaywrightAI.initialize({
  openaiApiKey: process.env.OPENAI_API_KEY,
  model: 'gpt-4o-mini',
  enableFlakyDetection: true,
});

Analyze Test Failures

import { analyzeTestFailures } from 'playwright-ai-insights';

const failures = [
  {
    testName: 'Login form should accept valid credentials',
    error: 'Timeout waiting for element with id "login-button"',
    domSnapshot: '<html>...</html>',
    screenshotPath: '/path/to/screenshot.png',
  },
];

const report = await analyzeTestFailures(failures);

console.log(report.summary);
// {
//   totalFailures: 1,
//   locatorIssues: 1,
//   timeoutIssues: 0,
//   assertionIssues: 0,
//   averageConfidence: 82
// }

console.log(report.suggestions);
// [
//   {
//     testName: 'Login form...',
//     failedLocatorGuess: '#login-button',
//     suggestedLocator: '[data-testid="submitButton"]',
//     confidence: 82,
//     reason: 'data-testid selectors are more stable than IDs...'
//   }
// ]

Usage Examples

CLI Commands

# Initial setup (automatic, or run manually)
npx playwright-ai setup

# Reconfigure or re-run setup
npx playwright-ai setup --force

# Generate AI-Powered Test Failure Report (HTML)
npx playwright-ai report

# Generate report with custom paths
npx playwright-ai report --results=./custom-results.json --output=./custom-report.html

# Add AI section to README
npx playwright-ai add-docs

# Clean up and remove AI integration
npx playwright-ai cleanup

# Remove but keep configuration files
npx playwright-ai cleanup --keep-config

# Analyze test failures
npx playwright-ai analyze --results=./test-results.json

# Get version
npx playwright-ai version

Generating AI-Powered Test Reports

After your tests run and generate a results JSON file, generate an intelligent HTML report:

# Generate report with default paths
# Expects: reports/test-results.json
# Creates: reports/ai-insights-report.html
npx playwright-ai report

# Generate report with custom input/output paths
npx playwright-ai report --results=path/to/results.json --output=path/to/report.html

Report Features:

📊 Test failure statistics and trends
🤖 AI-powered root cause analysis
💡 Intelligent locator suggestions with confidence scores
📈 Test run history with pass/fail status
🎯 Failure type classification (Assertion, Timeout, etc.)
🔴 Recent run status with hover tooltips

Report Output: reports/ai-insights-report.html (open in browser)

Environment Variables

# Skip automatic setup on installation
PLAYWRIGHT_AI_SKIP_SETUP=true npm install

# Required: Your OpenAI API key
OPENAI_API_KEY=sk-xxx

# Optional: Choose model
OPENAI_MODEL=gpt-4o-mini

# Optional: Enable flaky detection
ENABLE_FLAKY_DETECTION=true

# Optional: Enable screenshots in analysis
ENABLE_SCREENSHOTS=true

HTML Reports - AI-Powered Test Failure Analysis

Generate beautiful HTML reports with AI-powered insights for all test failures:

Basic Usage (Recommended):

# Prerequisites: 
# - Your test results in reports/test-results.json
# - OPENAI_API_KEY environment variable set

npx playwright-ai report
# Output: reports/ai-insights-report.html

Advanced Usage:

# Custom input and output paths
npx playwright-ai report \
  --results=path/to/your/results.json \
  --output=path/to/your/report.html

What's in the Report:

📊 Summary Statistics - Total failed tests, flaky test count, report date
🤖 AI Root Cause Analysis - Why each test failed (Assertion, Timeout, etc.)
💡 Smart Suggestions - AI-recommended fixes with confidence scores
📈 Test History - Pass/fail trends with recent run status
🎯 Confidence Scores - AI's confidence in each suggestion (0-100%)
🔴 Status Indicators - Visual markers for each test failure type

Example Report Output:

AI-Powered Test Failure Report
Intelligent Test Failure Diagnostics

1 Failed Tests | 1 Potentially Flaky | Report Date: 11-Mar-2026

Test Failure Analysis
┌─────────────────────────────────────────────────────────┐
│ #  │ Test Name            │ Type       │ AI Suggestion │ ⭐  │
├─────────────────────────────────────────────────────────┤
│ 1  │ Login Test           │ ⏱ Timeout  │ Use waitFor() │ 85% │
│ 2  │ Submit Form          │ 🟠 Assert  │ Update CSS    │ 92% │
└─────────────────────────────────────────────────────────┘

Default Paths:

Input: reports/test-results.json (standard Playwright test results)
Output: reports/ai-insights-report.html (open in browser)

Using AI Fixtures in Tests

// tests/hooks.js
import { test as base } from '@playwright/test';
import { analyzeTestFailures } from 'playwright-ai-insights';
import PlaywrightAI from 'playwright-ai-insights';

PlaywrightAI.initialize({
  openaiApiKey: process.env.OPENAI_API_KEY,
});

export const test = base.extend({
  captureFailure: async ({}, use) => {
    const failures = [];
    
    await use((data) => failures.push(data));
    
    if (failures.length > 0) {
      const report = await analyzeTestFailures(failures);
      console.log('AI Report:', report);
    }
  },
});

Directly Analyze a Failure

import { getFailureInsight } from 'playwright-ai-insights';

const failure = {
  testName: 'My Test',
  error: 'Element not found: .sidebar-nav > li',
  domSnapshot: pageHTML,
};

const insight = await getFailureInsight(failure);
console.log(insight.rootCause);
console.log(insight.suggestions);

Detect Flaky Tests

import { checkTestFlakiness } from 'playwright-ai-insights';

const history = [
  { status: 'PASSED', timestamp: '2024-01-01' },
  { status: 'FAILED', timestamp: '2024-01-02', failureType: '⏱ Timeout' },
  { status: 'PASSED', timestamp: '2024-01-03' },
];

const { isFlaky, confidence } = checkTestFlakiness(history);
console.log(`Flaky: ${isFlaky}, Confidence: ${confidence}`);

Get Locator Suggestions

import { suggestLocatorFixes } from 'playwright-ai-insights/agents';

const suggestions = await suggestLocatorFixes([
  {
    testName: 'Search test',
    error: 'Locator "#search_box" not found',
    dom: pageHTML,
  },
]);

suggestions.forEach((s) => {
  console.log(`${s.testName}:`);
  console.log(`  Failed: ${s.failedLocatorGuess}`);
  console.log(`  Suggested: ${s.suggestedLocator}`);
  console.log(`  Confidence: ${s.confidence}%`);
});

CLI Tool

Analyze test failures from the command line:

# Set your API key
export OPENAI_API_KEY=sk-...

# Analyze failures from JSON file
npx playwright-ai analyze --results=./test-failures.json

# Save report to custom location
npx playwright-ai analyze --results=failures.json --output=insights.json

# Enable debug logging
npx playwright-ai analyze --results=failures.json --debug

API Reference

`initialize(config: Partial<PlaywrightAIConfig>)`

Initialize the library with configuration.

Options:

openaiApiKey (required): Your OpenAI API key
model: 'gpt-4o' | 'gpt-4o-mini' (default) | 'gpt-4-turbo'
enableFlakyDetection: boolean (default: true)
maxDomSnapshotChars: number (default: 6000)
temperature: number (default: 0.2)
debug: boolean (default: false)

`analyzeTestFailures(failures: TestFailure[]): Promise<AnalysisReport>`

Analyze multiple test failures and generate a comprehensive report.

Returns:

{
  timestamp: string;
  failures: AIAnalysisResult[];
  suggestions: LocatorSuggestion[];
  summary: {
    totalFailures: number;
    locatorIssues: number;
    timeoutIssues: number;
    assertionIssues: number;
    averageConfidence: number;
  };
}

`getFailureInsight(failure: TestFailure): Promise<AIAnalysisResult>`

Get AI analysis for a single test failure.

`checkTestFlakiness(testHistory): { isFlaky: boolean; confidence: number }`

Check if a test is flaky based on its run history.

`utils`

Access to utility functions:

classifyFailure(error: string): FailureType
isLocatorRelated(error: string): boolean
isFlaky(history): boolean
calculateFlakyConfidence(history): number
logger, setDebugMode(enabled: boolean), setConfig()

Types

interface TestFailure {
  testName: string;
  error: string;
  errorType?: string;
  stack?: string;
  domSnapshot?: string;
  screenshot?: string;
}

interface AIAnalysisResult {
  testName: string;
  rootCause: string;
  suggestions: string[];
  confidence: number;
  failureType: FailureType;
}

interface LocatorSuggestion {
  failedLocatorGuess: string;
  suggestedLocator: string;
  confidence: number;
  reason: string;
  testName?: string;
}

Generated Project Structure

After running npx playwright-ai setup, your project will have:

your-project/
├── src/
│   ├── config/
│   │   └── aiConfig.js        # AI configuration (loads from .env)
│   ├── fixtures/
│   │   └── aiHooks.js         # Enhanced fixtures with AI context
│   └── utils/
│       └── selfHealer.js      # Self-healing locator suggestions
├── scripts/
│   └── analyzeFailures.js     # Post-test analysis script
├── .env                        # Your API keys (local, not committed)
├── .env.example               # Template for team (committed)
└── PLAYWRIGHT_AI_SETUP.md     # Integration guide

Environment Variables

OPENAI_API_KEY - Your OpenAI API key (required)
OPENAI_MODEL - AI model to use (default: gpt-4o-mini)
ENABLE_FLAKY_DETECTION - Enable flaky test detection (default: true)
ENABLE_SCREENSHOTS - Include screenshots in analysis (default: false)
DEBUG - Enable debug logging (default: false)
PLAYWRIGHT_AI_SKIP_SETUP - Skip setup on installation (default: false)

Pricing & Costs

Uses OpenAI's gpt-4o-mini model by default (most cost-effective):

~$0.015 per 1M input tokens
~$0.06 per 1M output tokens

Typical cost per failure analysis: $0.001 - $0.003

Best Practices

Batch Analysis: Send multiple failures at once for efficiency
Capture DOM: Include DOM snapshots for better locator suggestions
Set Debug Mode: Use debug: true during development
Cache Results: Store analysis results to avoid re-analyzing
Use data-testid: Tests using data-testid attributes get better suggestions
Commit .env.example: Share the example file with team, never commit .env
Run setup once: Setup creates the project structure once, use --force to regenerate

Troubleshooting

"This does not appear to be a Playwright project"

The setup wizard checks for @playwright/test in your dependencies. Make sure it's installed:

npm install --save-dev @playwright/test
npx playwright-ai setup

Files already exist, setup skipped

By default, setup won't overwrite existing files. Use --force to regenerate:

npx playwright-ai setup --force

Setup didn't run automatically after install

If postinstall.js didn't run, manually trigger setup:

npx playwright-ai setup

Or set the environment variable to skip automatic setup:

PLAYWRIGHT_AI_SKIP_SETUP=true npm install

OpenAI API key not found

Ensure .env file has been created and contains:

OPENAI_API_KEY=sk-your-actual-key-here

Test connection:

npx playwright-ai version

Cleanup command removed my files

The cleanup command is reversible - just run setup again:

npx playwright-ai cleanup --force
npx playwright-ai setup --force

Use --keep-config to preserve configuration files during cleanup:

npx playwright-ai cleanup --keep-config

Limitations

DOM snapshots capped at 6000 characters
OpenAI API rate limits apply
Requires internet connectivity
Works best with descriptive error messages

License

MIT

Support

GitHub: https://github.com/playwright-ai/insights
Issues: https://github.com/playwright-ai/insights/issues
Docs: https://playwright-ai.dev

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

playwright-ai-insights

Features

Installation

Quick Start

Automatic Setup (Recommended)

Manual Setup (Alternative)

1. Add Your API Key

2. Update Your Tests

3. Analyze Test Failures & Generate Reports

API Usage Examples

Initialize Configuration

Analyze Test Failures

Usage Examples

CLI Commands

Generating AI-Powered Test Reports

Environment Variables

HTML Reports - AI-Powered Test Failure Analysis

Using AI Fixtures in Tests

Directly Analyze a Failure

Detect Flaky Tests

Get Locator Suggestions

CLI Tool

API Reference

initialize(config: Partial<PlaywrightAIConfig>)

analyzeTestFailures(failures: TestFailure[]): Promise<AnalysisReport>

getFailureInsight(failure: TestFailure): Promise<AIAnalysisResult>

checkTestFlakiness(testHistory): { isFlaky: boolean; confidence: number }

utils

Types

Generated Project Structure

Environment Variables

Pricing & Costs

Best Practices

Troubleshooting

"This does not appear to be a Playwright project"

Files already exist, setup skipped

Setup didn't run automatically after install

OpenAI API key not found

Cleanup command removed my files

Limitations

License

Support

`initialize(config: Partial<PlaywrightAIConfig>)`

`analyzeTestFailures(failures: TestFailure[]): Promise<AnalysisReport>`

`getFailureInsight(failure: TestFailure): Promise<AIAnalysisResult>`

`checkTestFlakiness(testHistory): { isFlaky: boolean; confidence: number }`

`utils`