playwright-ai-assist

AI-powered assistant for Playwright test analysis using a local LLM (Ollama) via LangChain.js.

Parses Playwright JSON reports, extracts failed tests, and sends them to a local LLM for root cause analysis and fix suggestions.

Prerequisites

• Node.js 20+
• Ollama installed and running

Pull a model:

ollama pull gemma4

Installation

npm install -g playwright-ai-assist

Or use directly with npx:

npx playwright-ai-assist analyze report.json

Usage

1. Generate a Playwright JSON report

Add the JSON reporter to your Playwright config:

// playwright.config.ts
export default defineConfig({
  reporter: [['json', { outputFile: 'report.json' }]],
});

Or run tests with:

npx playwright test --reporter=json > report.json

2. Analyze the report

playwright-ai-assist analyze report.json

Save the analysis to a markdown file:

playwright-ai-assist analyze report.json --output analysis.md

Analyze in Ukrainian:

playwright-ai-assist analyze report.json --lang uk

Use a different model:

playwright-ai-assist analyze report.json --model llama3

Options

| Option | Description | Default | |--------|-------------|---------| | --model <model> | Ollama model name | gemma4 | | --url <url> | Ollama endpoint | http://localhost:11434 | | --output <file> | Save analysis to file (markdown) | stdout | | --lang <language> | Analysis language | en |

Environment variables

Options can also be set via environment variables (CLI flags take priority):

| Variable | Maps to | |----------|---------| | OLLAMA_URL | --url | | OLLAMA_MODEL | --model | | PLAYWRIGHT_AI_LANG | --lang |

What it does

1. Reads a Playwright JSON report
2. Extracts failed tests with error messages, stack traces, and code snippets
3. Checks that Ollama is reachable
4. Sends the failures to a local LLM for analysis
5. Returns for each failure: root cause, fix suggestion, and priority
6. Optionally saves the output as a structured markdown file

Programmatic API

import { parsePlaywrightJson, analyzeChain, resolveConfig } from 'playwright-ai-assist';

const config = resolveConfig({ model: 'gemma4' });
const parseResult = parsePlaywrightJson('report.json');
const analysis = await analyzeChain(parseResult, config);

console.log(analysis);

License

MIT