playwright-flaky-tracker

A zero-runtime-dependency Playwright reporter that detects, tracks, and visualises flaky tests across runs with persistent history and a rich HTML report.

Installation

npm install playwright-flaky-tracker --save-dev

Quick Start

1. Set the tracking flag

FLAKY_TRACKING=true

You can provide FLAKY_TRACKING=true|false from your shell, CI environment, or a .env file that you load into process.env.

2. Register the reporter in playwright.config.ts

import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: 2, // retries MUST be > 0 for flakiness detection to work
  reporter: [
    ['list'],
    ['playwright-flaky-tracker', {
      enabled: undefined,          // config value wins over FLAKY_TRACKING when set
      // All options are optional — defaults shown below
      outputDir:          'flaky-report',  // where reports are written
      runId:              undefined,       // optional run folder name under flaky-report/runs/
      maxRunsPerTest:     50,              // rolling history window
      minRunsToClassify:  5,               // min runs before HIGH/MEDIUM/LOW assigned
      highFlakyThreshold: 0.4,            // ≥ 40% flaky rate → HIGH
      medFlakyThreshold:  0.2,            // ≥ 20% flaky rate → MEDIUM
    }]
  ]
});

retries must be greater than 0, otherwise Playwright never produces the retry signal needed to detect flaky tests.

options.enabled takes precedence over FLAKY_TRACKING when both are present.

3. Open the HTML report

The package already includes a suggested script:

{
  "scripts": {
    "report:flaky": "playwright-flaky-report"
  }
}

Run:

npm run report:flaky

If flaky-report/flaky-report.html does not exist yet, run Playwright with FLAKY_TRACKING=true or set reporter option { enabled: true } first.

The included opener uses native OS commands (open, xdg-open, or Windows start) so it does not download open-cli or its deprecated transitive dependencies.

Output Files

| File | Description | |------|-------------| | flaky-report/flaky-history.json | Persistent history — grows across every run | | flaky-report/flaky-summary.json | Latest cumulative analysis snapshot | | flaky-report/flaky-report.html | Latest cumulative visual report | | flaky-report/runs/<run-id>/flaky-summary.json | Immutable snapshot for one execution | | flaky-report/runs/<run-id>/flaky-report.html | Immutable HTML report for one execution | | flaky-report/runs/<run-id>/run-metadata.json | Paths and metadata for that execution |

By default, <run-id> is a filesystem-safe ISO timestamp, for example 2026-04-13T13-12-44-123Z. You can override it with runId when CI already has a build number or run identifier; unsupported path characters are converted to -.

Risk Classification

| Level | Flaky Rate | Meaning | |-------|-----------|---------| | 🔴 HIGH | ≥ 40% | Unreliable — fix immediately | | 🟠 MEDIUM | ≥ 20% | Needs attention | | 🟡 LOW | > 0% | Occasional — monitor | | 🟢 STABLE | 0% | Consistently passing |

Classification only activates after minRunsToClassify runs (default: 5), to avoid false positives on new tests.

How Flakiness Is Detected

Playwright calls onTestEnd() for every retry attempt, not just the final result.

A test is flaky when:

• It fails on attempt 0 (or any earlier retry), AND
• It passes on a subsequent retry

The tracker records that as one flaky test run, not multiple retry attempts, so your flaky rate reflects overall suite runs per test.

This is the only reliable signal, and it requires retries > 0 in your Playwright config.

Package Exports

The package exposes:

• The default reporter export used by Playwright module loading
• FlakyTestReporter as a named export
• Utility exports: FlakyTestAnalyser, FlakyTestStore, FlakyTestHtmlReport, formatDate, timeAgo

Local Validation

npm test
npm run test:e2e

The test command rebuilds the package and runs Node's built-in test runner against the compiled dist output.

The e2e command rebuilds the package, runs the Playwright smoke test in e2e/, and writes flaky-report/flaky-report.html.

You can also run the smoke test directly after building:

FLAKY_TRACKING=true npx playwright test

CI Usage

Commit flaky-report/flaky-history.json to your repo (or cache it in CI) to build history across pipeline runs.

# GitHub Actions example
- name: Cache flaky history
  uses: actions/cache@v3
  with:
    path: flaky-report/flaky-history.json
    key: flaky-history-${{ github.ref }}
    restore-keys: flaky-history-

License

MIT