playwright-test-health

Flake triage and test-health analytics for any Playwright suite. Reads Playwright results, keeps a history, and tells you which tests are actually flaky, which are simply broken, which failures share one cause, and which have quietly got slower.

No runtime dependencies, and nothing is imported from @playwright/test — the history outlives suite upgrades, so it must still parse after one.

Adding it to a suite

npm i -D playwright-test-health   # or a git/file URL, see Installing below
npx test-health init              # writes test-health.config.json

The package is playwright-test-health; the command it installs is the shorter test-health (playwright-test-health works too, if something else has already claimed the short one).

Then record runs by adding the reporter alongside the ones already in playwright.config.ts:

reporter: [
  ["html", { open: "never" }],
  ["playwright-test-health/reporter", { label: process.env.APP_ENV }],
],

and read the result:

npx test-health report --html test-health.html

That is the whole integration. Everything suite-specific lives in test-health.config.json, so nothing in the tool needs editing.

Configuration

test-health.config.json, committed next to playwright.config.ts:

{
  "repo": "acme-web-e2e",
  "dataDir": ".test-health",
  "circleci": {
    "slug": "gh/acme/acme-web-e2e",
    "branch": "main",
    "artifactPath": "test-health/report.json"
  },
  "analysis": { "scoreThreshold": 20, "minRuns": 3 },
  "signatureRules": [{ "pattern": "\\border [A-Z]{3}-\\d{6}\\b", "replacement": "order <ORDER>" }]
}

| Key | Meaning | | ---------------- | -------------------------------------------------------------------- | | repo | Suite name, shown on reports and recorded on every run | | dataDir | History directory, relative to this file (default .test-health) | | circleci | Coordinates for fetch; only needed if you pull CI artefacts | | analysis | Analyser thresholds — every one is also a CLI flag | | signatureRules | Extra error-message normalisation, applied before the built-in rules | | top | Rows per table |

The file is found by walking up from the working directory, stopping at the first package.json — so test-health report gives the same answer from a subdirectory as from the root, and a package in a monorepo does not inherit its parent's suite name. A testHealth key in package.json works instead of a separate file. Run npx test-health config to see what was resolved and where it came from.

Three sources, highest first: CLI flags, then TEST_HEALTH_* environment variables (TEST_HEALTH_REPO, TEST_HEALTH_DIR, TEST_HEALTH_CIRCLE_SLUG, TEST_HEALTH_BRANCH, TEST_HEALTH_ARTIFACT, TEST_HEALTH_JOB_NAME), then the config file. The environment layer exists for CI, which often has no checked-in config to read.

repo is mixed into every run id, so renaming a configured suite starts a fresh history rather than merging into the old one. That is deliberate — one dataDir can hold two suites' runs and has to be able to tell them apart — but it means choosing the name once is worth more than choosing it well.

Why not just read the HTML report

The HTML report answers "what happened in this run". It cannot answer "is this test flaky", because that is a question about many runs — and it buries its data in a base64 zip inside index.html, which is why this tool keeps its own history instead of parsing it.

What it measures

Flake score (0–100) — the share of runs in which a test showed instability, from two independent signals:

| Signal | What it catches | History needed | | -------------- | ---------------------------------------------- | --------------------- | | Retry-flake | Failed, then passed on retry inside one run | None — works on run 1 | | Cross-run flip | Passed yesterday, failed today, no code change | Yes |

A retry-flake counts as passing for flip purposes, so it is not scored twice.

Broken ≠ flaky. A test that fails every single time is not flaky, and mixing the two is what makes most flake dashboards useless. They are reported separately: quarantining a test that never passes only hides it.

Failure clusters — failures grouped by error message with the volatile parts (timestamps, UUIDs, URLs, ports, paths, numbers) normalised out. A cluster spanning three or more distinct tests is flagged environment: one cause, many symptoms. That is the signal that stops someone debugging a test that was never broken.

Suite-specific identifiers can be named through signatureRules, which run before the built-in rules. The generic rules already collapse such values into <N> or <HEX>, so a rule only buys a more readable placeholder — but a cluster labelled order <ORDER> is easier to recognise than one labelled order <N>. Signatures are computed at ingest and stored, so a new rule affects new runs only.

Duration regressions — a recent window against the earlier baseline, not run-over-run, so one slow CI agent does not read as a regression. Both a relative (≥30%) and an absolute (≥2 s) threshold must be crossed.

Getting data in

Three ways in. They write to the same history and can be mixed across runs — but use only one per run, or that run is recorded twice under two ids.

The drop-in reporter — local and CI runs

Writes straight to the history store, so there is no separate step and no intermediate file:

reporter: [
  ["html", { open: "never" }],
  ["playwright-test-health/reporter", { label: process.env.APP_ENV }],
],

Options: label (a branch, environment or build number), repo, dataDir, quiet. Anything not given comes from test-health.config.json, found by walking up from the suite's root.

The reporter never throws — losing a run of history is recoverable, a red build caused by an analytics side-effect is not. An unreadable config degrades to defaults with a warning rather than failing the run.

If the tool is checked out beside the suite rather than installed, register it behind an fs.existsSync check: Playwright treats an unresolvable reporter path as a hard failure before any test runs.

ingest — a report you already have

npx playwright test --reporter=json > report.json
npx test-health ingest report.json

Idempotent: run ids are derived from the report's content, so re-ingesting the same file is a no-op rather than a duplicate run that would skew every rate.

fetch — CI runs on CircleCI

CI cannot keep history itself: each job starts from an empty workspace. Have it store the JSON report as a build artefact, then pull those down:

# .circleci/config.yml
- store_artifacts:
    path: report.json
    destination: test-health/report.json

export CIRCLE_TOKEN=...          # CircleCI → User Settings → Personal API Tokens
npx test-health fetch                                # last 30 jobs on the configured branch
npx test-health fetch --branch master --builds 60
npx test-health fetch --dry-run                      # list, download nothing

Needs circleci.slug in the config (<vcs>/<org>/<project>, gh for GitHub, bb for Bitbucket) or --slug. Jobs already examined are recorded in .test-health/fetched-jobs.json and skipped next time. Ingest is idempotent anyway, so this is about egress cost, not correctness — CircleCI bills transfer out.

Other CI providers have no fetch path yet. On those, download the artefact with whatever the provider offers and pipe it through ingest, or commit dataDir from the job.

Reporting

# console summary
npx test-health report

# plus a self-contained HTML dashboard
npx test-health report --html out/test-health.html

# last 20 runs only
npx test-health report --last 20

# gate a pipeline on it
npx test-health report --fail-on-flake

| Flag | Meaning | | ----------------- | -------------------------------------------------------------------- | | --last <n> | Only the n most recent runs | | --html <path> | Also write the dashboard (one file, no external assets) | | --top <n> | Rows per table | | --threshold <n> | Flake score needed for quarantine (default 20) | | --min-runs <n> | Runs required to trust cross-run evidence (default 3) | | --fail-on-flake | Exit 1 if any test is over the threshold | | --repo <name> | Override the suite name | | --data <dir> | History directory (default $TEST_HEALTH_DIR or the configured one) |

Reading the history strip: oldest to newest, . pass, F fail, ~ retry-flake, s skip. So .F..F. is a cross-run flipper and ~.~.~. is a test that needs a retry every second run.

Using it as a library

For a job that wants the numbers rather than the report — posting a flake count to Slack, or failing a pipeline on your own rule:

import { loadConfig, readHistory, analyse } from "playwright-test-health";

const config = loadConfig();
const analysis = analyse(readHistory(config.dataDir), { repo: config.repo });

console.log(analysis.quarantine.map((test) => `${test.flakeScore} ${test.title}`));

renderConsole, renderHtml, parsePlaywrightJson, appendRun, fetchAndIngest and the analysers are exported too. The CLI is a thin wrapper over exactly these.

Installing

npm i -D playwright-test-health                                            # from npm
npm i -D git+https://github.com/darvinpatel/playwright-test-health.git     # from git
npm i -D file:../playwright-test-health                                    # a sibling checkout

The npm install ships dist/ already built. Git and file installs build on install, because prepare runs tsc.

To hack on it, npm link it, or keep it checked out beside the suite and use relative paths (["../test-health/src/reporter", …] plus ts-node ../test-health/src/cli.ts) — that is what the fs.existsSync guard above is for.

Contributing and releasing

npm install          # also builds, via prepare
npm run typecheck
npm run demo         # end-to-end against the fixture suite
npm run format

Releases go out from CI, not from a laptop. Bump version, commit, then publish a GitHub Release tagged v<version>: .github/workflows/publish.yml typechecks, runs the demo, checks the tag against package.json, and publishes via npm trusted publishing — an OIDC exchange, so no token exists to leak. Provenance is attached automatically, linking the tarball on npm back to the commit and workflow run that built it.

Bump the version every time: npm never lets the same version be republished, and only allows unpublishing within 72 hours.

A local npm publish still works and runs typecheck plus scripts/check-publishable.js first, but needs an interactive 2FA challenge — and npm is removing direct publishing from bypass-2FA tokens in January 2027.

Licence

MIT. See LICENSE.

Storage

Append-only JSONL in dataDir:

• runs.jsonl — one line per run (wall-clock, totals, label)
• tests.jsonl — one line per test per run
• fetched-jobs.json — CircleCI jobs already examined

Append-only because ingest must never rewrite what is already there: a truncated line costs one record, not the whole history. Gitignore it to keep history per-machine, commit it to share history across a team, or point --data at a network share.

A test's identity is file::title::project. Renaming a test therefore starts a fresh history, which is the honest behaviour — it is a different test.

Try it without touching a real suite

npm install
npm run demo

The fixture suite in fixtures/ contains one of each thing the analysers look for — a retry-flake, a cross-run flipper, a permanently broken test, three tests failing on one shared cause, and a duration regression — so the output is checkable against a known answer. Six runs should yield five quarantine candidates (67, 50, 33, 33, 33), one broken test, one environmental cluster of three tests, and one regression.

fixtures/test-health.config.json is also the smallest working example of a consuming repo's config: the demo runs the CLI from fixtures/ with no flags and picks up the suite name, history directory and signature rules from there.

Layout

src/
  config.ts             per-suite config: discovery, environment, defaults
  init.ts               `init` — writes a starter config
  types.ts              normalised records, version-independent
  ingest/
    playwright-json.ts  parses the built-in json reporter
    history.ts          append-only JSONL store
  fetch/
    circleci.ts         minimal CircleCI API v2 client
    index.ts            pipelines → workflows → jobs → artifacts → ingest
    seen.ts             jobs already examined, so egress is paid once
  analyse/
    flake.ts            flake score, classification, quarantine
    duration.ts         percentiles, regression detection
    cluster.ts          failure grouping
    signature.ts        error-message normalisation
    report.ts           assembles one Analysis
  report/
    console.ts          terminal tables
    html.ts             self-contained dashboard
  reporter.ts           drop-in Playwright reporter
  index.ts              public API
  cli.ts

Notes on the dashboard

One HTML file, no CDN, no build step — email it, attach it to a build, or open it from a share. Charts are server-side SVG, so nothing depends on JavaScript to lay out; JS only adds hover tooltips, and every value is also in a table.

Outcome colours come from a status palette validated for colour-vision deficiency (worst adjacent pair ΔE 11.3 against a target of 8). The flaky yellow sits below 3:1 on the light surface by design, so it always ships with a labelled legend and a table view — the colour never carries meaning alone.

Known limitations

• Playwright only. Suites on other runners need their own ingest path; the normalised records in types.ts are the seam to write it against.
• fetch is CircleCI-only. Other providers work via ingest or by committing dataDir.
• Cross-run signal needs runs. With fewer than three, tests are reported as insufficient-data rather than guessed at. Retry-flakes are exempt — one run already proves those.
• History is wherever you ran it. Each machine builds its own unless dataDir is committed or pointed at a share.