Playwright Failed Extractor

Get the tests that are still failing out of a Playwright JSON report, correctly

Reads a Playwright JSON report and returns the tests that are still failing: retries, flaky tests, multi-project runs and merged rerun reports all handled. Then turns them into a --grep pattern so your CI can rerun exactly those.

Zero runtime dependencies. Playwright itself is not required at runtime, because this reads the report, not your test suite.

The Problem

The usual way to pull failures out of a report is a jq one-liner:

jq -r '.suites[].suites[].specs[] | select(.ok == false) | .title' results.json

Run that against a real report and you get:

jq: error (at results.json:755): Cannot iterate over null (null)

Exit code 5, mid-stream, with whatever it printed so far. If the script only captures stdout, the rerun step sees an empty list and reruns nothing, while the build still goes green.

Three separate things are wrong with that selector:

| Problem | What happens | | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | .suites[].suites[] assumes exactly two levels | Playwright omits the suites key entirely for a spec file with no describe, so jq iterates null and aborts. Describe-less file first gives an empty list; later gives a silently truncated one | | spec.ok goes stale | Merging appends test entries to a spec without recomputing ok, so a test that failed and then passed on a rerun still reads ok: false | | No notion of a rerun | After merge-reports combines an initial run with a --grep rerun, every test that ever failed is reported, including the ones the rerun cleared |

The Solution

import { extractFailedTests, toGrepPattern } from 'playwright-failed-extractor';

const failed = extractFailedTests('results.json');
const pattern = toGrepPattern(failed, { idPattern: /[A-Z][A-Z0-9]*-\d+/ });
// "CBRO-02|CAM2-03"

Selection is on the net outcome per (test, project), walking the suite tree to any depth. A test is reported only when every attempt recorded for it anywhere in the report was unexpected. Passed on a rerun, or on a retry, means it is not still failing, so it is not in the list.

Quick Start

Requirements

• Node.js 20.0.0 or higher
• ESM only

npm install --save-dev playwright-failed-extractor

In a CI rerun step

# 1. Run the suite, keeping the JSON report
npx playwright test --reporter=json > results.json || true

# 2. Ask for the still-failing tests as a grep pattern
npx playwright-failed-extractor results.json \
  --grep --id-pattern '[A-Z][A-Z0-9]*-\d+' --out failed_tests.txt

# 3. Rerun only those
[ -s failed_tests.txt ] && npx playwright test --grep "$(cat failed_tests.txt)"

The CLI exits 0 even when tests failed. It reports, it does not gate, so it will not fail your build on your behalf.

CLI

playwright-failed-extractor <report.json> [options]

| Flag | Meaning | | ---------------------- | ------------------------------------------------------------------------------------ | | --grep | Print a --grep pattern instead of JSON | | --id-pattern <regex> | Extract a short id from each test's own title (bare pattern, no delimiters or flags) | | --separator <sep> | Separator between grep alternatives (default \|) | | --include-flaky | Include flaky (failed, then passed) tests | | --out <file> | Write to a file instead of stdout | | --pretty | Pretty-print JSON output (ignored with --grep) |

| Exit code | Meaning | | --------- | --------------------------------------------- | | 0 | Ran normally, including when tests failed | | 1 | Report missing or malformed | | 2 | Bad arguments |

A non-empty top-level errors[] in the report is printed to stderr and does not change the exit code.

API

extractFailedTests(source, options?): FailedTest[]

source is a path to a JSON report, or an already-parsed report object.

interface FailedTest {
  title: string; // full path joined by ' > '
  titlePath: string[]; // the same path unjoined; the test's own title is last
  file?: string;
  line?: number;
  column?: number;
  project?: string;
  tags: string[]; // without the leading '@'
  error?: string; // first line of the error message
  errorStack?: string; // full message including stack
  duration?: number; // of the most recent attempt, in ms
  retries: number; // total attempts across the whole report, minus 1
  status: 'failed' | 'timedOut' | 'interrupted' | 'passed' | 'skipped';
}

| Option | Default | Meaning | | -------------- | ------- | --------------------------------------------- | | includeFlaky | false | Include tests that failed and then passed | | projects | all | Only report failures from these project names |

Three behaviours worth knowing before you rely on it:

• status: 'passed' is a failure. A test.fail()-annotated test that unexpectedly passed is reported, because the outcome was unexpected.
• retries counts attempts across the whole report. A test that ran once in the initial run and once in a merged rerun reports retries: 1, exactly like a test retried in-process. The two are indistinguishable to a rerun step, so they are reported the same way.
• One entry per failing (test, project). A test failing on two projects appears twice, with different project values. toGrepPattern dedupes.

toGrepPattern(failed, options?): string

Builds a single regex alternation for playwright test --grep.

| Option | Default | Meaning | | ------------- | --------- | ------------------------------------------------------------------------------ | | idPattern | - | Extract a short id from each test's own title. Omit to use escaped full titles | | separator | \| | Joiner between alternatives | | unique | true | Drop duplicate ids | | onUnmatched | 'title' | When idPattern matches nothing: 'title', 'skip' or 'throw' |

idPattern is applied to the test's own title, never the joined path, so it cannot accidentally match part of a file path or a describe title.

onUnmatched: 'title' is the safe default against dropping a test, but an escaped full title is long and widens the alternation. Use 'throw' if your suite guarantees every test carries an id.

extractSummary(source): ExtractSummary

The report's own stats block: expected, unexpected, flaky, skipped, duration, startTime.

stats.unexpected counts every unexpected entry individually, while extractFailedTests groups them. They agree on a plain report and are expected to disagree once a rerun has been merged in, so do not use stats as a cross-check there.

extractReportErrors(source): ReportErrorSummary[]

The report's top-level errors[]: config and compile failures, such as a spec file that fails to import.

Check this one. Such a run produces zero failed specs, so code that only looks at extractFailedTests concludes a catastrophically broken build had no failures and reruns nothing.

const errors = extractReportErrors('results.json');
if (errors.length) {
  console.error('Build is broken, not merely failing:', errors[0].message);
}

How it is tested

This package's own test suite contains no stored JSON reports. Every report it asserts against is produced on each run by really invoking Playwright, including the real blob reporter and the real merge-reports CLI, across nine scenarios: describe-less files, four-level nesting, timeouts, flaky retries, abandoned retries, interrupted attempts, multi-project merges and rerun merges.

A committed snapshot would stop testing the installed Playwright and start testing a frozen copy of what it used to emit, so a schema change in an upgrade would pass silently. Here it fails the build.

See report-src/README.md, which also documents an upstream merge-reports bug this library has to work around.

License

MIT (c) Harry Tran