@mergifyio/playwright

Find the Playwright tests that pass and fail on the same code, and quarantine them so they stop failing your CI. Requires a Mergify account (set MERGIFY_TOKEN in CI); free for open source. Why tests go flaky in Playwright, and how to fix them: mergify.com/learn/flaky-tests/playwright.

A Playwright reporter that integrates seamlessly with Mergify, uploading OpenTelemetry traces of test executions to Mergify Test Engine and absorbing failures of the tests it has quarantined.

More information at https://mergify.com

Installation

Install the package as a dev dependency alongside @playwright/test (>= 1.56.0):

npm install --save-dev @mergifyio/playwright

It needs Node.js 22 or later. Its CI detection is a native binding, prebuilt for Linux x86_64 and arm64 (glibc 2.28 or newer, as Node.js 22 itself needs, and musl), macOS arm64, and Windows x64. The macOS x86_64 (Intel) binding is best effort: published, but not tested before a release.

Usage

Wrap your playwright.config.ts with withMergify and import test / expect from @mergifyio/playwright instead of @playwright/test:

// playwright.config.ts
import { defineConfig } from '@playwright/test';
import { withMergify } from '@mergifyio/playwright';

export default withMergify(defineConfig({
  projects: [{ name: 'chromium', use: { /* ... */ } }],
}));

// tests/example.spec.ts
import { test, expect } from '@mergifyio/playwright';

test('flaky thing', async ({ page }) => {
  // ...
});

Set MERGIFY_TOKEN in your CI environment. Without it, the integration stays silent and tests run normally.

withMergify registers the reporter that uploads test-run traces to Mergify Test Engine, plus a globalSetup that loads the quarantine list and the contexts of flaky detection and test retry. The test export is Playwright's base test extended with an auto-fixture: when a test's name is on the quarantine list AND it fails, the fixture sets testInfo.expectedStatus = 'failed' so Playwright reports the outcome as passing. Quarantined tests that pass are reported as passing unchanged (no "unexpected pass" penalty — matches pytest's xfail(strict=False)).

At the end of the run, a summary is printed on stderr, worded as every Mergify test client words it:

Mergify CI ────────

🛡️ Quarantine
- Repository: acme/app
- Branch: main
- Quarantined tests fetched from API: 3

- 🔒 Quarantined:
    · tests/auth.spec.ts > Login > submits form (failure absorbed)

- Unused quarantined tests:
    · tests/api.spec.ts > retries once
    · tests/data.spec.ts > builds payload

Test run ID: 1f0c2a9b8e7d6c5b (use it when contacting Mergify support)

Every quarantined test that ran is listed, and its span carries cicd.test.quarantined; a failure the quarantine absorbed is still uploaded as failed, so Mergify keeps seeing the test fail. On a shard, entries the shard did not run are listed as "not run by this worker".

Gotcha: wrapping the config with withMergify but forgetting to change the test import leaves the quarantine list fetched but never applied — quarantined tests that fail still fail the run, and their entries show up under "Unused quarantined tests".

Flaky detection (preview)

Flaky detection is opt-in per repository from the Mergify dashboard. Once a repository has opted in, the reporter:

1. Loads the API context in globalSetup, and reruns by the run's shape:
   • new mode on PR-like runs (a base ref is detected): newly-added tests are candidates; phase-1 failures stand (no absorption).
   • unhealthy mode on push or scheduled runs: API-listed unhealthy tests are candidates; phase-1 failures stand too, and the reruns only measure flakiness.
2. Records each candidate's phase-1 outcome and duration during the normal test run.
3. After the main run, spawns a single Playwright subprocess (playwright test --test-list '<candidates>' --repeat-each=N --workers=1) that re-runs each candidate N times with native fresh fixtures, one attempt after another, so that a test's span, which lasts all its attempts, ends within the run's. The subprocess joins the main process's run, from the file globalSetup published it to, and hands each attempt back to it.
4. Counts phase-1 and phase-2 attempts per candidate. Mixed pass/fail → the candidate is flagged flaky and four attributes are emitted on its span: cicd.test.flaky_detection, cicd.test.new, cicd.test.flaky, cicd.test.rerun_count.
5. In new mode, fails the run on a candidate it flagged flaky, as every Mergify test client does, so that a flaky test does not merge. A phase-2 failure never reaches Playwright's own results: its summary still lists the test as passed, and the exit code is the reporter's. In unhealthy mode the reruns never change how the run ends.
6. Prints what it found on stderr, worded as every Mergify test client words it.

Each phase-2 rerun is a fresh Playwright test invocation, so all fixtures (including user-defined test.extend(...) ones) are re-initialised between attempts — this matches Playwright's normal test-isolation guarantees.

Caveats

• Cost. Phase 2 spawns an extra playwright test invocation, which runs the attempts one after another: it lasts as long as they add up to, which the budget caps.
• Runtime test.skip(condition) inside a candidate body can produce ambiguous outcomes — the test is recorded as skipped, but rerun iterations may behave differently from the first.
• Aggregation only counts phase-2 attempts as rerunCount. Phase 1's attempt is included in the flakiness decision but not in the count.

Test retry

On a push to the repository's default branch, for a repository with test retry on, the reporter runs a failing test again once Playwright's own retries of it are over, up to the number of retries the repository sets, within one time budget for the whole run, and stops at the first pass. It goes through Playwright's own retries: as the test fails, the reporter raises its retries by one when Mergify grants another attempt, and Playwright runs it again as it runs any retry. A test that passed on a retry is flaky to Playwright, which passes the run; its span still says its first attempt failed, and carries cicd.test.auto_retry. A test flaky detection reruns is retried in the run like any other, and flaky detection still reruns it after the run.

Not retried: a quarantined test, a test Mergify classifies broken, a test several projects run under one name (set MERGIFY_PLAYWRIGHT_INCLUDE_PROJECT_IN_TEST_NAME to tell them apart), a test of a serial group, and every test under failOnFlakyTests. Pull requests, merge-queue runs and other branches are never retried.

A job that sets MERGIFY_TEST_RETRY_ENABLE=false retries nothing, whatever the repository sets.

Bisection steps: failures first

It is on by default. A job that must always run its suite in its usual order sets MERGIFY_FAIL_FIRST_ENABLE=false, and never queries the endpoint. Only a recognised no (false, no, off, 0) turns it off. MERGIFY_TEST_SELECTION_ENABLE, its former name, is still read while the new one is unset or empty, and the summary at the end of the run says to switch. It is also enabled per organization on Mergify's side: until yours has it, every run is served the full suite, says nothing about it, and still sends its results, which show what Fail First would do there.

It only ever asks in a build of a merge-queue batch, the only builds a bisection step can be: on GitHub Actions, the event of the draft pull request the queue opened; on any other build, the queue's git note on the branch, one git ls-remote against origin that runs while the tests are being collected. Every other build asks nothing and sends nothing. Of a batch's builds, only those below a failed batch, and not retrying their own checks, ask: the others cannot be a bisection step, and send their results only when the batch holds two pull requests or more, so a bisection step may read them, or when they failed. A build where git cannot answer (no origin, credentials the test step does not hold) asks, as it would without this.

When Mergify's merge queue splits a failed batch, each bisection step only needs to tell whether it is red. The reporter asks the API whether this run is such a step — from Playwright's Reporter.preprocess() hook, once the collection is known, because the request carries a fingerprint of the tests this run collected. If it is, the files holding the tests this job failed in the run the step was split from run first, and the run stops at the first of those tests that fails again (Fail First), so the queue can split again without waiting for the suite. No test is ever skipped, so a green step still comes from a full suite. Any other answer — a run that is not a bisection step, an API error or timeout, an answer this version does not understand — runs the full suite in its usual order. A run that is not a bisection step says nothing about it; one that could not be asked, or was answered something this version does not understand, says why in one sentence:

⏩ Fail First

Mergify couldn't be asked which tests to run first, so the full suite ran.
Error: Mergify API request timed out

At the end of the run the reporter also files the session's verdict with Mergify — which tests failed after Playwright's own retries, and how many ran — before uploading the trace, so the next bisection step can be answered seconds after this one ended. A verdict that does not land never fails the run.

Setup and teardown projects always run in full: Playwright makes their tests read-only. They are not part of the fingerprinted collection, but their results are part of the verdict.

Sharded jobs

Each --shard=k/N leg is its own job for Mergify: it asks with its own collection, is answered with its own failures, and files its own verdict. Two things follow:

• The legs share one job name. Mergify tells them apart by what they collected: two legs of one job run different slices, so they carry different fingerprints. It identifies a test by its name and its job's name, so a name per leg, such as e2e-${{ matrix.shard }}, splits each test's history between the legs it ran in. Set MERGIFY_TEST_JOB_NAME once per environment (an operating system, a Node.js version), never per shard.
• The reporter asks Playwright which tests this leg owns once the job opted in. preprocess runs before Playwright shards, so the reporter runs the same command again with --list and a reporter of its own, which reads the ids off the suite Playwright has by then sharded. Your legs get the shares they always got — --grep, --project, PWTEST_SHARD_WEIGHTS and every other filter included, because the listing replays your own command line — and a future release that groups tests differently is simply obeyed. The extra collection costs well under a second (0.66 s on a suite of 811 tests) and starts neither your web server nor your globalSetup.
• That listing also reports the whole suite it saw, and the reporter checks it against what it collected itself, as a set, in both directions. Anything the listing has and this run does not — or, just as important, anything this run has and the listing missed — means the two are looking at different suites, and a slice drawn over the wrong one would exclude tests that belong to this leg from every leg at once.
• A setup or teardown project is left out on both sides. Playwright re-attaches those projects to every leg after sharding and their tests always run in full, so counting them would make a sharded run with an auth-setup project — the common shape — look like a mismatch and run in full.
• If the listing fails — Playwright exits non-zero, times out, answers something the reporter cannot read, or describes a suite this run does not have — the leg runs its own share in full: sharding stays with Playwright, Mergify is not asked, and the block says so. A slice is never guessed. One known limit follows from --list not running globalSetup: a suite whose collection depends on it lands here.
• A leg's slice must be stable from one step to the next — same suite, same weights, same number of legs. If it changes, the leg's fingerprint no longer matches the step above's and Mergify serves it the full suite in its usual order. Keeping the slice stable is the job's responsibility, not something Mergify guesses at.

Only one reporter may take sharding over; a second one calling skipSharding() makes Playwright abort the run.

Requires @playwright/test 1.62 or later, where Reporter.preprocess() was added. On older versions the hook is never called and the full suite runs in its usual order.

Multi-project test names

When your config defines named projects (e.g. one per browser), each project runs the same tests. To keep their results distinct in Test Engine, Mergify prefixes the project to the test name, following Playwright's JUnit includeProjectInTestName convention:

[chromium] > tests/login.spec.ts > logs in
[firefox] > tests/login.spec.ts > logs in

This is opt-in (off by default, to preserve existing test history). Set MERGIFY_PLAYWRIGHT_INCLUDE_PROJECT_IN_TEST_NAME=true to enable it. Tests with no project name are never prefixed.

Reporter options

withMergify takes the reporter's options as its second argument:

export default withMergify(defineConfig({ /* ... */ }), {
  token: process.env.MY_MERGIFY_TOKEN,
});

| Option | Description | Default | |---|---|---| | token | Mergify API authentication token | MERGIFY_TOKEN | | apiUrl | Mergify API endpoint | MERGIFY_API_URL | | doubles | For testing: Mergify played in memory. The run keeps what it would have sent (the verdict, the trace) instead of sending it, and appends it to the file doubles.record names, one JSON line each; verdict and trace can make a send meet a refusal ({ error }) or a route Mergify does not serve (null). It reports only where it would anyway: in CI, or with MERGIFY_CI_ENABLE. | — |

Environment variables

| Variable | Description | Default | |---|---|---| | MERGIFY_TOKEN | Mergify API authentication token | (required) | | MERGIFY_API_URL | Mergify API endpoint | https://api.mergify.com | | MERGIFY_CI_ENABLE | Force-enable outside CI | false | | MERGIFY_PLAYWRIGHT_INCLUDE_PROJECT_IN_TEST_NAME | Prefix the project to multi-project test names as [project] > … | false | | MERGIFY_FAIL_FIRST_ENABLE | Set to false to never run first what a merge-queue bisection step of this job failed before; only a recognised no turns it off | true | | MERGIFY_TEST_RETRY_ENABLE | Set to false to have this job retry no failing test, whatever the repository sets | true | | MERGIFY_TEST_JOB_NAME | The name this job reports under; set one per environment, never per shard | provider job name | | MERGIFY_CI_DEBUG | Print spans to console instead of uploading | false | | MERGIFY_TRACEPARENT | W3C distributed trace context | — | | MERGIFY_PLAYWRIGHT_HANDOFF_FILE | The run, as globalSetup publishes it for the processes Playwright starts afterwards to join, the flaky-detection subprocess included (set internally; do not set manually) | — |

PLAYWRIGHT_MERGIFY_ENABLE and PLAYWRIGHT_MERGIFY_INCLUDE_PROJECT_IN_TEST_NAME, the names MERGIFY_CI_ENABLE and MERGIFY_PLAYWRIGHT_INCLUDE_PROJECT_IN_TEST_NAME replace, are deprecated: they still work when the new name is not set, and the summary at the end of the run says which name to switch to.

For detailed documentation, see the official guide.

Development

Clone the repo and install dependencies:

pnpm install

Available scripts (from this package's directory or with pnpm --filter @mergifyio/playwright):

| Command | What it does | |---|---| | pnpm test | Run the test suite once (vitest run) | | pnpm run build | Bundle the package with tsdown |