@mergifyio/jest

Find the Jest tests that pass and fail on the same code, and quarantine them so they stop failing your CI. Requires a Mergify account (set MERGIFY_TOKEN in CI); free for open source. Why tests go flaky in Jest, and how to fix them: mergify.com/learn/flaky-tests/jest.

A Jest reporter for Mergify: it uploads OpenTelemetry traces of your test runs to Mergify Test Engine, with no JUnit file to write and upload afterwards, keeps the failures of the tests it has quarantined from failing your run, reruns new tests to catch a flaky one before it merges, retries a failing test on your default branch, and runs a merge-queue bisection step's failures first.

More information at https://mergify.com

Installation

Install the package as a dev dependency alongside jest (>= 29.0.0):

npm install --save-dev @mergifyio/jest

It needs Node.js 22 or later. Its CI detection is a native binding, prebuilt for Linux x86_64 and arm64 (glibc 2.28 or newer, as Node.js 22 itself needs, and musl), macOS arm64, and Windows x64. The macOS x86_64 (Intel) binding is best effort: published, but not tested before a release.

Usage

Wrap your Jest config with withMergify:

// jest.config.js
const { withMergify } = require('@mergifyio/jest');

module.exports = withMergify({
  testEnvironment: 'node',
});

// jest.config.ts
import type { Config } from 'jest';
import { withMergify } from '@mergifyio/jest';

const config: Config = {
  testEnvironment: 'node',
};

export default withMergify(config);

Set MERGIFY_TOKEN in your CI environment. Without it, the reporter says it has nowhere to upload to, and your tests run normally. Outside CI it stays silent unless you set MERGIFY_CI_ENABLE.

withMergify adds the Mergify reporter after the reporters your config lists, or after Jest's default reporter when it lists none. It gives your config, and each project it lists inline, Mergify's test runner: Jest's own, jest-circus, with the quarantine applied to each test, a failing test retried, and flaky detection's reruns run through it. It gives them Mergify's runner too: Jest's own, jest-runner, which starts the files Fail First puts first and stops the run when Fail First says so. It gives the run Mergify's sequencer: Jest's own, which notes the files the run is about to start, for Test Selection. It also turns on testLocationInResults, so each test reaches Test Engine with the line it is declared on.

withMergify is the only supported setup:

• A config in package.json or in jest.config.json has no function to call: move it to a jest.config.js (or .ts, .mjs, .cjs) and wrap it there.
• projects listed inline are covered by the root config's withMergify. A project listed by path is loaded from its own file: wrap that file too.
• --reporters on the command line replaces the reporters of the config, the Mergify one included. Set the config's reporters to the reporters it names, in order, then drop the flags: jest --reporters=default --reporters=jest-junit becomes reporters: ['default', 'jest-junit']. Dropping them without moving them stops what they produce, such as a JUnit file another step reads.
• A testRunner other than jest-circus (jest-jasmine2, your own) is kept, and neither the quarantine, flaky detection nor test retry applies to its tests.
• A runner other than jest-runner is kept: Fail First neither starts that project's files first nor stops them.
• A testSequencer of your own is kept, and Fail First does not apply: the run asks nothing and runs in full.

At the end of the run, the reporter names each project whose tests the quarantine and flaky detection could not reach, and how many failures test retry could not run again.

Quarantine

Before any test runs, the reporter fetches the list of tests quarantined in Test Engine. When one of them fails, Jest counts it as passed, so it does not fail your run, and it reaches Test Engine as failed and quarantined. A quarantined test that passes is reported as passing. This works the same whether Jest runs your files in its workers or in its own process (--runInBand), and the reporter prints a summary of what the quarantine did.

A test quarantined while you uploaded your results through jest-junit is quarantined under the name jest-junit gave it, and the reporter matches that name too.

Flaky detection

When your repository has flaky detection on in Test Engine, the reporter fetches its settings before any test runs. On a pull request, every test the target branch does not have yet is new: once your run is over, the reporter reruns the new tests, within the time budget Test Engine sets, and when a test both passed and failed, it is flaky and the run fails, so that it does not merge. On a push to a branch with no target, it reruns the tests Test Engine knows as unhealthy instead, to report whether they still are, without failing the run.

The reruns happen in Jest's workers, each file the way your run ran it: its hooks run around each attempt, its snapshots are checked and never written, and Jest's own retries are off. Jest's summary and the other reporters only see your run: when the reruns find a new test flaky, the reporter's summary says so, and the run exits with a failure.

They do not happen in watch mode, after you interrupt the run, or once --bail stopped it. A test.concurrent test is never rerun: jest-circus starts it with the others of its file, and cannot run it again on its own.

Test retry

When your repository has test retry on in Test Engine, a run that builds your default branch (a push to it, never a pull request or a merge-queue run) retries each test that fails: Mergify's test runner runs it again, right after its failed attempt, in the worker that ran it or in Jest's own process, up to the number of retries Test Engine sets, within one time budget for the whole run, which failures spend in the order they happen. A retry that passes leaves the test passed, and your run green on it; the test still reaches Test Engine as failed, its first attempt, marked as retried. The reporter's summary says which tests passed on a retry and which failed every attempt.

A quarantined test, and one Test Engine knows to be broken, are not retried. Jest's own retries come first: test retry runs a test again once they all failed, which it can only do when Jest retries it right after it (jest.retryTimes(n, { retryImmediately: true })). A test Jest retries at the end of its describe block (jest.retryTimes without retryImmediately, the only way in Jest 29) or with its whole describe block is not retried, nor is a test.concurrent test: the reporter says how many it left.

A job that sets MERGIFY_TEST_RETRY_ENABLE=false retries nothing, whatever the repository sets.

Bisection steps: failures first

When the merge queue splits a failed batch, each bisection step only needs to tell whether it is red. The reporter asks Mergify, before Jest starts any file, whether the current run is such a step and, if so, the files holding the tests this job failed in the run the step was split from run first; once they all ran, the run stops as soon as one of them failed again (Fail First), so the queue can split the batch again without waiting for the rest of the suite. In Jest's workers, the stop ends the workers still running a file, which Jest reports as a worker that "failed to exit gracefully". No test is ever skipped, so a green step still comes from a full suite.

It is on by default. A job that must always run its suite in its usual order sets MERGIFY_FAIL_FIRST_ENABLE=false, and never asks. Only a recognised no (false, no, off, 0) turns it off. MERGIFY_TEST_SELECTION_ENABLE, its former name, is still read while the new one is unset or empty, and the summary at the end of the run says to switch. It is also enabled per organization on Mergify's side: until yours has it, every run is served the full suite, says nothing about it, and still sends its results, which show what Fail First would do there.

It only ever asks in a build of a merge-queue batch, the only builds a bisection step can be: on GitHub Actions, the event of the draft pull request the queue opened; on any other build, the queue's git note on the branch, one git ls-remote against origin that runs while the tests are being collected. Every other build asks nothing and sends nothing. Of a batch's builds, only those below a failed batch, and not retrying their own checks, ask: the others cannot be a bisection step, and send their results only when the batch holds two pull requests or more, so a bisection step may read them, or when they failed. A build where git cannot answer (no origin, credentials the test step does not hold) asks, as it would without this.

• --shard is supported: each shard reports its own collection, by the files it runs.
• Jest knows a file's tests only once it ran the file, so once Fail First stopped the run, the reporter runs the files the stop left unrun again, in Jest's workers, under a test name none of their tests matches: Jest imports each and runs none of its tests, and the run counts every test, as one that went on would. Your global setup is still up then, and does not run again.
• Any other answer, an error or a timeout runs the full suite in its usual order.
• If the run fails outside any test (a file that cannot run, a snapshot left obsolete), its results are not sent to Mergify, so the next bisection step runs its full suite in its usual order.

At the end of the run, the reporter sends Mergify what it concluded (the counts and the failing tests), which the next bisection step of the job is answered from.

Test names

Each test reaches Test Engine named by its file, relative to Jest's rootDir, the describe blocks around it, and its own title:

src/app.test.js > App > header > renders

The file is part of the name because tests declared outside any describe often share a title across files (renders without crashing), and would otherwise be one test to Mergify.

If you uploaded your Jest results through jest-junit before, your tests had other names, so their history in Test Engine starts again under these ones.

Reporter options

withMergify takes the reporter's options as its second argument:

module.exports = withMergify(config, {
  token: process.env.MY_MERGIFY_TOKEN,
});

| Option | Description | Default | |---|---|---| | token | Mergify API authentication token | MERGIFY_TOKEN | | apiUrl | Mergify API endpoint | MERGIFY_API_URL | | doubles | For testing: Mergify played in memory. The run keeps what it would have sent (the trace) instead of sending it, and appends it to the file doubles.record names, one JSON line each; quarantine is the body Mergify answers the quarantine list with, flakyContext the one it answers flaky detection's settings with, testRetryContext the one it answers test retry's settings with, pullRequests the one it answers the open pull requests at the run's commit with, and trace can make the upload meet a refusal ({ error }) or a route Mergify does not serve (null). It reports only where it would anyway: in CI, or with MERGIFY_CI_ENABLE. | — |

Environment variables

| Variable | Description | Default | |---|---|---| | MERGIFY_TOKEN | Mergify API authentication token | (required) | | MERGIFY_API_URL | Mergify API endpoint | https://api.mergify.com | | MERGIFY_CI_ENABLE | Force-enable outside CI | false | | MERGIFY_FAIL_FIRST_ENABLE | Set to false to never run first what a merge-queue bisection step of this job failed before | true | | MERGIFY_TEST_RETRY_ENABLE | Set to false to have this job retry no failing test, whatever the repository sets | true | | MERGIFY_TEST_JOB_NAME | The name this job reports under; set one per environment, never per shard | provider job name | | MERGIFY_CI_DEBUG | Print spans to console instead of uploading | false | | MERGIFY_TRACEPARENT | W3C distributed trace context | — | | MERGIFY_JEST_HANDOFF_FILE | The run, as the reporter publishes it for Jest's workers to join, and for the ones that rerun tests (set internally; do not set manually) | — |

For detailed documentation, see the official guide.

Development

Clone the repo and install dependencies:

pnpm install

Available scripts (from this package's directory or with pnpm --filter @mergifyio/jest):

| Command | What it does | |---|---| | pnpm test | Run the unit tests once (vitest run) | | pnpm run test:integration | Build the package, then run Jest on the fixture project in tests/fixtures | | pnpm run build | Bundle the package with tsdown |