@mergifyio/mocha

A Mocha plugin for Mergify: it uploads OpenTelemetry traces of your test runs to Mergify Test Engine, with no JUnit file to write and upload afterwards, keeps the failures of the tests it has quarantined from failing your run, reruns new tests to catch a flaky one before it merges, retries a failed test on your default branch, runs a merge-queue bisection step's failures first, and leaves your reporter as it is.

More information at https://mergify.com

Installation

Install the package as a dev dependency alongside mocha (>= 8.2.0, the first Mocha with global fixtures):

npm install --save-dev @mergifyio/mocha

It needs Node.js 22 or later. Its CI detection is a native binding, prebuilt for Linux x86_64 and arm64 (glibc 2.28 or newer, as Node.js 22 itself needs, and musl), macOS arm64, and Windows x64. The macOS x86_64 (Intel) binding is best effort: published, but not tested before a release.

On Mocha 8.2, run serially: its own parallel mode (--parallel) crashes in its worker pool on current Node.js, with or without the plugin. Mocha 8.3 and later run in either mode.

On Node.js 26, use Mocha 10.6 or later: Mocha 8.3 to 10.5 pin yargs 16.2.0, which Node.js 26 loads as an ES module, so Mocha crashes on start, with or without the plugin.

Usage

Have Mocha require the plugin, in your Mocha config:

// .mocharc.js
module.exports = {
  require: ['@mergifyio/mocha'],
};

# .mocharc.yml
require:
  - '@mergifyio/mocha'

or on the command line:

mocha --require @mergifyio/mocha

Set MERGIFY_TOKEN in your CI environment. Without it, the plugin says it has nowhere to upload to, and your tests run normally. Outside CI it stays silent unless you set MERGIFY_CI_ENABLE.

The plugin registers Mocha's global setup and teardown fixtures, which Mocha runs once per run, in its main process, whether it runs your files there or in its workers (--parallel). It takes up no reporter: keep spec, mocha-junit-reporter, mochawesome or any other. Mocha merges the require lists of its config and of its command line, so a --require of your own keeps the plugin.

Code that runs Mocha through its API instead of its command line does not read --require: pass the plugin's fixtures yourself.

import { mergify } from '@mergifyio/mocha';
import Mocha from 'mocha';

const { mochaGlobalSetup, mochaGlobalTeardown } = mergify();
const mocha = new Mocha({ globalSetup: mochaGlobalSetup, globalTeardown: mochaGlobalTeardown });

Mocha 12 is an ES module: import it, as above. Node 22's require hands it over as a namespace rather than as Mocha's constructor.

The plugin reports nothing in watch mode, nor for a dry run (--dry-run, Mocha 9 and later), where Mocha reports every test as passed without running it, and says so.

What reaches Test Engine

Every test of the run, as it ended: passed, failed with its error, or skipped.

A failing hook stops Mocha from running tests, and Mocha reports only the hook. The plugin reports each test the hook kept from running as failed, with the hook's error, as Mocha's --fail-hook-affected-tests does:

• a failing before hook: every test of its describe, nested ones included, that --grep would have run;
• a failing beforeEach hook: the test it ran for, and the rest of its describe;
• a failing afterEach hook: the test it ran after, and the rest of its describe.

A failing after hook fails no test: it fails the run.

A test Mocha's retries ran again reaches Test Engine as its first run, how it ended and what it raised, with every run after it counted, and the time they all took.

Quarantine

Before any test runs, the plugin fetches the list of tests quarantined in Test Engine. When one of them fails, Mocha counts it as passed, so it does not fail your run, nor stop it under --bail, and your reporter shows it passed; it reaches Test Engine as failed and quarantined. A quarantined test that passes is reported as passing. This works the same whether Mocha runs your files in its main process or in its workers (--parallel), and the report the plugin prints under Mergify CI once the run is over says what the quarantine did.

A failing hook's failure is kept from failing your run when every test it kept from running is quarantined. A failing after hook keeps no test from running, so the quarantine never covers it.

A test quarantined while you uploaded your results through a JUnit file is quarantined under the name it had there, as mocha-junit-reporter or Mocha's xunit reporter name a test by default, and the plugin matches that name too.

Flaky detection

On a pull request, a test the target branch does not have yet is new: the plugin runs it again right after its own run, with its beforeEach and afterEach hooks around each run, as many times as fit in the time budget Mergify sets. When one of those runs fails, the test is flaky: Mocha reports it failed, with that run's error, so that it does not merge. The runs Mocha's retries made of it count too: a new test they rescued is flaky, and Mocha reports it failed with its first run's error. They count neither against the budget nor against the cap on the reruns. On a push, the plugin reruns the tests Mergify knows as unhealthy, and their reruns only inform Test Engine. Your reporter hears of no rerun, and Mocha's retries do not apply to them. The report under Mergify CI says what the reruns found.

In a parallel run (--parallel), the reruns wait for the run's tests to be over: then, before Mocha's global teardown, Mocha's workers run again each file holding a test to rerun, whole, and that test as many times as fit. Your reporter has printed the run by then, so it shows a new test found flaky as passed; the report under Mergify CI names it, and the run fails.

Test retry

On a push to your repository's default branch, a test that fails runs again right after the run that failed, with its beforeEach and afterEach hooks, as Mocha runs a test again for its retries: until a run passes, up to the number of retries your repository sets in Mergify, and within one time budget for the whole run, which failures spend in the order they happen. A test a retry passed does not fail your run, and your reporter shows it passed, as it shows a test its retries rescued; it reaches Test Engine as its first run, which failed. Mocha's own retries come first: test retry runs a test again once they all failed. In a parallel run (--parallel), each worker retries the tests it runs, and the workers spend the budget together. The report under Mergify CI says what test retry did.

A test the quarantine holds, or one Mergify classifies as broken, is not retried, nor is any test of a pull request, a merge-queue run or another branch. Mocha has no way to opt one test out.

In a serial run, a test flaky detection reruns on that push, one Mergify knows as unhealthy, is not retried: its reruns, right after it, answer for its failure, and once one of them passed, it does not fail your run either. In a parallel run, where flaky detection reruns tests once the run is over, test retry retries it as any other.

A job that sets MERGIFY_TEST_RETRY_ENABLE=false retries nothing, whatever the repository sets.

Bisection steps: failures first

When the merge queue splits a failed batch, each bisection step only needs to tell whether it is red. The plugin asks Mergify, before Mocha runs any test, whether the current run is such a step and, if so, the files holding the tests this job failed in the run the step was split from run first; once they all ran, the run stops as soon as one of them failed again (Fail First), so the queue can split the batch again without waiting for the rest of the suite. No test is ever skipped, so a green step still comes from a full suite.

It is on by default. A job that must always run its suite in its usual order sets MERGIFY_FAIL_FIRST_ENABLE=false, and never asks. Only a recognised no (false, no, off, 0) turns it off. MERGIFY_TEST_SELECTION_ENABLE, its former name, is still read while the new one is unset or empty, and the summary at the end of the run says to switch. It is also enabled per organization on Mergify's side: until yours has it, every run is served the full suite, says nothing about it, and still sends its results, which show what Fail First would do there.

It only ever asks in a build of a merge-queue batch, the only builds a bisection step can be: on GitHub Actions, the event of the draft pull request the queue opened; on any other build, the queue's git note on the branch, one git ls-remote against origin that runs while the tests are being collected. Every other build asks nothing and sends nothing. Of a batch's builds, only those below a failed batch, and not retrying their own checks, ask: the others cannot be a bisection step, and send their results only when the batch holds two pull requests or more, so a bisection step may read them, or when they failed. A build where git cannot answer (no origin, credentials the test step does not hold) asks, as it would without this.

• Each run is matched to the step above by the tests it runs: in parallel mode, the same files under the same --grep. A job that splits its files between several runs gives each its own MERGIFY_TEST_JOB_NAME.
• In parallel mode, the run stops as --bail stops it: the files Mocha's workers are still running finish, and Mocha drops their results. Mocha's main process never loads a parallel run's files, so the plugin then loads the files the stop left unrun there, as Mocha's workers load theirs, before your global teardown: none of their tests runs, and the run counts every test, as one that went on would. In a serial run, the tests declared outside any describe run before the others, whatever their file.
• Any other answer, an error or a timeout runs the full suite in its usual order.
• If the run fails outside any test (a failing hook, an error no test raised), its results are not sent to Mergify, so the next bisection step runs its full suite in its usual order.

At the end of the run, the plugin sends Mergify what it concluded (the counts and the failing tests), which the next bisection step of the job is answered from.

Test names

Each test reaches Test Engine named by its file, relative to the directory Mocha runs in, the describe blocks around it, and its own title:

test/api.spec.js > users > creates one

The file is part of the name because tests declared outside any describe often share a title across files (works), and would otherwise be one test to Mergify.

If you uploaded your Mocha results through a JUnit reporter before, your tests had other names, so their history in Test Engine starts again under these ones.

Plugin options

mergify takes the plugin's options; export its fixtures from a file of your own, and require that file instead:

// mergify.js
const { mergify } = require('@mergifyio/mocha');

module.exports = mergify({ token: process.env.MY_MERGIFY_TOKEN });

// .mocharc.js
module.exports = {
  require: ['./mergify.js'],
};

| Option | Description | Default | |---|---|---| | token | Mergify API authentication token | MERGIFY_TOKEN | | apiUrl | Mergify API endpoint | MERGIFY_API_URL | | doubles | For testing: Mergify played in memory. The run keeps what it would have sent (the trace) instead of sending it, and appends it to the file doubles.record names, one JSON line each; quarantine, flakyContext, selection and pullRequests are the bodies Mergify answers the quarantine list, the flaky-detection context, Test Selection and the open pull requests at the run's commit with, and trace can make the upload meet a refusal ({ error }) or a route Mergify does not serve (null). It reports only where it would anyway: in CI, or with MERGIFY_CI_ENABLE. | — |

Environment variables

| Variable | Description | Default | |---|---|---| | MERGIFY_TOKEN | Mergify API authentication token | (required) | | MERGIFY_API_URL | Mergify API endpoint | https://api.mergify.com | | MERGIFY_CI_ENABLE | Force-enable outside CI | false | | MERGIFY_TEST_JOB_NAME | The name this job reports under; recommended per leg of a matrix | provider job name | | MERGIFY_FAIL_FIRST_ENABLE | Set to false to never run first what a merge-queue bisection step of this job failed before | true | | MERGIFY_TEST_RETRY_ENABLE | Set to false to have this job retry no failing test, whatever the repository sets | true | | MERGIFY_CI_DEBUG | Print spans to console instead of uploading | false | | MERGIFY_TRACEPARENT | W3C distributed trace context | — | | MERGIFY_MOCHA_HANDOFF_FILE | The run, as the plugin publishes it for Mocha's workers to join in parallel mode (set internally; do not set manually) | — |

For detailed documentation, see the official guide.

Development

Clone the repo and install dependencies:

pnpm install

Available scripts (from this package's directory or with pnpm --filter @mergifyio/mocha):

| Command | What it does | |---|---| | pnpm test | Run the unit tests once (vitest run) | | pnpm run test:integration | Build the package, then run Mocha on the fixture project in tests/fixtures, serial and in parallel | | pnpm run build | Bundle the package with tsdown |