@midleman/playwright-reporter

A Playwright reporter that sends test results to the E2E Test Insights API, plus duration-balanced test sharding built on the same data.

npm install @midleman/playwright-reporter

Setup

// playwright.config.ts
export default defineConfig({
    reporter: [
        ['html'],
        ['@midleman/playwright-reporter', { mode: 'prod', repoName: 'positron' }],
        // Optional: duration-balanced sharding. Needs Playwright >= 1.62.
        ['@midleman/playwright-reporter/sharding', {}],
    ],
});

CONNECT_API_KEY (or E2E_CONNECT_APIKEY) authenticates both.

Reporting options

| Option | Default | Notes | | ---------- | -------------------- | ---------------------------------------------- | | mode | required | dev (localhost:8000), prod, or disabled | | repoName | GITHUB_REPOSITORY | Repo id | | verbose | false | Debug logging |

mode: 'prod' only reports from CI (GITHUB_ACTIONS or CI set); running it locally is a no-op so you can't pollute production data.

| Variable | Purpose | | -------- | ------- | | CONNECT_API_KEY | Posit Connect API key (required for prod) | | REPORTER_REPO_NAME | Overrides repoName for both reporters | | REPORT_URL | Base URL for Playwright HTML report links |

Predictive sharding

Playwright's native --shard splits by test count, so one shard can run twice as long as another. This splits by measured duration instead: the API returns a partition computed by LPT bin-packing over 14 days of per-spec medians, filtered to the OS and browser you're actually running on.

Since 0.55.0 this is the only way to shard. The e2e-insights run-shard CLI and its SHARD_TOTAL env var are gone; run plain playwright test --shard and add the reporter below.

Add the sharding reporter (see Setup) and shard as normal:

- run: npx playwright test --shard=${{ matrix.shard }}/4

It takes no options in CI. repoName, OS, browser and the API URL are all detected, and repoName resolves exactly as it does for the reporter above — which matters, because the two have to agree: sharding looks up duration history under that id and the reporter files results under it. Set it in both places, or neither.

How it works

1. Playwright calls the reporter's preprocess hook with the resolved config and the full suite.
2. The hook posts every spec path to POST /shards/compute, along with the shard index, shard count, and detected OS/browser.
3. The API bin-packs the specs by historical median duration and returns this shard's list.
4. The hook calls skipSharding() to opt out of Playwright's own split, then exclude()s every test not in its list.

Shard names and run IDs

Each shard needs a distinct identity so results don't overwrite each other in S3. The run ID is {githubRunId}-{attempt}-{browser}-{os}-{shardIndex}:

12345-1-electron-ubuntu-1
12345-1-electron-ubuntu-2

The shard index comes from config.shard, which survives skipSharding(), so the hook needs no help to number its shards. Set SHARD_NAME only when two sibling jobs would otherwise share the same (browser, os, index) triple.

OS and browser detection

OS: mac, win, or the specific Linux distro (ubuntu, debian, rhel, sles, opensuse) read from /etc/os-release, falling back to linux only when the distro can't be identified. One detector serves the shard name, the stored os column, and the duration filter, so they can't disagree.

Browser: the hook reads the running project from the suite and maps it by keyword (e2e-electron → electron). A project matching no known browser (e.g. e2e-windows) leaves the filter unset rather than guessing.

Coordination

Shards start at slightly different times, so they'd each compute a partition over a slightly different snapshot of history — and independently-computed partitions can overlap or leave gaps.

The API prevents that with an atomic create-if-absent write to S3, keyed on the workflow run: the first shard to ask computes the partition and publishes it, and every other shard in that run reads back the same object. One partition per run, by construction.

When it degrades

Both cases are deliberately quiet — an unbalanced run beats a broken one:

• API unreachable, or no history for these specs → warns and leaves Playwright's native split in place.
• Playwright < 1.62 → preprocess is never called. The reporter notices at onBegin and warns that balancing is off.

Watch for these two lines to confirm it's working:

[predictive-sharding] shard 2/4: 37/149 specs, ~412s
[predictive-sharding] shard 2/4 done: est 412s, actual 300s (0.73x)

The ratio says whether the estimate held. Near 1.0 is a good partition; off the same way on every shard means the duration history is stale; off on some shards only means those runners were slower or faster than the rest.

Publishing

npm test && npm run build
npm version patch   # or minor/major
npm publish --access public
git push