@midleman/playwright-reporter

A Playwright reporter that sends test results to the E2E Test Insights API, plus a CLI for predictive test sharding.

Installation

npm install @midleman/playwright-reporter

Quick Start

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
    reporter: [
        ['html'],
        ['@midleman/playwright-reporter', { mode: 'prod' }],
    ],
});

Set the CONNECT_API_KEY environment variable (required for mode: 'prod').

Reporter Configuration

Mode

| Mode | API Endpoint | Use Case | | ---------- | ------------------------------------------------ | ----------------- | | dev | http://localhost:8000 | Local development | | prod | https://connect.posit.it/e2e-test-insights-api | CI/Production | | disabled | None | Skip reporting |

Note: mode: 'prod' only works in CI (GITHUB_ACTIONS=true or CI=true). Running locally with mode: 'prod' disables reporting to avoid polluting production data.

Options

['@midleman/playwright-reporter', {
    mode: 'prod',           // Required: 'dev', 'prod', or 'disabled'
    repoName: 'my-repo',    // Optional: defaults to GITHUB_REPOSITORY
    verbose: true,          // Optional: enable debug logging
}]

Environment Variables

| Variable | Description | Required | | -------------------- | ---------------------------------------- | -------- | | CONNECT_API_KEY | Posit Connect API key | For prod | | REPORTER_REPO_NAME | Repository name override | No | | REPORT_URL | Base URL for Playwright HTML report links| No |

Predictive Sharding CLI

Distributes test specs across CI shards using historical duration data for optimal load balancing.

How It Works

┌────────────────────────────────────────────────────────────────────────────┐
│                              CI WORKFLOW                                   │
│                                                                            │
│  ┌──────────────────────────────────────────────────────────────────────┐  │
│  │  npx e2e-insights run-shard --shard-index=2 --shard-total=6 ...      │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
│                                    │                                       │
│                                    ▼                                       │
│  ┌──────────────────────────────────────────────────────────────────────┐  │
│  │  1. playwright test --list --reporter=json                           │  │
│  │     → Discovers all spec files in project                            │  │
│  │     → ["auth.spec.ts", "editor.spec.ts", "console.spec.ts", ...]     │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
│                                    │                                       │
│                                    ▼                                       │
│  ┌──────────────────────────────────────────────────────────────────────┐  │
│  │  2. POST /shards/compute                                             │  │
│  │     {                                                                │  │
│  │       "specs": ["auth.spec.ts", "editor.spec.ts", ...],              │  │
│  │       "shard_index": 1,                                              │  │
│  │       "num_shards": 6,                                               │  │
│  │       "repo": "posit-dev/positron"                                   │  │
│  │     }                                                                │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
│                                    │                                       │
│                                    ▼                                       │
│  ┌─────────────────────────────────────────────────────────────────────┐   │
│  │                         E2E INSIGHTS API                            │   │
│  │                                                                     │   │
│  │   ┌─────────────────┐    ┌─────────────────┐    ┌───────────────┐   │   │
│  │   │ Load test data  │ →  │ Filter & group  │ →  │ LPT algorithm │   │   │
│  │   │ (14-day window) │    │ by spec/os/etc  │    │ bin-packing   │   │   │
│  │   └─────────────────┘    └─────────────────┘    └───────────────┘   │   │
│  │                                                          │          │   │
│  │   Response:                                              ▼          │   │
│  │   {                                                                 │   │
│  │     "specs": ["editor.spec.ts", "console.spec.ts"],  ← shard 2      │   │
│  │     "estimated_duration_ms": 180000                                 │   │
│  │   }                                                                 │   │
│  └─────────────────────────────────────────────────────────────────────┘   │
│                                    │                                       │
│                                    ▼                                       │
│  ┌──────────────────────────────────────────────────────────────────────┐  │
│  │  3. playwright test editor.spec.ts console.spec.ts                   │  │
│  │     → Runs ONLY the specs assigned to this shard                     │  │
│  │     → Reporter sends results back to API                             │  │
│  └──────────────────────────────────────────────────────────────────────┘  │
│                                                                            │
└────────────────────────────────────────────────────────────────────────────┘

GitHub Actions Example

jobs:
  e2e-tests:
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: npm ci
      - name: Run E2E Tests
        env:
          CONNECT_API_KEY: ${{ secrets.CONNECT_API_KEY }}
        run: |
          npx e2e-insights run-shard \
            --shard-index=${{ matrix.shard }} \
            --shard-total=4 \
            --repo=${{ github.repository }} \
            --project=e2e-electron

CLI Options

| Option | Required | Description | | --------------- | -------- | -------------------------------------------------------- | | --shard-index | Yes | Shard index (1-based) | | --shard-total | Yes | Total number of shards | | --repo | Yes | Repository identifier (e.g., posit-dev/positron) | | --browser | No | Override browser (auto-inferred from --project) | | --fallback | No | native (default) or fail if API unavailable |

Additional arguments are passed through to Playwright:

npx e2e-insights run-shard \
  --shard-index=1 --shard-total=4 --repo=org/repo \
  --workers=4 --retries=2 --project=e2e-electron

Auto-Detection

OS: Detected from runner platform (mac, win, ubuntu, linux).

Browser:

• CLI: Parses the --project flag for keywords (e.g., e2e-electron → electron). Cannot auto-detect since Playwright hasn't started yet. If no match, GITHUB_JOB is used for shard naming.
• Reporter: Auto-detects at runtime from Playwright's project.use.browserName config, or infers from project name, or defaults to electron.

Shard Naming & Unique Run IDs

When running sharded tests, each shard needs a unique identifier so results don't overwrite each other in S3. The CLI handles this automatically by setting a SHARD_NAME environment variable before running Playwright.

Format: {browser}-{os}-{shardIndex}

Example: A 4-shard matrix on Ubuntu with --project=e2e-electron produces:

shard 1 → SHARD_NAME=electron-ubuntu-1 → runId: 12345-1-electron-ubuntu-1
shard 2 → SHARD_NAME=electron-ubuntu-2 → runId: 12345-1-electron-ubuntu-2
shard 3 → SHARD_NAME=electron-ubuntu-3 → runId: 12345-1-electron-ubuntu-3
shard 4 → SHARD_NAME=electron-ubuntu-4 → runId: 12345-1-electron-ubuntu-4

Each shard writes to its own S3 path, preventing collisions.

Fallback Behavior

If the API is unavailable or has no historical data, the CLI falls back to Playwright's native sharding:

npx playwright test --shard=1/4

Shard Coordination

When multiple shards run in parallel, they may query the API at slightly different times. To ensure all shards get consistent test assignments (no duplicates, no gaps), the API uses the GitHub workflow run ID to coordinate:

Shard 1 queries at 11:59  ──┐
Shard 2 queries at 12:00  ──┼──→  Same workflow_run_id  ──→  Same historical data  ──→  Same assignments
Shard 3 queries at 12:01  ──┘

How it works:

1. Each shard passes its workflow_run_id (from GITHUB_RUN_ID) to the API
2. The API uses this ID to derive a deterministic cutoff time for the historical data query
3. Since all shards have the same workflow_run_id, they all see the same historical data
4. Same data → same LPT algorithm result → consistent, non-overlapping assignments

Why this matters:

Without coordination, shards querying at different times might see slightly different historical data (e.g., a new test run completed between queries). This could cause the LPT algorithm to make different decisions for each shard, resulting in some tests being assigned to multiple shards or skipped entirely.

Comparison with Playwright Native

| Aspect | Playwright Native | E2E Insights | | ------------------- | ------------------- | ------------------------ | | Algorithm | Round-robin by test | LPT by spec file | | Balance | Even test count | Even duration | | Adaptation | Static | Uses historical data | | Platform awareness | No | Yes (OS/browser filter) |

Publishing

npm test
npm run build
npm version patch  # or minor/major
npm publish --access public
git push && git push --tags