test-eyes-reporter

Playwright reporter and CLI for Test Eyes analytics dashboard. Collects test results, tracks flaky tests, and deploys an analytics dashboard to GitHub Pages.

Features

• Playwright Reporter: Automatically collect test results during Playwright runs
• Flaky Test Detection: Tracks tests that fail then pass on retry
• JUnit XML Support: Also works with any test framework that outputs JUnit XML
• GitHub Pages Dashboard: Visualize test analytics over time
• Zero Config CI: Works out of the box with GitHub Actions

Installation

npm install test-eyes-reporter
# or
pnpm add test-eyes-reporter

Usage with Playwright

Add the reporter to your playwright.config.ts:

import { defineConfig } from '@playwright/test'

export default defineConfig({
  retries: 2, // Enable retries to detect flaky tests
  reporter: [
    ['list'],
    ['test-eyes-reporter', {
      dataBranch: 'gh-data',    // Branch to store test data
      githubComment: true       // Post PR comment with test insights
    }]
  ]
})

Then run your tests:

npx playwright test

The reporter will:

1. Collect all test results
2. Track which tests are flaky (failed then passed on retry)
3. Push data to the gh-data branch
4. Deploy the dashboard to gh-pages (if deploy: true)

Usage with JUnit XML (Any Test Framework)

For non-Playwright test frameworks, use the CLI:

# Collect test results
npx test-eyes collect --junit-path ./test-results.xml

# Collect and deploy dashboard
npx test-eyes collect --junit-path ./test-results.xml --deploy

# Deploy dashboard only
npx test-eyes deploy

CLI Commands

test-eyes collect

Collect test results from JUnit XML.

test-eyes collect --junit-path <path> [options]

Options:
  --junit-path <path>   Path to JUnit XML file (required)
  --data-branch <name>  Git branch for data storage (default: gh-data)
  --commit-sha <sha>    Commit SHA (default: current HEAD)
  --pr-number <num>     PR number (default: 0)
  --deploy              Also deploy dashboard after collecting

test-eyes deploy

Deploy the dashboard to GitHub Pages.

test-eyes deploy [options]

Options:
  --data-branch <name>    Source branch for data (default: gh-data)
  --deploy-branch <name>  Target branch for deployment (default: gh-pages)

GitHub Actions Setup

Add to your workflow:

name: Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install dependencies
        run: npm ci

      - name: Run Playwright tests
        run: npx playwright test
        env:
          CI: true

Make sure your repo has GitHub Pages enabled with source set to gh-pages branch.

Configuration Options

| Option | Type | Default | Description | |--------|------|---------|-------------| | dataBranch | string | 'gh-data' | Branch to store test data | | prNumber | number | auto | PR number (auto-detected from env) | | githubComment | boolean | false | Post a comment on the PR with test results | | commentTitle | string | '## 🔍 Test Eyes Report' | Title to identify and replace old comments |

PR Comments

When githubComment: true, the reporter posts a comment on the PR:

## 🔍 Test Eyes Report

### Summary
✅ 42 passed | ❌ 2 failed | 🔄 1 flaky

### What Needs Attention

- ⚠️ New slow test: `checkout payment` — 3.8s lands at P80 (in the slowest 20%)

  🐇 0.5s ░░░░░░░░░░░░░░░░▓▓▓▓ 4.0s 🐢
                              👆

- ❌ Ongoing flaky: `login session` — flaky 3 times before

Insights

| Insight | When shown | |---------|------------| | New slow test | Test not in history lands at P70+ (slowest 30%) | | Ongoing flaky | Test is flaky now AND was flaky before |

Insights require at least 10 tests in history to calculate baselines.

Required Permissions

Add to your GitHub Actions workflow:

permissions:
  contents: write
  pull-requests: write  # Required for PR comments

Limitations

Concurrent CI Runs

If two CI runs execute simultaneously on the same PR, the latest comment wins. To avoid this, use GitHub's concurrency setting:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Data Format

Test data is stored as JSON in the data branch:

{
  "runId": "2024-01-15_abc1234",
  "prNumber": 42,
  "commitSha": "abc1234...",
  "createdAt": "2024-01-15T10:00:00Z",
  "tests": [
    {
      "name": "login > should authenticate user",
      "durationMs": 1500,
      "status": "passed",
      "wasFlaky": false
    }
  ]
}

Dashboard

The dashboard shows:

• Test Overview: All tests with pass/fail counts
• Flaky Tests: Tests that have been flaky
• Slowest Tests: Tests sorted by p95 duration
• Search: Filter tests by name

License

MIT