@checkly/playwright-reporter

Technical Preview - This reporter is currently in technical preview. Features and APIs may change.

Official Playwright reporter for Checkly. Automatically upload test results, screenshots, videos, and traces to gain visibility into your end-to-end tests.

Fully compatible with Playwright's native JSON reporter - use it as a drop-in replacement that adds Checkly integration.

Installation

npm install --save-dev @checkly/playwright-reporter

Requirements:

• Node.js >= 18.0.0
• Playwright >= 1.40.0
• A Checkly account

Quick Start

1. Get Your Credentials

1. Go to Account Settings > General to find your Account ID
2. Go to User Settings > API Keys to create an API key

2. Configure Playwright

Add the reporter to your playwright.config.ts:

import { defineConfig } from '@playwright/test';
import { createChecklyReporter } from '@checkly/playwright-reporter';

export default defineConfig({
  reporter: [
    ['list'],
    createChecklyReporter(),
  ],
});

Tip: Using createChecklyReporter() provides full intellisense for configuration options. Alternatively, use the array syntax: ['@checkly/playwright-reporter', { ... }]

3. Set Environment Variables

export CHECKLY_API_KEY=your_api_key
export CHECKLY_ACCOUNT_ID=your_account_id

Or pass them inline:

CHECKLY_API_KEY=... CHECKLY_ACCOUNT_ID=... npx playwright test

4. Run Tests

npx playwright test

You'll see a session URL in the output:

View test results: https://app.checklyhq.com/test-sessions/abc123

[...test execution...]

✓ Check the results: https://app.checklyhq.com/test-sessions/abc123

Configuration Options

import { createChecklyReporter } from '@checkly/playwright-reporter';

createChecklyReporter({
  outputDir: 'test-results',
  sessionName: 'My Test Suite',
  dryRun: false,
  verbose: false,
})

| Option | Type | Default | Description | |--------|------|---------|-------------| | outputDir | string | Playwright's outputDir | Directory for assets, JSON, and ZIP | | sessionName | string \| function | Auto-generated | Custom session name | | dryRun | boolean | false | Create ZIP without uploading | | verbose | boolean | false | Enable debug logging |

Output files (written to outputDir):

• checkly-report.json - JSON test report
• checkly-report.zip - ZIP archive with report and assets

Deprecated options (will be removed in next major version): | Option | Migration | |--------|-----------| | outputFile | JSON now at {outputDir}/checkly-report.json | | testResultsDir | Use outputDir | | outputPath | ZIP now at {outputDir}/checkly-report.zip |

Environment Variables

| Variable | Description | |----------|-------------| | CHECKLY_API_KEY | Your Checkly API key | | CHECKLY_ACCOUNT_ID | Your Checkly account ID | | CHECKLY_REPORTER_VERBOSE | Set to true for detailed debug output |

What Gets Uploaded

• Test results and status (passed/failed/flaky)
• Execution duration and timing
• Screenshots (on failure or explicit capture)
• Video recordings
• Playwright traces
• Console logs and network requests (extracted from traces)

Flaky Test Detection

The reporter automatically detects flaky tests:

• Flaky test: A test that failed initially but passed after retry
• Degraded status: Set when there are flaky tests but no complete failures
• Overall status: passed if all tests eventually pass, failed otherwise

Running Locally (No Upload)

For local development without uploading to Checkly:

# Simply don't set CHECKLY_API_KEY or CHECKLY_ACCOUNT_ID
npx playwright test

What happens:

• Reporter creates checkly-report.zip locally
• Upload is automatically skipped
• Tests run normally
• You can inspect the ZIP file for debugging

Dry Run Mode

Create local JSON and ZIP files without uploading:

createChecklyReporter({
  dryRun: true,
})

Conditional dry run (skip uploads when no credentials):

createChecklyReporter({
  dryRun: !process.env.CHECKLY_API_KEY,
})

CI/CD Integration

GitHub Actions

name: Playwright Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright browsers
        run: npx playwright install --with-deps

      - name: Run Playwright tests
        env:
          CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }}
          CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }}
        run: npx playwright test

GitLab CI

test:
  image: mcr.microsoft.com/playwright:v1.57.0-jammy
  stage: test
  script:
    - npm ci
    - npx playwright test
  variables:
    CHECKLY_API_KEY: $CHECKLY_API_KEY
    CHECKLY_ACCOUNT_ID: $CHECKLY_ACCOUNT_ID

Setting up variables:

1. Go to your GitLab project
2. Navigate to Settings > CI/CD > Variables
3. Add CHECKLY_API_KEY (check "Mask variable")
4. Add CHECKLY_ACCOUNT_ID (check "Mask variable")

Multiple Reporters

Combine with other Playwright reporters:

import { createChecklyReporter } from '@checkly/playwright-reporter';

export default defineConfig({
  reporter: [
    createChecklyReporter(),
    ['html', { outputFolder: 'playwright-report' }],
    ['list'],
    ['junit', { outputFile: 'test-results/junit.xml' }],
  ],
});

Understanding Test Sessions

The reporter creates suite-level test sessions, not individual test file results.

When you run npx playwright test:

1. One test session is created, named after your directory
2. One test result for the entire run
3. All assets uploaded together in a single ZIP file

Benefits:

• Consolidated view of your entire test suite
• Efficient storage
• Track overall pass/fail rates over time

Security

Always use environment variables for credentials. Never commit API keys to version control.

// DON'T DO THIS
createChecklyReporter({
  apiKey: 'chk_api_...',  // Hardcoded credentials!
})

// DO THIS
createChecklyReporter()  // Reads from environment

Documentation

For detailed documentation, visit checklyhq.com/docs/detect/testing/playwright-reporter.

License

Apache 2.0