Better Bytes Reporter

A custom Playwright reporter designed for comprehensive test result tracking and reporting. This reporter collects detailed test information, manages test artifacts, and provides flexible output options for integration with CI/CD pipelines and reporting dashboards.

Features

• Comprehensive Test Tracking

  • Collects test case names with full hierarchy
  • Extracts case IDs from test tags (e.g., @TC001)
  • Tracks test results: pass, failed, skipped, interrupted
  • Records retry attempts for each test
  • Captures test execution duration

• Environment Integration

  • Captures CI/CD environment variables
  • Supports multiple deployment environments
  • Integrates with web-based reporting systems

• Test Artifact Management

  • Automatically moves test report ZIP files to web-accessible locations
  • Generates timestamped filenames for unique identification
  • Supports nginx or other web server deployments

• Flexible Output Options

  • Console summary for immediate feedback
  • JSON output for programmatic processing
  • Optional error message omission for webhook size limits
  • Verbose mode for detailed debugging
  • Webhook integration with Basic Authentication

Installation

npm install better-bytes-playwright-reporter

Usage

Configuration in playwright.config.ts

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [
    ['better-bytes-playwright-reporter', { 
      outputFile: 'test-results.json',
      verbose: true 
    }]
  ],
});

Reporter Options

• outputFile (optional): Path to save the JSON report
• verbose (optional): Enable detailed logging during test execution

Test Tagging

To include a case ID in your tests, use tags:

test('@TC001 Login test', async ({ page }) => {
  // Test implementation
});

The reporter will extract TC001 as the case_id.

Environment Variables

The reporter captures the following environment variables:

• JOB_NAME: Name of the CI/CD job
• ENV: Environment (e.g., development, staging, production)
• REPORTER_BASE_URL: Base URL for the reporter service
• BBCLIENT_ID: Client identifier for Better Bytes
• BBREPORT_OMIT_ERROR: If set to any value, error messages will be omitted from test results (useful for webhooks with size limits)

Test Report Moving (requires all three variables):

• BBREPORT_HTML_ENABLE: Set to 'true' to enable test report ZIP file moving
• BBREPORT_WWW_PATH: Path to the web server directory where test report ZIP files will be moved
• BBREPORT_BASE_URL: Base URL where the moved reports will be accessible

Webhook Integration (requires both variables):

• BBREPORT_WEBHOOK_URL: URL to send the test report JSON via POST request
• BBREPORT_CLIENT_ID: Client ID used for Basic Authentication (sent as base64 encoded token)

Example:

JOB_NAME='Nightly Tests' ENV='staging' REPORTER_BASE_URL='http://reporter.example.com' BBCLIENT_ID='client-123' npx playwright test

To omit error messages:

BBREPORT_OMIT_ERROR=1 npx playwright test

To enable test report ZIP file moving:

BBREPORT_HTML_ENABLE=true BBREPORT_WWW_PATH=/var/www/html/reports BBREPORT_BASE_URL=https://reports.example.com npx playwright test

This will move each test's report ZIP file (if available) to the specified directory with a timestamped filename like 2024-01-01_12-30-45_Login_test.zip. The filename is stored in the report_file_name field of each test result. You can construct the full URL by combining bbreport_base_url from metadata with the report_file_name

To enable webhook reporting:

BBREPORT_WEBHOOK_URL=https://api.example.com/test-results BBREPORT_CLIENT_ID=my-client-id npx playwright test

This will send the complete test report JSON to the specified webhook URL using POST with Basic Authentication

Output Format

The reporter generates a JSON output with the following structure:

{
  "metadata": {
    "status": "passed",
    "start_time": "2024-01-01T00:00:00.000Z",
    "duration": 5000,
    "total_tests": 10,
    "passed": 8,
    "failed": 1,
    "skipped": 1,
    "interrupted": 0,
    "job_name": "Nightly Tests",
    "environment": "staging",
    "reporter_base_url": "http://reporter.example.com",
    "client_id": "client-123",
    "bbreport_base_url": "https://reports.example.com"
  },
  "results": [
    {
      "case_name": "Login test",
      "case_id": "TC001",
      "result": "pass",
      "retried_num": 0,
      "duration_ms": 1500,
      "report_file_name": "2024-01-01_12-30-45_Login_test.zip"
    }
  ]
}

Development

Prerequisites

• Node.js 16 or higher
• TypeScript 5.0 or higher
• Playwright Test 1.0 or higher

Building from Source

# Install dependencies
npm install

# Build the project
npm run build

# Clean build artifacts
npm run clean

Running the Example

The example directory contains a complete working example with sample tests:

cd example
npm install

# Run tests with default configuration
npm test

# Run tests without error messages
npm run test:no-errors

# Run tests with HTML report moving enabled
npm run test:with-html

# Open Playwright Test UI
npm run test:ui

Project Structure

better-bytes-reporter/
├── src/
│   ├── reporter.ts    # Main reporter implementation
│   └── index.ts       # Module exports
├── dist/              # Compiled JavaScript (generated)
├── example/           # Working example with sample tests
│   ├── tests/         # Sample test files
│   └── playwright.config.ts
├── package.json
├── tsconfig.json
└── README.md

API Reference

Reporter Options

interface ReporterOptions {
  outputFile?: string;  // Path to save JSON report
  verbose?: boolean;    // Enable detailed logging
}

Test Info Structure

interface TestInfo {
  case_name: string;           // Full test name with hierarchy
  case_id: string | null;      // Extracted from first tag
  result: 'pass' | 'failed' | 'skipped' | 'interrupted';
  retried_num: number;         // Number of retries
  duration_ms: number;         // Test execution time
  error?: string;              // Error message (if failed)
  report_file_name?: string;   // Moved report filename
}

Integration Examples

CI/CD Pipeline Integration

# GitHub Actions example
- name: Run Playwright tests
  env:
    JOB_NAME: ${{ github.job }}
    ENV: staging
    REPORTER_BASE_URL: https://api.example.com
    BBCLIENT_ID: github-actions
    BBREPORT_HTML_ENABLE: true
    BBREPORT_WWW_PATH: /var/www/reports
    BBREPORT_BASE_URL: https://reports.example.com
    BBREPORT_WEBHOOK_URL: ${{ secrets.WEBHOOK_URL }}
    BBREPORT_CLIENT_ID: ${{ secrets.WEBHOOK_CLIENT_ID }}
  run: npx playwright test

Webhook Integration

// Process the JSON output for webhook posting
const report = JSON.parse(fs.readFileSync('test-results.json'));
const webhookPayload = {
  job: report.metadata.job_name,
  environment: report.metadata.environment,
  summary: {
    total: report.metadata.total_tests,
    passed: report.metadata.passed,
    failed: report.metadata.failed
  },
  failures: report.results.filter(r => r.result === 'failed')
};

Troubleshooting

Common Issues

1. Report files not being moved

   • Ensure all three environment variables are set: BBREPORT_HTML_ENABLE, BBREPORT_WWW_PATH, BBREPORT_BASE_URL
   • Check that the destination directory has write permissions
   • Verify that test attachments are being generated (check trace settings)

2. Large error messages in webhooks

   • Set BBREPORT_OMIT_ERROR=1 to exclude error messages from the output

3. Missing case IDs

   • Ensure tests are tagged with @CASEID format as the first tag

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT