@lytics/playwright-reporter

Adapter-based Playwright reporter with pluggable storage backends

Overview

CoreReporter is a flexible Playwright reporter that uses the adapter pattern to write test results to multiple storage backends simultaneously. It validates required annotations, tracks unique tests, handles retries, and generates comprehensive test run statistics.

Features

• ✅ Adapter Pattern - Write to multiple backends (filesystem, database, cloud storage, etc.)
• ✅ Annotation Validation - Enforces required testCaseId and journeyId annotations
• ✅ Unique Test Tracking - Handles retries and tracks latest test state
• ✅ Flaky Test Detection - Identifies tests that pass after retries
• ✅ Structured Errors - Parses and structures error information
• ✅ Environment Enrichment - Add custom environment data to test runs
• ✅ Parallel Adapter Execution - All adapters run concurrently

Installation

npm install @lytics/playwright-reporter @lytics/playwright-annotations
# or
pnpm add @lytics/playwright-reporter @lytics/playwright-annotations
# or
yarn add @lytics/playwright-reporter @lytics/playwright-annotations

Quick Start

// playwright.config.ts
import { defineConfig } from '@playwright/test';
import { CoreReporter } from '@lytics/playwright-reporter';
import { FilesystemAdapter } from '@lytics/playwright-adapters/filesystem';

export default defineConfig({
  reporter: [
    ['list'], // Built-in reporter
    [
      new CoreReporter({
        adapters: [
          new FilesystemAdapter({ outputDir: './test-results' })
        ]
      })
    ]
  ],
  // ... rest of config
});

Usage

Basic Configuration

import { CoreReporter } from '@lytics/playwright-reporter';
import { FilesystemAdapter } from '@lytics/playwright-adapters/filesystem';

const reporter = new CoreReporter({
  adapters: [
    new FilesystemAdapter({ outputDir: './test-results' })
  ]
});

export default reporter;

Multiple Adapters

import { CoreReporter } from '@lytics/playwright-reporter';
import { FilesystemAdapter } from '@lytics/playwright-adapters/filesystem';
import { SlackAdapter } from '@lytics/playwright-adapters/slack';

const reporter = new CoreReporter({
  adapters: [
    new FilesystemAdapter({ outputDir: './test-results' }),
    new SlackAdapter({ 
      webhookUrl: process.env.SLACK_WEBHOOK_URL,
      notifyOnFailure: true 
    })
  ]
});

export default reporter;

Environment Enrichment

Add custom environment data to test runs:

const reporter = new CoreReporter({
  adapters: [/* ... */],
  environmentEnricher: () => ({
    branch: process.env.GITHUB_REF,
    commit: process.env.GITHUB_SHA,
    author: process.env.GITHUB_ACTOR,
    buildNumber: process.env.GITHUB_RUN_NUMBER,
    nodeVersion: process.version,
  })
});

Core Types

CoreTestResult

Individual test result with full metadata:

interface CoreTestResult {
  testCaseId: string;
  journeyId: string;
  title: string;
  annotations: TestAnnotations;
  status: "passed" | "failed" | "timedOut" | "skipped" | "cancelled" | "interrupted" | "unknown";
  projectName: string;
  durationMs: number;
  error?: TestError;
  timestamp: Date;
  buildId: string;
  reportLink?: string;
}

CoreTestRun

Test run summary with aggregate statistics:

interface CoreTestRun {
  runId: string;
  timestamp: Date;
  overallStatus: "passed" | "failed" | "timedOut" | "skipped" | "cancelled" | "interrupted" | "unknown";
  totalTests: number;           // Unique tests (deduped)
  totalExecutions: number;      // Total test executions (including retries)
  passed: number;
  failed: number;
  skipped: number;
  durationMs: number;
  passRate: number;             // 0-1
  averageTestDuration: number;
  slowestTestDuration: number;
  flakyTests: number;           // Tests that passed after retries
  environment: Record<string, unknown>;
}

TestError

Structured error information:

interface TestError {
  matcher: string;              // e.g., "toHaveText", "toBeVisible"
  expected: string;
  actual: string;
  locator: string;
  location: {
    file: string;
    line: number;
    column: number;
  };
  message: string;
  snippet: string[];            // Code snippet around error
}

Creating Custom Adapters

Implement the ResultAdapter interface:

import { ResultAdapter, CoreTestResult, CoreTestRun } from '@lytics/playwright-reporter';

class MyCustomAdapter implements ResultAdapter {
  async initialize(): Promise<void> {
    // Connect to your storage backend
    console.log('Initializing adapter...');
  }

  async writeTestResult(result: CoreTestResult): Promise<void> {
    // Write individual test result
    console.log(`Test ${result.testCaseId}: ${result.status}`);
  }

  async writeTestRun(run: CoreTestRun): Promise<void> {
    // Write test run summary
    console.log(`Run ${run.runId}: ${run.passed}/${run.totalTests} passed`);
  }

  async close(): Promise<void> {
    // Cleanup connections
    console.log('Closing adapter...');
  }
}

// Use your custom adapter
const reporter = new CoreReporter({
  adapters: [new MyCustomAdapter()]
});

Annotation Requirements

The reporter requires tests to have proper annotations. Use @lytics/playwright-annotations:

import { pushSuiteAnnotation, pushTestAnnotations } from '@lytics/playwright-annotations';

test.describe('My Feature', () => {
  test.beforeEach(async ({}, testInfo) => {
    pushSuiteAnnotation(testInfo, 'MY_FEATURE');
  });

  test('validates user can perform action', async ({}, testInfo) => {
    pushTestAnnotations(testInfo, {
      journeyId: 'MY_FEATURE_ACTION',
      testCaseId: 'MY_FEATURE_ACTION_VALID',
    });

    // Test implementation...
  });
});

Tests missing required annotations will be skipped with a warning.

API Reference

CoreReporter

Constructor:

new CoreReporter(config: CoreReporterConfig)

Config:

interface CoreReporterConfig {
  adapters: ResultAdapter[];
  environmentEnricher?: () => Record<string, unknown>;
}

Methods:

• onBegin() - Called once before test suite starts
• onTestBegin(test) - Called before each test
• onTestEnd(test, result) - Called after each test
• onEnd(result) - Called once after test suite completes

parseError

Parse raw Playwright errors into structured format:

import { parseError } from '@lytics/playwright-reporter';

const error = parseError(result.errors[0]);
if (error) {
  console.log(`Expected: ${error.expected}`);
  console.log(`Actual: ${error.actual}`);
  console.log(`Locator: ${error.locator}`);
}

How It Works

Test Execution Flow

1. onBegin()
   └─> Initialize all adapters in parallel

2. For each test:
   onTestBegin(test)
   └─> Track test start
   
   onTestEnd(test, result)
   └─> Validate annotations
   └─> Track unique test state (handle retries)
   └─> Map to CoreTestResult
   └─> Write to all adapters in parallel

3. onEnd(result)
   └─> Calculate aggregate statistics
   └─> Map to CoreTestRun
   └─> Write to all adapters in parallel
   └─> Close all adapters

Retry Handling

The reporter intelligently handles test retries:

• Tracks unique tests by testCaseId
• Only the latest execution of each test is counted in final stats
• Detects flaky tests (passed after retry)
• Reports both totalTests (unique) and totalExecutions (including retries)

Example:

Test A: fail → retry → pass  ✅ (flaky)
Test B: fail → retry → fail  ❌
Test C: pass                  ✅

totalTests: 3
totalExecutions: 5
passed: 2
failed: 1
flakyTests: 1

Environment Variables

The reporter uses these environment variables (can be customized via environmentEnricher):

• GITHUB_RUN_ID - Used for runId and buildId
• ARTIFACT_BASE_URL - Used for reportLink

Adapters

See these packages for ready-to-use adapters:

• @lytics/playwright-adapters - Filesystem, Slack, Firestore

Troubleshooting

Tests are skipped with "missing annotation" warning

Solution: Ensure all tests have required annotations:

pushTestAnnotations(testInfo, {
  journeyId: 'YOUR_JOURNEY_ID',
  testCaseId: 'YOUR_TEST_CASE_ID',
});

Adapter initialization fails

Solution: Check adapter configuration and credentials. Adapters log errors to console.

Stats don't match expected values

Cause: Retries are handled automatically. The reporter tracks:

• totalTests = unique tests (deduped by testCaseId)
• totalExecutions = all test runs (including retries)

Contributing

See CONTRIBUTING.md in the repository root.

License

MIT

Related Packages

• @lytics/playwright-annotations - Annotation framework
• @lytics/playwright-adapters - Storage adapters (Filesystem, Slack, Firestore)