@flakiness-detective/playwright-parser

Production-grade Playwright test failure parser with optional metadata tracking

Parse Playwright test failures with rich error details and optional metadata tracking through annotations. Built from production-tested code and designed for AI-powered flakiness detection systems.

Features

• ✅ Zero Config - Works immediately without any setup
• 🎯 Rich Error Parsing - Extracts matcher, locator, expected/actual values, location, and code snippets
• 🔄 Hybrid Approach - Auto-generates deterministic test IDs when annotations are missing
• 📊 Optional Metadata - Add annotations for enhanced tracking (testCaseId, journeyId, testSuiteName)
• 🏭 Production Tested - Battle-tested parsing logic from real-world Playwright usage
• 📦 TypeScript First - Fully typed with comprehensive type definitions
• 🧪 Well Tested - 79 passing tests covering real Playwright error scenarios

Installation

# npm
npm install @flakiness-detective/playwright-parser

# pnpm
pnpm add @flakiness-detective/playwright-parser

# yarn
yarn add @flakiness-detective/playwright-parser

Quick Start

import { parsePlaywrightTest } from "@flakiness-detective/playwright-parser";
import type { TestCase, TestResult } from "@playwright/test/reporter";

// In your custom reporter or analysis tool
const parsed = parsePlaywrightTest(test, result);

console.log(parsed.error.matcher); // 'toBeVisible'
console.log(parsed.error.locator); // 'button[name="Submit"]'
console.log(parsed.error.expected); // 'visible'
console.log(parsed.error.actual); // 'hidden'
console.log(parsed.metadata.testCaseId); // 'auto-a3f7b9c4' (auto-generated)

Level Up: Add Annotations

For better tracking across test renames and historical analysis:

test("user can login", async ({ page }) => {
  // Add testCaseId for stable tracking
  test.info().annotations.push({
    type: "testCaseId",
    description: "TC-AUTH-001",
  });

  // Optional: group by user journey
  test.info().annotations.push({
    type: "journeyId",
    description: "authentication",
  });

  // Optional: organize into test suites
  test.info().annotations.push({
    type: "testSuiteName",
    description: "Login Tests",
  });

  // Your test code...
});

Why Add Annotations?

Without annotations (still works!):

• ✅ Pattern detection across similar errors
• ✅ Clustering by locator, matcher, etc.
• ✅ Basic flakiness detection
• ⚠️ Test renames break history

With annotations (10x better!):

• ✅ All the above PLUS:
• ✅ Track test history across renames
• ✅ Temporal pattern detection
• ✅ Journey-level insights
• ✅ Long-term stability trends
• ✅ Integration with test management systems

Complete Example

import { parsePlaywrightTest } from "@flakiness-detective/playwright-parser";
import type { TestCase, TestResult, Reporter } from "@playwright/test/reporter";

class FlakinessReporter implements Reporter {
  onTestEnd(test: TestCase, result: TestResult) {
    if (result.status !== "failed" && result.status !== "timedOut") {
      return; // Only process failures
    }

    const parsed = parsePlaywrightTest(test, result, {
      // Optional: enable strict mode (require annotations)
      strict: false,

      // Optional: customize auto-ID strategy
      autoIdStrategy: "hash", // 'hash' | 'uuid' | 'none'

      // Optional: warn users about missing annotations
      warnOnMissingAnnotations: true,

      // Optional: extract custom annotations
      customAnnotationTypes: ["priority", "owner", "tags"],
    });

    // Send to your flakiness detection system
    this.sendToAnalytics({
      testId: parsed.metadata.testCaseId,
      journey: parsed.metadata.journeyId,
      suite: parsed.metadata.testSuiteName,
      file: parsed.test.file,
      title: parsed.test.title,
      project: parsed.test.project,
      error: {
        matcher: parsed.error.matcher,
        locator: parsed.error.locator,
        expected: parsed.error.expected,
        actual: parsed.error.actual,
        message: parsed.error.message,
        location: parsed.error.location,
        snippet: parsed.error.snippet,
      },
      result: {
        status: parsed.result.status,
        duration: parsed.result.duration,
        retry: parsed.result.retry,
      },
      timestamp: new Date(),
    });
  }
}

export default FlakinessReporter;

API Reference

parsePlaywrightTest(test, result, options?)

Parses a Playwright test failure into structured data.

Parameters:

• test: TestCase - The Playwright test case
• result: TestResult - The test result (must be 'failed' or 'timedOut')
• options?: ParserOptions - Optional configuration

Returns: ParsedTestFailure

Throws:

• Error if test status is not 'failed' or 'timedOut'
• Error if no errors array or malformed error object
• Error if strict mode is enabled and required annotations are missing

Example:

const parsed = parsePlaywrightTest(test, result, {
  strict: false, // Don't throw on missing annotations
  warnOnMissingAnnotations: true, // Educate users (default)
  autoIdStrategy: "hash", // Generate deterministic IDs
  customAnnotationTypes: ["priority", "owner"],
});

ParserOptions

Configuration options for the parser.

interface ParserOptions {
  // Strict mode: throw error if required annotations missing
  strict?: boolean; // Default: false

  // Warn when annotations are missing (educates users)
  warnOnMissingAnnotations?: boolean; // Default: true

  // Strategy for auto-generating testCaseId
  autoIdStrategy?: "hash" | "uuid" | "none"; // Default: 'hash'

  // Custom annotation types to extract
  customAnnotationTypes?: string[];
}

ParsedTestFailure

The complete parsed test failure object.

interface ParsedTestFailure {
  // Structured error details
  error: PlaywrightErrorDetails;

  // Test metadata (annotations + auto-generated)
  metadata: TestMetadata;

  // Test information
  test: TestInfo;

  // Execution result
  result: TestResultInfo;
}

PlaywrightErrorDetails

Structured error information extracted from Playwright errors.

interface PlaywrightErrorDetails {
  matcher: string; // e.g., 'toBeVisible', 'toContainText', 'waitFor'
  expected: string; // Expected value in assertion
  actual: string; // Actual value received
  locator: string; // Playwright locator string
  location: {
    file: string; // File path
    line: number; // Line number
    column: number; // Column number
  };
  message: string; // Clean error message (no ANSI codes)
  snippet: string[]; // Code snippet lines
}

TestMetadata

Test metadata from annotations and auto-generation.

interface TestMetadata {
  testCaseId: string; // Unique test identifier
  testCaseIdSource: "annotation" | "auto-generated";
  journeyId: string | null; // User journey identifier
  testSuiteName: string | null; // Test suite name
  customAnnotations: Record<string, string>; // Custom annotations
}

TestInfo

Core test information.

interface TestInfo {
  title: string; // Test title
  file: string; // Test file path
  line: number | null; // Test definition line
  column: number | null; // Test definition column
  project: string; // Playwright project name
}

TestResultInfo

Test execution result information.

interface TestResultInfo {
  status: "failed" | "timedOut";
  duration: number; // Execution time in milliseconds
  retry: number; // Retry attempt number (0 for first run)
}

Utility Functions

parsePlaywrightError(error)

Parse raw Playwright error object into structured details.

import { parsePlaywrightError } from "@flakiness-detective/playwright-parser";

const errorDetails = parsePlaywrightError(result.errors[0]);
console.log(errorDetails.matcher); // 'toBeVisible'

extractTestMetadata(test, options)

Extract metadata from test annotations with auto-ID generation.

import { extractTestMetadata } from "@flakiness-detective/playwright-parser";

const metadata = extractTestMetadata(test, {
  autoIdStrategy: "hash",
  warnOnMissingAnnotations: false,
});

removeAnsiCodes(text)

Remove ANSI escape codes from strings.

import { removeAnsiCodes } from "@flakiness-detective/playwright-parser";

const clean = removeAnsiCodes("\x1B[31mError\x1B[0m: test failed");
// Result: 'Error: test failed'

Annotation Guide

Standard Annotations

testCaseId (Recommended)

Purpose: Unique, stable identifier for a test case.

Why use it: Auto-generated IDs change when test titles change. Explicit IDs enable tracking across refactors.

test("user can login", async ({ page }) => {
  test.info().annotations.push({
    type: "testCaseId",
    description: "TC-AUTH-001", // Your format: TC-*, JIRA-*, etc.
  });
  // ...
});

journeyId (Optional)

Purpose: Group tests by user journey or flow.

Why use it: Analyze flakiness patterns by user journey (auth, checkout, profile, etc.)

test.info().annotations.push({
  type: "journeyId",
  description: "authentication", // or 'checkout', 'onboarding', etc.
});

testSuiteName (Optional)

Purpose: Organize tests into logical suites.

Why use it: Higher-level grouping than journeys (e.g., "Smoke Tests", "Critical Path")

test.info().annotations.push({
  type: "testSuiteName",
  description: "Smoke Tests",
});

Custom Annotations

Add your own annotations for team-specific needs:

// In your test
test.info().annotations.push({ type: "priority", description: "high" });
test.info().annotations.push({ type: "owner", description: "team-auth" });
test.info().annotations.push({ type: "tags", description: "smoke,critical" });

// In your reporter
const parsed = parsePlaywrightTest(test, result, {
  customAnnotationTypes: ["priority", "owner", "tags"],
});

console.log(parsed.metadata.customAnnotations);
// { priority: 'high', owner: 'team-auth', tags: 'smoke,critical' }

Use Cases

1. Flakiness Detection System

Collect structured failure data for AI-powered flakiness analysis:

const parsed = parsePlaywrightTest(test, result);

await sendToFlakinessDetector({
  testId: parsed.metadata.testCaseId,
  journey: parsed.metadata.journeyId,
  errorSignature: {
    matcher: parsed.error.matcher,
    locator: parsed.error.locator,
    message: parsed.error.message,
  },
  timestamp: new Date(),
});

2. Firestore/Database Storage

Store structured failure data for analysis:

await db.collection("test_failures").add({
  testCaseId: parsed.metadata.testCaseId,
  testTitle: parsed.test.title,
  testFile: parsed.test.file,
  errorMessage: parsed.error.message,
  locator: parsed.error.locator,
  matcher: parsed.error.matcher,
  expected: parsed.error.expected,
  actual: parsed.error.actual,
  location: parsed.error.location,
  snippet: parsed.error.snippet,
  status: parsed.result.status,
  duration: parsed.result.duration,
  retry: parsed.result.retry,
  timestamp: new Date(),
});

3. CI/CD Integration

Generate failure reports in CI:

const failures = [];

class CIReporter implements Reporter {
  onTestEnd(test: TestCase, result: TestResult) {
    if (result.status === "failed") {
      const parsed = parsePlaywrightTest(test, result);
      failures.push(parsed);
    }
  }

  onEnd() {
    // Generate report
    console.log(`\n📊 Failure Report (${failures.length} failures)`);

    for (const failure of failures) {
      console.log(`\n❌ ${failure.test.title}`);
      console.log(`   ID: ${failure.metadata.testCaseId}`);
      console.log(`   Matcher: ${failure.error.matcher}`);
      console.log(`   Locator: ${failure.error.locator}`);
      console.log(`   Error: ${failure.error.message}`);
    }
  }
}

Advanced Configuration

Strict Mode

Enforce annotations for production systems:

const parsed = parsePlaywrightTest(test, result, {
  strict: true, // Throws error if testCaseId annotation is missing
  autoIdStrategy: "none", // Disable auto-generation
});

Custom ID Strategy

Choose your ID generation strategy:

// Deterministic hash (recommended for tracking across runs)
parsePlaywrightTest(test, result, { autoIdStrategy: "hash" });

// Random UUID (for one-off analysis)
parsePlaywrightTest(test, result, { autoIdStrategy: "uuid" });

// No auto-generation (annotations required)
parsePlaywrightTest(test, result, { autoIdStrategy: "none" });

Disable Warnings

For production environments where annotations are optional:

parsePlaywrightTest(test, result, {
  warnOnMissingAnnotations: false,
});

Integration with Flakiness Detective

This parser is designed to work seamlessly with the @flakiness-detective/core package:

import { FlakinessDetective } from "@flakiness-detective/core";
import {
  createDataAdapter,
  createEmbeddingProvider,
} from "@flakiness-detective/adapters";
import { parsePlaywrightTest } from "@flakiness-detective/playwright-parser";

// Parse failures in your reporter
class MyReporter implements Reporter {
  async onTestEnd(test: TestCase, result: TestResult) {
    if (result.status === "failed") {
      const parsed = parsePlaywrightTest(test, result);

      // Store in database for flakiness analysis
      await adapter.saveFailure(parsed);
    }
  }
}

// Later: Run flakiness detection
const detective = new FlakinessDetective(adapter, embeddingProvider);
const clusters = await detective.detect();

Related Packages

• @flakiness-detective/core - Core flakiness detection engine
• @flakiness-detective/adapters - Data adapters and AI providers
• @flakiness-detective/cli - Command-line interface
• @flakiness-detective/utils - Shared utilities

Testing

This package has comprehensive test coverage (79 tests):

pnpm test

Contributing

Contributions welcome! See CONTRIBUTING.md for guidelines.

License

MIT © Flakiness Detective

Links

• Documentation
• GitHub
• Issues
• npm