Quality Intelligence Engine

Deterministic Test Failure Classification & Analysis for Playwright

A rule-based intelligent system that automatically classifies test failures, provides root cause analysis, and delivers actionable insights—without relying on AI black boxes.

🎯 Overview

The Quality Intelligence Engine analyzes Playwright test failures and classifies them into:

• API_BUG: Backend/API failures (authentication errors, data mismatches, server errors)
• UI_BUG: Frontend/UI issues (visibility problems, rendering failures, element issues)

Each classification comes with:

• ✅ Confidence score (0-100%) - How certain we are about the classification
• ✅ Root cause analysis - What went wrong and why
• ✅ Evidence - Signals that led to the classification
• ✅ Diagnosis - Step-by-step breakdown of the failure
• ✅ Suggested actions - Concrete next steps to fix the issue

🚀 Quick Start

Installation

npm install

Run Analysis

# Run Playwright tests
npx playwright test

# Run Quality Intelligence Engine
npm start

The engine will:

1. Read test results from playwright-report/report.json
2. Classify failures using the taxonomy
3. Generate detailed analysis reports
4. Output results to output/ directory

📁 Project Structure

quality-intelligence-engine/
├── src/
│   ├── intelligence/         # Classification algorithms
│   ├── pipeline/             # Data processing
│   ├── utils/                # Utilities
│   ├── types.ts              # Type definitions
│   ├── normalizer.ts         # Input normalization
│   ├── configLoader.ts       # Configuration
│   └── reporter.ts           # Output generation
├── tests/                    # Example test cases
├── config/                   # Configuration files
└── playwright.config.ts      # Playwright configuration

⚙️ Configuration

Edit config/agent.config.json:

{
  "engine": {
    "mode": "standard",
    "confidenceThresholds": {
      "fail": 0.85,
      "passRisk": 0.6
    }
  }
}

🏷️ Taxonomy System

Tests are tagged with their expected failure type:

test('Data | NUMERIC_MISMATCH | inventory count wrong', async ({ page }) => {
  // Test code...
});

📊 Confidence Scoring

Deterministic, explainable confidence calculations:

• Base Confidence: 0.6 - 0.95
• Adjustments: +0.05 - +0.1 based on signals
• Maximum: 0.95 (never 100% certain)

🔧 Development

# Run tests
npx playwright test

# Type check
npx tsc --noEmit

Version: 1.0.0 (Refactored)
Last Updated: 2026-01-29