📊 QA Reporter Agent

Runs your Playwright tests and writes a plain English summary_report.md that any stakeholder can understand — with failed test explanations, recommendations, and next steps.

Part of the @agents-forge AIQA pipeline:

analyst → qa-planner → qa-engineer → qa-reviewer → qa-reporter

v1.0 uses Playwright's built-in JSON reporter. v2.0 will add Allure report integration for richer analytics.

Install

npm install @agents-forge/qa-reporter
npm install --save-dev @playwright/test
npx playwright install chromium

Quick Start

# Run all tests + generate stakeholder report
npx @agents-forge/qa-reporter

# Smoke tests only
npx @agents-forge/qa-reporter --grep @smoke

Authentication

Auto-detected — no config needed:

| Method | Setup | |---|---| | Anthropic API key | ANTHROPIC_API_KEY=sk-ant-... in .env | | Claude subscription | npm install -g @anthropic-ai/claude-code → claude login | | GitHub Copilot | gh auth login |

Features

• 🎭 Runs Playwright tests — with optional grep filter or spec file
• 📖 Reads JSON results — parses Playwright's built-in JSON reporter
• 📝 Writes summary_report.md — plain English for stakeholders
• ❌ Explains failures — error messages + likely cause + recommended fix
• 📋 Cross-references review_report.md — flags predicted vs actual failures
• 🔐 Multi-provider auth — API key, claude login, or GitHub Copilot
• 🔄 Results-only mode — skip test run, just report on existing results

CLI Usage

# Run all tests
npx @agents-forge/qa-reporter

# Filter by tag
npx @agents-forge/qa-reporter --grep @smoke
npx @agents-forge/qa-reporter --grep @regression

# Custom output
npx @agents-forge/qa-reporter --output sprint_23_report.md

# Single spec file
npx @agents-forge/qa-reporter --spec tests/login.spec.ts

# Read existing results (no test run)
npx @agents-forge/qa-reporter --results-only

# Different directory
npx @agents-forge/qa-reporter --dir ./my-project

Programmatic Usage

import { runQAReporter } from "@agents-forge/qa-reporter";

const result = await runQAReporter({
  grep: "@smoke",                    // optional tag filter
  specFile: "tests/login.spec.ts",   // optional specific file
  outputFile: "smoke_report.md",     // optional output name
  cwd: process.cwd(),                // optional directory
  resultsOnly: false,                // optional skip test run
});

console.log(`Passed: ${result.passed}, Failed: ${result.failed}`);

Output: summary_report.md

# Test Execution Report
Status: ✅ PASSED / ❌ FAILED / ⚠️ PARTIAL

## Results Summary
| ✅ Passed | ❌ Failed | ⏭️ Skipped | 🔄 Flaky | Total | Duration |

## Test Suite Breakdown
| Module | Passed | Failed | Skipped | Duration |

## ❌ Failed Tests
(with error message, likely cause, recommended fix per test)

## 🔄 Flaky Tests
## ⏱️ Performance (slowest tests)
## Browser Matrix Results
## Known Issues Cross-reference
## Recommendations
## Next Steps

Playwright Config Requirements

Your playwright.config.ts must include the JSON reporter:

reporter: [
  ["json", { outputFile: "test-results/results.json" }],
  ["html", { outputFolder: "playwright-report" }],
]

This is automatically configured if you used @agents-forge/qa-engineer.

Full Pipeline

npx @agents-forge/analyst https://my-app.com requirements.md
npx @agents-forge/qa-planner requirements.md test_plan.md
npx @agents-forge/qa-engineer test_plan.md --url https://my-app.com
npx @agents-forge/qa-reviewer
npx @agents-forge/qa-reporter --grep @smoke

Roadmap

| Version | Reporter | Status | |---|---|---| | v1.0 | Playwright built-in JSON | ✅ Current | | v2.0 | Allure Reports | 🔲 Planned |

Troubleshooting

| Error | Fix | |---|---| | Playwright not found | npm install --save-dev @playwright/test | | playwright.config.ts not found | Run @agents-forge/qa-engineer first | | No JSON results file found | Add JSON reporter to playwright.config.ts | | Authentication failed | Set ANTHROPIC_API_KEY or run claude login | | Tests fail on CI | Set BASE_URL env var in your CI config |

License

MIT