@sdet-agents/test-health-cli

v0.4.4

Published

9 days ago

Analyze Playwright test suite health - SDET Agent Suite #1

0High
0Medium
0Low

testvagrant

playwright testing test-automation code-analysis sdet quality-assurance

Test Health CLI

SDET Agent Suite - Agent #1

Analyze Playwright test suite health and get actionable recommendations to improve test reliability, reduce flakiness, and optimize test maintenance.

Features

Two-Pillar Architecture

The Test Health CLI analyzes your test suite across two pillars:

📐 Code Quality - General coding best practices (6 analyzers)
🧪 Test Quality - Test automation best practices (8 analyzers)

Code Quality Pillar (6 Analyzers)

All analyzers work without an API key, but AI enhancement improves recommendations when available.

✅ Configuration Analyzer - Playwright config, setup, timeouts (80% pattern, 20% AI)
- Detects missing config files, invalid timeout settings, missing baseURL
- AI provides context-aware configuration recommendations
✅ Error Handling Analyzer - Missing error handling, uncaught exceptions (60% pattern, 40% AI)
- Finds uncaught async operations, generic error messages, missing error assertions
- AI suggests specific error handling patterns for your codebase
✅ Organization Analyzer - File structure, naming, nesting depth (60% pattern, 40% AI)
- Identifies very large files, deep nesting, mixed concerns
- AI provides refactoring plans and organization strategies
✅ Reusability Analyzer - Code duplication, DRY violations (70% pattern, 30% AI)
- Detects duplicate code blocks across files, repeated patterns
- AI suggests extraction strategies and refactoring approaches
✅ Documentation Analyzer - JSDoc quality, comment clarity (40% pattern, 60% AI)
- Checks JSDoc presence, function documentation coverage, commented-out code
- AI evaluates documentation clarity and provides improvement suggestions
✅ Security Analyzer - Hard-coded secrets, vulnerabilities (70% pattern, 30% AI)
- Detects hard-coded API keys, passwords, tokens, and security anti-patterns
- AI identifies potential security vulnerabilities in test code

Test Quality Pillar (8 Analyzers)

Most analyzers work without an API key, but AI enhancement improves recommendations. Test Design and Maintainability analyzers are AI-heavy and provide deeper insights with an API key.

Pattern-Based with AI Enhancement (No API Key Required)

✅ Locator Analyzer - Brittle selectors, robust alternatives (60% pattern, 40% AI)
- Detects CSS class selectors, nth-child, XPath, and recommends data-testid/getByRole
- AI suggests better selector strategies based on your codebase patterns
✅ Sync Analyzer - Hard-coded waits, smart wait strategies (70% pattern, 30% AI)
- Finds waitForTimeout, missing waits after actions, suggests smart wait patterns
- AI recommends optimal wait strategies for specific scenarios
✅ Test Data Analyzer - Shared data conflicts, isolation issues (70% pattern, 30% AI)
- Detects hard-coded test data, shared data conflicts, missing cleanup hooks
- AI suggests test data factory patterns and isolation strategies
✅ Assertion Analyzer - Weak/missing assertions, assertion quality (50% pattern, 50% AI)
- Identifies weak assertions (toBeTruthy), missing assertions, assertion quality
- AI evaluates assertion strength and suggests more specific assertions
✅ Isolation Analyzer - Test dependencies, shared state (80% pattern, 20% AI)
- Detects test dependencies, shared state, order-dependent tests, serial suites
- AI identifies cross-file dependencies and suggests isolation improvements
✅ Performance Analyzer - Slow tests, optimization opportunities (80% pattern, 20% AI)
- Finds redundant waits, sequential operations, inefficient selectors, unnecessary page loads
- AI suggests parallelization opportunities and performance optimizations

AI-Powered (Requires API Key - BYOK) ⭐

✅ Test Design Analyzer - Coverage gaps, missing scenarios (20% pattern, 80% AI)
- AI analyzes test coverage, identifies missing scenarios, redundant tests
- Provides deep insights into test design quality and edge case coverage
✅ Maintainability Analyzer - Test fragility, refactoring needs (30% pattern, 70% AI)
- AI evaluates test fragility, tight coupling, complex logic, missing abstractions
- Suggests refactoring strategies and maintainability improvements

Output Formats

✅ HTML Dashboard (default) - Interactive web-based report with two-pillar breakdown
✅ Markdown Report - Human-readable report
✅ JSON Report - Machine-readable data for CI/CD integration

Roadmap

[ ] Parallel execution issues
[ ] Security vulnerability detection
[ ] Auto-fix mode
[ ] CI/CD integration

Installation

npm install -g @sdet-agents/test-health-cli

Or use without installation:

npx @sdet-agents/test-health-cli analyze

Usage

Basic Analysis (No API Key Required)

The tool works without an API key! All 14 analyzers run automatically, with 12 analyzers providing pattern-based analysis. The 2 AI-heavy analyzers (Test Design and Maintainability) provide basic pattern-based analysis without an API key, but deliver much deeper insights with AI enhancement.

cd your-playwright-project
test-health analyze  # Generates test-health-report.html by default

When run in an interactive terminal, you will be prompted to select which analyzers to run (or "Run all"). In CI or when stdin is not a TTY, all analyzers run by default so the command does not hang.

Or explicitly skip AI features:

test-health analyze --skip-ai

Enable AI Features (BYOK - Bring Your Own Key)

To enable AI-powered analysis and get enhanced recommendations from all analyzers (especially Test Design and Maintainability), provide your API key (Anthropic or OpenAI):

# Using command line option (auto-detects provider from key format)
test-health analyze --api-key sk-ant-xxxxx  # Anthropic
test-health analyze --api-key sk-xxxxx      # OpenAI

# Or set environment variable
export ANTHROPIC_API_KEY=sk-ant-xxxxx
# Or
export OPENAI_API_KEY=sk-xxxxx
test-health analyze

With Options

# Specify test directory
test-health analyze --path ./e2e-tests

# Enable AI features with API key
test-health analyze --api-key sk-ant-xxxxx

# Skip AI features (pattern-based analyzers only)
test-health analyze --skip-ai

# Choose output format (default: html)
test-health analyze                    # Generates HTML report by default
test-health analyze --output md        # Generate Markdown report
test-health analyze --output json      # Generate JSON report

Configuration File

Create .test-health.json in your project root:

{
  "testDir": "./tests",
  "framework": "playwright",
  "ignore": ["**/node_modules/**", "**/*.backup.ts"],
  "severity_thresholds": {
    "critical": 0,
    "high": 5,
    "medium": 20
  }
}

API Key Setup (BYOK - Bring Your Own Key)

The API key is optional. Without it, all 14 analyzers work fine with pattern-based analysis. With it, you get:

Enhanced recommendations from all analyzers (AI provides context-aware suggestions)
Deep AI analysis from Test Design and Maintainability analyzers (coverage gaps, test fragility, refactoring strategies)
Better issue prioritization and more actionable recommendations

Quick Setup (Recommended)

Create a .env file in your project root:

echo "ANTHROPIC_API_KEY=sk-ant-your-key-here" > .env

The CLI automatically loads this file. Remember to add .env to .gitignore!

Other Methods

Command line:

test-health analyze --api-key sk-ant-xxxxx  # Anthropic (auto-detected)
test-health analyze --api-key sk-xxxxx      # OpenAI (auto-detected)

Environment variable:

export ANTHROPIC_API_KEY=sk-ant-xxxxx  # Anthropic
# Or
export OPENAI_API_KEY=sk-xxxxx         # OpenAI
test-health analyze

Priority order: Command line → Environment variable → .env file → Not provided

Provider auto-detection: Keys starting with sk-ant- use Anthropic, others use OpenAI. Use --ai-provider to override.

📖 For detailed setup instructions, see API_KEY_SETUP.md

Get API keys from:

Anthropic: https://console.anthropic.com/
OpenAI: https://platform.openai.com/api-keys

Output

The tool generates:

test-health-report.html (default) - Interactive web-based dashboard with two-pillar breakdown, visual scores, and detailed recommendations
test-health-report.md - Human-readable Markdown report with recommendations
test-health-report.json - Machine-readable data for CI/CD integration
Terminal summary with health score and top issues

Example Report

The HTML report (default) provides an interactive dashboard with:

Two-Pillar Score Breakdown - Separate Code Quality and Test Quality scores
Visual Health Indicators - Color-coded scores and progress bars
Issue Categorization - Issues grouped by analyzer and severity
Actionable Recommendations - Context-aware suggestions with code examples
Quick Wins - High-impact, low-effort improvements

The report includes detailed analysis from all 14 analyzers across both pillars, with AI-enhanced recommendations when an API key is provided.

Development

# Clone repo
git clone https://github.com/your-org/test-health-cli
cd test-health-cli

# Install dependencies
npm install

# Run in development mode
npm run dev

# Build
npm run build

# Test
npm test

Architecture

Two-Pillar Structure

src/
├── cli.ts                    # CLI entry point
├── core/
│   ├── analyzer.ts           # Main analysis orchestrator (two-pillar)
│   ├── scorer.ts             # Two-pillar scorer (weighted scores)
│   ├── ai-coordinator.ts     # AI batching and coordination
│   └── config.ts             # Configuration loader
├── analyzers/
│   ├── base-analyzer.ts      # Abstract base class
│   ├── code-quality/         # Code Quality Pillar (6 analyzers)
│   │   ├── configuration-analyzer.ts
│   │   ├── error-handling-analyzer.ts
│   │   ├── organization-analyzer.ts
│   │   ├── reusability-analyzer.ts
│   │   ├── documentation-analyzer.ts
│   │   └── security-analyzer.ts
│   └── test-quality/         # Test Quality Pillar (8 analyzers)
│       ├── locator-analyzer.ts
│       ├── sync-analyzer.ts
│       ├── test-data-analyzer.ts
│       ├── assertion-analyzer.ts
│       ├── isolation-analyzer.ts
│       ├── test-design-analyzer.ts (AI-heavy)
│       ├── maintainability-analyzer.ts (AI-heavy)
│       └── performance-analyzer.ts
├── ai/
│   ├── context-builder.ts    # Project context for AI
│   └── prompts/              # AI prompt templates
├── reporters/
│   ├── markdown-reporter.ts  # MD output
│   ├── json-reporter.ts      # JSON output
│   └── html-reporter.ts      # HTML dashboard
└── utils/
    ├── ai-client.ts          # AI client abstraction
    └── file-scanner.ts       # File discovery

Score Calculation

Overall Score = Code Quality (35%) + Test Quality (65%)
Each pillar score is a weighted average of its analyzers
Individual analyzer scores range from 0-10

Contributing

This is Agent #1 in the SDET Agent Suite. Future agents will include:

Test Generator Agent
Flake Fixer Agent
Visual Regression Agent
Performance Test Agent

License

MIT

Support

For issues and questions:

GitHub Issues: https://github.com/your-org/test-health-cli/issues
Documentation: https://docs.sdet-agents.dev

Built with ❤️ by the SDET Agent Suite team