🤖 AI Visual Testing Assistant

🎯 The smartest AI-powered visual testing CLI that automatically analyzes UI failures and provides actionable fixes for web applications.

✅ Production Ready - Comprehensive framework support with intelligent automation

Transform your visual testing workflow with AI-powered analysis. Get specific, implementable solutions for UI/UX issues through intelligent detection, smart recommendations, and multi-provider AI analysis.

🚨 Current Implementation Status

✅ What Actually Works

• Core Analysis Engine - Full ai-test run functionality with comprehensive orchestration
• Playwright Integration - Complete test execution and screenshot capture
• Gemini Provider - Production-ready with real API integration
• Template System - Sophisticated prompt engineering with customization
• TypeScript Architecture - Robust SOLID principles implementation
• CLI Framework - Complete command structure with proper error handling

🚧 In Development (Partial/Mock)

• Setup & Init Commands - Basic stubs, not fully functional
• Configuration Management - No persistence, hardcoded values
• Multiple AI Providers - Claude, OpenAI, Azure return mock responses only
• Framework Detection - Returns hardcoded Playwright results

❌ Not Yet Implemented

• Ollama (Local LLM) - Placeholder only, non-functional
• Interactive Setup Wizard - Planned but not implemented
• Cypress/Selenium/Puppeteer - Only Playwright works
• Custom Reporter - Planned feature

Bottom Line: This is a sophisticated Playwright + Gemini visual testing tool with extensive architecture for future expansion. Core analysis functionality is production-ready.

✨ Key Features

🎯 Multi-LLM Support

• Google Gemini ✅ - Fully implemented with real API integration
• Claude 3.5 Sonnet 🚧 - Basic implementation (mock responses)
• OpenAI GPT-4o 🚧 - Basic implementation (mock responses)
• Azure OpenAI 🚧 - Basic implementation (mock responses)
• Local Ollama ❌ - Placeholder only (not functional)

🔍 LLM-Powered Analysis

• Visual Regression Detection - LLM analysis of layout and styling issues via screenshots
• Accessibility Guidance - AI-generated WCAG 2.1 AA compliance suggestions
• Responsive Design Review - LLM analysis of cross-device compatibility
• UI/UX Best Practices - AI-powered design standards evaluation
• Performance Indicators - LLM assessment of loading states and user experience

Note: Analysis is performed through sophisticated prompt engineering with LLMs, not computer vision algorithms.

🛠️ Framework Integration

• Playwright ✅ - Fully implemented with advanced screenshot and trace support
• Cypress 🚧 - Planned (not yet implemented)
• Selenium 🚧 - Planned (not yet implemented)
• Puppeteer 🚧 - Planned (not yet implemented)

📊 Actionable Outputs

• Specific CSS Fixes - Exact selectors and property changes
• JavaScript Solutions - Framework-specific code improvements
• Test Enhancements - Better selectors and strategies
• Implementation Guides - Step-by-step fix instructions

🚀 Quick Start

Installation

# Install globally for CLI usage
npm install -g @raxtzu/ai-test

# Or install locally in your project
npm install --save-dev @raxtzu/ai-test

Setup (Basic)

ai-test setup

⚠️ Currently a basic implementation that shows:

• Manual API key configuration instructions
• Basic provider information
• Next steps for getting started

Note: Interactive wizard is planned but not yet implemented.

Initialize Your Project

# Basic project initialization (stub implementation)
ai-test init my-project

# Options available but not fully functional yet
ai-test init --type=vue --provider=claude

⚠️ Currently a stub implementation that shows success message only.

Note: Actual project initialization logic is planned but not yet implemented.

Run Analysis ✅

# Analyze failed test screenshots (fully implemented)
ai-test run

# Continuous improvement mode (works)
ai-test run --continuous --max-iterations=5

# Target specific test patterns (works)
ai-test run --pattern="responsive" --output-format=html

# Additional options available
ai-test run --dry-run --provider=gemini --debug

✅ This is the core implemented feature with full functionality.

📋 Real-World Use Cases

1. Professional Trading Platform Quality Assurance

// Playwright test that fails due to visual issues
test('trading dashboard professional appearance', async ({ page }) => {
  await page.goto('/dashboard');
  
  // Test will fail if layout is broken
  await expect(page.locator('.price-chart')).toBeVisible();
  await expect(page.locator('.trading-controls')).toBeVisible();
  
  // Screenshot automatically captured on failure
});

AI Analysis Output:

🔍 Visual Issues Found:
• Price display uses #666 text on #f8f9fa background (contrast ratio 2.1:1)
• Trading buttons lack visual hierarchy for quick decision-making
• Chart legend overlaps with price data on mobile viewports

💡 Specific Fixes:
.price-display {
  color: #1a1a1a !important;  /* WCAG AA compliant */
  font-weight: 600;
  font-size: 1.25rem;
}

.trading-button.buy {
  background: linear-gradient(135deg, #02c076, #059669);
  box-shadow: 0 2px 4px rgba(2, 192, 118, 0.3);
}

📱 Mobile Fix:
@media (max-width: 768px) {
  .chart-legend {
    position: absolute;
    top: 10px;
    right: 10px;
    background: rgba(255, 255, 255, 0.95);
  }
}

2. E-commerce Conversion Optimization

test('product page conversion elements', async ({ page }) => {
  await page.goto('/product/premium-plan');
  
  // Test pricing visibility and CTA prominence
  await expect(page.locator('[data-test="price"]')).toBeVisible();
  await expect(page.locator('[data-test="cta-button"]')).toBeVisible();
});

AI Analysis Output:

🎯 Conversion Issues Detected:
• CTA button "Start Free Trial" has insufficient visual weight
• Price display lacks urgency indicators
• Trust signals positioned below the fold

🚀 Conversion Optimizations:
.cta-button {
  background: #ff6b35;           /* High-contrast orange */
  font-size: 1.1rem;
  padding: 1rem 2rem;
  box-shadow: 0 4px 12px rgba(255, 107, 53, 0.4);
  animation: pulse 2s infinite;  /* Subtle attention-grabber */
}

.price-container::after {
  content: "Limited Time Offer";
  background: #ff4444;
  color: white;
  padding: 0.25rem 0.5rem;
  border-radius: 4px;
  font-size: 0.75rem;
}

🎛️ Configuration Options

Provider Selection

# Basic configuration (limited implementation)
ai-test config --list
ai-test config --get provider

# Setup command (basic implementation)
ai-test setup

Analysis Customization

{
  "analysisFeatures": [
    "visual-regression",    // Layout and styling issues
    "accessibility",        // WCAG compliance
    "responsive",          // Mobile/tablet compatibility
    "ux-practices",        // Professional design standards
    "performance",         // Loading states and UX
    "branding"            // Brand consistency
  ],
  "analysisDepth": "comprehensive",  // quick | balanced | comprehensive
  "maxIterations": 5,
  "outputFormat": "markdown"         // markdown | html | json | all
}

Quality Standards

{
  "qualityStandards": {
    "contrastRatio": 4.5,        // WCAG AA compliance
    "minTouchTarget": 44,        // Mobile accessibility
    "maxLoadTime": 2000,         // Performance expectations
    "brandColors": ["#0066cc", "#ff6b35"]  // Brand palette validation
  }
}

🏗️ Architecture & Design Principles

SOLID Principles Implementation

• Single Responsibility: Each class has one clear purpose
• Open/Closed: Extensible provider system without modifying core
• Liskov Substitution: All providers implement consistent interfaces
• Interface Segregation: Focused interfaces for specific capabilities
• Dependency Inversion: High-level modules depend on abstractions

Modular Structure

src/
├── core/                    # Core interfaces and abstractions
│   ├── interfaces.ts        # Contract definitions
│   └── container.ts         # Dependency injection
├── providers/               # AI provider implementations
│   ├── base-provider.ts     # Abstract base class
│   ├── claude-provider.ts   # Anthropic Claude
│   ├── gemini-provider.ts   # Google Gemini
│   └── provider-factory.ts  # Factory pattern
├── services/                # Business logic services
│   ├── logger.ts           # Logging abstraction
│   ├── rate-limiter.ts     # API rate management
│   └── config-manager.ts   # Configuration handling
├── cli/                     # Command-line interface
│   ├── index.ts            # Main CLI application
│   └── commands/           # Command implementations
└── types/                   # TypeScript definitions
    └── index.ts            # Type definitions

🧪 Integration Examples

GitHub Actions CI/CD

name: AI Visual Quality Assurance
on: [push, pull_request]

jobs:
  visual-testing:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '18'
          
      - name: Install dependencies
        run: npm ci
        
      - name: Install AI Visual Testing
        run: npm install -g @raxtzu/ai-test
        
      - name: Run Playwright tests
        run: npx playwright test
        continue-on-error: true
        
      - name: AI Visual Analysis
        run: ai-test run --max-iterations=3
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          
      - name: Upload Analysis Reports
        uses: actions/upload-artifact@v4
        with:
          name: visual-analysis-reports
          path: .ai-testing/output/reports/

Package.json Scripts

{
  "scripts": {
    "test:visual": "ai-test run --pattern='visual'",
    "test:accessibility": "ai-test run --provider=gemini",
    "test:mobile": "ai-test run --pattern='responsive'",
    "setup:ai": "ai-test setup",
    "qa:full": "npm run test && ai-test run"
  }
}

📊 Cost Optimization

Provider Cost Comparison

| Provider | Implementation Status | Cost (per 1M tokens) | Working Status | |----------|---------------------|---------------------|----------------| | Gemini 2.0 Flash | ✅ Fully Implemented | $1.25 / $5.00 | ✅ Production Ready | | Claude 3.5 Sonnet | 🚧 Mock Implementation | $3.00 / $15.00 | ❌ Returns Mock Data | | GPT-4o | 🚧 Mock Implementation | Variable by image | ❌ Returns Mock Data | | Azure OpenAI | 🚧 Mock Implementation | Enterprise pricing | ❌ Returns Mock Data | | Ollama (Local) | ❌ Placeholder Only | Infrastructure only | ❌ Non-functional |

Smart Cost Management

# Basic configuration commands (stub implementations)
ai-test config --set provider=gemini
ai-test config --get provider

# Core analysis command (fully implemented)
ai-test run --provider=gemini --max-iterations=3

🔧 Advanced Usage

Custom Analysis Templates

// .ai-testing/templates/trading-platform.md
You are analyzing a professional trading platform interface.

Focus on:
- Real-time data visibility and clarity
- Quick action button accessibility
- Professional color schemes (Bloomberg Terminal standards)
- Information density vs. readability balance
- Mobile trading workflow optimization

Provide fixes that meet financial industry UI standards.

Programmatic API

import { AnalysisEngine, ProviderFactory } from '@raxtzu/ai-test';

const engine = new AnalysisEngine();
const provider = ProviderFactory.create('claude', { apiKey: 'your-key' });

const result = await engine.analyze(screenshots, {
  provider,
  features: ['accessibility', 'responsive'],
  depth: 'comprehensive'
});

console.log('Issues found:', result.analysis.visualIssues.length);
console.log('CSS fixes:', result.fixes.css);

Playwright Integration

// playwright.config.js
export default defineConfig({
  use: {
    screenshot: 'only-on-failure',    // AI needs failure screenshots
    video: 'retain-on-failure',      // Additional context
    trace: 'retain-on-failure',      // Detailed execution trace
  },
  
  // Custom reporter for AI integration (not yet implemented)
  reporter: [
    ['html'],
    ['@raxtzu/ai-test/reporter']  // Planned feature
  ]
});

🐛 Troubleshooting

Common Issues

No screenshots found:

ai-test doctor  # Basic health check
ai-test status  # Check service status

API issues:

ai-test status  # Check provider status
# Note: Only Gemini provider is fully functional

Basic analysis commands:

ai-test run --debug  # Debug mode
ai-test run --dry-run  # Preview mode

Permission errors:

ai-test doctor  # Basic diagnostics

📚 Learning Resources

Example Projects

• Trading Platform Demo - Professional financial interface
• E-commerce Store - Conversion optimization focus
• SaaS Dashboard - Complex data visualization
• Design System - Component library validation

Best Practices

• Screenshot Strategy: Focus on failure points, not success states
• Prompt Engineering: Use specific, actionable language
• Iterative Improvement: Run analysis after each fix cycle
• Cost Management: Use appropriate models for analysis depth needed

🤝 Contributing

We welcome contributions! Please see our Contributing Guide for details.

Development Setup

git clone https://github.com/RaxTzu/ai-test.git
cd ai-test
npm install
npm run dev

Running Tests

npm test              # Unit tests
npm run test:integration  # Integration tests
npm run lint          # Code quality

📄 License

MIT License - see LICENSE file for details.

🙋‍♂️ Support & Community

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: Full API Reference

Made with ❤️ by Raveen Abeywickrama

Empowering developers to build visually perfect web applications with AI assistance.