QAAtlas

AI-powered test coverage analyzer for Swift and Objective-C iOS projects. Generates test plans, risk assessments, coverage maps, and identifies test gaps.

Features

• Swift & Objective-C Support: Full support for iOS, macOS, watchOS, and tvOS projects
• Static Gap Analysis: Identifies symbols lacking test coverage without running code
• Semantic Test Coverage (v2.0+): 3-tier analysis that tracks actual object instantiations and method calls
• Test Plan Generation: AI-powered test case recommendations based on code changes
• Risk Assessment: Identifies potential risks and provides mitigation strategies
• Coverage Mapping: Maps changed files to recommended tests and identifies coverage gaps
• LLM Response Caching (v2.2+): Cache analysis results to save costs and ensure consistency
• Built-in Feedback (v2.1+): Report issues and request features directly from CLI
• Baseline Comparison: Track coverage improvements over time with delta reporting
• Multiple Output Formats: JSON, Markdown, PR comments, and coverage maps
• CI/CD Integration: Works seamlessly with GitHub Actions and other CI systems
• Flexible Configuration: YAML-based configuration with iOS presets

Supported Languages

| Language | Gap Analysis | Semantic Coverage | Test Detection | |----------|:------------:|:-----------------:|:--------------:| | Swift | ✅ | ✅ (Tier 1-3) | ✅ | | Objective-C | ✅ | ✅ (Tier 1-3) | ✅ | | TypeScript/JavaScript | — | — | ✅ | | Kotlin/Java | — | — | ✅ | | Python | — | — | ✅ | | Go | — | — | ✅ | | Rust | — | — | ✅ |

What's New in v2.2

LLM Response Caching

Cache analysis results to avoid redundant API calls:

qaatlas analyze --diff changes.patch --cache
qaatlas analyze --clear-cache  # Clear all cached results

Built-in Feedback System

Report issues directly from the CLI:

qaatlas feedback --type accuracy --title "Detection issue"
qaatlas feedback --type bug --title "Something broke"
qaatlas feedback --type feature --title "Add feature X"

Semantic Test Coverage Detection (v2.0)

QAAtlas v2.0 introduced a revolutionary 3-tier semantic test coverage system:

| Tier | Detection Method | Confidence | |------|-----------------|------------| | Tier 3 | Dependency graph - tracks transitive coverage | High | | Tier 2 | Body analysis - finds actual instantiations & calls | High | | Tier 1 | Name matching - file/method name patterns | Medium |

Example: If your test does this:

func testUserService() {
    let service = UserService()       // Tier 2: UserService detected
    let user = service.fetchUser()    // Tier 2: fetchUser() detected
    XCTAssertEqual(user.name, "John") // Tier 2: User.name detected
}

QAAtlas will correctly identify UserService, fetchUser, and User as covered - even if the test file isn't named UserServiceTests.swift.

Supports Swift and Objective-C

// Objective-C patterns detected:
[[UserService alloc] init]
[service fetchUserWithId:@"123"]

Mock Filtering

Automatically excludes test doubles:

• MockUserService, UserServiceMock, UserServiceStub, etc.

Installation

npm (Recommended)

npm install -g @synctek/qaatlas

Or use directly with npx:

npx @synctek/qaatlas analyze

Direct Binary Download

Pre-built binaries are available for all major platforms. No Node.js required.

macOS (Universal - Intel & Apple Silicon)

curl -L -o /usr/local/bin/qaatlas \
  https://github.com/SyncTek-LLC/qaatlas/releases/latest/download/qaatlas-macos
chmod +x /usr/local/bin/qaatlas

# Remove quarantine if needed
xattr -d com.apple.quarantine /usr/local/bin/qaatlas 2>/dev/null || true

Linux x64

curl -L -o /usr/local/bin/qaatlas \
  https://github.com/SyncTek-LLC/qaatlas/releases/latest/download/qaatlas-linux-x64
chmod +x /usr/local/bin/qaatlas

Linux ARM64

curl -L -o /usr/local/bin/qaatlas \
  https://github.com/SyncTek-LLC/qaatlas/releases/latest/download/qaatlas-linux-arm64
chmod +x /usr/local/bin/qaatlas

Windows x64

Download qaatlas-win-x64.exe from the latest release and add to your PATH.

Build from Source

git clone https://github.com/SyncTek-LLC/qaatlas.git
cd qaatlas
npm install
npm run build
npm link  # or: node dist/cli/index.js

Verify Installation

qaatlas --version
qaatlas --help

# Health check (validates SEA binary startup)
qaatlas --health

Quick Start

1. Set your API key:

   # For Claude (default)
   export ANTHROPIC_API_KEY=your-api-key
   
   # Or for OpenAI
   export OPENAI_API_KEY=your-api-key

2. Run analysis on your changes:

   # Analyze staged/unstaged git changes
   qaatlas analyze
   
   # Analyze a specific diff file
   qaatlas analyze --diff changes.patch
   
   # Pipe diff from stdin
   git diff main...HEAD | qaatlas analyze
   
   # Use OpenAI instead of Claude
   qaatlas analyze --provider openai --model gpt-4o

3. Check the output in ./qaatlas-output/:

   • report.json - Full structured report
   • testplan.md - Human-readable test plan
   • risk.md - Risk assessment document
   • pr-comment.md - Condensed PR comment
   • coverage-map.json - File-to-test mapping

For iOS Projects

# Run gap analysis on your iOS project
qaatlas gap-analysis --target ./Sources --preset ios

# Check coverage with baseline comparison
qaatlas gap-analysis --target ./Sources --baseline ./previous.json

Gap Analysis (iOS/Swift/Objective-C)

Static test gap analysis for iOS projects. Identifies symbols (classes, structs, enums, protocols, actors, functions) lacking test coverage by comparing production code against test files.

Supported:

• Swift 5.x (classes, structs, enums, protocols, actors, extensions)
• Objective-C (classes, protocols, categories)
• Mixed Swift/Objective-C projects

Basic Usage

# Analyze a Swift project
qaatlas gap-analysis --target ./MyApp

# Use iOS preset (excludes Pods, Carthage, DerivedData, generated code)
qaatlas gap-analysis --target ./MyApp --preset ios

# Analyze with baseline comparison (shows delta)
qaatlas gap-analysis --target ./MyApp --baseline ./previous-report.json

# Filter by priority level
qaatlas gap-analysis --target ./MyApp --min-priority medium

# Show only top 10 gaps
qaatlas gap-analysis --target ./MyApp --top 10

Options

qaatlas gap-analysis [options]

Options:
  -t, --target <path>        Target directory to analyze (required)
  --preset <name>            Use preset configuration (ios)
  -f, --formats <formats>    Output formats: json,markdown (default: "json,markdown")
  -o, --output-dir <path>    Output directory (default: "./qaatlas-output")
  --module <name>            Filter to specific module
  --min-priority <level>     Minimum priority: high,medium,low (default: "low")
  --baseline <path>          Compare against baseline JSON for delta
  --include-extensions       Include type extensions in analysis
  -v, --verbose              Enable verbose logging
  -h, --help                 Display help

Priority Levels

Gaps are automatically prioritized based on code location:

| Priority | Targets | |----------|---------| | High | Domain models, Services, business logic | | Medium | Data layer, Repositories, Networking | | Low | UI/Views, ViewModels, SwiftUI views |

Exit Codes

• 0 - PASS: No high or medium priority gaps
• 2 - FAIL: High priority gaps found
• 3 - WARN: Medium priority gaps (no high)

Use exit codes in CI to enforce coverage policies:

qaatlas gap-analysis --target ./MyApp --min-priority medium || exit 1

Example Output

The Markdown report includes:

# Test Gap Analysis Report

## Summary

| Metric | Value |
|--------|-------|
| Total Symbols | 45 |
| Tested | 32 |
| Untested | 13 |
| Coverage | 71.1% |

## High Priority Gaps

### Domain/Services
- `OrderService` - 40% covered (missing: processRefund, validateOrder)
- `PaymentValidator` - 0% covered

## Recommendations

1. Add tests for OrderService.processRefund (high priority, core business logic)
2. Create PaymentValidatorTests.swift for payment validation coverage

CI Integration

- name: Check Test Coverage Gaps
  run: |
    qaatlas gap-analysis \
      --target ./Sources \
      --preset ios \
      --min-priority medium
  # Fails if high-priority gaps exist

Configuration

Create a qaatlas.yml file in your project root:

version: "1"

provider:
  name: claude
  model: claude-sonnet-4-20250514
  maxTokens: 4096

analysis:
  maxFiles: 20
  maxLinesPerFile: 500
  includePatterns:
    - "**/*.ts"
    - "**/*.tsx"
    - "**/*.js"
  excludePatterns:
    - "**/node_modules/**"
    - "**/*.test.*"

output:
  dir: ./qaatlas-output
  formats:
    - report
    - testplan
    - risk
    - pr-comment
    - coverage-map

context:
  includeImports: true
  includeFunctionSignatures: true
  surroundingLines: 10

# Optional: Customize exit codes for CI
exitCodes:
  success: 0
  error: 1
  criticalRisk: 2           # Critical risk in analyze
  highPriorityGaps: 2       # High priority gaps in gap-analysis
  mediumPriorityGaps: 3     # Medium priority (warning)
  missingCriticalTests: 2   # Missing critical tests
  belowCoverageThreshold: 2 # Coverage below --min-coverage

CLI Options

qaatlas [options] [command]

Global Options:
  -V, --version            Output version number
  --health                 Quick startup health check (exit 0 if healthy)
  -h, --help               Display help

qaatlas analyze [options]

Options:
  -c, --config <path>      Path to config file (qaatlas.yml)
  -d, --diff <path>        Path to diff file
  -o, --output-dir <path>  Output directory for generated files
  -p, --provider <name>    LLM provider (default: claude)
  -m, --model <name>       Model to use for analysis
  -f, --formats <formats>  Comma-separated output formats
  -v, --verbose            Enable verbose logging
  -q, --quiet              Quiet mode - suppress logs, output only JSON
  --dry-run                Parse diff without calling LLM
  --baseline <path>        Compare against previous report.json
  --feedback               Prompt to submit feedback after analysis
  --cache                  Enable caching of LLM results
  --cache-dir <path>       Directory for cache files (default: .qaatlas-cache)
  --no-cache               Force fresh LLM call (bypass cache)
  --clear-cache            Clear all cached results and exit
  -h, --help               Display help

qaatlas gap-analysis [options]

Options:
  -t, --target <path>         Target directory to analyze (required)
  --test-dir <path>           Custom test directory
  --preset <name>             Use preset configuration (ios, swift-package)
  -f, --formats <formats>     Output formats: json,markdown (default)
  -o, --output-dir <path>     Output directory (default: ./qaatlas-output)
  --module <name>             Filter to specific module
  --min-priority <level>      Minimum priority: high, medium, low
  --min-coverage <percent>    Minimum coverage percentage threshold
  --fail-below                Fail if coverage is below --min-coverage
  --top <n>                   Show only top N gaps in summary (default: 20)
  --public-only               Only analyze public/open symbols
  --min-access-level <level>  Minimum: open, public, internal, fileprivate, private
  --baseline <path>           Compare against baseline JSON file
  --feedback                  Prompt to submit feedback after analysis
  -v, --verbose               Enable verbose logging
  -q, --quiet                 Quiet mode - suppress logs, output only JSON
  -h, --help                  Display help

qaatlas feedback [options]

Options:
  -t, --type <type>          Feedback type: accuracy, bug, or feature
  --title <title>            Issue title
  --description <desc>       Issue description
  --dry-run                  Preview without creating issue
  --include-version          Include QAAtlas version in report
  -h, --help                 Display help

Submit Feedback (v2.1)

QAAtlas includes a built-in feedback system to help improve detection accuracy:

# Interactive feedback
qaatlas feedback

# Report an accuracy issue
qaatlas feedback --type accuracy --title "False positive in CarPlay detection"

# Report a bug
qaatlas feedback --type bug --title "Crash on large diffs"

# Request a feature
qaatlas feedback --type feature --title "Add Python support"

# Preview without submitting
qaatlas feedback --dry-run

Integrated feedback after analysis:

# Prompt for feedback after gap-analysis
qaatlas gap-analysis --target ./src --feedback

# Prompt for feedback after analyze
qaatlas analyze --diff changes.patch --feedback

Requirements:

• GitHub CLI (gh) must be installed and authenticated
• Issues are created on the QAAtlas repository

LLM Response Caching (v2.2)

Cache LLM analysis results to avoid redundant API calls:

# Enable caching for repeated analyses
qaatlas analyze --diff changes.patch --cache

# Use custom cache directory
qaatlas analyze --cache --cache-dir ./my-cache

# Force fresh analysis (bypass cache)
qaatlas analyze --cache --no-cache

# Clear all cached results
qaatlas analyze --clear-cache

Benefits:

• Cost savings: Avoid duplicate LLM API calls
• Consistency: Same input = same output (deterministic)
• Speed: Instant results for unchanged diffs

Cache Key Includes:

• Diff content hash
• Model name
• QAAtlas version (auto-invalidates on upgrade)
• Config options (maxFiles, maxLinesPerFile, preprocess)

Cache Behavior:

• Stored in .qaatlas-cache/ (configurable)
• 7-day TTL by default
• JSON format with metadata for debugging

Tracking Coverage Improvements

QAAtlas can track test coverage improvements across PR updates using baseline comparisons.

First Run

qaatlas analyze --diff changes.patch -o ./first-run
# Output: Missing: 5 tests

After Adding Tests

qaatlas analyze --diff changes.patch --baseline ./first-run/report.json
# Output:
# 📈 Coverage: 3 → 7 tests covered (+4)
#    Risk Level: high → medium
# ✅ Test coverage improved since last run!

PR Comment Output

The PR comment includes coverage metrics:

### Test Coverage Status
✅ **PASS** - 5 of 5 test recommendations covered

| Status | Count |
|:-------|------:|
| Already covered | 5 |
| Partially covered | 0 |
| Missing | 0 |

### Baseline Comparison
📈 **Coverage:** 3 → 5 tests covered (+2)
- Risk level: HIGH → MEDIUM

🎉 *Test coverage improved since last run!*

GitHub Actions Integration

Basic Setup

name: QAAtlas Test Plan

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  analyze:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-node@v4
        with:
          node-version: '20'

      - run: npm install -g @synctek/qaatlas

      - run: |
          git diff origin/${{ github.base_ref }}...HEAD > diff.patch
          qaatlas analyze --diff diff.patch
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

      - uses: actions/upload-artifact@v4
        with:
          name: qa-reports
          path: ./qaatlas-output

      - name: Comment PR
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const comment = fs.readFileSync('./qaatlas-output/pr-comment.md', 'utf8');
            github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: comment
            });

With Binary Caching (Faster)

- name: Cache QAAtlas
  id: cache
  uses: actions/cache@v4
  with:
    path: /usr/local/bin/qaatlas
    key: qaatlas-2.2.1-${{ runner.os }}-${{ runner.arch }}

- name: Install QAAtlas
  if: steps.cache.outputs.cache-hit != 'true'
  run: |
    curl -L -o /usr/local/bin/qaatlas \
      https://github.com/SyncTek-LLC/qaatlas/releases/latest/download/qaatlas-linux-x64
    chmod +x /usr/local/bin/qaatlas

- name: Run QAAtlas
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
  run: |
    git diff origin/${{ github.base_ref }}...HEAD > diff.patch
    qaatlas analyze --diff diff.patch

iOS Project Gap Analysis

name: iOS Test Coverage Check

on:
  pull_request:
    paths:
      - '**/*.swift'
      - '**/*.m'
      - '**/*.h'

jobs:
  coverage:
    runs-on: macos-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install QAAtlas
        run: npm install -g @synctek/qaatlas

      - name: Check Test Coverage Gaps
        run: |
          qaatlas gap-analysis \
            --target ./Sources \
            --preset ios \
            --min-priority medium \
            --baseline ./previous-coverage.json || true

      - name: Upload Coverage Report
        uses: actions/upload-artifact@v4
        with:
          name: coverage-report
          path: ./qaatlas-output/

See examples/workflows/qaatlas.yml for a complete workflow example.

Output Formats

report.json

Complete structured report with all test cases, risks, and coverage data. Use this for programmatic access or further processing.

testplan.md

Human-readable test plan with:

• Test cases organized by priority
• Step-by-step test procedures
• Assertion checklists
• Quick checklist summary

risk.md

Risk assessment document with:

• Risk summary table
• Detailed risk descriptions
• Mitigation strategies
• Risk statistics

pr-comment.md

Condensed GitHub PR comment with:

• Risk level badge
• Important risks (critical/high)
• Priority test cases
• Collapsible details for full information

coverage-map.json

File-to-test mapping showing:

• Which tests cover each file
• Coverage status (full/partial/none/unknown)
• Uncovered areas

Exit Codes

• 0 - Success
• 1 - Error (config, parsing, API)
• 2 - Critical risk level detected

Environment Variables

• ANTHROPIC_API_KEY - Your Anthropic API key (for Claude provider)
• OPENAI_API_KEY - Your OpenAI API key (for OpenAI provider)
• NODE_ENV - Set to production for JSON logging

Note: Only one API key is required depending on which provider you use.

Troubleshooting

macOS Gatekeeper

If you see "cannot be opened because the developer cannot be verified":

xattr -d com.apple.quarantine /usr/local/bin/qaatlas

Or right-click the binary in Finder and select "Open".

Permission Denied

chmod +x /usr/local/bin/qaatlas

CI Cache Contains Bad Binary

Bust the cache by changing the cache key:

key: qaatlas-1.0.0-${{ runner.os }}-${{ runner.arch }}-v2

Binary Hangs on Startup

If the SEA binary hangs during startup (stuck on Unix domain sockets or IPC initialization):

1. Use health check to diagnose:

   qaatlas --health
   # Should output "QAAtlas health check: OK" within 5 seconds

2. Startup timeout protection: The binary automatically exits after 5 seconds if initialization hangs. You'll see:

   QAAtlas: Startup timeout after 5 seconds - exiting

3. Fallback to npm: If the binary consistently hangs, use npm instead:

   npx @synctek/qaatlas analyze
   # or
   npm install -g @synctek/qaatlas && qaatlas analyze

4. CI health check: Add a health check step before running analysis:

   - name: Verify QAAtlas
     run: qaatlas --health
     timeout-minutes: 1

Contributing

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Run tests (npm test)
4. Commit your changes (git commit -m 'Add amazing feature')
5. Push to the branch (git push origin feature/amazing-feature)
6. Open a Pull Request

Release Process

Releases are automated via GitHub Actions. To create a new release:

1. Update version: npm version patch|minor|major
2. Push the tag: git push --tags
3. The release workflow will:
   • Run tests
   • Build binaries for all platforms
   • Create a GitHub release
   • Publish to npm

Or trigger manually via workflow_dispatch.

License

MIT