🧬 TestMutant

AI Test Integrity Gate — catch tautological AI-generated tests before they ship.

The Problem

AI coding assistants generate tests that look comprehensive but catch nothing.

• CodeRabbit research: AI-authored code introduces 1.7x more bugs
• Daniel Vaughan: "100% line coverage but 4% mutation score = 96% of bugs missed"
• DEV.to postmortem: AI tests pass but miss the bug

Everyone measures coverage. Nobody measures whether AI tests actually catch bugs.

TestMutant fills that gap.

What It Does

TestMutant uses AST analysis to detect AI test anti-patterns that produce false confidence:

| Anti-Pattern | What It Catches | |---|---| | Assert-Nothing | expect(result).toBeDefined() without checking values | | Generic Naming | "should work correctly" — vague names hiding vague tests | | Repetitive Structure | Copy-paste test blocks with identical patterns | | Tautological Assertions | Tests that pass regardless of implementation | | Mock-Everything | >80% of dependencies mocked — testing mocks, not code | | No Edge Cases | Only happy-path tests, no error/boundary coverage |

Each file gets an Integrity Score (0-100) based on detected anti-patterns.

Quick Start

npm install -g @phoenixaihub/test-mutant

# Scan your test directory
test-mutant scan ./src

# Full integrity report
test-mutant report ./src --format json

# CI mode (exits non-zero if below threshold)
test-mutant ci ./src --threshold 70

CLI Commands

test-mutant scan [dir]       Scan for AI test anti-patterns
test-mutant report [dir]     Full integrity report
test-mutant ci [dir]         CI mode with exit codes
test-mutant init             Generate .testmutantrc.json config

Options

| Flag | Description | Default | |------|-------------|---------| | -t, --threshold <n> | Minimum integrity score to pass | 60 | | -f, --format <type> | Output: text, json, sarif, junit | text | | --json | Shorthand for JSON output | — |

Configuration

Run test-mutant init to generate .testmutantrc.json:

{
  "include": ["**/*.test.{ts,tsx,js,jsx}", "**/*.spec.{ts,tsx,js,jsx}"],
  "exclude": ["node_modules/**", "dist/**"],
  "threshold": 60,
  "reporters": ["text"]
}

CI Integration

GitHub Actions

- name: Test Integrity Check
  run: npx @phoenixaihub/test-mutant ci ./src --threshold 70

Pre-commit Hook

{
  "husky": {
    "hooks": {
      "pre-commit": "test-mutant ci ./src"
    }
  }
}

Output Formats

Text (default)

🧬 TestMutant — AI Test Integrity Report
══════════════════════════════════════════

✅ src/math.test.ts
   Integrity: 100/100 | Anti-patterns: 0

❌ src/api.test.ts 🤖 AI-generated (85%)
   Integrity: 54/100 | Anti-patterns: 4
   🟡 [generic-naming] L5: Generic test name: "should work correctly"
   🟠 [assert-nothing] L6: Weak assertion: expect(...).toBeDefined()
   💡 Use expect(...).toEqual(specificValue)

──────────────────────────────────────────
Files scanned: 2
AI-generated:  1
Avg integrity: 77/100
Result:        ❌ FAIL

SARIF

Integrates with GitHub Code Scanning, VS Code SARIF Viewer, and other SARIF-compatible tools.

JUnit XML

Drop into any CI system that reads JUnit reports.

Programmatic API

import { scan, report, DEFAULT_CONFIG } from '@phoenixaihub/test-mutant';

const result = await scan('./src', { ...DEFAULT_CONFIG, threshold: 70 });
console.log(report(result, { ...DEFAULT_CONFIG, reporters: ['json'] }));

// Access individual file results
for (const file of result.files) {
  if (file.isAiGenerated) {
    console.log(`${file.file}: AI confidence ${file.aiConfidence}`);
  }
}

How Scoring Works

Integrity Score = Anti-Pattern Score (0-100)

Each anti-pattern deducts points based on severity:

| Severity | Deduction | |----------|-----------| | Critical | -25 | | High | -15 | | Medium | -8 | | Low | -3 |

A file with zero anti-patterns scores 100. A file with 3 high-severity issues scores 55.

Contributing

See CONTRIBUTING.md for guidelines.

License

MIT — see LICENSE.