FlakeFence 🛡️

Detect and rank flaky tests from JUnit XML reports. Point it at your CI test results and instantly see which tests are unreliable.

Install

npm install -g @aaaaorg/flakefence

Or use with npx:

npx @aaaaorg/flakefence ./test-results/

Usage

# Analyze JUnit XML files in a directory (recursive)
flakefence ./test-results/

# JSON output of top 10 flakiest
flakefence ./test-results/ -f json -t 10

# Markdown report
flakefence ./test-results/ -f markdown

# CI mode — exit code 1 if flaky tests found
flakefence ./test-results/ --ci --threshold 0.5

# Generate quarantine list (test names only)
flakefence ./test-results/ --quarantine

Options

| Flag | Description | Default | |------|-------------|---------| | -f, --format <fmt> | Output format: table, json, markdown | table | | -t, --top <n> | Show top N flakiest tests | 20 | | --threshold <n> | Min flakiness score (0–1) | 0.1 | | --quarantine | Output just test names (for CI skip lists) | off | | --ci | Exit with code 1 if flaky tests found | off | | -h, --help | Show help | |

How It Works

1. Scans a directory for JUnit XML files (the standard CI test report format)
2. Parses all test results across multiple runs
3. Groups tests by name and classname
4. Identifies flaky tests — those that both pass AND fail across runs
5. Ranks by flakiness score (failure rate weighted by frequency)

A test that passes 7/10 times and fails 3/10 times is flaky. A test that always passes or always fails is not.

Flakiness Score

failRate = (failures + errors) / totalRuns
flakiness = 1 - |failRate - 0.5| × 2

Score ranges from 0 (always passes or always fails) to 1 (50/50 pass/fail — maximally flaky).

CI Integration

Add to your pipeline to catch flaky tests before they erode trust:

# GitHub Actions example
- name: Check for flaky tests
  run: npx @aaaaorg/flakefence ./test-results/ --ci --threshold 0.3

Supported Formats

• JUnit XML — the standard format produced by most test runners (Jest, pytest, JUnit, Go, etc.)
• Handles <testsuites> wrappers, nested suites, skipped tests, and error cases

License

MIT © aaaaorg