🛰️ flakeradar

Find, rank, and diagnose flaky tests in any language — straight from JUnit XML.

Run your suite N times (or point it at your CI history) and flakeradar tells you which tests are flaky, how often they fail, and why.

Why

Flaky tests are the most corrosive thing in a CI pipeline: they erode trust in the suite, waste hours on re-runs, and bury real bugs in the noise. Most tools just tell you a test sometimes fails. flakeradar goes further — it ranks flakes by how disruptive they are and guesses the root cause, turning "ugh, it failed again" into "here's the flaky test, here's how often, here's why."

• 🌐 Language-agnostic — anything that emits JUnit XML: pytest, Jest, Vitest, Go, Gradle/Maven, RSpec, PHPUnit, Mocha…
• 🧠 Root-cause hints — timing/async, resource, order-dependency, concurrency, external-dependency, or randomness
• 📊 Ranked, not just listed — worst offenders first
• 🐛 Separates flaky from broken — always-failing tests are real bugs, reported apart from the noise
• 📎 Shareable Markdown report — paste straight into a GitHub issue or PR
• 🚦 CI-ready — --fail-on-flaky gates merges; JSON output for dashboards
• 📦 Zero runtime dependencies — one small package, nothing transitive

Contents

• Quick start
• Two ways to use it
• Framework recipes
• Use it in CI
• How it works
• Output & exit codes
• Options
• Use as a library
• FAQ

Quick start

No install needed:

# Analyze existing CI reports (each XML file = one run):
npx flakeradar analyze "test-reports/**/*.xml"

# Or run a suite repeatedly and watch for flakiness:
npx flakeradar run -n 20 -- pytest --junitxml={out}

Or install it:

npm install -g flakeradar

Try it right now on the bundled example reports:

npx flakeradar analyze "examples/*.xml"

flakeradar — flakiness report

Runs analyzed: 5   Green runs: 0%   Crashed: 0
Unique tests: 5   Flaky: 3   Always-failing: 1

Flaky tests (ranked, worst first):
#  Test                                      Fails  Rate  Flips  Likely cause      Conf
1  checkout.CheckoutTest::test_places_order    2/5   40%      4  timing/async       95%
2  db.UserRepoTest::test_create_user           2/5   40%      3  order/state leak   95%
3  api.PaymentTest::test_charge_card           1/5   20%      2  resource/network   95%

Top offender: checkout.CheckoutTest::test_places_order
  likely: timing/async (95% confident)
  → Replace fixed sleeps with polling/awaits and raise timeouts.
  sample failures:
    • TimeoutError: timed out after 5000ms waiting for order confirmation

Always-failing (broken every run — likely a real bug, not flakiness):
  ✗ math.TaxTest::test_tax_rate  (5/5 failed)

Two ways to use it

1. run — reproduce flakiness locally

Run a test command many times back-to-back. Put the {out} token where your runner writes its JUnit XML; flakeradar swaps in a fresh path each run.

flakeradar run -n 25 -- pytest --junitxml={out}

If your runner writes to a fixed path (or many files), use --report instead:

flakeradar run -n 15 -r "build/test-results/test/*.xml" -- ./gradlew test

Stop as soon as a flake shows up:

flakeradar run -n 50 --stop-on-first-flaky -- go test ./... 2>&1

2. analyze — mine your CI history

Already have per-run JUnit reports archived from CI? Point flakeradar at them. Each file is treated as one run by default.

flakeradar analyze "ci-artifacts/**/junit.xml"

# One run per directory instead of per file:
flakeradar analyze --group-by dir "ci-artifacts/*/"

Framework recipes

| Framework | Command | |-----------|---------| | pytest | flakeradar run -n 20 -- pytest --junitxml={out} | | Jest | flakeradar run -n 20 -r junit.xml -- npx jest --reporters=default --reporters=jest-junit | | Vitest | flakeradar run -n 20 -r junit.xml -- npx vitest run --reporter=junit --outputFile=junit.xml | | Go | flakeradar run -n 20 -- sh -c "gotestsum --junitfile={out} ./..." | | Gradle | flakeradar run -n 15 -r "build/test-results/test/*.xml" -- ./gradlew test | | Maven | flakeradar run -n 15 -r "target/surefire-reports/*.xml" -- mvn -q test | | RSpec | flakeradar run -n 20 -- bundle exec rspec --format RspecJunitFormatter --out {out} | | PHPUnit | flakeradar run -n 20 -- phpunit --log-junit {out} |

Tip: for runners that only write to a fixed file, pass that path to --report (-r). For runners that accept an output path, use the {out} token.

Use it in CI

Detect flakiness that slips past a single run, and comment a report on the PR.

# .github/workflows/flaky.yml
name: flaky-test-check
on: [pull_request]

jobs:
  flaky:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }

      # ... set up your language + install deps ...

      - name: Hunt for flaky tests
        run: npx flakeradar run -n 15 --markdown -- pytest --junitxml={out} > flaky.md

      - name: Comment report on the PR
        if: always()
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          path: flaky.md

Or gate on your existing archived reports:

flakeradar analyze --fail-on-flaky "artifacts/**/*.xml"

How it works

Flakiness detection. A test is flaky when it both passes and fails across the runs analyzed. A test that fails every run isn't flaky — it's broken, and flakeradar reports those separately so you don't confuse a real bug with noise.

Each flaky test gets:

• Fails / Rate — how many runs failed, and the failure rate.
• Flips — how many times it switched between pass and fail across the run sequence. High flips = highly nondeterministic.
• Score — failures + flips, used to rank the worst offenders first.

Root-cause classification. flakeradar scans each test's failure output and matches it against tuned signatures to guess a category:

| Category | Typical signals | Suggested fix | |----------|-----------------|---------------| | timing/async | TimeoutError, deadline exceeded, waited for | Poll/await instead of fixed sleeps; raise timeouts | | concurrency/race | data race, deadlock, ConcurrentModification | Synchronize shared state; remove shared mutation | | resource/network | EADDRINUSE, ECONNREFUSED, too many open files | Isolate ports/temp files per test | | order/state leak | duplicate key, already exists, unique constraint | Reset fixtures/DB between tests; randomize order | | external dep | 503, 429 rate limit, bad gateway | Mock the dependency or add tolerant retries | | randomness/time | random, uuid, Date.now, timezone | Seed RNG; freeze the clock in tests |

The classifier is a heuristic — a strong hint about where to look, not a verdict.

Output formats

| Flag | Output | |------|--------| | (default) | Colored terminal report | | --markdown | GitHub-flavored Markdown (tables + collapsible failure samples) | | --json | Stable flakeradar/v1 JSON for dashboards and scripts |

Exit codes

| Code | Meaning | |------|---------| | 0 | Success (no flaky tests, or --fail-on-flaky not set) | | 1 | Usage error, or no reports/test data found | | 2 | Flaky or always-failing tests detected and --fail-on-flaky was set |

Options

flakeradar run [options] -- <test command>

| Option | Description | |--------|-------------| | -n, --runs <N> | Number of times to run the suite (default 10) | | -r, --report <glob> | Where the command writes its JUnit XML each run | | -o, --output <dir> | Directory for {out} report files (default .flakeradar/runs) | | --stop-on-first-flaky | Stop as soon as any flaky test is detected | | --keep-going | Keep running even if a run crashes (default: stop) | | --fail-on-flaky | Exit 2 if any flaky/always-failing test is found | | --json / --markdown | Choose output format | | --no-color | Disable ANSI colors |

flakeradar analyze [options] <glob...>

| Option | Description | |--------|-------------| | --group-by <file\|dir> | Treat each file (default) or each directory as one run | | --fail-on-flaky | Exit 2 if any flaky/always-failing test is found | | --json / --markdown | Choose output format | | --no-color | Disable ANSI colors |

Use as a library

The analysis engine is exported too, for custom integrations:

import { parseJUnitXml, analyze, runFromResults } from "flakeradar";

const runs = reportStrings.map((xml, i) => runFromResults(`run ${i + 1}`, parseJUnitXml(xml)));
const report = analyze(runs);

console.log(report.summary.flakyCount, "flaky tests");
for (const t of report.flaky) {
  console.log(t.id, t.failureRate, t.cause?.category);
}

FAQ

How many runs do I need? At least 2 to detect any flakiness. For confidence, 10–30 runs is a good range — the rarer the flake, the more runs you need to catch it.

Does it need my CI or a service? No. It's a local CLI. Nothing is uploaded.

My runner exits non-zero on failure — is that a problem? No. flakeradar reads the JUnit report regardless of exit code. A run only counts as crashed if it produces no report at all.

Contributing

Issues and PRs welcome.

npm install
npm test          # run the test suite (vitest)
npm run build     # compile TypeScript to dist/
node bin/flakeradar.js analyze "examples/*.xml"

License

MIT © shravnn