@flakytests/cli

Detect flaky tests, quarantine repeated flakes, and automatically recover stable tests without disabling them. Supports Playwright and Vitest 3.2, uses Git for state, and needs no external database. Requires Node.js 22+. MIT licensed.

npm install --save-dev @flakytests/cli

Core, both adapters, and the Git provider install automatically. Vitest users should also install @flakytests/adapter-vitest directly and use its @flakytests/adapter-vitest/reporter export instead of ordinary JSON reporting.

Create flaky.yml in your repository:

quarantine:
  flakyIncidents: 2
  window: 10
recovery:
  consecutivePasses: 10
state:
  provider: git
  branch: flaky-state
adapters:
  playwright:
    results: test-results/playwright.json

Configure Playwright's JSON reporter with retries, run your tests, then:

npx flaky evaluate
npx flaky status
npx flaky quarantine 'playwright:example.spec.ts::test title' --actor alice --reason 'Investigating'
npx flaky unquarantine 'playwright:example.spec.ts::test title' --actor alice

mark and unmark manage the known-flaky flag independently of quarantine. All manual actions require an actor and retain an audit record.

The evaluator creates flaky-state on origin automatically. It exits 1 for unquarantined hard failures, 0 for passes/flakes/quarantined failures, and 2 for invalid reports or operational errors. State must persist before CI passes.

In CI, allow the raw test command to fail, then use flaky evaluate as the final required gate. Pass the original exit code using --test-exit-code. Remove stale reports before testing, evaluate each report once, exclude flaky-state from workflow triggers, and restrict writes to automation. Do not expose write credentials to untrusted pull-request code.

See the project documentation for reporter configurations, GitHub Actions examples, and policy details.