flakytests

Detect flaky tests, quarantine repeated offenders, and recover stable tests without ever disabling them. One npm package, one command, no external database.

Status: [email protected] is prepared for publication. The unscoped package must be published before the registry installation below is available. Requires Node.js 22+, Git, and a writable remote.

Install

In the repository whose tests you want to manage:

npm install --save-dev flakytests

The package includes the CLI, policy engine, Playwright adapter, Vitest reporter and adapter, Git state store, and GitHub Actions helper. No additional @flakytests/* packages are needed. Your repository supplies its own Playwright or Vitest installation; those runners are not mandatory runtime dependencies.

Before publication, build this source checkout with npm ci && npm run build, run npm pack, then install the single resulting tarball in your target repository with npm install --save-dev /path/to/flakytests-0.1.0.tgz.

Configure

Create flaky.yml at your target repository root:

quarantine:
  flakyIncidents: 2
  window: 10
recovery:
  consecutivePasses: 10
state:
  provider: git
  branch: flaky-state
adapters:
  playwright:
    results: test-results/playwright.json

Configure Playwright's built-in JSON reporter with retries. For Vitest 3.2, use the included retry-aware reporter rather than standard Vitest JSON:

import { defineConfig } from "vitest/config";

export default defineConfig({
  test: {
    retry: 2,
    reporters: [
      "default",
      ["flakytests/vitest", { outputFile: "test-results/vitest.json" }],
    ],
  },
});

For Vitest, replace the adapter entry with vitest: { results: test-results/vitest.json }. Enable both only when you produce both reports.

npx --no-install flakytests validate
# Run your tests, then:
npx --no-install flakytests evaluate
npx --no-install flakytests status

validate checks configuration without requiring reports or a Git connection. evaluate creates flaky-state on the remote when it first records results, without switching branches or modifying your working tree/index. Use it as the final CI gate after allowing the raw test command to fail.

Behavior

| Result | Not quarantined | Quarantined | | ---------------------------- | ------------------------------ | ------------------------------ | | Passed first attempt | Pass CI; advance clean streak | Pass CI; advance clean streak | | Failed, then passed on retry | Pass CI; record flaky incident | Pass CI; record flaky incident | | Failed all attempts | Fail CI; reset clean streak | Pass CI; reset clean streak |

Two flaky incidents in a test's last ten recorded executions trigger quarantine. Ten consecutive clean passes clear both quarantine and the known-flaky flag. Thresholds are configurable. Skipped tests do not count as clean passes, and hard failures alone never trigger automatic quarantine.

Marking is not quarantine: marking aids triage but hard failures still block CI. Quarantining makes only that test's failures non-blocking. Tests always continue to run.

npx --no-install flakytests mark 'vitest:src/cache.test.ts::Cache::expires' --actor alice --reason 'Investigating'
npx --no-install flakytests unmark 'vitest:src/cache.test.ts::Cache::expires' --actor alice
npx --no-install flakytests quarantine 'vitest:src/cache.test.ts::Cache::expires' --actor alice --reason 'Tracked in issue 42'
npx --no-install flakytests unquarantine 'vitest:src/cache.test.ts::Cache::expires' --actor alice --reason 'Fixed'

Manual actions retain actor, timestamp, and optional reason. Git commits preserve previous versions. Missing/corrupt reports and persistence failures are errors, never passing gates.

One package, modular internals

| Import | Responsibility | | -------------------------------- | ------------------------------------------------------------ | | flakytests | Core contracts, policy, manual transitions, gate and helpers | | flakytests/cli | Programmatic CLI and adapter/store factory types | | flakytests/vitest | Default export: Vitest reporter | | flakytests/adapters/playwright | Playwright report adapter | | flakytests/adapters/vitest | Vitest report adapter | | flakytests/stores/git | Git state provider | | flakytests/github-actions | GitHub outputs/summary helper |

These are subpath exports of one package, not separate npm dependencies. The core has no framework, CI-provider, or storage-provider imports.

Documentation

• Configuration and CLI
• Architecture and policy
• Git state setup and concurrency
• GitHub Actions
• Adapters and reporter setup
• Developing adapters and state providers
• Publishing and release setup
• Migrating from scoped packages
• Playwright example
• Vitest example

Development

npm ci
npm run check
npm run release:check
node bin/flakytests.js --help

check runs strict TypeScript, ESLint, Prettier, unit tests, actual framework runner compatibility, CLI/Actions flows, and concurrent local Git writes. Core coverage thresholds are 95% statements/lines, 90% branches, and 100% functions. The included runner-only tests need no browser downloads.

release:check packs one tarball and installs it outside the repository, verifying exports, the reporter, the executable, and validate. It never publishes. Use the checked-in bin/flakytests.js after building when working from source; consumers use the installed flakytests executable.

V1 deliberately excludes a dashboard, hosted service, GitHub App, and external database. Each evaluation is a new observation; do not evaluate the same report twice. State is shared across source branches and ordered by ingestion.

Licensed under MIT.