repeatably

Run it again before you publish the number.

I built two repositories this month and nearly published a wrong number in both. Not a typo — a number that was true for the run I happened to look at and false on the next one.

$ repeatably -n 3 -f "invalidation window" -- npx tsx bench/invalidation-window.ts

NUMBERS (4 found, 0 identical in every run)
  label                 median            range   spread
! p95 (L5)                  84         82 – 148    78.6%
! max (L6)                  84         82 – 148    78.6%
! min (L7)                  12          10 – 16    50.0%
! p50 (L4)                  37          34 – 50    43.2%

DO NOT PUBLISH (4)
  line   5  p95 — 82 to 148 across 3 runs (78.6%)

RESULT: UNSTABLE -- 4 number(s) moved more than tolerance.
                    Quoting one run would be quoting an accident.

That p95 was in a README as 68 ms. A later verification run printed 139 ms. Same machine, same command, nothing changed. The benchmark was averaging over too few rounds and p95 was the second-slowest of twenty samples, so one scheduling hiccup moved it by seventy milliseconds.

I raised the round count, wrote "at forty it repeats to within a millisecond across runs", and published that. Then I pointed this tool at it:

| at 40 rounds | range across 3 runs | spread |
|--------------|---------------------|--------|
| p50          | 36 – 39 ms          |   7.9% |   reproducible
| p95          | 68 – 94 ms          |    37% |   not

The claim was written from two runs that happened to agree. A third disproved it. p50 was the only number in that benchmark worth quoting, and I had been quoting p95.

repeatably runs your command several times, lines the numbers up, and tells you which ones repeat.

npx repeatably -- npm run bench

What it does

Runs the command n times and reports, per number:

• stable — the range is within tolerance of the median. Safe to quote.
• unstable — it moved more than that. Quoting one run quotes an accident.
• constant — identical every time. Reported as a count, not a list.

Exit codes are 0 stable, 1 unstable, 2 untrusted, 3 usage — so it drops into CI as a gate on a number you have promised not to let drift.

How it aligns numbers with no schema

Mask every number in the output and compare what is left.

p50   34 ms          p50     ms
p95   51 ms    ──▶   p95     ms      same shape, so slot 0 is p50 in both

If the masked text matches, the nth number in one run is the nth number in the other, and no format has to be recognised. If it does not match, the runs printed different things — reported as a stronger finding than any single value moving, because there is no correspondence to report.

Numbers inside identifiers are left alone: sha256, utf8 and node v22 are not measurements, and treating them as such makes a run's shape depend on text that was never going to vary. ADR-0001.

It proves it can fail before it tells you it passed

Every run checks three things before reporting anything:

CONTROL
  ok    a value moved past the tolerance and the classifier saw it
  ok    a value that did not move was left alone
  ok    numbers were extracted from the output

A parser that silently read nothing and a tolerance loose enough to accept anything both produce a clean result — and a clean result is what you were hoping for, so nobody looks closely. Any control failing downgrades the verdict to untrusted no matter what the numbers said.

The first version of this used wall-clock duration as the control, reasoning that it always varies. It does not: a fast command finished within 7.5% of the same time every run, against a 10% tolerance, and a perfectly good measurement was declared untrustworthy. A control that depends on the environment behaving a certain way is a second thing that can be wrong. ADR-0002.

Real output has noise in it

Benchmarks print progress and then a summary, and the progress usually carries a run id or a timestamp that differs every time:

run 1:  round 1  window 45ms  (tyap59d7-1:43ms ...)
run 2:  round 1  window 59ms  (tyfd5kon-1:58ms ...)

Those runs have different shapes and cannot be aligned. --from starts at the summary; --drop removes noisy lines in place; --ignore keeps a number out of the verdict when you know it varies and do not care.

repeatably -n 5 -f "=== results" -i "elapsed|duration" -- npm run bench

Options

-n, --runs <n>        how many times to run it (default 7)
-w, --warmup <n>      discarded runs before measuring (default 1)
-t, --tolerance <p>   stable if the range is within this fraction of the
                      median (default 0.1)
-i, --ignore <regex>  do not report numbers whose label matches
-f, --from <regex>    ignore everything before the first matching line
-d, --drop <regex>    remove matching lines before analysing
-C, --cwd <path>      run the command here
    --json            machine-readable report

What it does not do

It checks reproducibility, not correctness. A benchmark that fails identically every run is perfectly reproducible and this will call it stable. The exit codes are surfaced for exactly that reason — a STABLE on a command that exited 1 three times is almost always someone measuring a broken benchmark.

It does not know which numbers matter. Every number in the output is a candidate, including ports and counts. Most are constant and cost one summary line; the rest are what --ignore is for.

It is not a statistics package. Range over median, one threshold. No confidence intervals, no outlier rejection, no distribution fitting — those need more samples than anyone runs a slow benchmark, and a number that survives three runs of range-checking is already better than the number most projects publish.

Library

import { runRepeatedly, analyse, formatReport } from 'repeatably';

const { outputs, durationsMs, exitCodes } = await runRepeatedly({
  command: ['npm', 'run', 'bench'],
  runs: 5,
});
const result = analyse({ outputs, durationsMs, exitCodes, tolerance: 0.1 });
if (result.verdict !== 'stable') process.exit(1);

Tests

npm install && npm test

29 tests. The important ones spawn the built binary against fixtures that are deliberately stable, deliberately unstable, and deliberately shape-shifting, and assert the exit code. If the unstable fixture ever passes as stable, every STABLE this tool has ever printed is worthless — so that assertion is the one to read first.

Built with Claude

Claude wrote most of this code. The design and the decisions in docs/decisions/ are mine.

It exists because of three specific mistakes, all mine, all in public repositories: a p95 published from twenty rounds that moved 78% on re-run; a comparison table published from a single run where two of four cells flipped when it was run again; and a sentence claiming the fix had made the number reproducible, which the first thing this tool was ever pointed at disproved in about four minutes.

Licence

MIT.