npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2026 – Pkg Stats / Ryan Hefner

@adlc/review-calibration

v1.13.0

Published

Measures reviewer recall via planted bugs — who reviews the reviewer (P5 meta-gate).

Readme

@adlc/review-calibration

ADLC phase: P5 meta-gate — "who reviews the reviewer"

Measures reviewer recall via planted bugs. Turns "we do adversarial review" from a vibe into a number, exposes category-level blind spots, and re-runs on every model change — catching the silent regressions everyone currently absorbs unknowingly.

This is mutation testing aimed at the reviewer instead of the code (ADLC C8).

How it works

  1. Target files: Files changed in --commit (git diff-tree) filtered to non-test, non-config source code. Falls back to --files if the commit touches no eligible files.
  2. Plant selection: mutate.generateMutants runs over each target file's full content. Up to --plants mutants are selected by round-robining across operators for coverage. Each plant carries a real bug category (off-by-one, logic-inversion, null-handling, …) and a defect description. --plants-file plants may also carry a witness.
  3. Equivalent-mutant filter: a plant with a witness must DISCRIMINATE (pass on the original, fail on the mutant). Plants whose witness does not discriminate are equivalent mutants — there is no bug to find, so they are excluded from the denominator rather than scored as missed. Each exclusion is printed to stderr with its reason; a witness that fails on the ORIGINAL (a broken witness rather than an equivalent mutant) includes the tail of that run's output, and one that cannot start says so instead of "timed out". Unparseable mutants and witnesses that never launched are discarded. JavaScript plants are checked with node --check; TypeScript-family plants (.ts, .mts, .cts, .tsx, .jsx) are parsed with the repository's own typescript through @adlc/core, using its 5.x/6.x JavaScript API and skipping a TypeScript 7 native package. A TypeScript plant with no usable typescript records syntax unknown and passes through. Other files also record unknown. An unparseable plant is discarded with the reason mutant does not parse, and a witness that cannot start (a spawn error, or shell exit 126/127) discards its plant instead of counting as a caught mutant. Discarded plants are outside the recall denominator; the scorecard prints a Discarded: line and --json carries plantsDiscarded: {total, unparseable, spawnFailed}. Node's --check accepts a .js file written in ES-module syntax even when it is malformed, so such a plant is not discarded; use .mjs targets for full coverage.
  4. Control self-test (scorer): before scoring, two reference reviewers run through the deterministic referenceJudge — an echoer (must score ~0) and an oracle (must score 1.0). If either is wrong the scorer itself is broken and the tool exits 1. This bounds the scorer's aggregation — that it has no non-semantic shortcut — and nothing else. 4b. Control self-test (judge): the number you actually get is produced by the configured judge, not referenceJudge, so the echo control runs a second time through that judge whenever it rendered at least one verdict. In the default judge mode an unbounded judge is an operational failure (exit 1) — a judge that cannot reject an echoing reviewer cannot certify a recall figure. --scorer string sets the judge to "match everything" by construction, so it can never be bounded: that run still completes, and reports configuredJudgeBounded: false so the uncertified number is visible to a machine consumer rather than only to a stderr warning. When no finding located a plant the judge was never consulted, and both configuredJudgeBounded and configuredJudgeEchoRecall are null — it contributed nothing to the score, so there was nothing to bound.
  5. Apply all plants, run --review-cmd ({base} → commit ref), restore (atomically, via finally + SIGINT/SIGTERM/SIGHUP handlers; a killed run is restored by the next one — see Safety).
  6. Parse findings: the reviewer's output is parsed as structured findings (adversarial-review --json shape, or a weak prose fallback).
  7. Score: a plant is CAUGHT only when a finding LOCATES it (file + line ±3) AND the configured judge (a cheap model) confirms it identifies the defect. A repro field on a finding is parsed but never run: the CLI does not execute commands supplied by the reviewer it is measuring. There is no string-match shortcut: a reviewer that echoes changed lines scores ~0. Recall = caught / valid plants. Precision = true / (true + spurious findings), where a finding that locates a plant without identifying it is spurious.
  8. Gate: Exit 2 if recall < --min-recall (or precision < --min-precision when set); exit 0 otherwise. Exit 1 on operational error or a failed control.

Scorer modes (--scorer)

  • judge (default) — cheap-model semantic match. Requires an LLM provider (ANTHROPIC_API_KEY / OPENAI_API_KEY / GEMINI_API_KEY). With no provider the tool fails closed (exit 1) rather than emit an untrustworthy number. Every locating finding is judged, including one that carries a repro.
  • string — LEGACY location-only matching. Gameable by a reviewer that echoes changed lines; prints a warning and is not a trustworthy recall number. Provided only as an offline escape hatch.

See REDESIGN.md in the repository for the full design and the rationale for each decision.

Safety

  • Refuses to run on a dirty working tree (opError, exit 1). Commit or stash first.
  • Plants are written and restored atomically (temp file + rename), so no source file is ever left truncated.
  • On a normal exit, an error or Ctrl-C, files are restored by the runner's finally block (and the CLI's SIGINT/SIGTERM/SIGHUP handlers).
  • A run killed mid-review (SIGTERM from a CI cancel, docker stop, SIGKILL) cannot restore anything itself — the review runs inside spawnSync, where no signal handler runs. Before planting, the CLI writes an in-flight record of each file's original and planted contents to the git dir (.git/adlc-review-calibration-inflight.json). The next run restores those files before its dirty-tree check and says so on stderr. It only restores a file that still holds exactly the planted content; if the file has changed since, it refuses (exit 1), writes nothing and keeps the record, which then holds the only copy of the original. A record owned by a process that is still running is left alone (exit 1).
  • Exit codes from the review command: 0 and 2 are valid (pass / gate-fail). Any other exit code is treated as a crash (opError, exit 1).

Usage

review-calibration --review-cmd "cmd with {base} placeholder" [options]

Options

| Flag | Default | Description | |---|---|---| | --review-cmd <cmd> | (required) | Command template to run the reviewer (tokenized, not run through a shell — shell: false, no pipes, redirects, or variable expansion). {base} is substituted with the commit ref. On Windows, Node.js refuses to spawn .cmd/.bat shims with shell: false; invoke the underlying script with node <script.mjs> directly. | | --review-timeout <seconds> | 300 | Timeout in seconds for the --review-cmd subprocess. | | --commit <ref> | HEAD | Commit whose changed files are used as plant targets. | | --plants <n> | 8 | Number of bugs to plant. | | --min-recall <f> | 0.5 | Minimum recall fraction required to pass (0–1). | | --min-precision <f> | — | Optional minimum precision fraction to also require (0–1). | | --min-plants <n> | 4 | Minimum valid-plant count required to report a recall measurement (refuses with exit 1 below this floor). | | --scorer <mode> | judge | judge (cheap-model semantic match; requires an LLM API key) or string (legacy location-only offline mode; prints a warning). | | --files <list> | — | Comma-separated fallback file list when commit has no eligible code files. | | --plants-file <path> | — | JSON array of authored plants ([{file,line,original,mutated,category?,defect?,witness?}]). | | --tier <tier> | cheap | Model tier for the LLM judge (cheap, mid, or frontier). | | --review-provider <name> | — | Provider/model family used by --review-cmd (anthropic, openai, gemini); compared against the judge provider for independence. | | --strict | false | Gate-fail (exit 2) instead of warning when --review-provider matches the judge's resolved provider (requires --review-provider). | | --json | false | Machine-readable JSON output. | | --prompt-only | false | Print a representative judge prompt and exit 0 (no API key or git repo needed). | | --help | — | Show help. |

Exit codes

| Code | Meaning | |---|---| | 0 | Gate passes — recall (and precision, if --min-precision is set) meet thresholds | | 1 | Operational error — dirty tree, fewer than --min-plants valid plants, review command crashed (exit ∉ {0,2}), no judge available, or control self-test failed | | 2 | Gate fails — recall < --min-recall, precision < --min-precision, or --strict provider match |

Examples

Basic calibration with adversarial-review:

review-calibration \
  --review-cmd "adversarial-review --base {base} --json" \
  --commit HEAD \
  --plants 8 \
  --min-recall 0.6

JSON output for orchestrators:

review-calibration \
  --review-cmd "my-reviewer {base}" \
  --json

Custom file fallback (when commit has no eligible source files):

review-calibration \
  --review-cmd "my-reviewer {base}" \
  --commit HEAD \
  --files "src/auth.mjs,src/api.mjs"

Testing with a trivial fake reviewer (offline CI smoke test using --scorer string):

review-calibration \
  --review-cmd "node ./fake-reviewer.mjs {base}" \
  --scorer string \
  --min-recall 0 \
  --json

JSON output shape

{
  "recall": 0.625,
  "caught": 5,
  "total": 8,
  "precision": 0.83,
  "truePositives": 5,
  "falsePositives": 1,
  "minRecall": 0.5,
  "minPrecision": null,
  "gatePass": true,
  "scorer": "judge",
  "judgeAgreement": null,
  "equivalentExcluded": 0,
  "plantsDiscarded": { "total": 0, "unparseable": 0, "spawnFailed": 0 },
  "commit": "HEAD",
  "reviewExitCode": 0,
  "perCategory": {
    "off-by-one":      { "caught": 2, "total": 2, "recall": 1.0 },
    "logic-inversion": { "caught": 2, "total": 4, "recall": 0.5 },
    "null-handling":   { "caught": 1, "total": 2, "recall": 0.5 }
  },
  "plants": [
    {
      "file": "src/auth.mjs",
      "line": 42,
      "category": "logic-inversion",
      "operator": "invert-comparison",
      "status": "caught",
      "original": "  if (user.role === 'admin') {",
      "mutated":  "  if (user.role !== 'admin') {"
    }
  ]
}

precision is real: truePositives / (truePositives + falsePositives), where a false positive is a finding that locates no plant (in a clean base + only-our-plants tree, nothing else is broken) or one that locates a plant but identifies no plant's defect — an unsubstantiated claim such as an echo of the changed line. Every locating finding gets its own verdict, so padding a real catch with echoes lowers precision. unsubstantiated reports how many of the false positives were of the second kind. When truePositives + falsePositives === 0 (the reviewer produced no findings), precision is null (not 1.0) because precision cannot be evaluated; configuring --min-precision will fail the gate if precision is null.

Scoring logic

Caught — a plant is caught only when a finding does BOTH:

  1. Locates it — mentions the file's basename and a line within ±3.
  2. Identifies it — a cheap-model judge confirms the finding describes this defect. (scorePlants accepts an injected verifyRepro for library callers; the CLI does not supply one.)

There is no "output contains a substring of the changed line" rule: that is exactly what let a line-echoing reviewer score 1.0. Echoing locates but does not identify, so it scores ~0 — enforced by the built-in echo control on every run.

Per-category recall — the catch rate broken down by real bug category (off-by-one, logic-inversion, null-handling, …), so low recall on a category points at a specific reviewer blind spot.

Relationship to sibling tools

| Tool | Role | |---|---| | hollow-test (C4) | Mutation testing aimed at tests — kills mutants via the test suite | | review-calibration (C8) | Mutation testing aimed at the reviewer — measures reviewer recall | | adversarial-review | The reviewer under test; wire its output to --review-cmd | | gate-manifest (C11) | Records the calibration score alongside each review verdict |

Core gaps

None — all required functionality (mutate.generateMutants, mutate.applyMutant, isDirty, isGitRepo, git, parseArgs, pass, gateFail, opError, printJson) is available in @adlc/core.