diffhawk

Measure whether the AI code reviewer running on your repo is actually working.

npx diffhawk score kubeedge/kubeedge

  Gemini on kubeedge/kubeedge  · 2026-04-28 → 2026-07-27

  65% of 26 decided comments led to a code change  (17 acted-on, 16 pending)
  census global for Gemini: 44%  — this repo is above average  ·  sharp — earning its place

  by severity
    critical     ████████░░░░░░░░   50%  2/4
    high         █████████████░░░   80%  4/5
    unknown      ██████████░░░░░░   65%  11/17

Why compare at all

Across 1,112 public repositories, AI reviewer comments led to a code change 43% of the time on average. That average is the problem, not the finding: per repository it ranges from 14% to 70%, and about 1 in 5 repos sit at 25% or below — some near zero. Their teams cannot tell, because the average is the only number they have seen.

The study, its method and its raw data.

Usage

npx diffhawk score <owner>/<repo> [options]

  --reviewer <name>   Score one reviewer (default: the most active)
  --prs <n>           Pull requests to inspect (default 120)
  --days <n>          Reporting window in days (default 90)
  --json              Emit the raw scorecard as JSON

Auth uses GITHUB_TOKEN, falling back to gh auth token. Read-only: it reads public pull requests and writes nothing.

How far to trust the number

"Acted on" means the code a comment points at changed afterwards, from GitHub's own thread state. That is a proxy for influence, not proof of correctness, and it is wrong in two known directions: a rebase touching the same lines counts when it should not, and a fix elsewhere in the file does not count when it should. Both biases are printed on every scorecard.

An attempt to validate that signal against commit history failed — the validation signal proved noisier than the thing it was validating. That is written up rather than hidden: EVAL.md.

Comments on still-open pull requests are excluded as undecided rather than counted as failures, and no verdict is issued below 10 decided comments.

Also available

• GitHub Action — a weekly scorecard comment on your repo
• Self-hosted — NestJS API + BullMQ workers + Postgres for fleet scale

Both in the main repository.

Apache-2.0.