@dispatch-triage/eval

Calibration and accuracy measurement.

Part of Dispatch Triage — programmable triage for GitHub. Behind dispatch eval; import it directly to measure a decision log yourself.

npm i @dispatch-triage/eval

This is the feature that needs probabilities to mean something. A model claiming 0.8 should be right about 80% of the time; if it is not, every threshold in every config is decorative and the gate ladder is theatre.

What it measures

• Reliability diagram — predictions bucketed by confidence, claimed accuracy against observed. Empty bins stay in the output: a diagram with gaps is honest about where there is no evidence.
• Expected calibration error — the average gap, weighted by bin population. Reported alongside the worst bin, because ECE happily hides one badly-wrong bucket.
• Per question — sorted most over-confident first, so the question to fix is at the top.
• Threshold recommendation — the lowest confidence at which accuracy still clears a target, never below the lowest confidence actually observed, and never on fewer than 20 predictions.

The bias, stated plainly

Dispatch has no ground truth. What it has is maintainers overriding it: removing a label it applied, or adding one it held back. A prediction counts as correct when nobody corrected it — which means silence counts as agreement.

That overstates accuracy on items nobody looked at. The honest reading of any number this package produces is "accuracy among items a maintainer actually reviewed", and the CLI says so under the report whenever there are no corrections yet.

Why a recommendation can be refused

Three ways, all deliberate:

• insufficient_data — fewer than 20 predictions. A threshold derived from six samples is a guess wearing a number.
• never_reaches_target — accuracy never clears the target at any threshold. That usually means the rules are asking the wrong questions, not asking them badly.
• A drift smaller than 0.05 is not reported at all. Nudging a maintainer from 0.85 to 0.84 every week trains them to ignore the report.

CLI reference · Documentation · Repository

MIT