@kas0235/calibration

Measure and fix the calibration of classifier probabilities: log-loss, Brier, ECE, and reliability bins, plus Platt scaling and isotonic (PAV) recalibration. Pure TypeScript, zero runtime dependencies, tree-shakeable.

A model is calibrated when its stated probabilities match observed frequencies: of all the events it called 70% likely, about 70% should actually happen. This package scores how far a set of (probability, outcome) pairs departs from that ideal, and corrects it with two classic, deterministic methods.

Install

npm install @kas0235/calibration

Works in ESM and CommonJS, ships its own type declarations, and pulls in no runtime dependencies.

Usage

1. Measure: ECE and a reliability diagram from probabilities and labels

import {
  computeMetrics,
  expectedCalibrationError,
  reliabilityBins,
} from "@kas0235/calibration";

const predictions = [
  { p: 0.92, y: 1 },
  { p: 0.81, y: 1 },
  { p: 0.65, y: 0 },
  { p: 0.55, y: 1 },
  { p: 0.30, y: 0 },
  { p: 0.12, y: 0 },
] as const;

// Headline bundle: log-loss, Brier, ECE, accuracy, and n in one pass.
const metrics = computeMetrics(predictions);
console.log(metrics);
// { logLoss, brier, ece, accuracy, n: 6 }

// Just the Expected Calibration Error, over 10 equal-width bins by default.
console.log(expectedCalibrationError(predictions, 10));

// The bins behind a reliability diagram: predicted vs observed per bucket.
for (const bin of reliabilityBins(predictions, 5)) {
  console.log(
    `[${bin.lo.toFixed(2)}, ${bin.hi.toFixed(2)}) ` +
      `n=${bin.count} predicted=${bin.meanPredicted.toFixed(3)} ` +
      `observed=${bin.observedFrequency.toFixed(3)}`,
  );
}

2. Fix: fit isotonic regression on a calibration set, then apply it

import {
  isotonicRegressionPAV,
  applyCalibration,
  logLoss,
} from "@kas0235/calibration";

// Held-out calibration set: raw model probabilities paired with true outcomes.
const calibrationSet = [
  { p: 0.10, y: 0 },
  { p: 0.20, y: 0 },
  { p: 0.35, y: 1 },
  { p: 0.40, y: 0 },
  { p: 0.60, y: 1 },
  { p: 0.75, y: 1 },
  { p: 0.90, y: 1 },
];

// Fit a monotone step map from raw probability to calibrated probability.
const model = isotonicRegressionPAV(calibrationSet);

// Apply it to new raw probabilities from the same model.
const rawScores = [0.18, 0.42, 0.83];
const calibrated = applyCalibration(model, rawScores);
console.log(calibrated);

// Confirm the in-sample fit did not make log-loss worse.
const before = logLoss(calibrationSet);
const after = logLoss(
  calibrationSet.map((d, i) => ({
    p: applyCalibration(model, [d.p])[0],
    y: d.y,
  })),
);
console.log({ before, after });

Prefer Platt scaling when you want a smooth two-parameter sigmoid fit; use recalibrate(data, "platt" | "isotonic") to fit, apply, and get before/after metric bundles in one call.

API

| Export | Signature | Description | | --- | --- | --- | | logLoss | (data: readonly LabeledPrediction[], eps?: number) => number | Mean negative log-likelihood (cross-entropy). Clamps confident misses to a finite penalty. | | brierScore | (data: readonly LabeledPrediction[]) => number | Mean squared error of probability vs outcome, a proper scoring rule in [0, 1]. | | accuracy | (data: readonly LabeledPrediction[], threshold?: number) => number | Hard-classification accuracy at a threshold (default 0.5). | | reliabilityBins | (data: readonly LabeledPrediction[], nBins?: number) => ReliabilityBin[] | Equal-width bins with per-bucket predicted vs observed frequency; empty bins kept. | | expectedCalibrationError | (data: readonly LabeledPrediction[], nBins?: number) => number | Count-weighted mean gap between observed and predicted across the bins (ECE). | | computeMetrics | (data: readonly LabeledPrediction[], nBins?: number) => CalibrationMetrics | The { logLoss, brier, ece, accuracy, n } bundle in one pass. | | plattScale | (data: readonly LabeledPrediction[], opts?: PlattOptions) => PlattParams | Deterministically fit sigmoid(a * logit(p) + b) by fixed-iteration gradient descent. | | isotonicRegressionPAV | (data: readonly LabeledPrediction[]) => IsotonicModel | Fit a monotone non-decreasing step map via Pool Adjacent Violators. | | applyCalibration | (model: PlattParams \| IsotonicModel, ps: readonly number[]) => number[] | Apply a fitted model to probabilities (overloaded by model shape). | | recalibrate | (data: readonly LabeledPrediction[], method: "platt" \| "isotonic") => RecalibrationResult | Fit, apply back, and report before/after metrics. | | parseCalibrationInput | (text: string) => ParseResult | Parse tolerant "p,y" or "p y" rows (header-aware, line-numbered errors). | | SYNTHETIC_SAMPLES | readonly SyntheticSample[] | Three seeded, reproducible demo datasets (overconfident, well calibrated, underconfident). |

Types

LabeledPrediction { p: number; y: 0 | 1 } · ReliabilityBin { lo, hi, count, meanPredicted, observedFrequency } · CalibrationMetrics { logLoss, brier, ece, accuracy, n } · PlattParams { a, b } · PlattOptions { iterations?, learningRate?, eps? } · IsotonicModel { x: number[]; y: number[] } · RecalibrationResult { calibrated, before, after } · ParseError { line, message } · ParseResult { rows, errors } · SyntheticSample { id, label, description, synthetic, data }.

Every function is pure and deterministic, with no React, DOM, or Node I/O. Degenerate inputs (empty, single-class, all-positive, fewer rows than bins, confident misses) return finite numbers rather than NaN or Infinity.

Provenance

This is the engine behind the Calibration Lab at https://karimnsemaan.me/calibration-lab and the KickCast calibration study. The same code that powers those interactive tools is what ships here.

License

MIT © 2026 Karim Semaan