@fengrru/confidence-gate

Experimental — API may break in minor versions. See STABILITY.md.

LLM output confidence calibration with ECE scoring and temperature scaling.

Install

npm install @fengrru/confidence-gate

Quick Start

import { ConfidenceGate } from "@fengrru/confidence-gate"

const gate = new ConfidenceGate()

// Fit calibration data
gate.fit([
  { confidence: 0.9, correct: true },
  { confidence: 0.7, correct: false },
  // ...
])

// Calibrate new prediction
const calibrated = gate.calibrate(0.85)
console.log(calibrated.confidence) // temperature-scaled value
console.log(calibrated.isOverconfident) // true if raw > calibrated

Features

• ECE scoring: Expected Calibration Error
• Brier score: probabilistic accuracy
• Temperature scaling: T>1 softens, T<1 sharpens
• Dynamic threshold: find optimal decision boundary
• Hallucination rate: % high-confidence wrong answers

Documentation

• API Reference — TypeDoc-generated API docs
• Source Code
• Examples

License

MIT