badgr-eval-check

Assert that AI model output contains what it should, excludes what it shouldn't, and matches the shape you expect — repeatable checks you can run in CI.

npx badgr-eval-check \
  --output "The answer is 42. Confidence: high." \
  --must-include "42" \
  --must-not-include "I cannot"

Free. No signup required. Runs entirely on your machine.

The problem it solves

LLM outputs are non-deterministic, so regressions are easy to miss. A model update silently starts refusing questions it used to answer, hallucinating success, or dropping required fields from JSON responses. badgr-eval-check lets you write simple assertions against model output and run them on every deploy — like unit tests for your prompts.

Quick start

# Check that output includes required text
npx badgr-eval-check --output "The answer is 42." --must-include "42"

# Check that output excludes banned phrases
npx badgr-eval-check --output "The answer is 42." --must-not-include "I cannot,sorry,I don't know"

# Check a JSON response has required fields
npx badgr-eval-check --output '{"answer": "42", "confidence": "high"}' --must-include answer,confidence

# Machine-readable JSON
npx badgr-eval-check --output "..." --must-include "42" --json

CLI flags

| Flag | Description | |------|-------------| | --output <str> | The model output string to check (required) | | --must-include <items> | Comma-separated strings that must appear in the output | | --must-not-include <items> | Comma-separated strings that must not appear in the output | | --json | Output machine-readable JSON |

Exit codes: 0 = all checks passed, 1 = one or more checks failed

Example output

badgr-eval-check

  ✓  must-include     "42" found in output
  ✓  must-include     "confidence" found in output
  ✗  must-not-include "I cannot" found in output — model refused to answer

  1 check failed.

badgr-eval-check

  ✓  must-include     "42" found in output
  ✓  must-not-include "I cannot" not found ✓
  ✓  must-not-include "sorry" not found ✓

  All checks passed.

TypeScript API

Use the programmatic API to run multiple fixtures as a batch:

import { runEvalCheck } from "badgr-eval-check";

const result = runEvalCheck([
  {
    name: "answer present",
    output: modelOutput,
    mustInclude: ["42", "confidence"],
    mustNotInclude: ["I cannot", "sorry", "I don't know"],
  },
  {
    name: "JSON fields present",
    output: modelOutput,
    requiredJsonFields: ["answer", "confidence"],  // dot-path: "data.items.0.name"
  },
]);

console.log(result.passed);  // true / false
console.log(result.checks);  // per-fixture results

Types:

interface EvalFixture {
  name: string;
  output: string;
  mustInclude?: string[];
  mustNotInclude?: string[];
  requiredJsonFields?: string[];  // supports dot-path, e.g. "data.items.0.name"
}

interface EvalCheckResult {
  checks: DiagnosticCheck[];
  passed: boolean;
  report: JsonReport;
}

Use in CI

# GitHub Actions example
- name: Run model eval checks
  run: |
    OUTPUT=$(curl -s https://api.openai.com/v1/chat/completions \
      -H "Authorization: Bearer $OPENAI_API_KEY" \
      -d '{"model":"gpt-4o","messages":[{"role":"user","content":"What is 6*7?"}]}' \
      | jq -r '.choices[0].message.content')
    npx badgr-eval-check --output "$OUTPUT" --must-include "42" --must-not-include "I cannot"

Requirements

• Node.js 18+