costeval

A cost-aware eval harness for real product pipelines.

Most eval frameworks assume the only thing that costs money is the model. If your pipeline calls paid APIs downstream of the LLM — Google Places, geocoding, weather, search — an eval run bills you on every one of them, every time, and nothing warns you first.

costeval treats spend as a first-class concern:

• Pre-run cost estimate with a hard abort. The run refuses to start if the estimate crosses your threshold, unless you explicitly say --yes.
• Runtime kill switch. If actual metered spend crosses a cap mid-run — which is exactly when your estimate was wrong — no new cases are dispatched.
• Record/replay fixtures for paid calls. Wrap any metered call in ctx.io(...); the first run records the response to a PR-diffable JSON fixture, every later run replays it at $0, and the report tells you how much replay saved.

It is TypeScript-native, evaluates your real pipeline as a function (not a prompt), and scores outputs against declarative per-case expectations from a golden dataset.

Born from a production incident: an 82-case eval suite that quietly billed CA$156 of Google Places API calls in two days. The estimate/abort/replay design here is what we wished existed.

Quickstart

npm install costeval

// eval/run.ts
import { defineEval, runCli, numericMax, complianceRatio, countInRange, fixtureKey } from 'costeval';
import { generateDay } from '../src/pipeline';   // your real pipeline

const config = defineEval<TripInput, Itinerary, Expectations>({
  cases: [
    {
      id: 'VEGAN-NYC-001',
      description: 'Vegan day in NYC under $60',
      input: { city: 'New York', budget: 60, dietary: 'vegan' },
      expect: { maxTotalCost: 60, dietary: { tag: 'vegan', minRatio: 'all' } },
    },
    {
      id: 'VEGAN-RURAL-001',
      description: 'Vegan in a small town — should degrade gracefully',
      input: { city: 'Dillon, MT', budget: 40, dietary: 'vegan' },
      expect: { dietary: { tag: 'vegan', minRatio: 'all' } },
      expectedToFail: true,   // if this starts passing, the system over-promised
    },
  ],

  // Your shipped pipeline, called per case. ctx.io wraps paid calls.
  pipeline: async (input, ctx) => {
    return generateDay(input, {
      searchPlaces: (params) =>
        ctx.io(fixtureKey('places', params), () => realPlacesSearch(params), {
          cost: 0.032,          // what one live call costs you
          meter: 'places',
        }),
    });
  },

  checks: [
    numericMax({ name: 'budget', value: o => o.totalCost, max: e => e.maxTotalCost, tolerance: 0.1 }),
    complianceRatio({
      name: 'dietary',
      items: o => o.stops.filter(s => s.kind === 'food'),
      predicate: (s, e) => s.tags.includes(e.dietary!.tag),
      minRatio: e => e.dietary?.minRatio,
    }),
    countInRange({ name: 'stops', items: o => o.stops, min: e => e.minStops, max: e => e.maxStops }),
  ],

  cost: {
    perCase: 0.15,      // pre-run estimate (number, or a per-case function)
    abortAbove: 10,     // refuse to start above this estimate without --yes
    stopAbove: 25,      // stop dispatching cases if ACTUAL spend crosses this
  },

  replay: { dir: 'eval/fixtures' },
  run: { concurrency: 2, timeoutMs: 90_000, minPassRate: 0.85 },
});

runCli(config);

npx tsx eval/run.ts --dry-run          # validate + print the estimate, $0
npx tsx eval/run.ts                    # first run records fixtures
npx tsx eval/run.ts                    # re-runs replay paid calls at $0
npx tsx eval/run.ts --cases VEGAN-NYC-001 --yes

Sample output:

Pass rate: 100.0% (4/4 judged)
Cost: $0.00 actual vs $0.20 estimated
  io: 0 live, 4 replayed — replay saved $0.20

Per-check satisfaction:
  budget     pass 100%  avg score 100%  (n=3)
  dietary    pass 100%  avg score 100%  (n=1)
  stops      pass  50%  avg score  50%  (n=2)

A runnable demo lives in examples/travel/ — no API keys needed.

Concepts

Cases are declarative

A case is structured input plus expectations — no assertions in the case file. Checks read the expectations and decide pass/fail, so the golden dataset stays reviewable by non-engineers and diffable in PRs. expectedToFail: true marks graceful-degradation regressions: the case passes when it fails, and an unexpected pass is reported as UNEXPECTED_PASS. Pipeline errors are never inverted — a crash is an infra problem, not graceful degradation.

The three cost layers

1. Estimate + abort (cost.perCase, cost.abortAbove): computed before anything runs; breaching the threshold throws unless overridden with --yes.
2. Actual metering (ctx.cost.add(meter, usd) and ctx.io(..., { cost })): the report shows actual vs estimated, broken down per meter and per case.
3. Kill switch (cost.stopAbove): judged on actual spend, so it catches the runs where your estimate was wrong. Skipped cases are reported as skipped_budget and excluded from the pass rate denominator.

Replay only what should be frozen

ctx.io is designed for the non-LLM half of your pipeline. Freezing the model's output and the downstream validation together makes a broken downstream path keep "passing" — so keep the model live (or cache it with your provider's own tooling) and replay the metered lookups. Fixtures are plain JSON, keyed deterministically (fixtureKey sorts object keys), safe to commit, and a corrupt fixture falls through to a live call instead of failing the run.

Hard rate and soft rates

summary.passRate is the hard satisfaction rate: cases where every applicable check passed. report.checks gives the soft rate per check — pass rate and mean score across the cases where that check applied. A pass rate alone hides whether one case failed badly or many failed slightly; the per-check table is where regressions get names.

Reports are the record

The full JSON report is persisted to run.resultsDir before any console rendering — console tables get truncated by non-TTY pipes, and at real API prices the disk copy is the record.

CI

runCli sets a non-zero exit code when the cost guard refuses, validation fails, or the pass rate lands under run.minPassRate:

- run: npx tsx eval/run.ts --yes
  env:
    MY_API_KEY: ${{ secrets.MY_API_KEY }}

With committed fixtures, CI re-runs are $0 and deterministic for the replayed calls. To re-record, delete the fixture dir (or run with --no-replay) locally and commit the diff.

When you should use something else

Honesty section. promptfoo is excellent, TypeScript, and far more featureful — model matrices, red-teaming, a web UI, ~30 assertion types. If you're evaluating prompts/models and cost isn't your binding constraint, use it. Use costeval when:

• your eval calls a real product pipeline whose non-LLM API spend dominates,
• you want a run to be refused before it spends, not reported on after,
• you want paid downstream calls recorded once and replayed at $0 in CI.

For LLM-judge or embedding-similarity scoring, pair costeval with autoevals — any async function returning CheckResults is a valid check.

API surface

| Export | What it is | |---|---| | defineEval(config) | Type-pinning identity helper | | runEval(config, opts) | Programmatic runner → RunReport (never calls process.exit) | | runCli(config, argv?) | CLI wrapper: --cases, --concurrency, --dry-run, --yes, --no-replay, --json | | numericMax / numericMin / countInRange / complianceRatio / keywordMatch / noAdjacent / custom | Declarative check factories (all accessor-based, output-shape-agnostic) | | fixtureKey(ns, params) | Deterministic replay key (recursively key-sorted) | | FixtureStore, CostMeter, CostGuardError, estimateRun, renderReport | Lower-level pieces |

Roadmap

• Conditional expectations (only score check B when check A passed — ComplexBench-style dep)
• A promptfoo assertion adapter for the check pack
• Fixture TTL / staleness warnings
• Optional cost-estimate calibration from recorded actuals

License

MIT