@deepagents/evals

A general-purpose LLM evaluation framework with dataset loading, scoring, run persistence, model comparison, and console reporting.

Installation

npm install @deepagents/evals

Quick Start

import { dataset, evaluate, exactMatch } from '@deepagents/evals';

const summary = await evaluate({
  name: 'my-eval',
  model: 'gpt-4o',
  dataset: dataset([
    { input: 'What is 2+2?', expected: '4' },
    { input: 'What is 3+3?', expected: '6' },
  ]),
  task: async (item) => {
    const response = await callMyLLM(item.input);
    return { output: response };
  },
  scorers: { exact: exactMatch },
});

Modules

The package is organized into subpath exports for granular imports:

| Import | Description | | ------------------------------ | --------------------------------------- | | @deepagents/evals | Top-level convenience API (evaluate) | | @deepagents/evals/dataset | Dataset loading and transforms | | @deepagents/evals/scorers | Scorer functions and combinators | | @deepagents/evals/store | SQLite run persistence | | @deepagents/evals/engine | Eval engine with concurrency and events | | @deepagents/evals/comparison | Run diffing and regression detection | | @deepagents/evals/reporters | Console and file reporters |

Dataset

Load data from inline arrays or local files (JSON, JSONL, CSV):

import { dataset } from '@deepagents/evals/dataset';

// Inline array
const ds = dataset([{ input: 'hello', expected: 'world' }]);

// From file
const ds = dataset('./data/questions.json');
const ds = dataset('./data/questions.jsonl');
const ds = dataset('./data/questions.csv');

Transforms

Chainable, lazy transforms on datasets:

dataset('./large-dataset.jsonl')
  .filter((row) => row.difficulty === 'hard')
  .map((row) => ({ input: row.question, expected: row.answer }))
  .shuffle()
  .limit(100);

| Transform | Behavior | | ------------ | -------------------------------------------- | | map(fn) | Lazy — transforms each element | | filter(fn) | Lazy — excludes non-matching elements | | limit(n) | Lazy — caps output at n elements | | shuffle() | Eager — buffers all, randomizes order | | sample(n) | Eager — buffers all, picks n random elements | | toArray() | Consumes into a plain array |

Dataset Utilities

The top-level package also exports dataset helpers for Hugging Face ingestion and deterministic record selection:

import {
  dataset,
  filterRecordsByIndex,
  hf,
  parseRecordSelection,
  pickFromArray,
} from '@deepagents/evals';

const rows = dataset(
  hf({
    dataset: 'openai/gsm8k',
    config: 'main',
    split: 'test',
    rows: 50,
  }),
);

const { indexes } = parseRecordSelection('1,3-5');
const sampled = pickFromArray(await rows.toArray(), indexes);

for await (const row of filterRecordsByIndex(rows, indexes)) {
  console.log(row);
}

| Utility | Description | | --------------------------------------- | ---------------------------------------------------------- | | hf(options) | Stream rows directly from the Hugging Face datasets server | | parseRecordSelection(spec) | Parse 1,3-5 style selectors into zero-based indexes | | pickFromArray(items, indexes) | Apply a parsed selection to an in-memory array | | filterRecordsByIndex(source, indexes) | Apply a parsed selection to an async dataset stream |

Scorers

All scorers return { score: number (0..1), reason?: string }.

Deterministic Scorers

import {
  exactMatch,
  includes,
  jsonMatch,
  levenshtein,
  regex,
} from '@deepagents/evals/scorers';

| Scorer | Description | | ---------------- | ----------------------------------- | | exactMatch | Strict string equality | | includes | Substring check | | regex(pattern) | RegExp test | | levenshtein | Normalized edit distance similarity | | jsonMatch | Deep JSON structural equality |

LLM-Based Scorers

import { factuality, sql } from '@deepagents/evals/scorers';

const fact = factuality({ model: 'gpt-4.1-mini' });
const sqlEquivalence = sql({ model: 'gpt-4.1-mini' });

factuality compares factual content against the expected answer. sql compares submitted SQL with expert SQL for semantic equivalence while ignoring formatting, output column names, and result ordering. Both scorers expect an OpenAI-compatible model id.

Combinators

import { all, any, weighted } from '@deepagents/evals/scorers';

// Weakest-link (minimum score)
const strict = all(exactMatch, includes);

// Best-of (maximum score)
const lenient = any(exactMatch, includes);

// Weighted average
const balanced = weighted({
  accuracy: { scorer: exactMatch, weight: 2 },
  grounding: { scorer: factuality({ model: 'gpt-4.1-mini' }), weight: 1 },
});

Run Store

SQLite-backed persistence for runs, cases, and scores:

import { RunStore } from '@deepagents/evals/store';

const store = new RunStore('.evals/store.db');

// Create a suite for grouping runs
const suite = store.createSuite('text2sql-accuracy');

// Query results
const runs = store.listRuns(suite.id);
const failing = store.getFailingCases(runId, 0.5);
const summary = store.getRunSummary(runId);

Engine

The engine orchestrates dataset iteration, task execution, scoring, and persistence:

import { EvalEmitter, runEval } from '@deepagents/evals/engine';

const emitter = new EvalEmitter();
emitter.on('case:scored', (data) => console.log(data.index, data.scores));

const summary = await runEval({
  name: 'my-eval',
  model: 'gpt-4o',
  dataset: ds,
  task: myTask,
  scorers: { exact: exactMatch },
  store,
  emitter,
  maxConcurrency: 10,
  timeout: 30_000,
  trials: 1,
  threshold: 0.5,
});

Events

| Event | Payload | When | | ------------- | ------------------------------------- | ----------------------------------------- | | run:start | { runId, totalCases, name, model } | Run begins | | case:start | { runId, index, input } | Case execution starts | | case:scored | { runId, index, scores, latencyMs } | Case scored (always fires, even on error) | | case:error | { runId, index, error } | Task threw an error | | run:end | { runId, summary } | All cases complete |

Comparison

Compare two runs case-by-case to detect improvements and regressions:

import { compareRuns } from '@deepagents/evals/comparison';

const result = compareRuns(store, baselineRunId, candidateRunId, {
  tolerance: 0.01,
  regressionThreshold: 0.05,
});

console.log(result.regression.regressed); // true if any scorer regressed
console.log(result.scorerSummaries); // per-scorer mean deltas and counts
console.log(result.costDelta); // latency and token differences

Console Reporter

Subscribe to engine events for terminal output:

import { consoleReporter } from '@deepagents/evals/reporters';

consoleReporter(emitter, {
  verbosity: 'normal', // 'quiet' | 'normal' | 'verbose'
  threshold: 0.5,
});

File Reporters

Persist run output in machine-readable or shareable formats:

import {
  csvReporter,
  htmlReporter,
  jsonReporter,
  markdownReporter,
} from '@deepagents/evals/reporters';

const reporters = [
  jsonReporter({ outputDir: '.evals/reports', pretty: true }),
  csvReporter({ outputDir: '.evals/reports' }),
  markdownReporter({ outputDir: '.evals/reports' }),
  htmlReporter({ outputDir: '.evals/reports' }),
];

| Reporter | Output | | -------------------- | ----------------------------------------------------- | | jsonReporter() | Writes per-case .jsonl plus a final summary .json | | csvReporter() | Writes a flat case-by-case CSV file | | markdownReporter() | Writes a Markdown summary with per-case scores | | htmlReporter() | Writes a shareable HTML report with summary tables |

License

MIT