@kognitivedev/evals

Provider-agnostic evaluation framework with model-graded, rule-based, and composite scorers.

Installation

bun add @kognitivedev/evals @kognitivedev/adapter-ai-sdk @ai-sdk/openai zod

Quick Start

import { EvalDatasetBuilder, modelGradedScorer, ruleBased, runEvals } from "@kognitivedev/evals";
import { createAISDKStructuredGenerationAdapter } from "@kognitivedev/adapter-ai-sdk";
import { openai } from "@ai-sdk/openai";

const judge = createAISDKStructuredGenerationAdapter({
  model: openai("gpt-4o-mini"),
});

const report = await runEvals({
  generateResponse: async (messages) => {
    return "Hello";
  },
  dataset: EvalDatasetBuilder.fromArray([
    { input: [{ role: "user", content: "Hello" }] },
  ]),
  scorers: [
    ruleBased.containsKeyword("hello"),
    modelGradedScorer({ runtime: judge, criteria: "Is this helpful?" }),
  ],
});

console.log(report.summary.avgScore);

Scorers

• modelGradedScorer via a StructuredGenerationAdapter
• visionScorer for multimodal judging
• rule-based scorers
• compositeScorer