@assay-ai/ai-sdk

Vercel AI SDK adapter for LLM evaluation with Assay

Documentation · Metrics · API Reference

Installation

pnpm add @assay-ai/core @assay-ai/ai-sdk     # pnpm
npm install @assay-ai/core @assay-ai/ai-sdk   # npm
yarn add @assay-ai/core @assay-ai/ai-sdk      # Yarn

Quick Start

import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
import { fromGenerateText } from "@assay-ai/ai-sdk";
import { evaluate, FaithfulnessMetric, AnswerRelevancyMetric } from "@assay-ai/core";

const result = await generateText({
  model: openai("gpt-4o"),
  prompt: "What is the capital of France?",
});

const testCase = fromGenerateText(result, {
  input: "What is the capital of France?",
  context: ["France is a country in Europe. Its capital is Paris."],
});

const evalResults = await evaluate(
  [testCase],
  [new FaithfulnessMetric(), new AnswerRelevancyMetric()],
);

console.log(`Pass rate: ${evalResults.summary.passRate.toFixed(1)}%`);

Part of the Assay monorepo