@orqo/foundationmodels-eval

Evaluation harness for Apple Foundation Models — datasets, assertions, and prompt versioning.

Install

pnpm add @orqo/foundationmodels-eval

Usage

import { runEval, exact, contains, PromptRegistry } from "@orqo/foundationmodels-eval";
import { createFoundationModels } from "@orqo/foundationmodels";

const fm = await createFoundationModels();

// Run an evaluation against a dataset of classify cases.
const report = await runEval(
  fm,
  "classify",
  {
    name: "sentiment-v1",
    cases: [
      { input: { input: "I love this!", labels: ["positive", "negative"] }, expected: ["positive"], meta: { name: "positive-case" } },
      { input: { input: "This is terrible.", labels: ["positive", "negative"] }, expected: ["negative"], meta: { name: "negative-case" } },
    ],
  },
  (actual, expected) => contains(actual, expected)
);

console.log(`${report.passed}/${report.total} passed`);
console.log(report.results);

// Prompt versioning.
const registry = new PromptRegistry();
registry.register({
  id: "summarize-news",
  version: "1.0.0",
  instructions: "Summarize the following article in 3 bullet points.",
  createdAt: new Date().toISOString(),
});

const prompt = registry.resolve("summarize-news");
console.log(prompt?.instructions);

Exported assertion helpers: exact (deep equality via JSON), contains (substring or structural subset), jsonMatch (JSON string equality).

Supported EvalPrimitive values: "classify", "extract", "rank", "summarize".