@introspection-ai/evalite

The supported Evalite boundary for local Introspection Recipes.

import { evalite } from "evalite";
import { createRecipeTask, recipeCases } from "@introspection-ai/evalite";

evalite("support", {
  data: recipeCases([
    {
      input: { prompt: "Where is my order?" },
      expected: { outcome: "requests order number" },
    },
  ]),
  task: createRecipeTask({ runtime: "support-agent", agent: "agent" }),
  scorers: [],
});

agent is the agent YAML's declared name, not its filename. This makes an inherited variant such as agents/agent2.yaml (name: agent2, from: agent) selectable with agent: "agent2" without duplicating the base agent's tools, extensions, or instructions. The selected agent YAML is also the only source of its model; create an inherited agent variant to evaluate another model.

Run the suite with introspection eval run --runner evalite. A production-derived run injects one prompt while keeping the first authored case's expected value as the scorer contract. The adapter forwards an opaque replay-context path to introspection local; it does not inspect the canonical trajectory or create a native Pi session.

The task result's events array is one canonical Pi event stream. Every event has a run scope identifying the root or delegated child that emitted it; child messages and tool calls are not copied into a second trajectory field. Recipes transports each child event as a non-context Pi custom entry, and the adapter unwraps that single entry_appended record into the original scoped child event. usage aggregates model tokens and cost across all scopes, while output and completion continue to describe the root run. Tool calls carry the same run scope so scorers and optimizers can attribute child behavior directly.