@yansirplus/evals

Purpose

Authoring DSL for generated app behavior evals. The package lets product repos declare evals/**/*.eval.ts cases, symbolic provider needs, public driver facades, and deterministic assertions over generated app behavior.

Public API Status

0.6.x active development. The package is an authoring surface for eval definitions; execution, app serving, provider resolution, and artifact persistence remain owned by the CLI/runtime/client seams.

Audience

| Entrypoint | Audience | Capability | | ----------------- | ---------------- | -------------------------------- | | @yansirplus/evals | default-direct | generated app eval authoring DSL |

Invariant

@yansirplus/evals is declaration-only. It owns eval definitions, datasets, symbolic config, assertion declarations, and public facade types. It does not assemble runtime graphs, construct model providers, serve generated apps, import runtime internals, or read eval result artifacts as source facts.

Minimal Usage

import { defineEval, evalAssertion } from "@yansirplus/evals";

export default defineEval({
  path: import.meta.url,
  cases: [{ id: "turn", input: { sessionRef: "session-1", turnRef: "turn-1" } }],
  assertions: [evalAssertion.completed()],
  run: async ({ case: testCase, sessions }) => {
    await sessions.submitTurn({
      intent: "answer",
      context: {},
      sessionRef: testCase.input.sessionRef,
      turnRef: testCase.input.turnRef,
    });
  },
});

Verification

pnpm --filter @yansirplus/evals test
pnpm run agentos eval --cwd /path/to/generated-app --target local