botruntime Evals

Evaluation definitions and runner for brt-based botruntime agents.

Author evals with a small declarative API (Eval), then run them against a live agent through the native platform eval transport and a trace collector. The runner projects spans into grader-friendly turn data and graders assert on responses, tool calls, state mutations, workflow spans, and timing — including an optional LLM-judge grader backed by @holocronlab/botruntime-zai.

The authoring contract also supports private file fixtures, synthetic actors routed to related conversations, delivery/mode assertions, same-target parallel turns, and isolated-development clock/fault controls.

Durable table fixtures use the ordinary Tables API and are platform-owned:

Hosted runners also provide a durable effect transport for table seeds, isolated controls, and native event turns. Every mutation carries a stable identity: replay returns the original committed result, while the same identity with a different payload fails loudly before another side effect is applied. The immutable eval report is checkpointed before hosted outcome and verdict writes. Those writes use independent durable checkpoints, so a lost response replays the same result coordinates and verdict; an already acknowledged write is not sent again. Store errors expose only safe operation, status, kind, and ambiguous diagnostics and never copy the response body. Network failures, HTTP 408, 425, 429, and 5xx responses are ambiguous and may retry the immutable write; a 409 is definitive (ambiguous: false) and still fails loudly because the same identity was previously committed with different content. The workflow checkpoint persists this allowlisted envelope and one safe cause level across generations. A rehydrated EvalProgressSinkError also restores its sinkCause, preserving ownership diagnostics without persisting raw bodies.

const order = new Eval({
  name: 'order-is-durable',
  setup: {
    tables: [{ table: 'OrderTable', rows: [{ externalId: 'eval-{{eval.id}}', status: 'pending' }] }],
  },
  conversation: [
    {
      message: 'Check the test order',
      assert: {
        tables: [{ table: 'OrderTable', row_exists: { externalId: { equals: 'eval-{{eval.id}}' } } }],
      },
    },
  ],
  outcome: {
    tables: [
      {
        table: 'OrderTable',
        row_count: { equals: 1 },
        where: { externalId: { equals: 'eval-{{eval.id}}' } },
      },
    ],
  },
})

The runner creates setup rows before the first turn and deletes only their exact row IDs after grading. {{eval.id}} is one execution-scoped identity shared by setup values and table assertions. Reports contain match counts, not row contents. Partial creation or cleanup fails loudly with a stable error code. This is eval manifest schema v2; an older runtime rejects it rather than silently skipping durable setup.

Install

npm install @holocronlab/botruntime-evals

Usage

import { Eval } from '@holocronlab/botruntime-evals'
import { runEval } from '@holocronlab/botruntime-evals/runner'
import { Client } from '@holocronlab/botruntime-client'

const greeting = new Eval({
  name: 'greeting',
  conversation: [
    {
      message: 'hello',
      assert: {
        response: [{ llm_judge: 'Greets the user back politely' }],
      },
    },
  ],
})

const client = new Client({ token: process.env.BP_TOKEN! })

const report = await runEval(greeting, {
  client,
  botId: process.env.BOT_ID!,
})

Local fixture paths are authoring-only. brt eval uploads referenced files with private integration access and stores only file id, name, MIME, size and sha256 in the hosted manifest; signed URLs and contents are never persisted in eval results.

Entry points

• . — Eval authoring API + shared types
• ./runner — runEval / runEvalSuite, the trace-driven execution engine
• ./loader — discover *.eval.ts files from an agent directory
• ./graders, ./graders/* — individual grader functions (response, tools, state, workflow, timing, outcome, LLM judge)
• ./stores, ./stores/vortex — persistence for eval run history (local SQLite via bun:sqlite, or a remote Vortex-backed store)
• ./spans, ./sse-collector, ./trace — the trace/span primitives the runner and graders operate on
• ./client — a send-only chat session for driving eval conversations
• createNativeEvalChatClient — adapter over the authenticated platform chat API; uses synthetic incoming messages and requires no integration/provider key
• ./transformer — projects raw trace spans into grader-friendly turn data
• ./manifest, ./types, ./definition — shared manifest/type/definition building blocks

License

MIT — see LICENSE.