@tangle-network/agent-eval

Run agent evaluations, compare changes on the same cases, and decide whether a candidate has enough evidence to release.

Eval runs in your TypeScript process. You supply agent execution, judges, and model transports. It records outputs, failures, costs, and evidence for each comparison.

Install

Use Node.js 20.19 or newer.

pnpm add @tangle-network/agent-eval

Quickstart

This complete example runs offline. Save it as eval.mts.

import { defineAgentEval } from '@tangle-network/agent-eval/contract'

interface SupportCase {
  id: string
  kind: 'support'
}

const evalKit = defineAgentEval<SupportCase, string>({
  scenarios: [
    { id: 'refund', kind: 'support' },
    { id: 'shipping', kind: 'support' },
    { id: 'cancel', kind: 'support' },
  ],
  agent: async (prompt, scenario) =>
    String(prompt).includes('ticket') ? `Ticket ${scenario.id}: on it.` : 'On it.',
  judge: {
    name: 'ticket-id',
    dimensions: [{ key: 'present', description: 'The answer includes the ticket id' }],
    score: ({ artifact, scenario }) => {
      const present = artifact.includes(scenario.id) ? 1 : 0
      return { dimensions: { present }, composite: present, notes: '' }
    },
  },
  baselineSurface: 'Answer politely.',
  expectUsage: 'off',
})

const baseline = await evalKit.evaluate()
const candidate = await evalKit.evaluate({
  surface: 'Answer politely and cite the ticket id.',
})

console.log('baseline:', baseline.aggregates.byJudge['ticket-id']?.mean)
console.log('candidate:', candidate.aggregates.byJudge['ticket-id']?.mean)

Run it with a TypeScript runner:

pnpm add --save-dev tsx
pnpm exec tsx eval.mts

baseline: 0
candidate: 1

The baseline scores 0; the candidate scores 1 on all three cases. These scores describe the three examples. They do not establish a release decision or performance on new tasks.

A case is one task. A surface is the prompt, skill, or configuration being changed. A judge scores the agent's result.

expectUsage: 'off' applies because this example makes no paid calls. Set expectUsage: 'assert' for paid agents so missing dispatch receipts become execution failures. The runnable example uses the same evaluation. The existing-agent example shows how to connect your agent and record model usage.

Choose a workflow

| Intent | Start with | Result | |---|---|---| | Score one change | defineAgentEval() from /contract | Cell results, failures, score distributions, and measured cost. | | Search for a better surface | selfImprove() from /contract | A selected surface, final comparison, and gateDecision. | | Compare search methods | compareOptimizationMethods() from /campaign | Paired final comparisons, uncertainty, coverage, and costs under declared budgets. | | Register evidence and decision rules | defineEvaluationClaim() and sealExperiment() from /experiment | A declared population, independent unit, optional practical effect, and sealed rules. | | Check the evaluator | auditEvaluator() and calibration tools from /meta-eval | Error rates, admission evidence, bias diagnostics, and outcome associations. | | Analyze completed work | analyzeRuns() from /contract; trace analysts from /analyst | Comparisons and findings with links to recorded evidence. |

defineAgentEval() also exposes improve() when the same agent, cases, judge, and baseline should share configuration. Use direct campaign controls for scheduling, durable caches, model matrices, or custom release rules. The example index covers fixtures, trace intake, code verification, replay, and training-data exports.

Make automated improvement accountable

Use reusable evaluations for development feedback. For a direct edit, compare the baseline and candidate on the same cases. Claims, evaluator audits, and final-evidence tracking are optional. Add stronger controls when a result must support performance on new tasks or an adaptive release decision.

1. Pass a claim describing the population, sampling frame, and independent unit to the comparison. Declare minimumEffect when the decision concerns a useful improvement.
2. When introducing an evaluator, check known good and known bad controls with auditEvaluator().
3. Give search separate training and selection cases.
4. For fresh confirmation, supply finalEvidence with a shared ledger, request ID, and evaluator digest. This reserves final units before search and records exposure before measurement.
5. Inspect the final comparison, gate contributions, exclusions, uncertainty, cost, and search history before releasing.

Repeated attempts on one task do not create new independent tasks. The top-level claim controls unit aggregation for reusable comparisons. Power checks assess the declared minimum effect. Optional finalEvidence binds fresh confirmation to that claim and refuses reused final units across campaigns sharing the ledger.

The host must enforce access isolation and author/auditor separation. A digest records identity; it cannot prove secrecy or that a benchmark represents future users. Custom gates remain responsible for their decision rules. See evaluation integrity for the complete API and its boundaries.

These controls check the evidence behind a result. They do not establish that an optimizer beats a direct edit or simple search. The historical evidence audit records prior gains, failed transfer, and missing comparisons.

Set searchHistoryPolicy: 'require-complete' when every attempted search slot must be accounted for before final evidence is exposed. The search-history receipt binds the planned denominator to Eval's existing search ledger.

A gateDecision is ship, hold, need_more_work, model_ceiling, or arch_ceiling. Gate contributions distinguish missing evidence from measured failures and successful checks. Concepts explains these decisions and how gates compose.

Configure model calls

Pass a ChatClient to model judges, analysts, and adapters. Eval obtains credentials from the values you supply; it does not search your environment.

import { createChatClient } from '@tangle-network/agent-eval/contract'

const chat = createChatClient({
  transport: 'openai-compatible',
  baseUrl: 'https://router.example/v1',
  apiKey: process.env.MY_ROUTER_KEY,
  defaultModel: process.env.EVAL_MODEL_ID,
})

Use your deployed model identifier and preserve the returned servedModel identity and cost receipt. For an existing SDK, use transport: 'custom' with your chat callback and an explicit maximumAttempts. Agent Runtime callers can bind profileChatClient() from @tangle-network/agent-runtime/kernel. Eval has no dependency on Runtime.

Official GEPA, SkillOpt, and DSPy integrations use a Python bridge. Their maintained installation instructions and execution contracts are in campaign proposers. The Python client and wire protocol support other-language consumers.

Public imports and evidence

Use /contract for a product integration, /campaign for execution controls, /experiment for registered decisions, and /meta-eval for evaluator checks. Root Scenario, JudgeScore, and GateDecision are the same types as /contract. Product judging retains the explicit root names ProductScenario and DimensionJudgeScore beside its functions. HeldOutGate.evaluate() returns HeldOutGateDecision.

Specialist subpaths and their examples are listed in the surface map. Current canonical envelopes are required for seals, attestations, and profile identities. Retired or incomplete formats fail verification; historical reports retain their recorded identities.

Published measurements live in the evidence registry. The benchmark-book review records the source analysis and reproduced defects behind these integrity changes. The charter defines package ownership and the remaining research boundaries.

Development

pnpm install
pnpm build
pnpm typecheck
pnpm typecheck:examples
pnpm typecheck:scripts
pnpm lint
pnpm test
pnpm verify:package

Build before checking examples because they resolve the package's generated declarations. The Python development guide gives the locked commands for each optimizer environment.

License

MIT.