@kontinent/etalon

Model evals for AI agents, routed through the Kontinent gateway. An eval file checks three things about an agent run: which tools were called, which tools were avoided, and how good the answer was. It then runs the same cases across candidate models and prints a comparison table with cost in EUR and an EU sovereignty tier.

Every result is a Score (name, value, dataType, source, comment), so assertions, evaluator output, judge verdicts and metrics land in one uniform stream that the reporter, the CI gates and the stored artifacts all read.

Install

bun add -d @kontinent/etalon

Runs on Bun. Eval files are executed by bun test. You need a Kontinent API key in KONTINENT_API_KEY, either in the environment or in a .env file. The gateway defaults to https://api.kontinent.ai; override it with KONTINENT_BASE_URL.

An eval file

// my-agent.eval.ts, run with: bun test ./my-agent.eval.ts
import {
  defineEval,
  inProcess,
  contains,
  passRate,
  avgJudge,
  latencyPercentile,
} from "@kontinent/etalon";
import { runMyAgent } from "./src/agent";

defineEval({
  name: "my-agent",
  agent: inProcess((input, { model }) => runMyAgent(input, { model })),

  candidates: [
    "mistral/mistral-small-latest", // current production model
    "mistral/mistral-large-latest",
    "scaleway/llama-3.3-70b",
  ],
  baseline: "mistral/mistral-small-latest",
  judgeModel: "mistral/mistral-large-latest", // pinned, never varies per run
  authoredBy: process.env.ETALON_AUTHORED_BY, // coding agent that wrote this

  // Aggregate gates, which is what CI should block on.
  runEvaluators: [passRate(0.9), avgJudge(7), latencyPercentile(95, 20_000)],

  cases: [
    {
      id: "refund-requires-lookup",
      input: "I want a refund for order NK-1042.",
      assert: (run) => {
        run.toComplete();
        run.tool("lookup_order").toBeCalled();
        run.tool("delete_account").toNotBeCalled();
      },
      expectedOutput: "NK-1042",
      evaluators: [contains()], // deterministic, free, zero variance
      judge: { criteria: "Confirms the refund politely.", minScore: 6 },
    },
  ],
});

You get one bun test result per model and case, a comparison table, and a JSON artifact under ./runs/:

| model                                  | sovereignty | cases | judge/10 | p50  | €/case  | result |
| -------------------------------------- | ----------- | ----- | -------- | ---- | ------- | ------ |
| mistral/mistral-large-latest           | strict      | 5/5   | 8.4      | 6.1s | €0.0031 | pass   |
| mistral/mistral-small-latest (current) | strict      | 4/5   | 6.8      | 3.2s | €0.0004 | FAIL   |

Numbers above are illustrative. Run your own suite to get real ones.

Assertions

Assertions run against the transcript, so they describe behaviour rather than wording:

run.toComplete();                       // produced a final answer
run.tool("x").toBeCalled();
run.tool("x").toNotBeCalled();
run.tool("x").toBeCalledWith((args) => args.amount <= 100, "amount within limit");
run.expect(run.raw.finalText.includes("NK-1042"), "answer mentions the order id");

Evaluators

Reach for deterministic checks before the judge. They are free, instant and have zero variance. Keep the judge for genuinely subjective criteria.

import {
  contains,
  notContains,
  exactMatch,
  matchesRegex,
  matchesSchema,
  answerLength,
} from "@kontinent/etalon";

evaluators: [
  contains(),                    // uses expectedOutput when called bare
  notContains(["admin mode"]),   // leakage and injection guard
  matchesSchema({ type: "object", required: ["status"] }),
]

Aggregate gates run once per candidate model:

import {
  passRate,
  avgJudge,
  latencyPercentile,
  eurPerCase,
  neverCallsTool,
} from "@kontinent/etalon";

runEvaluators: [
  passRate(0.9),                  // at least 90% of cases pass
  avgJudge(7),                    // mean judge score of at least 7 out of 10
  latencyPercentile(95, 20_000),  // p95 under 20 seconds
  eurPerCase(0.01),               // budget per case
  neverCallsTool("issue_refund"), // never, in any case
]

Without a threshold these are observational: reported, not gating.

Prefer gating on aggregates over individual cases. In a nondeterministic system one flaky case is noise, while a two point drop in the average is signal.

Adapters

The agent under test can be written in any language.

import { http, inProcess, subprocess } from "@kontinent/etalon";

inProcess((input, { model }) => runMyAgent(input, { model }));

subprocess({ cmd: ["python3", "agent.py"], modelEnv: "AGENT_MODEL" });

http({ url: "http://localhost:8080/agent-run" });

Out of process agents receive the candidate model through an environment variable (AGENT_MODEL) and the user message through argv, stdin or a POST body. They reply with the wire transcript, printed as the last line of stdout for subprocess agents or returned as the response body for HTTP agents:

{
  "final_text": "Your order shipped yesterday.",
  "tool_calls": [{ "name": "lookup_order", "arguments": { "order_id": "NK-3310" } }],
  "usage": { "input_tokens": 812, "output_tokens": 74 },
  "completed": true,
  "error": null
}

Both snake_case and camelCase keys are accepted. latency_ms is optional, since the harness measures wall clock time when it is absent.

CI semantics

Each run is classified, and the verdict is printed as KONTINENT_EVAL_RESULT suite=... classification=... and stored in the artifact:

| Classification | Meaning | CI should | |---|---|---| | regression | The model got worse on merit | Block the merge | | harness_error | Gateway 5xx, spawn failure, timeout | Retry |

Aggregate gates are ignored for any model whose cases hit infrastructure errors, because a 0% pass rate caused by a dead gateway is not a quality regression. A real regression always outranks a concurrent harness error.

bun test exits 1 for every kind of failure. The etalon CLI wraps it and maps these classifications to distinct exit codes, which is what makes the difference actionable in CI.

Reproducibility

• The judge model is pinned in the eval definition, runs at temperature 0, and its rubric prompt is versioned (RUBRIC_VERSION, recorded in every artifact).
• Cases and candidates live in code, so a score change between runs is attributable to the candidate model.
• A winner is only meaningful when baseline, your current production model, is in the candidate list.

Related

• etalon CLI: cost estimates, aggregation, exit codes
• Worked example: a TypeScript agent and a Python agent sharing one suite

License

Apache-2.0. See LICENSE.