@aroha-sdk/evals

Evaluation harness for Aroha agents — verify mandate-respecting behavior across test datasets before shipping.

Install

npm install -D @aroha-sdk/evals

Quick example

import { runEvals, mandateEvalSuite } from "@aroha-sdk/evals";
import { issueIntentMandate } from "@aroha-sdk/credentials";

// Built-in suite: tests that mandate bounds are respected
const results = await runEvals({
  agentUrl: "http://flight-agent/v1/run",
  issuerPublicKey: humanPublicKey,
  suites: [
    mandateEvalSuite({
      // Tests that the agent rejects over-scope mandates
      overScopeCases: [
        {
          mandate: await issueIntentMandate(humanDID, agentDID, { spendLimitUsd: 100 }, humanPrivateKey, 60_000),
          action: { requestedSpend: 200 },
          expectedOutcome: "rejected",
        },
      ],
      // Tests that the agent accepts valid in-scope mandates
      validCases: [
        {
          mandate: await issueIntentMandate(humanDID, agentDID, { spendLimitUsd: 300, allowedActions: ["book-flights"] }, humanPrivateKey, 60_000),
          action: { requestedSpend: 299, action: "book-flights" },
          expectedOutcome: "accepted",
        },
      ],
      // Tests that forged mandates are rejected
      forgedCases: [
        {
          forgedToken: "eyJhbGciOi...tampered",
          expectedOutcome: "rejected",
        },
      ],
    }),
  ],
});

console.log(results.summary);
// Passed: 12 / 12  Mandate violations: 0

Why this exists in the mandate chain

Shipping a mandate-aware agent without evals risks silent regressions — an agent that passes valid mandates but also passes forged ones. @aroha-sdk/evals provides a structured harness that tests mandate acceptance, rejection, and forgery handling before any code ships.

API

• runEvals(opts) — run one or more eval suites against an agent URL.
• mandateEvalSuite(opts) — built-in suite covering over-scope rejection, valid mandate acceptance, forged mandate rejection, and TTL expiry.
• customEvalCase(opts) — define a single eval case with a message, mandate, and expected outcome.
• EvalReport — structured result with pass/fail counts, mandate violation count, and per-case details.

License

MIT