AgentSpec

Regression testing for non-deterministic AI agents — behavioral contracts, adaptive sampling, GitHub Action ready.

Why?

AI agents are non-deterministic. Traditional tests (assert output == expected) don't work. You run the same input twice and get different outputs — both might be correct.

AgentSpec brings behavioral contracts from academic research into a practical CLI tool:

• Test behavior, not output — "Agent must call search_flights tool" instead of "output must equal X"
• Adaptive sampling — run 3 times instead of 30, save 70%+ token costs (based on AgentAssay research)
• Behavioral fingerprinting — detect regressions even when tests pass (e.g., token usage +50%, new error paths)
• Statistical confidence — "95% confident pass rate ≥ 80%" instead of "passed once, ship it"

Quick Start

# Install globally
npm install -g @bensonday/agent-spec

# Or use npx (no install needed)
npx @bensonday/agent-spec init

# Initialize in your project
agentspec init

# Run tests
agentspec test

Define a Contract

# agent-spec.yaml
agent: openai
agentConfig:
  model: gpt-4o
  tools:
    - type: function
      function:
        name: search_flights
        parameters:
          type: object
          properties:
            from: { type: string }
            to: { type: string }
            date: { type: string }

contracts:
  - name: "Search flights and return results"
    input: "Find flights from Beijing to Shanghai tomorrow"
    assertions:
      - must_call_tool: "search_flights"
      - must_contain_any: ["航班", "flight", "机票"]
      - must_not_error: true
      - completes_within: "30s"
      - token_budget: 5000
    sample: 5
    passRate: 0.8
    adaptive: true

Run Tests

# Run all contracts
agentspec test

# Filter by name
agentspec test --filter "search"

# Update baseline (on main branch)
agentspec test --update

# JSON output for CI
agentspec test --json report.json

# Skip regression check
agentspec test --no-regression

Real-World Examples

See examples/ for complete, runnable contracts:

| Example | What it tests | Key assertions | |---|---|---| | Customer Support | E-commerce bot must search KB, not over-create tickets | must_call_tool, must_not_call_tool | | RAG Pipeline | Doc QA must cite sources, must not hallucinate | must_contain_any, must_not_contain, token_budget | | Multi-Tool Agent | Smart assistant must pick the right tool for each task | must_call_tool + must_not_call_tool combos |

Each example works with mock (no API key) or real API (DeepSeek/OpenAI/Claude/Gemini).

Assertion Types

| Assertion | Description | |---|---| | must_call_tool: "name" | Agent must call this tool | | must_not_call_tool: "name" | Agent must not call this tool | | must_contain_any: ["a", "b"] | Output must contain at least one keyword | | must_contain_all: ["a", "b"] | Output must contain all keywords | | must_not_contain: ["x"] | Output must not contain keywords | | must_not_error: true | Agent must not error | | completes_within: "30s" | Must complete within time limit | | token_budget: 5000 | Token usage must not exceed budget |

GitHub Action

Add to .github/workflows/agent-tests.yml:

name: Agent Regression Tests
on:
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: bensonday/agent-spec@v1
        with:
          config: agent-spec.yaml
          fail-on-drift: true       # fail CI on behavioral drift
          comment-on-pr: true       # post results as PR comment
        env:
          DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }}
          # OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          # ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          # GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}

Action Inputs

| Input | Default | Description | |---|---|---| | config | agent-spec.yaml | Path to contract file | | filter | "" | Only run matching contracts | | agent | "" | Override adapter (mock/openai/claude/gemini) | | update-baseline | false | Update & commit baseline (use on main branch) | | no-regression | false | Skip regression check | | fail-on-drift | false | Fail CI on high-severity behavioral drift | | comment-on-pr | false | Post test summary as PR comment |

The Action automatically:

• Installs AgentSpec via npm
• Runs all contracts with adaptive sampling
• Uploads JSON report as artifact (30-day retention)
• Generates GitHub Actions step summary
• Optionally comments on PR with results
• Optionally commits baseline file on main branch

See examples/github-action-workflow.yml for a complete CI setup with baseline updates.

How Adaptive Sampling Works

Based on the AgentAssay paper:

1. Run 3 times (minimum sample)
2. Extract behavioral fingerprints — tool sequence, step count, token bucket, error state
3. If fingerprints are consistent → behavior is deterministic, stop early (save 70%+ tokens)
4. If fingerprints vary → keep sampling up to N times, compute statistical confidence

Behavioral Fingerprint Format

OK|search_flights,select_flight,book_flight|S3|TM|LF
│  │                                  │  │  └─ latency bucket (Fast/Mid/Slow)
│  │                                  │  └──── token bucket (Low/Mid/High)
│  │                                  └─────── step count
│  └────────────────────────────────────────── tool call sequence
└───────────────────────────────────────────── error status (OK/ERR)

Only the pattern is compared, not exact values — so natural variation in token count or latency won't trigger false alarms.

Agent Adapters

| Adapter | Providers | Install | |---|---|---| | mock | Built-in, no API needed | — | | openai | OpenAI, DeepSeek, Moonshot, Qwen, 智谱GLM, MiniMax, Yi, Baichuan, SiliconFlow | npm install openai | | claude | Anthropic Claude | npm install @anthropic-ai/sdk | | gemini | Google Gemini | npm install @google/generative-ai |

Provider Presets

Use provider field to auto-configure baseURL and API key for OpenAI-compatible services:

agent: openai
agentConfig:
  provider: deepseek  # auto-sets baseURL + reads DEEPSEEK_API_KEY
  # No need to manually set baseURL or apiKey

| Provider | Display Name | Env Var | |---|---|---| | openai | OpenAI | OPENAI_API_KEY | | deepseek | DeepSeek | DEEPSEEK_API_KEY | | moonshot | Moonshot (Kimi) | MOONSHOT_API_KEY | | qwen | 通义千问 (Qwen) | DASHSCOPE_API_KEY | | zhipu | 智谱 GLM | ZHIPU_API_KEY | | minimax | MiniMax | MINIMAX_API_KEY | | yi | 零一万物 (Yi) | YI_API_KEY | | baichuan | 百川 (Baichuan) | BAICHUAN_API_KEY | | siliconflow | SiliconFlow | SILICONFLOW_API_KEY |

Run agentspec list to see all available adapters and providers.

Custom Adapter

import { AgentAdapter, AgentTrace, registerAdapter } from "@bensonday/agent-spec";

class MyAgent implements AgentAdapter {
  name = "my-agent";
  async run(input: string): Promise<AgentTrace> {
    // ... your agent logic
    return { toolsCalled, output, tokens, latency, error, steps };
  }
}

registerAdapter("my-agent", () => new MyAgent());

Programmatic API

import { adaptiveSample, evaluateContract, extractFingerprint } from "@bensonday/agent-spec";

// Run a contract with adaptive sampling
const result = await adaptiveSample(
  async (seed) => myAgent.run("Find flights"),
  contract,
  { minSamples: 3, maxSamples: 10, confidenceThreshold: 0.95, passRateThreshold: 0.8, adaptive: true }
);

console.log(result.passRate);    // 0.9
console.log(result.confidence);  // 0.55 (Wilson lower bound)
console.log(result.stoppedEarly); // true (saved 70% tokens)

License

MIT