agentcheck

Smoke test runner for LLM agents.

Write YAML. Run assertions on outputs, latency, and cost. Catch regressions before they reach production — across Claude, GPT-4o, Copilot Studio, AWS Bedrock, and any HTTP agent.

npm install -g agentcheck
agentcheck run ./smoke/*.yaml --ci --report report.html

Quick start

# 1. Install
npm install -g agentcheck

# 2. Set your API key
export ANTHROPIC_API_KEY=sk-ant-...

# 3. Scaffold a test file
agentcheck init

# 4. Run
agentcheck run ./smoke/example.yaml

Writing tests

suite: Leave Request Agent
provider: anthropic          # anthropic | openai | copilot-studio | bedrock | http
model: claude-sonnet-4-6

defaults:
  timeout: 15000

tests:
  - name: Handles leave request
    retry: 2                 # retry up to 2× on failure
    messages:
      - role: user
        content: I need 3 days off starting Monday.
    assert:
      - not_empty: true
      - contains: "request"
      - contains_none: ["error", "I can't help"]
      - latency_ms: "<10000"
      - cost_usd: "<0.01"

  - name: Multi-turn context retained
    messages:
      - role: user
        content: My name is Alex.
      - role: assistant
        content: Nice to meet you, Alex!
      - role: user
        content: What is my name?
    assert:
      - contains: "Alex"

Assertion types

| Assertion | Example | Description | |---|---|---| | not_empty | not_empty: true | Output is not blank | | contains | contains: "hello" | Output includes this string (case-insensitive) | | contains_none | contains_none: ["error", "sorry"] | Output includes none of these strings | | equals | equals: "yes" | Exact match | | matches | matches: "(?i)order\|status" | Regex match | | latency_ms | latency_ms: "<5000" | Response time under limit | | cost_usd | cost_usd: "<0.01" | API cost under limit |

Providers

Anthropic (Claude)

provider: anthropic
model: claude-sonnet-4-6
# Env: ANTHROPIC_API_KEY

OpenAI (GPT)

provider: openai
model: gpt-4o
# Env: OPENAI_API_KEY

Microsoft Copilot Studio

provider: copilot-studio
config:
  secret: ${COPILOT_DIRECT_LINE_SECRET}
defaults:
  timeout: 45000   # allow time for Power Automate flows

Get your Direct Line secret from Copilot Studio → Publish → Channels → Direct Line.

Captures bot response text plus any Power Automate flow error events automatically.

AWS Bedrock Agents / Strands

provider: bedrock
config:
  agent_id: ${BEDROCK_AGENT_ID}
  agent_alias_id: ${BEDROCK_AGENT_ALIAS_ID}
  region: us-east-1
  # Optional: tail Lambda log groups after each test
  log_groups:
    - /aws/lambda/MyActionGroup
    - /aws/lambda/AnotherLambda

Full agent traces are captured automatically (enableTrace: true) — reasoning, tool calls, tool outputs, knowledge base hits, and failure reasons.

Generic HTTP

provider: http
config:
  url: ${AGENT_HTTP_URL}
  response_path: output.text   # dot-notation path into JSON response
  headers:
    X-Api-Key: ${AGENT_HTTP_API_KEY}

CLI reference

agentcheck run [files...]       Run test suites

  -m, --model <model>           Override model for all tests
  -p, --provider <provider>     Override provider for all tests
  --snapshot                    Compare outputs against golden snapshots
  --snapshot-update             Save current outputs as new golden snapshots
  --report [file]               Generate HTML report  (default: agentcheck-report.html)
  --junit  [file]               Generate JUnit XML    (default: agentcheck-results.xml)
  --notify                      Send Slack/Teams notification on completion
  --json                        Output raw JSON
  --ci                          Exit 1 on any failure (for CI pipelines)

agentcheck init                 Scaffold a starter smoke test file

Retry

Add retry: N to any test to retry up to N times on failure with exponential back-off (1s, 2s, 3s). Useful for non-deterministic agents.

tests:
  - name: Flaky flow test
    retry: 2
    messages: ...

Snapshots

# Save golden outputs
agentcheck run ./smoke/*.yaml --snapshot-update

# On next run, fail if output changed
agentcheck run ./smoke/*.yaml --snapshot

Snapshots are stored in .agentcheck-snapshots/ — commit them to git.

Notifications

Set env vars or add a .agentcheckrc file to your project root:

{
  "notify": {
    "slack": "https://hooks.slack.com/services/...",
    "teams": "https://your-org.webhook.office.com/...",
    "on": "failure"
  }
}

Or via environment:

export SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
export TEAMS_WEBHOOK_URL=https://your-org.webhook.office.com/...

Notifications fire automatically on failure when webhooks are configured. Use --notify to force notification regardless of result.

CI / CD

GitHub Actions

- name: Run smoke tests
  run: agentcheck run ./smoke/*.yaml --ci --report report.html --junit results.xml
  env:
    ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    COPILOT_DIRECT_LINE_SECRET: ${{ secrets.COPILOT_DIRECT_LINE_SECRET }}
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: agentcheck-report
    path: report.html

- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: test-results
    path: results.xml

Azure DevOps

- script: agentcheck run ./smoke/*.yaml --ci --report report.html --junit $(Build.ArtifactStagingDirectory)/results.xml

- task: PublishTestResults@2
  condition: always()
  inputs:
    testResultsFormat: JUnit
    testResultsFiles: '$(Build.ArtifactStagingDirectory)/results.xml'

- task: PublishBuildArtifacts@1
  condition: always()
  inputs:
    pathToPublish: report.html
    artifactName: agentcheck-report

Programmatic API

import { loadSuite, runSuite } from 'agentcheck';

const suite  = loadSuite('./smoke/my-agent.yaml');
const result = await runSuite(suite);

console.log(`${result.passed}/${result.passed + result.failed} passed`);
if (result.failed > 0) process.exit(1);

Environment variables

| Variable | Provider | Description | |---|---|---| | ANTHROPIC_API_KEY | anthropic | Claude API key | | OPENAI_API_KEY | openai | OpenAI API key | | COPILOT_DIRECT_LINE_SECRET | copilot-studio | Direct Line secret from Copilot Studio | | BEDROCK_AGENT_ID | bedrock | Bedrock Agent ID | | BEDROCK_AGENT_ALIAS_ID | bedrock | Bedrock Agent Alias ID | | AWS_REGION | bedrock | AWS region (default: us-east-1) | | AWS_ACCESS_KEY_ID | bedrock | AWS credentials | | AWS_SECRET_ACCESS_KEY | bedrock | AWS credentials | | AGENT_HTTP_URL | http | Agent REST endpoint | | AGENT_HTTP_API_KEY | http | Bearer token for HTTP agent | | SLACK_WEBHOOK_URL | notify | Slack Incoming Webhook URL | | TEAMS_WEBHOOK_URL | notify | Microsoft Teams Webhook URL |

License

MIT