crosscheck

CI for MCP servers. Catch it when a schema change or a new AI-model release silently breaks your agent-facing tools — before your users' agents do.

⚡ 30-second quickstart

npx crosscheck init        # writes crosscheck.config.json + a sample task suite
#  → edit crosscheck.config.json and point "server" at your MCP server
npx crosscheck snapshot    # record your current tool contract (commit this file)
npx crosscheck check       # fail the build if a change would break agents

That's the whole contract-check loop. No API key, no cost — check is pure local schema diffing. Add behavioral evals with a free local model when you want them (see below).

What is this?

Your MCP server is a production API whose client is a stochastic model that changes every month, and today it ships with no tests. Two things break it silently:

1. You change your own code — rename a field, make an argument required, drop a tool. Every agent that learned the old shape now fails.
2. A new model version ships — same tools, but the new Claude/GPT/Gemini starts calling them wrong. You changed nothing; your success rate just dropped.

crosscheck is the missing test suite, in two layers:

• check — snapshots your tool schemas and fails CI on breaking drift (removed tools, new required params, type changes, narrowed enums, tightened additionalProperties). Local, instant, free.
• eval — drives real models through real tasks against your live server and grades what they did: which tools, which arguments, in what order. Catches behavioral regressions a schema check can't — including the "a new model broke us" case. Local-first (free via Ollama / LM Studio); cloud optional.

Why it matters

Companies are exposing their software to AI agents through MCP, and when those tools break, the author finds out from support tickets and churned agent traffic. crosscheck moves that failure left — into CI, where a red build stops it before it ships.

Configuration

{
  "server": { "command": "node", "args": ["dist/my-mcp-server.js"] },
  "snapshot": ".crosscheck/snapshot.json",
  "suites": ["suites"],
  "eval": { "models": ["ollama:qwen3"], "runs": 3, "passRate": 0.8 }
}

Remote servers work too: "server": { "url": "https://mcp.example.com/mcp", "headers": { "authorization": "Bearer ..." } }

Commands

crosscheck snapshot   # record the current tool contract (commit this file)
crosscheck check      # diff live contract vs snapshot — fails on breaking changes
crosscheck eval       # run task suites with real models against your server
crosscheck run        # check + eval: the full CI gate

Exit codes: 0 clean · 1 breaking change or eval below threshold · 2 config/usage error. Add --json to any command for machine-readable output.

Task suites

A suite is a YAML file of real jobs an agent should do with your tools:

suite: booking
tasks:
  - name: book a meeting
    prompt: "Book a 30-minute meeting with Alex Chen on 2026-07-21. Check availability first."
    expect:
      - tools_in_order: [list_availability, create_booking]
      - tool_called: { name: create_booking, with: { duration_minutes: 30 } }
      - tool_not_called: { name: cancel_booking }
      - answer_matches: "book"
      - max_tool_calls: 4

| Assertion | Meaning | |---|---| | tool_called: {name, with?} | tool was called; with matches args partially ({$regex}, {$exists} supported) | | tool_not_called: {name} | tool was never called | | tools_in_order: [a, b] | the named tools appear in this order (subsequence) | | answer_matches: "regex" | final answer matches (case-insensitive) | | max_tool_calls: n | the model didn't flail |

Each task runs runs times per model (models are non-deterministic — one run proves nothing); it passes when its pass rate meets passRate.

Models

| Spec | Cost | Notes | |---|---|---| | ollama:qwen3 | free | local via Ollama; any tool-calling model | | lmstudio:<model> | free | local via LM Studio's OpenAI-compatible server | | openai:<model>@http://localhost:8080/v1 | free | any OpenAI-compatible local server | | anthropic:claude-sonnet-5 | paid | needs ANTHROPIC_API_KEY | | openai:gpt-5.2 | paid | needs OPENAI_API_KEY | | google:gemini-2.5-flash | paid | needs GEMINI_API_KEY |

The signal is the matrix: run the same suite across models and you learn whether a failure is your server's regression or one model's quirk — and you catch it when a model release changes behavior your tools relied on.

Use in CI (GitHub Action)

# .github/workflows/crosscheck.yml
name: crosscheck
on: [push, pull_request]
jobs:
  contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm ci
      - run: npx crosscheck check   # breaking changes fail the build, annotated on the PR

Development

npm install
npm test          # unit + e2e (spins up the demo server over real stdio)
npm run build

License

MIT © 2026 Lakshay