You built an MCP server. You tested it by clicking around the Inspector — once. Then you changed a tool description, shipped it, and found out from users that the server hangs on startup.

The Inspector is a great GUI and a terrible regression suite. It can't run in CI, it can't assert anything, and it won't tell you that your npx entry is missing -y or that two tools have identical descriptions so the model picks between them at random.

mcpcanary is the missing command line:

npx mcpcanary probe -- npx -y @modelcontextprotocol/server-everything   # does it even start?
npx mcpcanary init -- node ./build/server.js                            # scaffold a drill file
npx mcpcanary run drills/                                               # do the tools still work?
npx mcpcanary lint --probe                                              # will clients choke on my config?

Why you want this

• Probe replaces "let me open the Inspector and see". One command answers: does it start, what protocol version, what capabilities, how many tools, are any descriptions empty? Exit code tells CI the truth.
• Drills are YAML, not code. A tool call plus assertions in six lines. No test framework to wire up, no client library to learn.
• Lint catches the failure modes nobody warns you about: npx without -y hanging every headless client, mcpservers instead of mcpServers silently ignored, hardcoded API keys, remote servers over plain http://, prompt-injection phrasing in third-party tool descriptions, duplicate tool descriptions…
• Built for CI first: stable exit codes, --json output, GitHub Actions annotations (::error) out of the box.

Install

npm install -D mcpcanary        # as a dev dependency
# or just run it
npx mcpcanary --help

Node ≥ 20.

Quick start

# 0. Let init write your first drill file from a live probe:
mcpcanary init -- node ./build/server.js

created mcp.drill.yaml with 6 drill(s) scaffolded from your server
next: review the TODO expectations, then run:
  mcpcanary run mcp.drill.yaml

init lists your real tool names, scaffolds arguments from each tool's input schema, and leaves clearly-marked TODO expectations for you to tighten. (If you skip this step, the YAML below is everything there is.)

# 1. Does my server start and what does it expose?
mcpcanary probe -- node ./build/server.js

tiny-fixture @ 1.2.3 — connected in 104ms
capabilities: tools, resources

tools (2)
  · add — Adds two numbers and returns the sum.
  · echo — Echoes back whatever it receives.

# 2. Write a drill file…

# mcp.drill.yaml
server:
  command: node
  args: [./build/server.js]

timeoutMs: 10000

drills:
  - name: adds numbers
    tool: add
    arguments: { a: 2, b: 3 }
    expect:
      textEquals: "5"
      latencyMsAtMost: 2000

  - name: errors surface properly
    tool: divide
    arguments: { a: 1, b: 0 }
    expect:
      isError: true
      textContains: zero

  - name: structured JSON is intact
    tool: stats
    expect:
      jsonPath: $.total
      jsonEquals: 42

  - name: the toolbox is complete
    listTools:
      contains: [add, divide, stats]

# 3. Run it (add it to `npm test`)
mcpcanary run mcp.drill.yaml

mcp.drill.yaml — [email protected]
  ✓ adds numbers 3ms
  ✓ errors surface properly 1ms
  ✓ structured JSON is intact 2ms
  ✓ the toolbox is complete

ok: 1 file, 4 drills, 4 passed, 118ms

When something breaks you get the assertion, the actual value, and the server's last stderr lines — which is usually the whole debugging session:

  ✗ bad math
    · text equals "3" — got "2"

ok: …

  ✗ unreachable
    server failed to start or respond: connect ECONNREFUSED
stderr tail:
  | boom: missing OPENAI_API_KEY

What lint checks

mcpcanary lint scans .mcp.json, .cursor/mcp.json, .vscode/mcp.json, Claude Desktop / Claude Code configs and server.json manifests it finds.

| Rule | Severity | Catches | |------|----------|---------| | C001 | error | mcpservers / MCP_SERVERS key typos that clients silently ignore | | C002 | info | config files with no server entries | | C003–C004 | error | malformed entries; neither command nor url | | C005 | error | npx/bunx/uvx without -y — startup hangs at the install prompt in every headless client | | C006 | warning | relative paths that only work on your machine | | C007–C008 | warn/error | Windows cmd /c pitfalls; malformed env | | C009 | error | hardcoded API keys (sk-…, AWS, GitHub, Slack) in config files | | C010–C011 | error/warning | invalid URLs; remote servers over plain http:// | | M001–M007 | error/warning | registry manifest problems: missing fields, non-reverse-DNS names, invalid npm identifiers | | D001–D008 | error/warning | live tool audits (--probe): missing descriptions, name-only descriptions, duplicate descriptions across tools, invalid tool names, oversized context-burning descriptions, prompt-injection phrasing |

Add it to review CI and stop bad configs at the PR:

mcpcanary lint --probe

Drills reference

A drill file has an optional server: block (stdio via command/args/env/cwd, or HTTP via url/headers), an optional default timeoutMs, and a drills: list.

| Drill type | Fields | Asserts on | |------------|--------|------------| | tool call | tool, arguments, expect | result content | | listing | listTools / listResources / listPrompts | advertised names |

Expectations for tool calls:

| Key | Meaning | |-----|---------| | textContains | string or array of strings, all must appear | | textEquals | exact (trimmed) text match | | textMatches | regular expression source | | isError | whether the tool result is an error | | jsonPath (+ jsonEquals / jsonContains / jsonExists) | JSONPath into structuredContent, falling back to JSON embedded in text output | | latencyMsAtMost | wall-clock budget for the call |

Listing expectations: contains, excludes, countExactly, countAtLeast, countAtMost.

Exit codes: 0 everything passed · 1 failures/errors · 2 usage problem (nothing to run, unreadable file).

GitHub Actions

Annotations are emitted automatically when GITHUB_ACTIONS=true; failures show up inline on the PR:

name: mcp-drills
on: [push, pull_request]

jobs:
  drill:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci && npm run build
      - run: npx mcpcanary run drills/
      - run: npx mcpcanary lint --probe

How it compares

| | Inspector (official) | GUI test platforms | mcpcanary | |---|---|---|---| | Runs in CI | ✗ (interactive GUI) | partial | ✓ | | Declarative assertions | ✗ | some | ✓ YAML | | Config linting (npx -y, secrets, typos) | ✗ | ✗ | ✓ | | Tool-description audit | manual | manual | ✓ automated | | Setup | open app | account/API keys | one npm package, zero config | | Output for bots | ✗ | ✗ | --json + GH annotations |

mcpcanary doesn't replace the Inspector for interactive exploration — it replaces re-opening the Inspector every time you change something.

Roadmap

See ROADMAP.md for the full plan. Highlights:

• v0.2 — init scaffolding ✅, standalone GitHub Action, --watch, session reuse
• v0.3 — snapshot testing, SSE transport, schema linting depth, token-cost report
• v0.4 — registry health sweep, pass/fail badges, preset drill packs

PRs welcome — see CONTRIBUTING.md.

License

MIT