mcp-probe

Conformance and robustness tests for MCP servers. Built to run in CI.

npx @beeeeen/mcp-probe -- node build/index.js

No install, no config, no dependencies. Point it at a server, get a verdict.

Why this exists

The official Inspector is a visual tool -- you click through it by hand. The official conformance suite runs in CI, but it tests servers over HTTP only, and it checks the spec, not quality.

Most published MCP servers are stdio processes launched by the host -- and stdio is where the deadliest failure mode lives, one no HTTP-based test can even observe:

console.log('Server started')   // <- stdout IS the protocol channel.

That one line corrupts the JSON-RPC stream. The host cannot parse it, so it disconnects — or worse, silently drops every message after it. No stack trace, no error, no log. The server "just doesn't work in Claude Desktop" and you spend an afternoon on it.

Every MCP client library discards bytes it cannot parse, which is why nothing reports this. mcp-probe keeps the discarded bytes and shows them to you.

That is one check out of 16, across roughly 30 distinct findings — all of them things that have shipped in real, published servers. Beyond spec conformance, mcp-probe also grades what the spec cannot: whether a model can actually use the server -- description quality, schemas whose required fields exist, argument validation that runs before side effects, and the token weight of the tool list.

What it finds

Run against a server with the usual set of mistakes:

  mcp-probe  bad-fixture
  node server.js  |  protocol 2025-06-18

  protocol conformance
    WARN  initialize returns serverInfo.version
          "bad-fixture" reports no version.
    PASS  Declared "tools" capability answers tools/list            5 tools
    FAIL  Declared "resources" capability answers resources/list
          resources/list returned no "resources" array.
    FAIL  Unknown method returns -32601
          Server answered a method that does not exist with a success result.

  tool schemas
    FAIL  Tool name is host-compatible (search files)
          "search files" contains characters hosts reject.
    FAIL  Tool names are unique (dup)
          "dup" is listed 2 times.
    FAIL  Tool has a description (mystery)
          No description.
    FAIL  required fields exist in properties (search files)
          required lists "directory", which is not in properties.
    PASS  Tool list does not dominate the context window            ~189 tokens across 5 tools

  robustness
    FAIL  Calling a tool that does not exist is rejected
          An unknown tool name returned a success result.
    FAIL  Missing required arguments are rejected (delete_everything)
          Ran with none of its 1 required argument and reported success.

  transport hygiene
    FAIL  stdout carries only JSON-RPC
          1 non-JSON line written to stdout.

  ----------------------------------------------------------------
  11 failed  |  8 warnings  |  9 passed  |  1 skipped   0.42s

Every failure comes with the offending payload and an explanation of what breaks:

  FAIL  stdout carries only JSON-RPC
        hygiene.stdout_purity  https://modelcontextprotocol.io/specification/2025-06-18/basic/transports
        1 non-JSON line written to stdout.
        stdout is the protocol channel for stdio transport. Every one of these
        lines corrupts the stream:

          > bad-server starting up...

        Fix: send all human-readable output to stderr (console.error, or a
        logger configured with stderr as its sink).

Usage

# stdio server
npx @beeeeen/mcp-probe -- node build/index.js
npx @beeeeen/mcp-probe -- npx -y @modelcontextprotocol/server-filesystem /tmp

# streamable HTTP
npx @beeeeen/mcp-probe --url http://localhost:3000/mcp

# a server you already have configured
npx @beeeeen/mcp-probe --config ~/.claude.json --server github

--config reads both the mcpServers shape (Claude Desktop, Claude Code, Cursor) and the servers shape (VS Code), so you can probe a server without retyping how it launches.

mcp-probe's own flags come first; everything after -- is the server's command line, untouched.

| flag | | |---|---| | --strict | treat warnings as failures | | --json / --json-out <file> | machine-readable report | | --junit <file> | JUnit XML for CI test reporting | | --only / --skip | select checks by id or group | | --verbose | expand the explanation for warnings | | --timeout <ms> | per-request timeout (default 10000) | | --call-tools | invoke tools — see Safety |

Exit codes: 0 clean · 1 findings · 2 could not run.

In CI

- uses: Beeeeen/mcp-probe@v0
  with:
    command: node build/index.js
    strict: true

Failures become inline annotations, and a table lands in the job summary. Outputs failures, warnings and report for later steps.

Or without the action:

- run: npx @beeeeen/mcp-probe --junit results.xml -- node build/index.js

What gets checked

Protocol conformance — the handshake returns a usable protocolVersion and serverInfo; declared capabilities are actually implemented and return the right shape; unknown methods produce -32601 rather than a crash or a fake success; ping answers; an older protocolVersion does not take the process down.

Tool schemas — names are unique and host-compatible; descriptions exist and are not TODO; inputSchema is a real JSON Schema rooted at type: object; required never names a field that is missing from properties; parameters carry descriptions; and the whole tool list is measured in tokens, because it is re-sent on every request and nothing else tells you what that costs.

Robustness — unknown tool names are rejected; omitting required arguments does not silently execute the tool anyway; wrongly typed arguments are not coerced into a wrong answer; malformed JSON-RPC on the wire does not wedge or kill the read loop; tool results are not large enough to evict the conversation.

Transport hygiene — stdout carries only JSON-RPC; stderr has no unhandled exceptions; the process is still alive at the end.

Every check has a stable dotted id (schema.input_schema.orphan_required), so --skip and --only work at any granularity, and a finding you have decided to live with can be silenced precisely.

Safety

mcp-probe does not invoke your tools unless you ask it to.

The default run only ever calls tools with invalid arguments — missing required fields, wrong types. A correct server rejects those at validation, before any side effect, and that rejection is exactly what is being measured. A server that performs work anyway is the bug being reported.

To measure real responses, opt in explicitly:

npx @beeeeen/mcp-probe --safe-tool list_directory -- node server.js   # just this one
npx @beeeeen/mcp-probe --call-tools -- node server.js                 # all of them

There is a delete_everything tool in the test fixtures for a reason.

Programmatic use

import { run, exitCodeFor } from 'mcp-probe'

const report = await run(
  { kind: 'stdio', command: 'node', args: ['build/index.js'] },
  { strict: true },
)

for (const r of report.results.filter((r) => r.status === 'fail')) {
  console.error(`${r.id}: ${r.message}`)
}
process.exit(exitCodeFor(report, true))

The check list is exported too, so you can run a subset or add your own:

import { allChecks, selectChecks } from 'mcp-probe'

Design notes

No runtime dependencies, and no MCP SDK. The JSON-RPC layer is hand-rolled on purpose. Client libraries normalise responses and throw away what they cannot parse — which is precisely the evidence a conformance tester needs. mcp-probe reads the raw bytes so it can report what the SDK would have hidden.

A broken server is a result, not an exception. A server that will not start, will not handshake, or dies mid-run produces a report saying so. CI can always render something.

Findings explain themselves. Each one carries the payload that triggered it, a link to the relevant part of the spec, and a sentence on what actually breaks. A finding you cannot act on is noise.

Contributing

New checks are welcome, especially ones drawn from a bug you actually hit. A check is one object in src/checks/, and the bar is:

• it must be actionable — the message says what to change
• it must not false-positive on the reference servers (@modelcontextprotocol/server-everything and server-filesystem are probed in CI)
• add the defect to fixtures/bad-server.js and assert on it in test/probe.test.js

npm install && npm run build && npm test

See also

The rest of the toolchain, built on the same zero-dependency MCP client:

• mcp-wtf -- your MCP server won't connect; find out why in 10 seconds.
• context-xray -- what every configured MCP server costs you in context-window tokens on every request.

mcp-wtf answers "why won't it connect", context-xray answers "what is it costing me", mcp-probe answers "will it break my users".

Newsletter

Agent Receipts -- weekly, first-hand numbers from building and shipping tools like this one: what I measured, what broke, and what I got wrong. Subscribe.

License

MIT