MCP Doctor

The easiest way to score MCP agent readiness.

Open-source CLI — inspect, benchmark, and eval MCP servers before agents hit production.

"Prove that models can execute your MCP, then see where schemas create friction."

Real benchmark results

State of MCP Quality 2026 (v0) — historical 2026-07-10 snapshot: 10 public servers attempted, 5 connected and scored. It is schema-readiness evidence, not behavioral proof:

| Server | Grade | Tools | Tokens | |--------|-------|-------|--------| | MCP Filesystem | A | 14 | 1,997 | | MCP Sequential Thinking | A | 1 | 996 | | MCP Everything | A | 13 | 1,236 | | MCP Memory | A | 9 | 1,040 | | MCP Puppeteer | B | 7 | 612 |

Per-server reports: examples/reports/

npx --yes --package @coefficient-work/[email protected] mcp-doctor benchmark -o ./reports

Public package: @coefficient-work/mcp-doctor. Do not install the unrelated unscoped mcp-doctor package.

Three pillars

| Pillar | Command | Status | |--------|---------|--------| | Static scorecard | inspect, test | v0.4 | | Execution proof | eval (BYOK) | v0.4 | | Agent friction | included in eval | v0.4 |

Plus: Recommended Improvements, Replay Timeline, Model Compatibility Matrix (multi-model eval).

Quick start

Node 22 or Node 24 LTS is recommended. MCP Doctor 0.4.7 remains compatible with Node 20+, but Node 20 is no longer maintained by the Node.js project.

# Benchmark public MCPs
npx --yes --package @coefficient-work/[email protected] mcp-doctor benchmark

# Inspect your Cursor MCP server (also reads ./mcp.json)
npx --yes --package @coefficient-work/[email protected] mcp-doctor list
npx --yes --package @coefficient-work/[email protected] mcp-doctor inspect <name> -o report.md

# Cross-provider agent eval (BYOK — credential values stay local)
export OPENROUTER_API_KEY=...
npx --yes --package @coefficient-work/[email protected] mcp-doctor eval memory \
  --task "List all tools and describe them" \
  --models openrouter/openai/gpt-5.6-sol,openrouter/anthropic/claude-sonnet-5,openrouter/google/gemini-3.7-flash \
  -o eval-report.md

Portable eval credentials (macOS/Linux)

For evals that run from temporary sandboxes or on more than one Mac, keep the same private file at ~/.config/mcp-doctor/evaluation.env on each machine:

install -d -m 700 ~/.config/mcp-doctor
install -m 600 examples/evaluation.env.example ~/.config/mcp-doctor/evaluation.env

Edit that file and add only the provider keys you use. The CLI never overrides an already exported environment variable. Without an explicit file, precedence is ./.env.local followed by ~/.config/mcp-doctor/evaluation.env. Select a different private file with mcp-doctor eval --env-file /path/to/evaluation.env or MCP_DOCTOR_ENV_FILE; explicitly selected files must exist and be mode 600. Only known model-provider variables are loaded, and values are parsed as data — not executed as shell code.

OpenRouter uses its OpenAI-compatible endpoint through the Vercel AI SDK. Prefix an OpenRouter-routed model with openrouter/, followed by the normal OpenRouter model ID:

export OPENROUTER_API_KEY=...
npx --yes --package @coefficient-work/[email protected] mcp-doctor eval memory \
  --task "List all tools and describe them" \
  --models openrouter/openai/gpt-5.6-sol,openrouter/anthropic/claude-sonnet-5,openrouter/google/gemini-3.7-flash \
  -o eval-report.md

eval proves execution only when at least one MCP tool returns a non-error result. It does not formally prove that an arbitrary natural-language task was semantically completed. Credential values are not stored or printed, but eval necessarily sends the task, tool schemas, calls, and tool results to the selected model provider. inspect is local except for connecting to the MCP endpoint; benchmark launches or contacts every server in its catalog.

AI Gateway setup (one-time)

vercel login
cd mcp-doctor && vercel link
vercel ai-gateway api-keys create --name mcp-doctor-local --budget 5 --refresh-period monthly
# Key saved to .env.local (gitignored) — CLI auto-loads it for eval

Pain Interview (before sending to friends)

See docs/FRIEND-GUIDE.md.

Do not ask friends to beta-test until you've run a 30-min workflow interview.

Commands

| Command | Description | |---------|-------------| | benchmark | State of MCP Quality — score catalog servers (writes files only with --out) | | list | MCP servers in ./mcp.json or ~/.cursor/mcp.json | | inspect <name> | Live connect + scorecard + suggested fixes (missing inputSchema -> Grade F, exit 2) | | eval <name> --task "..." | BYOK execution proof + friction + replay (OpenRouter, OpenAI, Anthropic, Gateway, or Ollama) | | test --demo | Static scorecard on OpenAPI fixture | | build --demo --out <dir> | Write optimized MCP tool bundle (requires --out) | | competitors | Adjacent MCP tooling map |

Run npm run prepublish-gate before every publish. Run npm run blind-eval when you want a no-context agent pass against a packed tarball.

Roadmap

| Priority | Feature | Status | |----------|---------|--------| | 1 | Public benchmark reports | v0.4 | | 2 | Pain Interview (Jonty / Anders) | In progress | | 3 | BYOK eval + friction | v0.4 | | 4 | Suggested fixes | v0.4 | | 5 | Model matrix | v0.4 | | 6 | Scale to 50 MCPs + awards | Next | | 9 | OpenAPI drift | Deprioritized |

License

MIT