@yawlabs/mcp-compliance

Test any MCP server for spec compliance. 88-test suite covering transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security against the MCP specification. Works against HTTP endpoints (https://my-server.com/mcp) and stdio servers (npx @modelcontextprotocol/server-filesystem /tmp) alike. CLI, MCP server, and programmatic API.

Built and maintained by Yaw Labs.

One click adds this to your local Yaw MCP config so it's available in every Yaw Terminal session. Or install manually below.

Why this tool?

MCP servers are multiplying fast — but most ship without compliance testing. Broken transport handling, missing error codes, malformed schemas, and silent capability violations are common. Hand-rolling test scripts is tedious and incomplete.

This tool solves that:

• 88 tests across 8 categories — transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security. No gaps. (HTTP runs all 85 transport-applicable tests; stdio runs ~75 — HTTP-specific tests like CORS, TLS, session headers, and rate limiting are gated out.)
• Capability-driven — tests adapt to what the server declares. If it says it supports tools, tool tests become required. No false failures for features the server doesn't claim.
• Graded scoring — A-F letter grade with a weighted score (required tests 70%, optional 30%). One number to communicate compliance.
• CI-ready — --strict mode exits with code 1 on required test failures. Drop it into any pipeline.
• Spec-referenced — every test links to the exact section of the MCP specification it validates. No ambiguity about what's being tested or why.
• Three interfaces — CLI for humans, MCP server for AI assistants, programmatic API for integration.
• Published methodology — the testing methodology and rule catalog are open (CC BY 4.0) so anyone can build compatible tooling or fork the rules.

Quick start

Remote HTTP server:

npx @yawlabs/mcp-compliance@latest test https://my-server.com/mcp

Local stdio server (the vast majority of MCP servers on npm):

# Pass the command directly, Inspector-style
npx @yawlabs/mcp-compliance@latest test npx @modelcontextprotocol/server-filesystem /tmp

# Or a local build
npx @yawlabs/mcp-compliance@latest test node ./dist/server.js

# With env vars
npx @yawlabs/mcp-compliance@latest test -E GITHUB_TOKEN=$GITHUB_TOKEN -- npx @modelcontextprotocol/server-github

Install globally:

npm install -g @yawlabs/mcp-compliance
mcp-compliance test https://my-server.com/mcp

That's it. You'll get a colored terminal report with a letter grade (A-F), per-test pass/fail, and a compliance score.

CLI usage

HTTP targets

# Terminal output with colors and grade
mcp-compliance test https://my-server.com/mcp

# JSON / SARIF for scripting + GitHub Code Scanning
mcp-compliance test https://my-server.com/mcp --format json
mcp-compliance test https://my-server.com/mcp --format sarif > compliance.sarif

# Strict mode for CI — exits 1 on required-test failure
mcp-compliance test https://my-server.com/mcp --strict

# Auth (shorthand or full header)
mcp-compliance test https://my-server.com/mcp --auth "Bearer tok123"
mcp-compliance test https://my-server.com/mcp -H "X-Api-Key: abc"

# Focus the run
mcp-compliance test https://my-server.com/mcp --only transport,lifecycle
mcp-compliance test https://my-server.com/mcp --skip prompts,resources
mcp-compliance test https://my-server.com/mcp --verbose

stdio targets

Pass the command and its args as positional arguments (MCP Inspector-style). Use -- to disambiguate when the target needs flags that collide with ours.

# npm-distributed stdio servers
mcp-compliance test npx -y @modelcontextprotocol/server-filesystem /tmp
mcp-compliance test uvx mcp-server-git

# Local build
mcp-compliance test node ./dist/server.js

# With env vars (repeatable -E, or --env-file)
mcp-compliance test -E API_KEY=secret -E REGION=us-east-1 -- npx my-server
mcp-compliance test --env-file .env -- node ./server.js

# Set working directory
mcp-compliance test --cwd ./services/mcp -- node ./dist/server.js

# Target uses a flag that collides with ours — use `--` to separate
mcp-compliance test --verbose -- node ./server.js --verbose

On Windows, npx and other .cmd shims are handled automatically by spawning through the shell.

Options

| Option | Applies to | Description | |--------|-----------|-------------| | --format <format> | both | Output format: terminal, json, sarif, github, or markdown (default: terminal) | | --config <path> | both | Load defaults from a config file (default: mcp-compliance.config.json in cwd) | | --output <file> | both | Write a local SVG badge to the given path after the run | | --list | both | Print test IDs that would run given current filters, then exit (no connection) | | --transport <kind> | both | Filter by http or stdio (only used with --list when no target is provided) | | --strict | both | Exit with code 1 on any required test failure (for CI) | | --min-grade <grade> | both | Exit with code 1 if grade is below this threshold (A–F) | | -H, --header <h> | HTTP | Add header to all requests, format "Key: Value" (repeatable) | | --auth <token> | HTTP | Shorthand for -H "Authorization: <token>" | | -E, --env <var> | stdio | Set env var for stdio command, format "KEY=VALUE" (repeatable) | | --env-file <path> | stdio | Load env vars from a file (one KEY=VALUE per line) | | --cwd <dir> | stdio | Working directory for the stdio command | | --timeout <ms> | both | Request timeout in milliseconds (default: 15000) | | --preflight-timeout <ms> | HTTP | Preflight connectivity check timeout (HTTP only) | | --retries <n> | both | Number of retries for failed tests (default: 0) | | --only <items> | both | Only run tests matching these categories or test IDs (comma-separated) | | --skip <items> | both | Skip tests matching these categories or test IDs (comma-separated) | | --concurrency <n> | both | Max parallel-safe tests in flight (default: 1; raising reduces wall time but can perturb timing-sensitive servers) | | --verbose | both | Print each test result as it runs (also forwards stdio stderr) |

CI integration

GitHub Action (drop into any .github/workflows/*.yml):

- uses: YawLabs/mcp-compliance@v0
  with:
    target: 'node ./dist/server.js'   # or a URL like https://my-server.com/mcp
    format: github                     # ::error / ::warning annotations on the PR
    strict: 'true'                     # exit non-zero if any required test fails
    min-grade: 'A'                     # also exit if grade slips

Manual CLI invocation:

# GitHub Actions: emits ::error / ::warning annotations inline on the PR
mcp-compliance test https://my-server.com/mcp --format github --strict

# Slack/Linear/PR comment: drop the body straight into a comment
mcp-compliance test https://my-server.com/mcp --format markdown > report.md

# HTML report (self-contained, share anywhere — issue comments, S3, GitHub Pages)
mcp-compliance test https://my-server.com/mcp --format html > report.html

# Block release if grade slips below B
mcp-compliance test https://my-server.com/mcp --min-grade B

# Preview which tests will run before connecting (handy for --only/--skip authoring)
mcp-compliance test --list --transport stdio --skip security

# Diff two runs — exit 1 if anything that was passing is now failing
mcp-compliance test https://my-server.com/mcp --format json > current.json
mcp-compliance diff baseline.json current.json

# Watch mode for stdio dev loop — re-runs on file changes in cwd
mcp-compliance test --watch -- node ./dist/server.js

# Latency benchmark
mcp-compliance benchmark -- node ./dist/server.js -r 200 -c 4

Docker:

docker run --rm ghcr.io/yawlabs/mcp-compliance test https://my-server.com/mcp

Scaffold a config

mcp-compliance init

Interactive prompts walk you through transport (http/stdio), command/url, env vars, timeout, and strict mode — then write a mcp-compliance.config.json you can commit.

Config file

Check in a mcp-compliance.config.json so CI and your dev loop can run mcp-compliance test with no arguments. Supported locations (searched in order): mcp-compliance.config.json, .mcp-compliancerc.json, .mcp-compliancerc, and the "mcp-compliance" field of package.json. Pass --config <path> to load an explicit file.

HTTP:

{
  "target": {
    "type": "http",
    "url": "https://my-server.com/mcp",
    "headers": { "Authorization": "Bearer tok123" }
  },
  "timeout": 20000,
  "strict": true
}

stdio:

{
  "target": {
    "type": "stdio",
    "command": "node",
    "args": ["./dist/server.js"],
    "env": { "LOG_LEVEL": "error" }
  },
  "skip": ["security"],
  "strict": true
}

Precedence: CLI flags > config file > defaults. Any field can be overridden on the command line.

Local SVG badge

Write a local SVG reflecting the real grade and commit it to your repo:

mcp-compliance test https://my-server.com/mcp --output badge.svg
mcp-compliance test node ./dist/server.js --output badge.svg

Then embed it in your README:

![MCP Compliance](./badge.svg)

What the 88 tests check

HTTP-only (13):

• transport-post — Server accepts HTTP POST requests (required)
• transport-content-type — Responds with application/json or text/event-stream (required)
• transport-notification-202 — Notifications return exactly 202 Accepted
• transport-content-type-reject — Rejects non-JSON request Content-Type
• transport-session-id — Enforces MCP-Session-Id after initialization
• transport-session-invalid — Returns 404 for unknown session ID
• transport-get — GET returns SSE stream or 405
• transport-delete — DELETE accepted or returns 405
• transport-batch-reject — Rejects JSON-RPC batch requests (required)
• transport-content-type-init — Initialize response has valid content type
• transport-get-stream — GET with session returns SSE or 405
• transport-concurrent — Handles concurrent requests
• transport-sse-event-field — SSE responses include required event: message field

stdio-only (3):

• stdio-framing — Newline-delimited JSON framing (required)
• stdio-unicode — UTF-8 unicode roundtrip preserves non-ASCII payloads
• stdio-unknown-method-recovers — Returns -32601 for unknown methods and keeps serving

• lifecycle-init — Initialize handshake succeeds (required)
• lifecycle-proto-version — Returns valid YYYY-MM-DD protocol version (required)
• lifecycle-server-info — Includes serverInfo with name
• lifecycle-capabilities — Returns capabilities object (required)
• lifecycle-jsonrpc — Response is valid JSON-RPC 2.0 (required)
• lifecycle-ping — Responds to ping method (required)
• lifecycle-instructions — Instructions field is valid string if present
• lifecycle-id-match — Response ID matches request ID (required)
• lifecycle-string-id — Supports string request IDs (JSON-RPC 2.0)
• lifecycle-version-negotiate — Handles unknown protocol version gracefully
• lifecycle-reinit-reject — Rejects second initialize request
• lifecycle-logging — logging/setLevel accepted (required if logging capability declared)
• lifecycle-completions — completion/complete accepted (required if completions capability declared)
• lifecycle-cancellation — Handles cancellation notifications
• lifecycle-progress — Handles progress notifications gracefully
• lifecycle-list-changed — Accepts listChanged notifications for declared capabilities
• lifecycle-progress-token — Supports progress tokens in requests via SSE
• lifecycle-sampling-capability — Advisory check for server-side use of the client sampling capability
• lifecycle-roots-capability — Advisory check for server-side use of the client roots capability
• lifecycle-elicitation-capability — Advisory check for the 2025-11-25 client elicitation capability
• lifecycle-meta-tolerance — Server ignores unknown _meta fields on incoming requests

• tools-list — tools/list returns valid array (required if tools capability declared)
• tools-call — tools/call responds with correct format
• tools-pagination — tools/list supports cursor-based pagination
• tools-content-types — Tool content items have valid types

• resources-list — resources/list returns valid array (required if resources capability declared)
• resources-read — resources/read returns content items
• resources-templates — resources/templates/list works or returns method-not-found
• resources-pagination — resources/list supports cursor-based pagination
• resources-subscribe — Resource subscribe/unsubscribe (required if subscribe capability declared)

• prompts-list — prompts/list returns valid array (required if prompts capability declared)
• prompts-get — prompts/get returns valid messages
• prompts-pagination — prompts/list supports cursor-based pagination

• error-unknown-method — Returns JSON-RPC error for unknown method (required)
• error-method-code — Uses correct -32601 error code
• error-invalid-jsonrpc — Handles malformed JSON-RPC (required)
• error-invalid-json — Handles invalid JSON body
• error-missing-params — Returns error for tools/call without name
• error-parse-code — Returns -32700 for invalid JSON
• error-invalid-request-code — Returns -32600 for invalid request
• tools-call-unknown — Returns error for nonexistent tool name
• error-capability-gated — Rejects methods for undeclared capabilities
• error-invalid-cursor — Handles invalid pagination cursor gracefully

• tools-schema — All tools have valid name and inputSchema (required if tools capability declared)
• tools-annotations — Tool annotations are valid if present
• tools-title-field — Tools include title field (2025-11-25)
• tools-output-schema — Tools with outputSchema are valid (2025-11-25)
• prompts-schema — Prompts have valid name field (required if prompts capability declared)
• resources-schema — Resources have valid uri and name (required if resources capability declared)

• security-auth-required — Rejects unauthenticated requests
• security-www-authenticate — 401 responses include WWW-Authenticate header
• security-auth-malformed — Rejects malformed auth credentials
• security-tls-required — Enforces HTTPS/TLS
• security-session-entropy — Session IDs are high-entropy
• security-session-not-auth — Session ID does not bypass auth
• security-oauth-metadata — Protected Resource Metadata endpoint exists (RFC 9728)
• security-token-in-uri — Rejects auth tokens in query string
• security-cors-headers — CORS headers are restrictive
• security-origin-validation — Validates Origin header for DNS rebinding protection
• security-command-injection — Resists command injection in tool params
• security-sql-injection — Resists SQL injection in tool params
• security-path-traversal — Resists path traversal in tool params
• security-ssrf-internal — Resists SSRF to internal networks
• security-oversized-input — Handles oversized inputs gracefully
• security-extra-params — Rejects or ignores extra tool params
• security-tool-schema-defined — All tools define inputSchema
• security-tool-rug-pull — Tool definitions are stable across calls
• security-tool-description-poisoning — Tool descriptions free of injection patterns
• security-tool-cross-reference — Tools do not reference other tools by name
• security-error-no-stacktrace — Error responses do not leak stack traces
• security-error-no-internal-ip — Error responses do not leak internal IPs
• security-rate-limiting — Rate limiting is enforced

Grading

| Grade | Score | |-------|--------| | A | 90-100 | | B | 75-89 | | C | 60-74 | | D | 40-59 | | F | 0-39 |

Required tests are worth 70% of the score, optional tests 30%. See the full scoring algorithm in the methodology doc.

CI integration

# GitHub Actions example
- name: MCP Compliance Check
  run: npx @yawlabs/mcp-compliance@latest test ${{ env.MCP_SERVER_URL }} --strict

# With JSON output for parsing
- name: MCP Compliance Check
  run: |
    npx @yawlabs/mcp-compliance@latest test ${{ env.MCP_SERVER_URL }} --format json > compliance.json
    cat compliance.json | jq '.grade'

# With retries for flaky network conditions
- name: MCP Compliance Check
  run: npx @yawlabs/mcp-compliance@latest test ${{ env.MCP_SERVER_URL }} --strict --retries 2 --timeout 30000

# SARIF output for GitHub Code Scanning
- name: MCP Compliance Check
  run: npx @yawlabs/mcp-compliance@latest test ${{ env.MCP_SERVER_URL }} --format sarif > compliance.sarif
- name: Upload SARIF
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: compliance.sarif

MCP server (for Claude Code, Cursor, etc.)

This package also exposes an MCP server with 3 tools that can be used from Claude Code, Cursor, or any MCP client.

Setup

Claude Code (one-liner):

claude mcp add mcp-compliance -- npx -y @yawlabs/mcp-compliance@latest mcp

Or create .mcp.json in your project root:

macOS / Linux / WSL:

{
  "mcpServers": {
    "mcp-compliance": {
      "command": "npx",
      "args": ["-y", "@yawlabs/mcp-compliance@latest", "mcp"]
    }
  }
}

Windows:

{
  "mcpServers": {
    "mcp-compliance": {
      "command": "cmd",
      "args": ["/c", "npx", "-y", "@yawlabs/mcp-compliance@latest", "mcp"]
    }
  }
}

Tip: This file is safe to commit — it contains no secrets.

Restart your MCP client and approve the server when prompted.

Tools

• mcp_compliance_test — Run the full 88-test suite against a URL or stdio command. Supports auth, custom headers, env vars, timeout, retries, and category/test filtering. Returns grade, score, and detailed results.
• mcp_compliance_explain — Explain what a specific test ID checks and why it matters.

All tools have MCP tool annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint) so MCP clients can skip confirmation dialogs for safe operations.

Programmatic usage

import { runComplianceSuite } from '@yawlabs/mcp-compliance';

const report = await runComplianceSuite('https://my-server.com/mcp');
console.log(`Grade: ${report.grade} (${report.score}%)`);

// With options
const report2 = await runComplianceSuite('https://my-server.com/mcp', {
  headers: { 'Authorization': 'Bearer tok123' },
  timeout: 30000,
  retries: 1,
  only: ['transport', 'lifecycle'],
});

// Live progress for streaming UIs (e.g. server-sent-events to a browser)
await runComplianceSuite('https://my-server.com/mcp', {
  onTestComplete: (result) => {
    // result has the full TestResult: id, name, category, required,
    // passed, details, durationMs, specRef. Push it to your client.
    sendToClient(result);
  },
});

Report schema

The JSON output of the test suite is a stable, versioned contract. Every report includes a schemaVersion field at the top level. The full JSON Schema lives at schemas/report.v1.json and is shipped with the npm package.

{
  "schemaVersion": "1",        // bumped on breaking changes to the report shape
  "specVersion": "2025-11-25", // MCP spec version tested against
  "toolVersion": "0.10.0",     // mcp-compliance version that produced the report
  "url": "...",
  "timestamp": "...",
  "grade": "A",
  "score": 92.5,
  "tests": [ ... ],
  // ...
}

Consumer guidance:

• Pin against schemaVersion. Reject reports with an unknown version rather than guessing at the shape.
• The schema validates with any Draft 2020-12 validator (e.g. ajv).
• Within a major version, additions are non-breaking. Renames, removals, or type changes bump the version.
• Two runs against the same server produce equivalent grade, score, and per-test pass/fail (modulo timings/timestamps).

Methodology & docs

The testing methodology is published openly so the grading is auditable:

• Testing methodology — test execution model, scoring algorithm, all 88 test rules with pass/fail criteria (CC BY 4.0)
• Machine-readable rule catalog — JSON Schema-compliant catalog for programmatic consumption
• Why mcp-compliance — the problem, existing alternatives, what this tool does differently
• Fixing common failures — recipes for the most frequent test failures with code snippets
• Spec version migration policy — how this tool evolves with MCP spec releases
• Performance deep-dive — why the suite is sequential and what parallel execution would cost
• Spec PR drafts — our proposed MCP spec clarifications for ambiguous cases we've hit

The methodology is not an authoritative conformance standard — it's one tool's choices, published so they can be inspected, adopted, or forked. The official MCP specification defines what servers must do; this document describes how @yawlabs/mcp-compliance verifies it.

Requirements

• Node.js 18+

Contributing

git clone https://github.com/YawLabs/mcp-compliance.git
cd mcp-compliance
npm install
npm run build
npm test

Development commands:

| Command | Description | |---------|-------------| | npm run build | Compile with tsup | | npm run dev | Watch mode | | npm test | Run tests (vitest) | | npm run lint | Check with Biome | | npm run lint:fix | Auto-fix with Biome | | npm run typecheck | TypeScript type checking | | npm run test:ci | Build + test (CI-safe) |

Links

• MCP Specification
• Testing methodology
• Yaw Labs

License

MIT — see LICENSE.