mcp-protocol-conformance

Conformance test harness for Model Context Protocol servers. Validates JSON-RPC 2.0 wire compliance, spec-version handshake, transport behaviour, OAuth 2.1 PKCE, tool schemas, capability advertisement, smoke roundtrip, and annotation hygiene against MCP spec 2024-11-05, 2025-03-26, and 2025-06-18.

This is a Foundation build of the StudioMeyer MCP Factory: every other Factory build runs through this harness before promotion (npm publish, marketplace submit, upstream PR).

Install

npm install --save-dev mcp-protocol-conformance

The package ships both a CLI (mcp-conformance) and a TypeScript library entry (import ... from "mcp-protocol-conformance").

CLI usage

Run against a stdio server

mcp-conformance run \
  --target stdio \
  --cmd node \
  --cmd-arg dist/server.js \
  --spec 2025-06-18 \
  --suite all \
  --format terminal

Run against an HTTP server (Streamable HTTP, 2025-03-26+)

mcp-conformance run \
  --target http \
  --url https://memory.studiomeyer.io/mcp \
  --header "Authorization:Bearer ${TOKEN}" \
  --spec 2025-06-18 \
  --suite full \
  --format json --out report.json

Run only a subset of suites

mcp-conformance run --target stdio --cmd ./server --spec 2025-06-18 \
  --suite jsonrpc,version,schema

Compare two manifests

mcp-conformance compare \
  --expected manifests/v1.json \
  --actual   manifests/v2.json

Assert no breaking changes between two reports

mcp-conformance assert-no-breaking \
  --baseline reports/main.json \
  --current  reports/pr-42.json

Exit codes: 0 clean, 1 failures, 2 invocation error.

Library usage

import {
  runFullSuite,
  generateReport,
} from "mcp-protocol-conformance";

const report = await runFullSuite(
  { kind: "stdio", cmd: "node", args: ["dist/server.js"] },
  "2025-06-18",
  { suite: "all" },
);

console.log(generateReport(report, "terminal"));
if (report.status === "fail") process.exit(1);

MCP server usage

The harness is itself an MCP server. Start it over stdio and any MCP client can call its 12 tools:

node dist/server.js

Tools (all read-only, all destructiveHint: false):

| # | Tool | Purpose | |---|------|---------| | 1 | runJsonRpcCompliance | JSON-RPC 2.0 error-code suite | | 2 | runSpecVersionAssertion | Verify advertised protocolVersion | | 3 | runTransportSuite | Transport-layer ping + session-id | | 4 | runOauthPkceFlow | OAuth 2.1 PKCE S256 (mock-AS or real-tenant) | | 5 | runToolSchemaValidation | inputSchema is valid JSON-Schema | | 6 | runCapabilityIntrospection | initialize.capabilities matches behaviour | | 7 | runRoundtripSmoke | One tools/call per advertised tool | | 8 | runAnnotationsAudit | readOnlyHint / destructiveHint hygiene | | 9 | runFullSuite | All suites + summary | | 10 | generateReport | Render JUnit / JSON / terminal | | 11 | compareManifests | Diff two tool manifests | | 12 | assertNoBreakingChanges | Diff two FullReports |

Compatibility matrix

| | 2024-11-05 | 2025-03-26 | 2025-06-18 | |--------------------|:----------:|:----------:|:----------:| | jsonrpc | yes | yes | yes | | version (handshake)| yes | yes | yes | | transport (stdio) | yes | yes | yes | | transport (http) | n/a | yes | yes | | oauth (mock) | n/a | yes | yes | | oauth (real) | n/a | yes | yes | | schema | yes | yes | yes | | capability | yes | yes | yes | | smoke | yes | yes | yes | | annotations | warn-only | warn-only | yes |

yes = suite runs and produces actionable results. n/a = capability not in spec; suite skips automatically. warn-only = suite runs but the spec does not formally require the feature.

Integration in Factory builds

In every Factory build's package.json:

{
  "scripts": {
    "factory:conformance": "mcp-conformance run --target stdio --cmd 'node dist/server.js' --spec 2025-06-18 --suite full"
  }
}

In CI:

- run: npm run factory:conformance

Spec references

• JSON-RPC 2.0: https://www.jsonrpc.org/specification
• MCP 2024-11-05: https://modelcontextprotocol.io/specification/2024-11-05
• MCP 2025-03-26: https://modelcontextprotocol.io/specification/2025-03-26
• MCP 2025-06-18: https://modelcontextprotocol.io/specification/2025-06-18
• RFC 7636 (PKCE): https://datatracker.ietf.org/doc/html/rfc7636

License

MIT — Copyright (c) 2026 Matthias Meyer (StudioMeyer)