mcptuner

MCP Tuner — make MCP servers measurably better.

Tool selection quality is driven by metadata that is easy to get wrong and hard to test: tool names, descriptions, parameter schemas, and how much of the tool list an agent must read before deciding. MCP Tuner turns "does renaming this tool help?" into a measurable experiment instead of a guess.

What it does

| Mode | What it does | Needs a model? | |---|---|---| | Audit | Scores a server's tool/prompt/resource metadata against a rule catalog — description quality, schema completeness, annotations, docs. | No | | Observe | Transparent stdio proxy logs every JSON-RPC frame to JSONL without altering untargeted traffic. | No | | Tune | A/B tests variants of a tool's name, description, or parameter schema; scores selection accuracy, task success, token cost, latency. | Yes | | Compare | Runs two servers with overlapping capability side by side to see which an agent prefers, and why. | Yes |

Audit and Observe work today with no API key. Tune and Compare need an agent driver.

Quick start

cd proxy && npm install && npm run build

# Score a server's metadata (no model required)
node dist/cli.js audit -- npx -y @modelcontextprotocol/server-everything

# Log all traffic between a host and a server
node dist/cli.js log --out traffic.jsonl -- npx -y @modelcontextprotocol/server-everything

audit exits non-zero if any Critical or High rule fails, so it works as a CI gate. --format json emits machine-readable findings.

Docs

• AGENTS.md — canonical agent instructions and hard rules
• docs/rubric-tool-metadata.md — the rule catalog audits score against
• docs/experiments.md — A/B methodology
• docs/decisions.md — numbered decision record

Status

Early. Audit and Observe implemented; Tune and Compare specified but not built.