tooltax
v0.1.0
Published
Token-budget linter and public leaderboard for MCP tool schemas.
Downloads
80
Maintainers
Readme
tooltax
Every MCP server charges you rent on your context window. tooltax measures it.
MCP servers inject their tool definitions into every single request. You pay for them before the user has typed anything, on every turn, whether or not the model uses a single tool.
The numbers are not small. Anthropic's engineering team, writing about advanced tool use:
At Anthropic, we've seen tool definitions consume 134K tokens before optimization.
and, for a merely ordinary setup:
That's 58 tools consuming approximately 55K tokens before the conversation even starts.
This is a known problem in the protocol. It has been raised on the MCP spec repository in SEP-1576, "Mitigating Token Bloat in MCP" and in issue #2808, "MCP spec should address tool schema token overhead", which measured production tools costing roughly 1,000 tokens each. Both issues are now closed without the protocol changing, which leaves the cost where it has always been: with whoever installed the server.
What has been missing is measurement. You cannot budget what nobody counts.
tooltax does three things:
- Scores a tool schema with a real BPE tokenizer, not
length / 4. - Lints it, explaining which parts are expensive and what to change.
- Ranks MCP servers publicly, from a registry anyone can add one file to.
What we measured
The leaderboard is generated by CI, which launches each server, performs the MCP handshake, and tokenizes what comes back. From the current run:
| Server | Tools | Tokens | Context @200k | |---|---:|---:|---:| | Fetch | 1 | 237 | 0.1% | | Git | 12 | 1,129 | 0.6% | | Everything | 13 | 1,171 | 0.6% | | Filesystem | 14 | 2,383 | 1.2% | | Playwright | 24 | 3,423 | 1.7% | | Kubernetes | 23 | 5,066 | 2.5% | | MongoDB | 29 | 6,375 | 3.2% | | Firecrawl | 26 | 16,818 | 8.4% | | Notion | 24 | 17,430 | 8.7% |
Across the 20 servers scored live: 290 tools, 73,323 tokens. Installing all of them would consume 36.7% of a 200k context window before a single user message.
The spread is the interesting part. Notion costs 726 tokens per tool; Puppeteer costs 78. That is a factor of nine, and it is a design choice, not a law of nature.
Install
npm install -g tooltaxOr without installing:
npx tooltax score npm:@modelcontextprotocol/server-everythingNode 20 or newer.
Use it
Score a server
Point it at a live server and it does the MCP handshake for you. This is real output:
$ tooltax score npm:@modelcontextprotocol/server-everything
Source npx -y @modelcontextprotocol/server-everything
Tokenizer gpt-tokenizer (o200k_base)
1,171 tokens across 13 tool(s) -- 0.6% of a 200,000-token context window.
Mean 90.1 tokens/tool, 5,321 bytes on the wire.
TOOL TOKENS SHARE DESC/SCHEMA
gzip-file-as-resource 204 17.4% 45/149 ########################
get-structured-content 153 13.1% 12/131 ##################
simulate-research-query 134 11.4% 49/75 ################
get-annotated-message 105 9.0% 14/80 ############
get-resource-reference 87 7.4% 11/66 ##########
trigger-long-running-operation 86 7.3% 11/65 ##########
get-resource-links 80 6.8% 12/58 #########
get-sum 72 6.1% 6/56 ########
echo 57 4.9% 6/43 #######
toggle-simulated-logging 53 4.5% 14/27 ######
toggle-subscriber-updates 49 4.2% 11/27 ######
get-env 47 4.0% 11/27 ######
get-tiny-image 44 3.8% 7/27 #####DESC/SCHEMA splits each tool's cost between its prose and its JSON Schema,
which tells you which one to go after.
Sources can be a live server, a file, or a pipe:
tooltax score npm:@playwright/mcp # npx
tooltax score pypi:mcp-server-git # uvx
tooltax score cmd:node ./my-server.js # any stdio command
tooltax score https://example.com/mcp # MCP over HTTP, or a static schema
tooltax score ./tools.json # a file
cat tools.json | tooltax score - # stdinFind out why it is expensive
lint runs rules that name the problem and the fix. Real output, trimmed:
$ tooltax lint npm:@notionhq/notion-mcp-server
17,430 tokens across 24 tool(s) -- 8.7% of a 200,000-token context window.
Mean 726.3 tokens/tool, 74,641 bytes on the wire.
TOOL TOKENS SHARE DESC/SCHEMA
API-update-page-markdown 1,278 7.3% 62/1201 ########################
API-post-search 1,093 6.3% 14/1068 #####################
API-patch-page 889 5.1% 14/863 #################
...
------------------------------------------------------------------------
52 finding(s): 0 error, 46 warn, 6 info. Estimated recoverable: ~765 tokens (4.4% of the payload).
[WARN ] deep-nesting -- Input schema nests deeper than models need (24 finding(s), ~316 tokens)
API-update-page-markdown: schema nests 5 levels deep (max 3) at inputSchema.properties.update_content.anyOf[0].properties.content_updates.items.properties.old_str; 7 node(s) sit below the limit.
API-create-a-comment: schema nests 5 levels deep (max 3) at inputSchema.properties.rich_text.items.anyOf[0].properties.text.properties.content; 4 node(s) sit below the limit.
fix: Flatten the schema. Accept an opaque string or a single object of primitives where the nested tree only exists to mirror an internal data model, and validate the detail inside the tool instead.
[WARN ] verbose-description -- Description is longer than it needs to be (18 finding(s), ~321 tokens)
API-post-search: inputSchema.properties.sort.anyOf[0].description is 66 tokens (budget 25): "A set of criteria, `direction` and `timestamp` keys, that o..." -- trim to roughly 25 tokens.
fix: Cut descriptions to one sentence that says what the tool does and when to pick it. Move examples, caveats and formatting notes into the error messages the tool returns, or into your README.
[WARN ] duplicate-param-shape -- The same parameter shape is redefined across tools (1 finding(s), ~40 tokens)
parameter "properties" has an identical shape in 4 tools (~19 tokens each); hoist it into $defs and $ref it.
fix: Hoist the shared shape into $defs and reference it with $ref, or flatten it into a small number of primitive parameters that do not need repeating.Eight rules ship today. tooltax rules prints them all with rationale and fix:
| Rule | Severity | Catches |
|---|---|---|
| missing-description | error | No description, or one that just restates the tool name |
| verbose-description | warn | Prose longer than it needs to be |
| repeated-boilerplate | warn | The same sentence pasted across many tools |
| duplicate-param-shape | warn | The same parameter object redefined instead of $refd |
| enum-bloat | warn | Every allowed value listed inline |
| oversized-example | warn | A full sample payload embedded in the schema |
| deep-nesting | warn | Schemas nested past about three levels |
| mergeable-tools | info | Near-identical tools that could be one |
Gate it in CI
tooltax score npm:your-server --max-tokens 5000Exits 1 with a diff when the budget is blown, so a schema change that doubles your context cost fails the build:
$ tooltax score npm:@modelcontextprotocol/server-everything --max-tokens 1000
...
FAIL: 1,171 tokens exceeds --max-tokens 1,000 by 171 (17.1% over budget).
$ echo $?
1--max-findings <n> does the same for lint findings. Everything supports
--json.
Add your server in 3 steps
The leaderboard is a registry. One server is one small file, and CI does the rest.
1. Create servers/<owner>__<name>.json:
{
"name": "Your Server",
"repo": "https://github.com/you/your-server",
"source": { "type": "npm", "package": "your-mcp-server" },
"category": "dev-tools"
}Five fields, three of them required. source can be npm, pypi, command,
url, or file -- see CONTRIBUTING.md.
2. Check it:
node dist/cli.js validate3. Open a pull request.
CI validates the file against schema/server.schema.json,
scores your server, comments the result on your PR, and regenerates
LEADERBOARD.md on merge. You never edit the leaderboard by
hand, and nobody has to take your word for a number.
If your server needs credentials to start, add it anyway. It lands in the pending section with the reason attached, which is honest and still useful.
Get a badge for your server
Every registry entry gets a shields.io endpoint generated
by CI at badges/<owner>__<name>.json. Put it in your own README:
[](https://github.com/duskel/tooltax/blob/main/LEADERBOARD.md)It renders as tool tokens | 2.4k (14 tools), coloured green through red by
cost, and updates itself when the nightly run re-scores you. Generate the markdown
with:
tooltax badge npm:your-mcp-server --slug your-org__your-serverA lean schema is worth advertising. That is the point.
Methodology
Being explicit about this matters, because tooltax publishes numbers about other people's work.
Tokenizer. Real BPE via gpt-tokenizer,
defaulting to o200k_base. Verified against published tiktoken reference values:
hello world is 2 tokens, tiktoken is great! is 6. Swap encodings with
--tokenizer cl100k. There is also an approx heuristic that does length / 4
purely so reports can show how wrong that is; anything scored with it is labelled
as an approximation.
What gets counted. Each tool's name, description, inputSchema and
outputSchema, serialized to canonical JSON with sorted keys so key ordering
cannot change a score. Vendor extensions and _meta are excluded because they
are not sent to the model.
Context share assumes a 200,000-token window. Change it with
--context-window.
Limits, stated plainly
- Providers serialize tools differently. tooltax uses one canonical JSON form for every server. It is a close proxy for real cost and a fair basis for ranking, not a byte-exact reproduction of any one vendor's wire format.
- Tokenizers differ by vendor. Claude does not use
o200k_base. The ranking is reproducible under one method; absolute numbers will shift under another. - Prompt caching changes the billing, not the budget. Cached tool definitions cost less money after the first call. They occupy exactly the same context window every time. tooltax measures the window.
- Some servers vary their tool list by configuration or credentials. Where
that applies it is in the entry's
notes. estimatedSavingsis an estimate, deliberately conservative, and is labelled as such everywhere it appears.
If you think a score is wrong, that is a bug.
Contributing
Three contribution surfaces, all designed so CI does the reviewing:
- Add a server -- one JSON file.
- Add a rule -- one directory under
src/rules/, with fixtures. A discovery-based contract test enforces the contract for you. - Add a tokenizer -- one file under
src/tokenizers/.
Looking for somewhere to start? docs/GOOD-FIRST-ISSUES.md has ten concrete tasks, from a CSV output mode to a real Anthropic token-counting tokenizer.
git clone https://github.com/duskel/tooltax.git
cd tooltax && npm install && npm run build && npm testBuilt and maintained by Duskel
We build AI automation and MCP integrations -- custom servers, agent tooling, and the unglamorous context engineering that decides whether any of it works in production.
tooltax exists because we kept measuring this by hand for clients and got tired of it. If you are shipping an MCP server, or trying to keep an agent's context budget under control, we can help: duskel.com.
License
MIT (c) Duskel
Built and maintained by Duskel — an AI automation & software studio. We build MCP servers and AI tooling; tooltax came out of keeping our own tool schemas lean. Work with us.
