Static linters check that your tool descriptions exist. toolscore checks that they work: it generates realistic tasks for your toolset (with ground-truth labels built in by construction), runs them against a real model, and tells you which tools the model confuses, which parameters it gets wrong, and what to fix.

Why

Agent toolsets fail silently. Research on production agents shows tool-selection accuracy collapses as toolsets grow (down to roughly 13% on large tool menus), and a wrong parameter cascades into a wrong final answer most of the time. Audits of the MCP ecosystem grade the overwhelming majority of published tools F, mostly for descriptions a model cannot disambiguate.

The teams who fix this today do it by hand: building custom eval harnesses, iterating on tool descriptions until selection accuracy climbs. One documented case took tool selection from 60% to 100% purely by rewriting descriptions. toolscore turns that loop into one command.

Install

# No Node.js required. Standalone binary, checksum-verified at install time.
curl -fsSL https://raw.githubusercontent.com/Aryan-MP/toolscore/main/install.sh | bash

# Or, if you are already in a Node/npm project
npm install -g @aryan_mangod/toolscore

Quick start

# Optional: interactive setup, saves defaults to .toolscorerc.json
npx @aryan_mangod/toolscore init

# Scan your MCP server (BYOK, uses your ANTHROPIC_API_KEY)
npx @aryan_mangod/toolscore scan --stdio "node ./my-mcp-server.js"

# No API key? Use your Claude subscription via the Claude Code CLI
npx @aryan_mangod/toolscore scan --stdio "node ./my-mcp-server.js" --provider claude-code

# Or try the pipeline against the bundled flawed demo server, fully offline
npm run demo

# Then let a model rewrite the descriptions it got wrong, and prove the delta
npx @aryan_mangod/toolscore fix --stdio "node ./my-mcp-server.js" --provider claude-code

scan and fix show live progress (connecting, generating tasks, running, scoring) and print nothing but clean JSON on stdout when run with --json, so they are safe to pipe in CI.

Real output from a real model (Claude, via --provider claude-code) scanning the bundled demo server:

  toolscore · tsx demo/flawed-server.ts
  7 tools · 21 tasks · claude-code/default-prompted

    Grade: B+  (89/100)

  ├─ Tool selection       86%
  └─ Parameter accuracy   94%  (when the right tool was chosen)

  Lexical baseline: 0%  (bag-of-words matcher on the same tasks)

  Confusion pairs  (expected → actually called)
    find_content → search_docs  ×1
    add_document → create_note  ×1
    get_user → fetch_account    ×1

The demo server has three deliberately planted flaws: three near-identical search tools, a vague note/document split, and two user-lookup tools with interchangeable descriptions. The model stumbled on exactly those three.

How it works

1. Discover. Connects to your MCP server over stdio and lists its tools.
2. Generate. Creates realistic user tasks from each tool, so the correct answer is known by construction: pick tool T, write a request that objectively requires T with concrete parameter values embedded in the text. Extra tasks deliberately target your most confusable tool pairs (ranked by description similarity), hunting where failures actually live instead of sampling randomly.
3. Run. Presents the full toolset and each task to the model, records which tool it picks and with what arguments.
4. Score. Selection accuracy (60% weight) plus parameter accuracy among correct selections (40% weight) produce a 0 to 100 score and an A to F grade, plus a confusion-pair diagnosis pointing at the exact tools to fix.

The lexical baseline: keeping the benchmark honest

A tool benchmark can silently flatter you: if generated tasks reuse your tools' vocabulary ("search the docs for…" leading to search_docs), any model passes by keyword matching and the grade means nothing. toolscore defends against this two ways:

• Task generation is anti-leakage by instruction: tasks describe the user's goal in the user's own words and must never echo a tool's name or distinctive vocabulary.
• Every scan also runs a bag-of-words matcher over the same tasks. If that dumb baseline scores high, the report warns you that the grade is inflated. (On the demo scan above: model 86%, baseline 0%. The tasks genuinely require understanding intent.)

toolscore fix

fix runs a scan, has the model rewrite the descriptions of every tool involved in a failure, then re-runs the identical task set against the rewritten toolset and shows the before and after delta, including whether the resolved confusion pairs actually resolved, and whether the new descriptions cheated by echoing task vocabulary (the lexical baseline is compared too). You get copy-pasteable descriptions with a verified score improvement, or an honest report that they did not help.

Reproducibility

Generated task sets are cached in .toolscore/, keyed by a hash of your toolset's schemas. Same server, same task set, same model produces comparable grades run to run, which is what makes regression gating in CI meaningful. Pass --regenerate after changing your tools.

CLI reference

toolscore init                        interactive setup, saves defaults to .toolscorerc.json
toolscore scan --stdio "<command>"    benchmark the toolset, print the graded report
toolscore fix  --stdio "<command>"    propose description rewrites and verify them

  --stdio <command>      command that starts your MCP server (stdio transport)   [required unless set by `init`]
  --provider <name>      anthropic | claude-code | mock       (default: anthropic)
  --model <id>           model id for the provider            (default: provider's default)
  --tasks-per-tool <n>   tasks generated per tool             (default: 4)
  --regenerate           rebuild the cached task set
  --json                 machine-readable output for scripting/CI

Providers:

• anthropic: raw Claude API via ANTHROPIC_API_KEY, native tool-use. The reference provider.
• claude-code: drives the Claude Code CLI, billed to your Claude subscription. No API key needed. Tool selection is prompted (toolset shown as JSON) rather than the native tool-use API, so grades are comparable across claude-code runs but not directly against raw-API runs; the report labels the model *-prompted accordingly.
• mock: keyless bag-of-words matcher. Powers the offline demo and the lexical baseline.

Exit codes: scan returns 0 for grade D or better, 1 for grade F, 2 on error. fix returns 0 when the score improved or held, 1 when rewrites made it worse, 2 on error. Usable in CI as-is.

Project structure

src/
  cli.ts             command-line entry point (scan, fix, init)
  ui.ts              terminal UI: spinners, intro/outro framing, brand mark
  config.ts          .toolscorerc.json read/write
  mcp/client.ts      stdio MCP connection and tool discovery
  providers/         model backends behind one interface (anthropic, claude-code, mock)
  taskgen/           task generation, confusable-pair targeting, cached task sets
  runner.ts          executes tasks against the provider, checks calls
  scoring.ts         metrics, weights, A-F grading, confusion pairs
  fix.ts             description-rewrite proposals and verification toolset patching
  report.ts          terminal reports (scan and fix before/after)
demo/
  flawed-server.ts   deliberately bad MCP server: overlapping tools, vague
                     descriptions, so you can see a failing grade in 60 seconds
assets/
  logo-light.svg, logo-dark.svg    full lockup for light/dark backgrounds
  mark-light.svg, mark-dark.svg    icon only, for favicons and small placements

Design principles

• Zero-config. Point it at a server, get a grade. No test files, no YAML, no scorer code.
• Prescriptive. A score without a diagnosis is trivia. Confusion pairs and parameter mismatches name the exact thing to fix.
• BYOK. Your keys, your data, your machine. Nothing leaves your environment.
• Provider-pluggable. A model backend is one small interface: raw APIs, CLI harnesses, or local models all fit.
• Self-skeptical. A benchmark that cannot detect its own inflation is marketing. The lexical baseline runs on every scan and calls out leaky task sets in the report.

Roadmap

• Providers: ollama (local, keyless), OpenAI, Gemini
• toolscore ci: baseline comparison and regression gating on every commit
• Distractor scaling: accuracy vs. toolset-size curves, to answer which tools you should even expose
• Embedding-based pair targeting: replace lexical similarity for sharper adversarial tasks
• HTML report: shareable scan reports

Development

npm install
npm run build     # compile TypeScript to dist/
npm run demo      # end-to-end scan of the flawed demo server, no API key needed
npx tsx src/cli.ts scan --stdio "..." --provider anthropic

Requires Node 22.12 or later.

Building standalone binaries

Releases are compiled with Bun to self-contained native executables (no Node.js runtime needed to run them) for Linux, macOS, and Windows, x64 and arm64:

bun run compile          # current platform, into dist-bin/
bun run compile:all      # cross-compile every target from one machine

CI builds every target on tag push, publishes checksums.txt alongside the binaries in the GitHub release, and install.sh verifies the checksum before installing. See .github/workflows/release.yml for details.

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md for how the project is organized and how to run it locally.

License

MIT. See LICENSE.