agent-tool-spec

An open spec for measuring whether an AI agent can pick and call your tools correctly — whether those tools are exposed over MCP or as an HTTP API.

This repository holds the rubric, the defect taxonomy, and the report format used to produce the public scores at gradable.dev. It exists so that a maintainer who disagrees with a score can check the arithmetic instead of taking anyone's word for it.

Why this is public

Scores are published for third-party software. That is a judgement made in public about work other people did, and it is only defensible if the rules behind it are inspectable and contestable. An opaque ranking is easy to dismiss and deserves to be.

So: every formula, weight, and threshold is written down in SPEC.md, including the parts that are proxies, the parts that double-count, and the things the score does not measure at all.

What is actually measured

A tool surface is read — names, descriptions, input and output schemas — and a set of natural-language tasks is generated from it. A judge model is then asked, per task, which tool it would call and with what arguments.

The implementation is never executed and never read. Every finding comes from the declared surface plus observed selection behaviour. This measures how legible your tools are to a model, not whether they work.

That constraint has a consequence worth stating plainly: the suggested description rewrites in a report are hypotheses derived from the schema, not corrections. Applying them unchecked can raise a score while making descriptions less accurate. A fix is only a fix if it matches what the handler actually guarantees.

Contents

| Path | What it is | |---|---| | SPEC.md | Scoring rubric: categories, weights, formulas, grades, and stated limits | | schema/agent-tool-report.v1.schema.json | JSON Schema for a conforming report |

The report format

The schema is published so that reports are not something only one tool can produce. Any scanner may emit a conforming report; any consumer may render or rank one.

This matters for tools that cannot be scanned anonymously. Many of the most widely used integrations require credentials just to enumerate their tools, which puts them out of reach of a public crawler. A maintainer can run a conforming scan against their own instance and publish the result, without handing anyone credentials.

Comparability

A score is a measurement under a stated configuration, not a verdict. Every report records the judge that produced it, and scores from different judge configurations are not directly comparable — judge models differ in how strictly they read a description and how reliably they select a tool.

If you think a score is wrong, re-run it against your own judge and open an issue with both reports. A disagreement between judges is a finding about judge sensitivity, and this repo is the right place for it.

Contributing

Disagreement with the rubric is the point. Useful issues include: a weight that misrepresents real agent failure, a defect type that is missing, a capability the score claims to cover but does not, or evidence that a formula behaves badly on a real surface.

License

MIT.