@loadline/tool-selection

A merge gate for LLM tool selection. Part of loadline.

npx @loadline/tool-selection --baseline .loadline/main.json

The question it answers

Every other test asks whether a tool works. This asks whether a model reaches for the right one — and whether it invents identifiers on the way.

A tool can be correct, fast, fully typed and thoroughly covered, and still never be chosen, because its description is vague or overlaps another. Nothing else will tell you that. The tool passes its tests and quietly never runs.

What it checks

| Check | Fails when | |---|---| | tool_choice | The first tool called is not one the case expects | | no_forbidden_tools | A tool the case rules out was called at all | | arguments | The right tool was called with the wrong arguments | | no_identifier_invented | An identifier was passed that appears nowhere in the question | | no_decoy_identifier | A value the case named as a decoy was used as an identifier |

The identifier checks are the ones that matter most. An identifier with no checksum — a company number, an order reference, an account id — has nothing about a wrong one that looks wrong, so a model supplying one from memory gets a real record for the wrong subject and nothing downstream flags it.

Two variants exist because one is not enough. forbidUngrounded catches a value absent from the question. forbidAnyIdentifier catches the harder case: a question carrying a VAT number, a postcode or an order reference, where the wrong value is in the question and grounding waves it straight through. A model defeated an earlier decoy list by taking nine VAT digits, dropping one and padding with a zero — so the rule became "this question contains no identifier, therefore any identifier is wrong", which cannot be beaten by a derivation nobody anticipated.

Gating on regression

Exits non-zero only when a case that held on the baseline no longer holds. A lower total, a new failing case, or a wobble at the margin are reported and do not block. See the root README for the measurement that decided this.

# record a baseline on main
npx @loadline/tool-selection --repeat 3 --baseline .loadline/main.json

# gate a pull request against it
npx @loadline/tool-selection --repeat 3 --baseline .loadline/main.json --markdown comment.md

| Flag | Does | |---|---| | --config <path> | Config file. Defaults to loadline.config.{js,mjs,json} | | --baseline <path> | Compare against this; recorded automatically if absent | | --update-baseline | Overwrite the baseline with this run | | --repeat <n> | Repeats per case. 3 surfaces flakiness | | --case <id-or-category> | Run one case, or a whole category | | --provider anthropic\|openrouter | Force a provider | | --model <id> | Any model with tool support | | --json <path> / --markdown <path> | Write results / a PR comment | | --fail-on-still-failing | Also block on pre-existing failures | | --fail-on-flaky | Also block when a case flip-flops within a run |

Limits worth knowing before you start

• MCP over stdio only. No HTTP or SSE, and no path for tools defined any other way. If your agent is not MCP, this cannot help you today.
• First turn only. Nothing is executed; multi-turn behaviour is out of scope.
• Tool calls, not answers. A model can pick the right tool and still say something wrong. Invisible here, by construction.
• Writing the cases is the work. No starter suite, no generator. The worked example took an afternoon by hand — deliberately, since generated cases test the phrasings a model finds natural rather than the ones your users type.
• A baseline belongs to one subject. Switch models and you re-record; the gate refuses to act on a mismatched baseline rather than reporting model differences as regressions.

Writing cases

{
  id: 'grounding-profile',
  category: 'grounding',
  intent: 'name-to-number',       // optional: groups variations for agreement reporting
  question: 'Can you tell me about Royal Mail Group Limited?',
  expectTool: ['find_company'],   // [] means "call nothing"
  allowNoTool: true,              // declining is also acceptable
  forbidTools: ['get_company'],
  expectArgs: [{ path: 'company_number', equals: '04138203' }],
  forbidUngrounded: true,
  why: 'Given a name and no number, the only correct first move is to search.'
}

why is required. A case nobody can justify is a case nobody fixes when it fails — it gets deleted instead.

path: 'any' matches against every argument value rather than one named path, for cases accepting several tools that take differently-named arguments.

Use intent to group variations of one question. Phrasings that choose different tools is a finding about your descriptions even when each individual choice is defensible.

Credentials

Your MCP server needs no working credentials: it is started only so its tools and instructions can be read, and nothing is executed. A placeholder gets most servers past their own config validation.

You do need OPENROUTER_API_KEY or ANTHROPIC_API_KEY. OpenRouter is used by default when both are set. In CI those come from secrets; locally, if your key already lives in a .env somewhere, Node will load it for you rather than making you export it:

node --env-file=../path/to/.env node_modules/.bin/loadline-tool-selection --baseline .loadline/main.json