llm-tool-router

An agent with 100 tools sends 100 tool definitions on every iteration of its loop. On a turn that takes eight tool calls, that is eight times the whole catalogue — the dominant cost of the turn, the reason the context that matters gets pushed out, and the reason accuracy drops as the model picks between near-duplicate options.

This library puts a classifier in front of the loop. A small cheap model reads the request, picks 1-3 tool categories, and only those tools are sent. One extra ~100-token call replaces most of the definitions on every subsequent iteration.

"Has invoice 42 been paid?"  ──▶  classify  ──▶  ["billing"]  ──▶  4 tools instead of 100

It never throws, it degrades in steps, and it always tells you which step it used.

Install

npm install llm-tool-router

Node ≥ 20, ESM, no dependencies, no provider SDK. Bring your own model call.

Quickstart

import { createToolRouter } from "llm-tool-router";

const router = createToolRouter<MyTool, MyContext>({
  categories: {
    billing: {
      // Written FOR the classifier: what is in it, AND when to pick it.
      description:
        "Invoices, payments, quotes, expenses. Use for anything about money, " +
        "an invoice, a payment or cash flow.",
      groups: [billingTools, expenseTools],
    },
    files: {
      description: "Documents, uploads, folders. Use for anything about a file.",
      groups: [
        documentTools,
        // Gated: an agent able to call a tool from a hidden module is a back
        // door around whatever hid it.
        { tools: vaultTools, enabled: (ctx) => ctx.modules.includes("vault") },
      ],
    },
    // …
  },
  classifier: myClassifier,          // see adapters below
  fallbackCategories: ["files", "billing"],
  assistantDescription: "a back-office assistant",
  onRoute: (r) => metrics.record(r), // outcome, latency, cost — log this
});

const { tools, names, cost, outcome } = await router.route({
  message: userMessage,
  history: recentTurns,   // must NOT include userMessage
  context: { modules, plan },
});

// cost → { selected: 4, available: 100 }
runAgentLoop({ tools });

The classifier adapter

Classifier is one function: take a ClassifyRequest, call a small model, return its raw text. Retries and transport belong to your adapter — this library owns the prompt, the schema, the ladder and the parsing.

The schema field is handed to you in OpenAI's json_schema shape ({ name, strict, schema }), so most providers take it as-is:

const classifier: Classifier = async (req) => {
  const res = await openai.chat.completions.create({
    model: "gpt-5-nano",           // small and cheap: this is a classification
    temperature: 0,
    max_tokens: req.maxTokens,
    response_format: { type: "json_schema", json_schema: req.schema },
    messages: [
      { role: "system", content: req.system },
      ...req.history,
      { role: "user", content: req.user },
    ],
  }, { timeout: req.timeoutMs });
  return res.choices[0]?.message?.content ?? "";
};

Adapters are a few lines each and the field names differ per provider — the Mistral SDK, for instance, wants responseFormat.jsonSchema with the JSON Schema under schemaDefinition rather than schema. Check your SDK's current shape; req.schema.schema is the plain JSON Schema if you need to unwrap it.

Two things are not optional, whatever the provider:

• Use structured output. "Reply with JSON" returns JSON with a preamble often enough to matter at scale, and every one of those costs a retry.
• temperature: 0. This is a classification, not a completion.

Degradation, in order

Classification is an extra dependency in the hot path, so route() never rejects. Every result carries outcome, and degraded is the single field to alert on.

| outcome | What happened | degraded | | --- | --- | --- | | classified | First attempt returned usable categories. | false | | skipped | This caller has ≤ maxCategories categories — nothing to choose between, so no model call at all. | false | | retried | First attempt failed; a stricter prompt on a shorter deadline worked. | true | | fallback | Both attempts failed; fallbackCategories was used. | true |

errors lists what each failed attempt said, so a rise in degraded is diagnosable rather than mysterious.

Choose fallbackCategories deliberately. The tempting default — load everything when classification fails — reintroduces exactly the cost the router exists to avoid, on the requests that are already having a bad time. Pick the two or three categories your traffic actually lands in.

The failure mode this library exists to prevent

A tool that is in your registry but in no category is silently unreachable. The router only ever offers the tools of the categories it picked, so that tool is never proposed to the model, nothing errors, and no log line appears. It looks exactly like the model choosing not to use it. You find out weeks later, from a user.

The mirror image is just as quiet: a tool wired into a category but missing from the registry gets offered to the model and then cannot be executed.

it("registry and router agree", () => {
  router.assertConsistent(fullToolRegistry);
});

One test, both directions, with the offending names in the message. Add it the day you add the router.

Gating

Each group may carry enabled: (context) => boolean. It is applied in two places, and both matter:

• Before classification — a category with no reachable group is not even shown to the classifier. Offering it wastes prompt space and invites the model to pick a category that comes back empty.
• After classification — the subset is filtered again, so a hallucinated or stale category cannot reach past the gate.

context is your type: a tenant's enabled modules, a user's role, a plan tier, a feature-flag snapshot. The library never interprets it.

Limits

• One classification per turn, not per iteration. The category set is fixed for the whole agent loop. If a request genuinely needs tools from a fourth category discovered at iteration five, this router will not hand them over — raise maxCategories, or re-route on a follow-up turn.
• Category descriptions are the actual interface. They are what the small model sees, and a vague one silently mis-routes. Write them as instructions to a classifier, not as documentation for humans.
• Overlapping categories cost you. If two categories both plausibly own "send an invoice by email", the classifier will oscillate. Split by what the user talks about, not by your module boundaries.
• No latency win. This trades one small extra call for a much smaller prompt on every iteration. It is a cost and accuracy play; on a single-iteration turn it is a net loss.
• history must exclude the current message, or the model sees it twice.
• Nothing here calls a model. No SDK, no retry policy, no rate limiting — that is your adapter's job, and it is deliberate: the retry ladder here is about classification quality, not transport.

API

| Export | Purpose | | --- | --- | | createToolRouter(options) | The router: route, subsetOf, categoriesFor, uncategorised, assertConsistent. | | route({ message, history?, context }) | Classify then build the subset. Never rejects. | | subsetOf(categories, context) | Skip classification and take these categories. | | categoriesFor(context) | Categories with at least one reachable group. | | assertConsistent(registry) | Throws on either kind of registry/category mismatch. | | buildSubset / allowedCategories / uncategorisedTools / unknownCategorisedTools | The registry primitives, usable without a router. | | buildClassifierPrompt / buildCategorySchema / parseCategories | The prompt layer, if you want to drive it yourself. |

See it

npx tsx examples/basic.ts

Four requests through a stub classifier, printing the tool count each one selected (4/12, 6/12, 2/12), the fallback path on an unclassifiable message, and a gated-down tenant where the model call is skipped entirely.

Development

npm install
npm test
npm run typecheck
npm run build

37 tests, no network: the classifier is an injected function, so every path — including both failure rungs — is tested offline.

Provenance

Extracted from a production agent with ~120 tools across ~50 modules, where the router cut the per-iteration prompt by roughly an order of magnitude. Generalised on the way out rather than bug-fixed: the model client became an injected Classifier, the tenant gate became a predicate on your own context type, and the categories became yours instead of a hard-coded French domain.

Three behaviours changed deliberately:

• An empty or fully-hallucinated selection used to load every tool as a "safe" fallback. It now falls back to fallbackCategories, because loading everything is the failure this library exists to avoid.
• maxTokens for the classification is derived from maxCategories instead of a fixed 80, which truncates the JSON — and so fails the call — once category names get long.
• A caller with few enough categories now skips the model call (outcome: "skipped") instead of paying to be told what was already known.

License

MIT