llm-smart-routing

Cost-aware LLM gateway for structured extraction. Route each call to the cheapest capable model, escalate to a stronger one only when the cheap model isn't confident, fall back across providers on failure, cap your spend hard, and validate every output against your schema — with a typed, never-throwing result.

Works with any OpenAI-compatible or Anthropic-compatible HTTP endpoint. fetch is injectable, so your orchestration logic is fully unit-testable without a network.

npm install llm-smart-routing zod

zod is a peer dependency. Requires Node ≥18 (global fetch).

Why

Most routers only pick a model. This one manages the funnel and the bill:

• 🪜 Cascade cheap→strong — start on the cheap tier; escalate one hop to the strong tier only when the model's self-reported confidence is below your threshold. Pay for top-tier quality only when the cheap model isn't sure.
• 🛟 Multi-provider fallback — one logical call, tried across an ordered chain of providers; 5xx/429/timeout/invalid-JSON/schema-miss all advance to the next provider. Mix OpenAI-compat and Anthropic-compat in the same chain. Zero vendor lock-in.
• 💸 Hard budget cap — set budgetUsd; once cumulative cost crosses it, the gateway stops and returns { ok:false, error: BudgetExceededError, usage } instead of quietly spending more.
• 🧱 Schema-validated JSON — every response is parsed, code-fence-stripped, and run through your validator (zod.safeParse-compatible). Invalid output triggers fallback, never reaches your code.
• 🔀 JSON mode that works across vendors — response_format for OpenAI; assistant-prefill { trick for Anthropic (which has no response_format).
• 🔑 API key pool — round-robin multiple keys per provider to spread rate limits.
• 📋 Model discovery — provider.listModels() GETs /models (uniform across OpenAI- and Anthropic-compatible providers) and returns a normalized ModelInfo[].
• 📊 Usage metering — tokens, USD cost, and call count on every result (success and failure).
• ⛓️ Prompt-chaining primitive — compose multi-step LLM pipelines (research → generate → critique) with per-step observability, importable from llm-smart-routing/chain.
• ✅ Never-throws on expected failure — extract() returns a discriminated union so queue workers can skip cleanly without nested try/catch.

Quick start

import { createGateway } from 'llm-smart-routing';
import { z } from 'zod';

// Load config/llm.json (providers, tiers, thresholds, defaults) and assemble the gateway.
const { gateway, policy } = createGateway('config/llm.json');

const Sentiment = z.object({
  sentiment: z.enum(['positive', 'negative', 'neutral']),
  confidence: z.number(),
});

const res = await gateway.extract({
  input: 'I love this, best purchase ever!',
  stage: 'classify',               // classify → cheap tier first; distill → strong tier
  prompt: [
    { role: 'system', content: 'Reply with a single JSON object only.' },
    { role: 'user', content: 'Classify sentiment: "I love this, best purchase ever!"' },
  ],
  schema: Sentiment,               // any { safeParse } works
  policy: policy(),                // fresh stateless policy per call
});

if (res.ok) {
  console.log(res.data.sentiment); // 'positive'  — typed & validated
  console.log(res.decision.tier);  // 'cheap' or 'strong' (which tier answered)
  console.log(res.usage);          // { promptTokens, completionTokens, costUsd, calls }
} else {
  console.error('skip:', res.error); // LLMUnavailableError | BudgetExceededError | ...
}

In-memory config (no file, no node:fs)

import { createGatewayFromConfig } from 'llm-smart-routing';

const { gateway, policy } = createGatewayFromConfig({
  version: 1,
  providers: [{
    id: 'main', kind: 'openai-compat',
    baseUrl: 'https://api.example.com/v1',
    apiKeys: [process.env.LLM_KEY_1!, process.env.LLM_KEY_2!], // round-robin
    forceJsonMode: true,
    models: {
      cheap:  { name: 'small-model', costPer1kIn: 0.0005, costPer1kOut: 0.0015 },
      strong: { name: 'big-model',   costPer1kIn: 0.01,   costPer1kOut: 0.03 },
    },
  }],
  tiers: { cheap: ['main'], strong: ['main'] },
  thresholds: { classify: 0.55, distill: 0.5 },
  defaults: { timeoutMs: 60000, temperature: 0.2, maxEscalations: 1, maxTokens: 1024 },
  budgetUsd: 5.0, // optional hard cap
});

Budget cap

import { BudgetExceededError } from 'llm-smart-routing';

// Set `budgetUsd` in your config (see below). Once cumulative cost crosses it:
const res = await gateway.extract({ /* ... */ });
if (!res.ok && res.error instanceof BudgetExceededError) {
  // stopped before overspending; res.usage tells you what was spent
}

Prompt chaining

import { runChain } from 'llm-smart-routing/chain';
import type { Step } from 'llm-smart-routing/chain';

interface Ctx { text: string; topic?: string; headline?: string }

const extractTopic: Step<Ctx, Ctx, { topic: string }> = {
  name: 'extract-topic', stage: 'distill',
  buildPrompt: (c) => [{ role: 'user', content: `Topic of: ${c.text}. Reply {"topic":"..."}` }],
  schema: TopicSchema,
  mapOutput: (out, c) => ({ ...c, topic: out.topic }),
};

const res = await runChain<Ctx, Ctx>([extractTopic, writeHeadline], { text }, {
  gateway, policy, now: () => Date.now(),
});
// res.data.topic, res.data.headline, res.usage (accumulated), res.observations (per step)

Listing available models

import { buildRegistry, loadLlmConfig } from 'llm-smart-routing';

const cfg = loadLlmConfig('config/llm.json');
const registry = buildRegistry(cfg);

// Works for both OpenAI-compat and Anthropic-compat providers (same call, same shape).
const models = await registry.get('main').listModels();
for (const m of models) {
  console.log(m.id, m.displayName ?? '', m.ownedBy ?? '');
}

Configuration

Copy config/llm.example.json → config/llm.json and fill in apiKeys. Schema in config/llm.schema.json. Config is validated at load (zod + cross-field invariants: every tier id must exist as a provider and own a model for that tier; no duplicate ids) — it fails fast on startup rather than at first call.

⚠️ Never commit real keys. config/llm.json is gitignored; the published package ships only dist/ (see files in package.json).

API

| Import | What | |---|---| | createGateway(path?, fetch?) | Load file config + assemble { gateway, policy, config }. | | createGatewayFromConfig(cfg, fetch?) | Assemble from an in-memory LlmConfig. | | gateway.extract(params) | Promise<{ ok:true, data, usage, decision, confidence } \| { ok:false, error, usage }>. | | loadLlmConfig, validateConfig, LlmConfigSchema | Config loading/validation. | | buildRegistry, ProviderRegistry, buildProvider | Build/resolve providers from config. | | provider.listModels(timeoutMs?) | GET /models → normalized ModelInfo[]. | | OpenAICompatProvider, AnthropicCompatProvider, BaseChatProvider, parseModelList | Provider adapters + /models parser (advanced/custom adapters). | | CascadePolicy, buildTierChains | Routing policy + tier-chain assembly (advanced). | | UsageMeter, computeCost, KeyPool | Utilities. | | ProviderError, LLMUnavailableError, LLMParseError, BudgetExceededError | Error types. | | llm-smart-routing/chain → runChain, Step, emptyUsage, addUsage | Prompt-chaining primitive. |

Scripts

npm run build       # tsup → dist (ESM + CJS + .d.ts)
npm run typecheck   # tsc --noEmit
npm test            # vitest (offline, injected fetch)
npm run poc         # run all feature POCs (offline + live)

License

MIT