@idevconn/llm-router

Library-agnostic LLM router. Provider-neutral LlmStrategy interface + LlmRegistry with env-driven platform selection, BYOK support, and boot-time env-key audit. Opt-in adapters for Gemini, Claude, Grok, ChatGPT, and DeepSeek via subpath exports — install only the SDKs you actually use. A TaskRouter + Orchestrator on top can split a free-text task into subtasks and run each on whichever registered provider fits best.

Features

• Pure router core: zero SDK dependencies on the main entry. Just types + LlmRegistry.
• Subpath adapters: @idevconn/llm-router/gemini, /claude, /grok, /chatgpt, /deepseek. Each declares its SDK as an optional peer dependency, so consumers install only what they need.
• BYOK first-class: every strategy accepts a per-call apiKey that overrides the platform key for that one request.
• Platform-fallback fully optional: pass platform: null to LlmRegistry to require BYOK from every caller — useful for SaaS that doesn't subsidize AI usage.
• Streaming (onToken) and reasoning/thinking (thinking) request options, and call cancellation (signal) — provider-agnostic on LlmGenerateOptions, implemented per-provider where the underlying SDK actually supports it (see Streaming and reasoning/thinking).
• Typed errors: UnknownProviderError, NoPlatformProviderError, InvalidPlatformProviderError, LlmKeyValidationError, UnsupportedAttachmentError, TaskDecompositionError, NoAvailableProviderError, BudgetExceededError, UnsupportedThinkingModeError, InvalidThinkingConfigError, LlmAbortedError. No framework-specific exceptions.
• Cost control: withBudget decorator enforces per-call and total spend caps against a caller-supplied pricing table.
• Instrumentation: withInstrumentation decorator emits a call event (usage, latency, truncation, errors) to any logger you choose.
• Prompt injection defense: sanitizeUntrustedContent + detectPromptInjection (cheap heuristic gate) + detectPromptInjectionWithModel (opt-in LLM-based second opinion).
• RAG: Retriever + EmbeddingStrategy/VectorStore interfaces, with a pgvector-backed VectorStore adapter via a subpath export.

Install

npm install @idevconn/llm-router

# Then install only the SDKs for the providers you use:
npm install @google/generative-ai   # for Gemini (direct API / BYOK)
npm install @google/genai           # optional; only for GeminiStrategy({ connection: "vertex" })
npm install @anthropic-ai/sdk       # for Claude
npm install openai                  # for Grok, ChatGPT, and DeepSeek (all OpenAI-compatible)
npm install pg                      # for the pgvector VectorStore adapter

Quick start

import { LlmRegistry } from "@idevconn/llm-router";
import { GeminiStrategy } from "@idevconn/llm-router/gemini";
import { ClaudeStrategy } from "@idevconn/llm-router/claude";
import { GrokStrategy } from "@idevconn/llm-router/grok";

const registry = new LlmRegistry<"gemini" | "claude" | "grok">({
  strategies: [
    new GeminiStrategy({
      apiKey: process.env.GEMINI_API_KEY,
      defaultModel: process.env.GEMINI_MODEL,
      // connection: "vertex" → Vertex AI via ADC; BYOK still uses the API key
    }),
    new ClaudeStrategy({ apiKey: process.env.CLAUDE_API_KEY, defaultModel: process.env.CLAUDE_MODEL }),
    new GrokStrategy({ apiKey: process.env.XAI_API_KEY, defaultModel: process.env.GROK_MODEL }),
  ],
  platform: process.env.ML_STRATEGY as "gemini" | "claude" | "grok" | null,
  providerEnvKeys: {
    gemini: "GEMINI_API_KEY",
    claude: "CLAUDE_API_KEY",
    grok: "XAI_API_KEY",
  },
  env: process.env,
});

// Platform call
const platform = registry.getPlatform();
const result = await platform.generate({
  prompt: "Summarize this invoice.",
  attachments: [{ data: fileBuffer, mimetype: "application/pdf" }],
});

// BYOK call — same registry, user-supplied provider + key
const strategy = registry.get("claude");
const byok = await strategy.generate({
  prompt: "Summarize this invoice.",
  attachments: [{ data: fileBuffer, mimetype: "image/png" }],
  apiKey: user.claudeApiKey,
  model: user.preferredModel,
});

// Live key check (used in BYOK save flows)
await strategy.validateKey(user.claudeApiKey, user.preferredModel);

Multi-turn conversation history

LlmGenerateOptions.prompt is now optional. Pass messages: LlmMessage[] instead for multi-turn conversation history:

interface LlmMessage {
  role: "user" | "assistant";
  content: string;
}

Exactly one of prompt / messages must be set — both, or neither, throws InvalidGenerateOptionsError before any network call. When messages is used, attachments apply only to the last turn (same place they'd land if you were building that turn as a single-turn prompt call). All 5 built-in strategies (Claude, Gemini, ChatGPT, Grok, DeepSeek) implement messages natively — none of them throw UnsupportedMultiTurnError (that error exists for custom third-party strategies that choose not to support it).

Gemini maps roles to its own vocabulary: "assistant" → "model", "user" → "user" — an implementation detail only relevant if you're reading or extending GeminiStrategy directly; callers use the same LlmMessage.role values ("user"/"assistant") for every provider.

const result = await strategy.generate({
  messages: [
    { role: "user", content: "What's the capital of France?" },
    { role: "assistant", content: "Paris." },
    { role: "user", content: "And its population?" },
  ],
});

Streaming and reasoning/thinking

LlmGenerateOptions has three additive fields. A strategy that doesn't implement one of them simply ignores it — every existing caller keeps working unchanged.

• onToken?: (delta: string) => void — called with each incremental text delta as it arrives. When set, a strategy that supports streaming uses the provider's streaming endpoint internally, but still resolves the same Promise<LlmResponse> once the stream ends. A strategy MAY never invoke it (no streaming support) — don't assume it fires. A throwing callback is caught and logged, never propagates out of generate(). Keep it fast/sync-safe: it is not awaited between deltas, so slow async work inside it will not backpressure the provider stream.
• thinking?: {type:'adaptive'} | {type:'budget', tokens} | {type:'effort', level:'low'|'medium'|'high'} — provider-agnostic reasoning-depth request. Check the 'thinking' capability tag before requesting it, or catch UnsupportedThinkingModeError. budget.tokens/effort.level are runtime-validated against allow-lists before any network call — an out-of-range or unsupported value throws InvalidThinkingConfigError or UnsupportedThinkingModeError up front, never silently downgraded.
• signal?: AbortSignal — cancels the call, including an open stream.

LlmResponse.thinking?: string carries a returned reasoning trace, when the provider returned one. Treat it exactly like response.text — untrusted model output. Never re-feed it into another prompt (e.g. a "show your reasoning" UI that summarizes it via another LLM call) without running it through sanitizeUntrustedContent/detectPromptInjection first.

Per-provider support (check strategy.capabilities for 'streaming'/'thinking'):

| Provider | Streaming | Thinking | Multi-turn | |---|---|---|---| | Claude | ✅ messages.stream() | adaptive, budget (budget_tokens ≥1024 and < maxTokens). effort unsupported. | ✅ | | Gemini | ✅ both connection modes | Vertex only (GeminiStrategy({ connection: "vertex" })) — adaptive, budget. The direct API SDK has no thinking support at all; any thinking request on that connection throws UnsupportedThinkingModeError. | ✅ | | ChatGPT | ✅ stream:true | effort only (reasoning_effort). Chat Completions never returns a reasoning trace — response.thinking is always undefined for this strategy. | ✅ | | Grok | ✅ stream:true | Not implemented. xAI's reasoning-effort parameter contract could not be verified against primary documentation — any thinking request throws UnsupportedThinkingModeError until this is confirmed against docs.x.ai. | ✅ | | DeepSeek | ✅ stream:true | Not implemented as a request option, same reason as Grok. deepseek-reasoner's reasoning_content is instead surfaced passively via response.thinking whenever the model returns it — inherent to the model, not caller-controlled. | ✅ |

const stream = await strategy.generate({
  prompt: "Explain quantum entanglement.",
  onToken: (delta) => process.stdout.write(delta),
  thinking: { type: "budget", tokens: 2048 },
  signal: abortController.signal,
});
console.log(stream.thinking); // reasoning trace, if the provider returned one

Aborting a stream: LlmAbortedError

When a streaming call (onToken set) is aborted via signal, every built-in strategy rejects with LlmAbortedError instead of the provider's bare abort error, so you can keep what was already shown to the user and account for the tokens spent:

• partialText — exactly the concatenation of the deltas delivered to onToken (including one whose callback threw).
• partialThinking? — reasoning received so far, for strategies that stream it (Gemini Vertex thought parts, DeepSeek reasoning_content).
• usage / usageEstimated — provider-reported counts when the provider had already sent both input and output usage mid-stream (usageEstimated: false); otherwise the missing side is a deliberately rough Math.ceil(chars / 4) estimate (usageEstimated: true). In practice: Claude reports input tokens on message_start but output tokens only near the end; Gemini attaches usageMetadata to stream chunks when the server sends it; ChatGPT/Grok/DeepSeek report usage only in the final chunk, so a mid-stream abort is almost always estimated (usage is reported only if that final usage chunk had already arrived). Provider-reported numbers mean the same as on a normal response: Claude's message_start input_tokens excludes cache read/creation tokens, and Gemini's candidatesTokenCount excludes thought tokens. Only an estimated output side counts received thinking text.
• cause — the original provider/abort error (for ChatGPT/Grok/DeepSeek, whose SDK ends an aborted stream silently, this is signal.reason); providerName.

Its name is "AbortError", so existing err.name === "AbortError" checks keep working and withRetry / withCircuitBreaker treat it as an intentional abort (never retried, never counted as a provider failure). Use instanceof LlmAbortedError for the typed check. Non-streaming calls, calls aborted before the request starts (these throw the plain signal.reason, as before), and non-abort errors are unchanged. An abort that lands after the provider already signalled completion (e.g. a finish_reason arrived) resolves normally with the full response (usage may be unreported).

partialText and partialThinking are untrusted model output, exactly like response.text/response.thinking. Never re-feed them into another prompt without running them through sanitizeUntrustedContent/detectPromptInjection first. Both are enumerable own properties of the error, so loggers that serialize error properties (pino's error serializer, Nest setups that log the full error object, …) will write model output into your logs — redact them (e.g. pino redact: ["err.partialText", "err.partialThinking"]) if that matters.

import { LlmAbortedError } from "@idevconn/llm-router";

try {
  const res = await strategy.generate({ prompt, onToken: send, signal });
  await saveAnswer({ text: res.text, usage: res.usage });
} catch (err) {
  if (err instanceof LlmAbortedError) {
    await saveAnswer({
      text: err.partialText,
      usage: err.usage,
      usageEstimated: err.usageEstimated,
      aborted: true,
    });
    return;
  }
  throw err;
}

Adding a custom provider

Implement LlmStrategy and pass it to LlmRegistry. The SDK choice is yours — the pkg never imports it. Useful for Bedrock, Vertex, local models via Ollama, internal LLM gateways, etc.

import type { LlmStrategy } from "@idevconn/llm-router";

class OllamaStrategy implements LlmStrategy {
  readonly providerName = "ollama";
  readonly defaultModel = "llama3.1";

  async generate(opts) { /* call your gateway */ }
  async validateKey(apiKey, model) { /* ping endpoint */ }
}

hasPlatformKey() and capabilities are optional, so a strategy like the one above works fine for direct registry.get("ollama").generate(...) calls. It just won't participate in TaskRouter's automatic matching: without hasPlatformKey() the router treats it as unavailable, and without capabilities it never wins the capability-tag rule stage. To opt a custom strategy in without implementing either, pass a providerOverrides entry (a ProviderDescriptor: { provider, available?, capabilities? }) on the TaskRouter.route() / Orchestrator.run() call — or supply that provider's key in apiKeys, which marks it available for that call. Either way it then routes like a built-in. See Task orchestration below.

Task orchestration

Orchestrator splits a free-text task into subtasks and runs each on whichever registered provider is best suited, using only providers that have a usable key (platform or BYOK for this call):

import { LlmRegistry, Orchestrator } from "@idevconn/llm-router";
import { ClaudeStrategy } from "@idevconn/llm-router/claude";
import { GeminiStrategy } from "@idevconn/llm-router/gemini";

const registry = new LlmRegistry({
  strategies: [
    new ClaudeStrategy({ apiKey: process.env.CLAUDE_API_KEY }),
    new GeminiStrategy({ apiKey: process.env.GEMINI_API_KEY }),
  ],
  platform: "claude",
});

const orchestrator = new Orchestrator({ registry });
const result = await orchestrator.run(
  "Summarize this contract and flag any unusual liability clauses.",
  { synthesize: true },
);

for (const subtask of result.subtasks) {
  console.log(subtask.subtask.description, "->", subtask.decision?.provider, subtask.result);
}
console.log(result.final);

Routing is capability-tag matching first (TaskRouter's rule stage), falling back to a one-shot LLM classifier call when tags don't decide it. Each subtask runs through a bounded critique/retry loop (maxRounds, default 1) before being flagged unresolved: true in its result. One subtask's failure never aborts the run — see the task-orchestrator design doc for the full design.

Cost control

calculateCost turns LlmResponse.usage into a dollar figure using a pricing table you own and pass in (prices drift independently of this package's release cycle, so nothing is hardcoded). withBudget wraps any LlmStrategy to enforce spend limits across calls:

import { withBudget, type PricingTable } from "@idevconn/llm-router";

const pricing: PricingTable = {
  gemini: { "gemini-2.5-flash": { inputPer1M: 0.3, outputPer1M: 2.5 } },
};

const budgeted = withBudget(strategy, {
  pricing,
  maxCostPerCall: 0.5,
  maxCostTotal: 20,
  onCost: ({ provider, model, cost }) => console.log(provider, model, cost),
});

Cost is only known after a call returns (token counts come from the response), so maxCostPerCall is checked against the call that just finished, while maxCostTotal is checked up front against the accumulated total before the next call is allowed to start. Either limit being exceeded throws BudgetExceededError.

Instrumentation

withInstrumentation wraps any LlmStrategy and emits an LlmCallEvent on both success and failure — no bundled logger, you decide where events go:

import { compose, withBudget, withInstrumentation } from "@idevconn/llm-router";

const instrumented = compose(
  strategy,
  (s) => withBudget(s, { pricing }),
  (s) => withInstrumentation(s, { onCall: (event) => logger.info(event) }),
);

compose(s, a, b) behaves like b(a(s)), so decorators chain without manual nesting.

Resilience: retry, circuit breaker, rate limiting

Three more compose()-able decorators, same shape as withBudget/withInstrumentation. No new runtime dependency — all three are hand-rolled.

import { compose, withCircuitBreaker, withRateLimit, withRetry } from "@idevconn/llm-router";

const resilient = compose(
  strategy,
  (s) => withCircuitBreaker(s, { threshold: 5, samplingWindowMs: 60_000, resetTimeoutMs: 30_000 }),
  (s) => withRateLimit(s, { tokensPerSecond: 10, maxConcurrent: 5 }),
  (s) => withRetry(s, { maxAttempts: 3, initialBackoffMs: 200, maxBackoffMs: 2_000, multiplier: 2 }),
);

Put withRetry outermost (as above) — each retry attempt then re-enters the rate-limiter and circuit-breaker layers, so an open breaker or an exhausted rate-limit wait fails the whole call fast on the first attempt instead of wasting the remaining retry attempts. Note withRateLimit sits between the breaker and the provider in this order, so while the breaker is open, requests still consume limiter capacity before getting the fail-fast CircuitBreakerOpenError — that's the deliberate trade-off protecting the breaker's own failure count from being polluted by the limiter's throttling (both CircuitBreakerOpenError and RateLimitExceededError are excluded from what counts as a circuit-breaker failure, regardless of composition order). maxWaitMs on withRetry is a per-attempt limit, not a total call deadline — with the example config above, a fully-exhausted call can take up to roughly maxAttempts × (backoff + maxWaitMs).

• withCircuitBreaker — after threshold failures land inside the rolling samplingWindowMs, further calls throw CircuitBreakerOpenError immediately (no network call) until resetTimeoutMs elapses, then lets exactly one trial call through to test recovery. Caller-fault errors (InvalidGenerateOptionsError, InvalidThinkingConfigError, UnsupportedAttachmentError, UnsupportedThinkingModeError, UnsupportedMultiTurnError, BudgetExceededError) and an intentional signal abort never count as a failure.
• withRetry — exponential backoff (initialBackoffMs * multiplier^(attempt-1), capped at maxBackoffMs, jittered to 0.5-1.0x by default). Never retries a streaming call (onToken set) — a mid-stream failure means some tokens already reached the caller, and retrying would re-emit them from the start. Never retries the same caller-fault errors as above, nor CircuitBreakerOpenError/RateLimitExceededError, nor an aborted signal.
• withRateLimit — token bucket (tokensPerSecond, refilling continuously) plus an optional maxConcurrent in-flight cap. A call with no capacity blocks until capacity frees up, up to maxWaitMs (default 30s), then throws RateLimitExceededError.

All three respect signal — an abort during a backoff sleep or a capacity wait stops immediately rather than waiting out the full delay.

Prompt injection defense

Retrieved documents, tool output, and other untrusted text embedded in a prompt are a classic injection vector. sanitizeUntrustedContent wraps such text in explicit delimiters plus a data-only instruction; detectPromptInjection is a cheap synchronous heuristic gate meant to run before generation:

import { detectPromptInjection, sanitizeUntrustedContent } from "@idevconn/llm-router";

const { suspicious, reasons } = detectPromptInjection(userSuppliedText);
if (suspicious) {
  // log, reject, or route to stricter handling — reasons explains why
}

const prompt = `Answer using this context:\n${sanitizeUntrustedContent(retrievedDoc)}`;

For a probabilistic second opinion, detectPromptInjectionWithModel(text, strategy) runs the same check through an LlmStrategy — deliberately separate and opt-in, since it costs a model call.

RAG

EmbeddingStrategy mirrors LlmStrategy for embeddings providers, and VectorStore is a provider-neutral upsert/query/delete interface. Retriever ties them together and always runs retrieved chunks through sanitizeUntrustedContent before returning them:

import { Retriever } from "@idevconn/llm-router";
import { PgVectorStore } from "@idevconn/llm-router/embeddings/pgvector";
import { Pool } from "pg";

const vectorStore = new PgVectorStore({ pool: new Pool() });
const retriever = new Retriever({ embeddingStrategy, vectorStore });

const { chunks, sources } = await retriever.retrieve("What's our refund policy?", { topK: 5 });
const prompt = `Context:\n${chunks.join("\n\n")}\n\nQuestion: ...`;

PgVectorStore (subpath export, optional pg peer dependency) expects a table with id text primary key, embedding vector(n), and metadata jsonb columns — create the table and its pgvector index yourself; the adapter never runs DDL.

PgVectorStore is just one VectorStore implementation, not a hard dependency on Postgres — Supabase works out of the box (it's Postgres with pgvector under the hood, so it's just a Pool pointed at a Supabase connection string). A different backend (Mongo Atlas Vector Search, Firestore vector search, Pinecone, etc.) means writing a second implementation of the same three-method VectorStore interface (upsert/query/delete) and passing it to Retriever — the same opt-in-adapter pattern as adding a new LlmStrategy provider.

Error mapping

The pkg throws plain Error subclasses so it stays framework-agnostic. Wrap at the controller boundary:

// NestJS example
try {
  return await registry.getPlatform().generate(opts);
} catch (err) {
  if (err instanceof UnknownProviderError) throw new NotFoundException(err.message);
  if (err instanceof NoPlatformProviderError) throw new BadRequestException(err.message);
  if (err instanceof LlmKeyValidationError) throw new BadRequestException(err.message);
  if (err instanceof UnsupportedAttachmentError) throw new BadRequestException(err.message);
  if (err instanceof UnsupportedThinkingModeError) throw new BadRequestException(err.message);
  if (err instanceof InvalidThinkingConfigError) throw new BadRequestException(err.message);
  if (err instanceof BudgetExceededError) throw new HttpException(err.message, 402);
  if (err instanceof CircuitBreakerOpenError) throw new HttpException(err.message, 503);
  if (err instanceof RateLimitExceededError) throw new HttpException(err.message, 429);
  throw err;
}

Stability

Pre-1.0 — minor versions may break API. Pin a tilde range until the first real second consumer surfaces real-world feedback on the shape.

License

Apache-2.0