llm-fallback

Automatic LLM provider fallback with circuit breaker. Zero dependencies. TypeScript-first.

LLM APIs go down. Rate limits hit. Keys expire. When your app relies on a single provider, any outage becomes your outage.

llm-fallback automatically tries the next provider in your list — with a built-in circuit breaker so a broken provider doesn't slow down every request.

Install

npm install llm-fallback

Usage

import { createFallback, openai, anthropic, gemini } from 'llm-fallback'

const client = createFallback([
  openai({ apiKey: process.env.OPENAI_KEY }),
  anthropic({ apiKey: process.env.ANTHROPIC_KEY }),
  gemini({ apiKey: process.env.GEMINI_KEY }),
])

const result = await client.complete({
  messages: [{ role: 'user', content: 'Hello!' }],
})

console.log(result.content)   // "Hello! How can I help you today?"
console.log(result.provider)  // "openai" (or "anthropic" if openai failed)

If OpenAI fails → Anthropic is tried. If Anthropic fails → Gemini is tried. All transparent to your app.

How it works

1. Ordered fallback — providers are tried in the order you list them
2. Circuit breaker — after N failures, a provider is skipped for a cooldown period, then retried
3. Per-request timeout — each provider call has a timeout; slow providers don't block fallback
4. Retries per provider — optionally retry a provider before giving up on it

Request → OpenAI (timeout 10s)
  ↓ fails
Request → Anthropic (timeout 10s)
  ↓ succeeds
Response ✓

API

createFallback(adapters, options?)

const client = createFallback(
  [openai({ apiKey: '...' }), anthropic({ apiKey: '...' })],
  {
    timeout: 10000,           // ms per provider (default: 10000)
    retries: 1,               // retries per provider before fallback (default: 0)
    circuitBreaker: {
      threshold: 3,           // failures before circuit opens (default: 3)
      resetAfter: 60000,      // ms before circuit half-opens (default: 60000)
    },
  }
)

client.complete(options)

const result = await client.complete({
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: 'What is 2 + 2?' },
  ],
  maxTokens: 512,
  temperature: 0.7,
})

Returns:

{
  content: string        // LLM response text
  provider: string       // which provider responded ("openai" | "anthropic" | "gemini")
  model: string          // actual model used
  usage?: {
    promptTokens: number
    completionTokens: number
  }
}

client.getCircuitStates()

client.getCircuitStates()
// { openai: "closed", anthropic: "open", gemini: "closed" }

Useful for health-check endpoints and monitoring dashboards.

Built-in adapters

| Adapter | Default model | |---|---| | openai({ apiKey }) | gpt-4o-mini | | anthropic({ apiKey }) | claude-haiku-4-5-20251001 | | gemini({ apiKey }) | gemini-1.5-flash |

Override the model:

openai({ apiKey: '...', model: 'gpt-4o' })
anthropic({ apiKey: '...', model: 'claude-opus-4-7' })

Custom adapter

Any object with { name, complete } works:

import type { Adapter } from 'llm-fallback'

const myAdapter: Adapter = {
  name: 'my-provider',
  async complete(options, signal) {
    // call your LLM API here
    return { content: '...', provider: 'my-provider', model: 'my-model' }
  }
}

const client = createFallback([myAdapter, openai({ apiKey: '...' })])

FallbackError

Thrown when all providers fail:

import { FallbackError } from 'llm-fallback'

try {
  await client.complete({ messages: [...] })
} catch (err) {
  if (err instanceof FallbackError) {
    console.log(err.errors)
    // [
    //   { provider: 'openai', error: Error('OpenAI 429: rate limit') },
    //   { provider: 'anthropic', error: Error('Circuit breaker open') },
    // ]
  }
}

Production example

import { createFallback, openai, anthropic } from 'llm-fallback'

const llm = createFallback(
  [
    openai({ apiKey: process.env.OPENAI_KEY, model: 'gpt-4o' }),
    anthropic({ apiKey: process.env.ANTHROPIC_KEY, model: 'claude-sonnet-4-6' }),
  ],
  {
    timeout: 8000,
    retries: 1,
    circuitBreaker: { threshold: 5, resetAfter: 120000 },
  }
)

// Health check endpoint
app.get('/health/llm', (req, res) => {
  res.json(llm.getCircuitStates())
})

// Chat endpoint
app.post('/api/chat', async (req, res) => {
  const result = await llm.complete({ messages: req.body.messages })
  res.json({ reply: result.content, via: result.provider })
})

Why llm-fallback?

• Zero dependencies — uses native fetch (Node 18+)
• TypeScript-first — full types included
• Circuit breaker built in — failing providers don't slow down requests
• Provider-agnostic — bring your own adapter for any LLM
• Works anywhere — Node.js, Edge, Bun, Deno

License

MIT © Sufiyan Khan