llm-fallback-mcp

MCP server that completes prompts with automatic provider fallback across OpenAI, Anthropic Claude, and Google Gemini. Built for production: rate-limit aware, retries on transient failures, transparent attempt log.

Why

LLM APIs go down. They rate-limit you. One provider has an outage, the others usually don't. Production LLM apps need fallback. This server gives you the pattern as a single MCP tool.

• Try OpenAI → Anthropic → Gemini in order (configurable)
• Each provider retries once on 429 / 5xx / network errors with backoff
• Return first success with a full per-provider attempt log
• Skip providers without keys automatically
• Zero SDK dependencies — calls each provider's REST API directly

Install

npm install -g llm-fallback-mcp

Or npx:

npx llm-fallback-mcp

Use with Claude Desktop

Add to claude_desktop_config.json:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "llm-fallback": {
      "command": "npx",
      "args": ["-y", "llm-fallback-mcp"],
      "env": {
        "OPENAI_API_KEY": "sk-…",
        "ANTHROPIC_API_KEY": "sk-ant-…",
        "GEMINI_API_KEY": "…"
      }
    }
  }
}

You only need to set the keys for the providers you actually want to use. Missing keys = provider skipped.

Restart Claude Desktop. Ask:

"Use llm-fallback to complete: 'Summarise war and peace in 3 lines'"

Tools

complete

Complete a prompt with automatic provider fallback.

| Arg | Type | Required | Default | Notes | |---|---|---|---|---| | prompt | string | yes | — | The prompt to send | | chain | string[] | no | ["openai", "anthropic", "gemini"] | Provider order | | model_overrides | object | no | — | Per-provider model id override | | temperature | number | no | 0.5 | 0–2 | | max_tokens | number | no | 1024 | Max output tokens |

Default models:

| Provider | Default model | |---|---| | openai | gpt-4o-mini | | anthropic | claude-haiku-4-5-20251001 | | gemini | gemini-2.0-flash |

health_check

Check which providers are configured (have API keys in env).

[
  { "provider": "openai",    "configured": true,  "envKey": "OPENAI_API_KEY",    "defaultModel": "gpt-4o-mini" },
  { "provider": "anthropic", "configured": false, "envKey": "ANTHROPIC_API_KEY", "defaultModel": "claude-haiku-4-5-20251001" },
  { "provider": "gemini",    "configured": true,  "envKey": "GEMINI_API_KEY",    "defaultModel": "gemini-2.0-flash" }
]

Example response

{
  "text": "War and Peace in 3 lines:\n1. A Russian aristocrat searches for meaning during the Napoleonic era.\n2. Three families' lives entwine across war, marriage, and philosophy.\n3. Tolstoy ultimately argues history bends to ordinary lives, not great men.",
  "provider_used": "anthropic",
  "model_used": "claude-haiku-4-5-20251001",
  "attempts": [
    { "provider": "openai", "model": "gpt-4o-mini", "ok": false, "error": "OpenAI 429: rate_limit_exceeded", "status": 429, "durationMs": 412 },
    { "provider": "anthropic", "model": "claude-haiku-4-5-20251001", "ok": true, "durationMs": 1133 }
  ]
}

Use cases

• Production LLM apps that need provider resiliency
• Dev workflows where one provider is rate-limited but you need to keep going
• Cost optimisation — try cheapest first, fall back on failure
• A/B chaining — different chains for different use cases via the chain arg

Custom provider order

// Cheapest-first
{ "prompt": "...", "chain": ["gemini", "openai", "anthropic"] }

// Anthropic-only with retry
{ "prompt": "...", "chain": ["anthropic"] }

Local development

git clone https://github.com/KhushalB25/llm-fallback-mcp.git
cd llm-fallback-mcp
npm install
npm run build
OPENAI_API_KEY=sk-... npm start

Inspect:

npx @modelcontextprotocol/inspector node dist/index.js

Author

Khushal Bhandari · GitHub

License

MIT