opencode-failover

OpenCode plugin for automatic API-key failover and rotation across multiple provider keys.

opencode-failover is a OpenCode plugin that automatically rotates API keys across multiple provider credentials. When a key hits a rate limit, the plugin quarantines it and switches to the next available key -- zero downtime, zero manual intervention.

• Multiple keys per provider with weighted round-robin distribution
• Automatic quarantine on rate-limit (429) with exponential backoff
• Permanent disable on auth failure (401/403)
• Works with NVIDIA NIM, OpenRouter, Anthropic, OpenAI, and any OpenCode-compatible provider

Quick Start

opencode plugin opencode-failover

Set your API keys. Ask the LLM in the TUI (natural language):

Add these NVIDIA API keys for failover rotation: nvapi-xxx, nvapi-yyy, nvapi-zzz

The plugin saves keys to .env and activates automatically.

Or create .env manually:

NVIDIA_API_KEYS="nvapi-xxx,nvapi-yyy,nvapi-zzz"

Restart OpenCode. The plugin activates automatically and begins rotating keys.

Quick Prompt (copy & paste)

Open the opencode TUI and send this to the LLM:

Add these NVIDIA API keys for failover rotation: nvapi-key1, nvapi-key2, nvapi-key3

The plugin saves them to .env. Works immediately in the current session.

To check key status later, ask:

Show me the keychain status

To remove keys:

Remove all NVIDIA API keys from the keychain

Or remove a specific key:

Remove nvapi-key1 from NVIDIA

Most LLM providers enforce per-key rate limits. When you hit the limit, requests fail and you are stuck waiting.

opencode-failover solves this by:

• Rotating to the next available key on rate-limit (429)
• Quarantining exhausted keys with exponential backoff (60s to 300s)
• Disabling permanently on auth failure (401/403)
• Recovering quarantined keys automatically when their timer expires

One provider, multiple keys, zero downtime.

Features

• Weighted round-robin key rotation
• Exponential backoff quarantine (60s, 120s, 240s, 300s cap)
• retry-after header respect (milliseconds, seconds, HTTP-date formats)
• Permanent disable on auth errors (401/403) and billing errors (402)
• Temporary quarantine on server errors (5xx)
• Rate-limit pattern detection (Anthropic, OpenAI, and generic patterns)
• keychain-status tool for real-time key monitoring
• /keychain slash command for live key status in the TUI
• Debug logging via OPENCODE_FAILOVER_DEBUG=1
• Works with any OpenCode-compatible provider

Installation

Option 1: OpenCode CLI (recommended)

opencode plugin opencode-failover

This installs the package and updates your opencode.json automatically.

Option 2: npm

npm install opencode-failover

Then add to your opencode.json:

{
  "plugin": ["opencode-failover"]
}

Option 3: Local development

git clone https://github.com/bulutmuf/opencode-failover.git
cd opencode-failover
bun install

Then configure OpenCode to load it from the source directory. Add to your opencode.json:

{
  "plugin": ["/path/to/opencode-failover/src/index.ts"]
}

Configuration

Environment variables

Single provider (comma-separated keys):

export NVIDIA_API_KEYS="nvapi-key1,nvapi-key2,nvapi-key3"

Multiple providers (JSON):

export OPENCODE_FAILOVER_PROVIDERS='{
  "nvidia": {
    "keys": ["nvapi-key1", "nvapi-key2", "nvapi-key3"],
    "scheme": "Bearer"
  },
  "openrouter": {
    "keys": ["sk-or-key1", "sk-or-key2"],
    "header": "Authorization"
  }
}'

opencode.json options

{
  "plugin": [
    ["opencode-failover", {
      "providers": {
        "nvidia": {
          "keys": ["nvapi-key1", "nvapi-key2", "nvapi-key3"],
          "scheme": "Bearer"
        },
        "openrouter": {
          "keys": ["sk-or-key1", "sk-or-key2"],
          "header": "Authorization"
        }
      }
    }]
  ]
}

Config options

| Option | Type | Default | Description | |--------|------|---------|-------------| | keys | string[] | required | API keys for this provider | | header | string | "Authorization" | HTTP header to inject | | scheme | string | "Bearer" | Header value prefix | | weight | Record<string, number> | {} | Per-key rotation weights |

Precedence

1. OPENCODE_FAILOVER_PROVIDERS env (full JSON) overrides everything
2. OPENCODE_FAILOVER_KEYS env (JSON key map) overrides individual env vars
3. <PROVIDER>_API_KEYS env + opencode.json options (merged)
4. opencode.json options only

How It Works

Request    -->  Plugin intercepts via global fetch patch
             -->  Identifies provider by existing Authorization header
             -->  Picks next key (weighted round-robin)
             -->  Sets Authorization header
             -->  OpenCode makes LLM call
             -->  Success? Done.
             -->  Error?   Plugin classifies HTTP response:
                    429 / rate-limit  -->  Quarantine key (exponential backoff)
                    401 / 403 / 402  -->  Disable key permanently
                    5xx              -->  Quarantine key temporarily
                    ResourceExhausted -->  Overload action (2s backoff)
             -->  Next request picks a different key
             -->  Quarantined keys auto-release when timer expires

Error classification

| Error | Action | Behavior | |-------|--------|----------| | 429 | Quarantine | Exponential backoff: 60s, 120s, 240s, 300s cap | | 401 / 403 | Disable | Permanent, requires manual re-enable | | 402 | Disable | Billing error, permanent | | 5xx | Quarantine | Temporary, same backoff as 429 | | Rate-limit pattern | Quarantine | Detected in body/message text | | Overload pattern | Quarantine | 2s backoff (ResourceExhausted) | | Other | Ignore | No action taken |

Quarantine schedule

| Consecutive errors | Quarantine duration | |-------------------|-------------------| | 1 | 60 seconds | | 2 | 120 seconds | | 3 | 240 seconds | | 4+ | 300 seconds (cap) |

If the provider returns a retry-after header, that value overrides the schedule.

Tools (LLM natural language)

These are tools the LLM can call. Say them in natural language:

| Tool | What to say | Description | |------|-------------|-------------| | keychain-setup | "Add these API keys for nvidia: key1, key2" | Save API keys for a provider to .env (appends to existing) | | keychain-remove | "Remove all nvidia API keys" or "Remove key1 from nvidia" | Remove all or specific API keys for a provider from .env | | keychain-status | "Show me the keychain status" | Show all configured keys, their status, weights, and retry timers | | keychain-reset | "Reset the keychain" | Reset all quarantined keys back to active state immediately |

Example output:

## nvidia
  nvapi...abc [w=2] -- active
  nvapi...def -- QUARANTINED until 2026-07-04T12:35:00.000Z
  nvapi...ghi -- active
  [2 active, 1 quarantined, 0 disabled]

TUI Dashboard

Type /keychain in the prompt (or /failover, /opencode-failover) to open the keychain dashboard.

The dashboard displays live key counts and allows you to instantly search and select models from all your configured providers, automatically switching the current session.

The dashboard is read-only — use LLM tools (keychain-setup/keychain-remove) to manage keys.

The TUI plugin auto-registers on first start. A ~/.config/opencode/tui.json backup is created before any changes.

Supported Providers

Works with any provider that uses API key authentication:

| Provider | Env var | Default scheme | Aliases | |----------|---------|----------------|---------| | NVIDIA NIM | NVIDIA_API_KEYS | Bearer | nim | | OpenRouter | OPENROUTER_API_KEYS | Bearer | | | Anthropic | ANTHROPIC_API_KEYS | Bearer | | | OpenAI | OPENAI_API_KEYS | Bearer | | | Cloudflare Workers AI | CLOUDFLARE_WORKERS_AI_API_KEYS | Bearer | cf, cloudflare | | Any custom | <PROVIDER>_API_KEYS | Bearer | |

Note: For Cloudflare Workers AI, you must provide your Account ID as well: Add this cloudflare key: xxx with account_id: yyy.

The provider ID must match the providerID used in your OpenCode model configuration.

Development

git clone https://github.com/bulutmuf/opencode-failover.git
cd opencode-failover
bun install
bun test

Debug mode

OPENCODE_FAILOVER_DEBUG=1 opencode

Logs key injection, quarantine decisions, and provider pool initialization.

Project structure

src/
  index.ts          Plugin factory: hooks wiring + tools
  lib/
    config.ts         Env + options parser
    state.ts          KeyPool: rotation, quarantine, backoff
    classify.ts       Error classifier: status/body -> action
    auth.ts           Native key backup/restore from auth.json
    fetch-patch.ts    Global fetch monkey-patch interceptor
    shared.ts         Masked state file (~/.opencode/failover-state.json)
    version-check.ts  npm registry poll for new versions
    tui.tsx           TUI dashboard: /keychain slash command
  state.test.ts     7 tests
  classify.test.ts  19 tests
  config.test.ts    7 tests
  auth.test.ts      6 tests
documents/
  00-architecture.md
  01-error-patterns.md
  02-quarantine-strategy.md
  03-decisions.md
  04-provider-guides.md
  05-troubleshooting.md
  06-security.md
  07-changelog.md
  08-contributing.md
  09-faq.md

Architecture

See documents/ for detailed Architecture Decision Records:

| Document | Description | |----------|-------------| | Architecture Overview | Hook surface, module layout, OpenCode integration | | Error Classification | Decision table, pattern matching, retry-after parsing | | Quarantine Strategy | Exponential backoff, cap, recovery semantics | | Design Decisions | Naming, scope, config precedence, distribution | | Provider Guides | Per-provider setup (NVIDIA, OpenRouter, Anthropic, OpenAI, custom) | | Troubleshooting | Common issues, debug mode, key status inspection | | Security | Key masking, env safety, production risks | | Changelog | v0.1.0 release notes | | Contributing | Dev setup, commit format, test rules, PR flow | | FAQ | Provider compatibility, edge cases, debugging |

Contributing

Contributions are welcome. Please open an issue or submit a pull request.

License

This project is licensed under the MIT License.

Copyright (c) 2026 bulutmuf

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.