megallm-provider-router

A production-grade TypeScript NPM package for LLM provider health monitoring and intelligent provider selection with automatic key rotation, rate limit handling, and circuit breakers.

Features

• Provider Health Monitoring: Real-time tracking of provider health with scores (0-100)
• Intelligent Key Rotation: 7 strategies including round-robin, weighted, LRU, least-failures, random, priority-based, and latency-based
• Automatic Rate Limit Handling: Detects rate limits from headers and response bodies, automatically disables keys with exponential backoff
• Circuit Breaker Pattern: Prevents cascade failures by temporarily disabling unhealthy providers
• Per-Key Metrics: Success rate, latency, failure count, tokens per second, and more
• Request Queuing: Configurable concurrency limits per provider and per key
• Streaming Support: Full SSE and event-stream support for OpenAI and Anthropic formats
• Type-Safe Events: 13 strongly-typed events for monitoring
• Middleware System: Request and response transformation
• Hot-Reloading: Update configurations without restart
• Zero Dependencies: Minimal, well-maintained dev dependencies only
• Framework Agnostic: Works in Node.js and Bun

Installation

npm install megallm-provider-router

Quick Start

import { LLMProviderRouter } from 'megallm-provider-router';

// Setup router with providers
const router = new LLMProviderRouter({
  providers: [
    {
      type: 'openai',
      name: 'openai-main',
      baseUrl: 'https://api.openai.com',
      apiKeys: ['sk-...', 'sk-...'],
      rotationStrategy: 'round-robin',
    },
    {
      type: 'anthropic',
      name: 'anthropic-main',
      baseUrl: 'https://api.anthropic.com',
      apiKeys: ['sk-ant-...'],
    },
  ],
  logLevel: 'info',
});

// Get a specific provider
const openaiProvider = router.getProvider('openai-main');

// Use the provider directly (it handles key rotation, retries, rate limits automatically)
if (openaiProvider) {
  const response = await openaiProvider.createChatCompletion({
    model: 'gpt-4',
    messages: [{ role: 'user', content: 'Hello!' }],
  });
}

// Get the healthiest provider automatically
const healthyProvider = router.getHealthiestProvider();
if (healthyProvider) {
  // Use it...
}

// Get all healthy providers sorted by health
const healthyProviders = router.getHealthyProviders();
for (const provider of healthyProviders) {
  console.log(`${provider.getName()}: ${provider.getMetrics().healthScore}`);
}

Core Concept

This package monitors provider health and rotates API keys, but you decide which provider to use. No automatic model matching or fallback - just intelligent health-based provider selection.

// The router monitors health in the background
const router = new LLMProviderRouter({ providers: [...] });

// You decide which provider to use:
// Option 1: Get specific provider by name
const provider = router.getProvider('openai-main');

// Option 2: Get healthiest provider
const provider = router.getHealthiestProvider();

// Option 3: Get healthiest from a subset
const provider = router.getHealthiestProvider(['openai-main', 'openai-backup']);

// Then use the provider directly
const response = await provider.createChatCompletion({...});

API Methods

Router Methods

• getProvider(name) - Get a specific provider by name
• getHealthyProvider(name) - Get provider if healthy (checks circuit breaker)
• getHealthiestProvider(names?) - Get provider with highest health score
• getHealthyProviders() - Get all healthy providers sorted by health score
• getProviderMetrics(name) - Get metrics for specific provider
• getAllMetrics() - Get metrics for all providers
• getProviderNames() - List all provider names
• hasProvider(name) - Check if provider exists
• on/once/off(event, listener) - Event listeners
• use(middleware) - Add global middleware
• useProvider(name, middleware) - Add provider-specific middleware
• destroy() - Cleanup resources

Provider Methods (OpenAIProvider / AnthropicProvider)

• createChatCompletion(request, options?) - OpenAI chat completion
• createChatCompletionStream(request, options?) - OpenAI streaming
• createMessages(request, options?) - Anthropic messages
• createMessagesStream(request, options?) - Anthropic streaming
• getMetrics() - Get provider health metrics
• getName() - Get provider name

Configuration

Basic Configuration

const router = new LLMProviderRouter({
  providers: [
    {
      type: 'openai',
      name: 'my-provider',
      baseUrl: 'https://api.openai.com',
      apiKeys: ['sk-...'],
    },
  ],
});

Advanced Configuration

Every provider supports extensive configuration:

{
  type: 'openai' | 'anthropic',
  name: string,
  baseUrl: string | string[],
  apiKeys: string[] | ApiKeyConfig[] | Record<string, string[]>,

  // Key Rotation
  rotationStrategy?: 'round-robin' | 'weighted-round-robin' | 'least-recently-used'
                   | 'least-failures' | 'random' | 'priority-based' | 'latency-based',

  // Rate Limiting
  rateLimitCooldown?: number,           // Default: 60000ms

  // Circuit Breaker
  enableCircuitBreaker?: boolean,       // Default: true
  circuitBreakerThreshold?: number,     // Default: 5 failures
  circuitBreakerTimeout?: number,       // Default: 30000ms

  // Health Monitoring
  enableHealthCheck?: boolean,          // Default: true
  healthCheckInterval?: number,         // Default: 60000ms

  // Concurrency & Retries
  maxConcurrency?: number,              // Default: 10
  timeout?: number,                     // Default: 60000ms
  enableRetry?: boolean,                // Default: true
  maxRetries?: number,                  // Default: 3

  // Other
  headers?: Record<string, string>,
}

Examples

Multiple Keys with Weighted Rotation

const router = new LLMProviderRouter({
  providers: [
    {
      type: 'openai',
      name: 'openai-main',
      baseUrl: 'https://api.openai.com',
      apiKeys: [
        { key: 'sk-premium', weight: 3, priority: 1 },
        { key: 'sk-standard', weight: 2, priority: 2 },
        { key: 'sk-backup', weight: 1, priority: 3 },
      ],
      rotationStrategy: 'weighted-round-robin',
    },
  ],
});

const provider = router.getHealthyProvider('openai-main');

Multi-Region Setup

const router = new LLMProviderRouter({
  providers: [
    {
      type: 'openai',
      name: 'multi-region',
      baseUrl: ['https://api.openai.com', 'https://eu.api.openai.com'],
      apiKeys: {
        'https://api.openai.com': ['sk-us-1', 'sk-us-2'],
        'https://eu.api.openai.com': ['sk-eu-1', 'sk-eu-2'],
      },
    },
  ],
});

Azure OpenAI

The package automatically detects Azure endpoints and constructs the correct URL format:

const router = new LLMProviderRouter({
  providers: [
    {
      type: 'openai',
      name: 'azure-openai',
      baseUrl: 'https://your-resource.openai.azure.com',
      apiKeys: ['your-azure-key'],
      headers: {
        'api-key': 'your-azure-key', // Azure uses api-key header
      },
    },
  ],
});

// Automatically constructs: https://your-resource.openai.azure.com/openai/v1/chat/completions
const provider = router.getProvider('azure-openai');
const response = await provider.createChatCompletion({
  model: 'gpt-4',
  messages: [{ role: 'user', content: 'Hello!' }],
});

Azure Anthropic

Azure Anthropic endpoints are also automatically detected:

const router = new LLMProviderRouter({
  providers: [
    {
      type: 'anthropic',
      name: 'azure-anthropic',
      baseUrl: 'https://your-resource.services.ai.azure.com',
      apiKeys: ['your-azure-key'],
      headers: {
        'api-key': 'your-azure-key',
      },
    },
  ],
});

// Automatically constructs: https://your-resource.services.ai.azure.com/anthropic/v1/messages
const provider = router.getProvider('azure-anthropic');
const response = await provider.createMessages({
  model: 'claude-3-opus-20240229',
  messages: [{ role: 'user', content: 'Hello!' }],
  max_tokens: 1024,
});

Streaming Example

const provider = router.getHealthiestProvider();

if (provider instanceof OpenAIProvider) {
  for await (const chunk of provider.createChatCompletionStream({
    model: 'gpt-4',
    messages: [{ role: 'user', content: 'Tell me a story' }],
  })) {
    console.log(chunk.choices[0]?.delta?.content || '');
  }
}

Health Monitoring

// Get metrics for a provider
const metrics = router.getProviderMetrics('openai-main');
console.log({
  healthScore: metrics.healthScore,          // 0-100
  successRate: metrics.successRate,          // 0-1
  averageLatency: metrics.averageLatency,    // ms
  activeKeys: metrics.activeKeys,
  rateLimitedKeys: metrics.rateLimitedKeys,
  circuitBreakerOpen: metrics.circuitBreakerOpen,
});

// Get all healthy providers
const healthy = router.getHealthyProviders();
healthy.forEach(provider => {
  const m = provider.getMetrics();
  console.log(`${provider.getName()}: ${m.healthScore}/100`);
});

Event Monitoring

// Listen to health changes
router.on('provider:health_changed', (event) => {
  console.log(`${event.provider}: ${event.previousHealthScore} -> ${event.newHealthScore}`);
});

// Listen to rate limits
router.on('key:rate_limited', (event) => {
  console.log(`Key ${event.keyId} rate limited until ${event.resetTime}`);
});

// Listen to circuit breaker
router.on('circuit_breaker:opened', (event) => {
  console.log(`Circuit breaker opened for ${event.provider}`);
});

// Listen to key rotation
router.on('key:rotation', (event) => {
  console.log(`Rotated from ${event.previousKeyId} to ${event.newKeyId}`);
});

Error Handling

import {
  RateLimitError,
  AuthenticationError,
  ProviderDownError,
  TimeoutError,
} from 'megallm-provider-router';

const provider = router.getProvider('openai-main');

try {
  const response = await provider.createChatCompletion({...});
} catch (error) {
  if (error instanceof RateLimitError) {
    console.log(`Rate limited until ${error.resetTime}`);
    // Try another provider
    const backup = router.getHealthiestProvider();
  } else if (error instanceof AuthenticationError) {
    console.log('Invalid API key');
  } else if (error instanceof ProviderDownError) {
    console.log('Provider is down');
  }
}

Rotation Strategies

• round-robin: Cycle through keys in order
• weighted-round-robin: Select based on weights
• least-recently-used: Use the key that hasn't been used in longest time
• least-failures: Use the key with fewest failures
• random: Random selection
• priority-based: Use highest priority keys first (lowest priority number)
• latency-based: Use the key with lowest average latency

Supported Providers

The package works with any OpenAI or Anthropic compatible API:

OpenAI-compatible:

• OpenAI
• Azure OpenAI
• NVIDIA NIM
• DigitalOcean
• Together AI
• Groq
• Perplexity
• Any custom OpenAI-compatible endpoint

Anthropic-compatible:

• Anthropic
• Azure Anthropic
• AWS Bedrock Claude
• Vertex AI Claude
• Any custom Anthropic-compatible endpoint

License

MIT