ai-prod-guard

Production guard for AI workloads: hard spend caps, intelligent retries, automatic failover, and provider health tracking.

For AI businesses that need runtime protection across multiple services.

import { createRuntimeGuard } from "ai-prod-guard";
import OpenAI from "openai";
import Anthropic from "@anthropic-ai/sdk";

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

const guard = createRuntimeGuard({
  maxCostPerRequest: 1.0,
  maxCostPerSession: 50.0,
  maxRetries: 3,
  primaryProvider: {
    name: "openai",
    call: (req, signal) => openai.chat.completions.create(req, { signal }),
    estimateCost: () => 0.03,
  },
  fallbackProviders: [
    {
      name: "anthropic",
      call: (req, signal) => anthropic.messages.create({ ...req, signal }),
      estimateCost: () => 0.02,
    },
  ],
  onRetry: (e) => console.log(`Retry ${e.attempt} for ${e.provider}`),
  onFallback: (e) => console.log(`Switched to ${e.nextProvider}`),
  onLimitHit: (e) => console.log(`Hit ${e.limitType}`),
});

const response = await guard.call(async (provider, signal) => {
  return await provider.call({
    model: "gpt-4o",
    messages: [{ role: "user", content: "Generate report" }],
  }, signal);
});

const { confirmedUsd, potentialUsd } = guard.getSpendSummary();
console.log(`Confirmed: $${confirmedUsd}, Potential: $${potentialUsd}`);

Four Core Problems

1. Runaway Spend

Without hard limits, a single retry loop or misconfigured request can spiral into unexpected charges.

ai-prod-guard enforces per-request and per-session cost caps before the network call is made. No surprises on your invoice.

maxCostPerRequest: 1.0,    // Block requests over $1
maxCostPerSession: 50.0,   // Block if session exceeds $50

2. Retry / 429 / Cooldown

Rate limits and transient failures require intelligent backoff. Teams manually retry, lose requests, or give up.

ai-prod-guard automatically retries 429, 408, and 5xx errors with Retry-After backoff and exponential cooldown. Production-grade retry logic out of the box.

maxRetries: 3,     // Retry up to 3 times
cooldownMs: 500,   // Start with 500ms, exponential growth

3. Fallback-on-Failure

When the primary provider is down or rate-limited, requests fail. Teams need a fast failover.

ai-prod-guard seamlessly switches to backup providers when primary exhausts retries or returns a non-retryable error (401, 403, etc.). Keep your service running.

primaryProvider: openai,
fallbackProviders: [anthropic, cohere],  // Tried in order

4. Provider Health Memory

One provider goes down, all requests fail. Teams lack visibility into provider reliability.

ai-prod-guard tracks provider health per instance: marks unhealthy providers, skips them on retry, and auto-recovers after a configurable interval. Local visibility into what's failing.

healthCheckIntervalMs: 60_000,  // Auto-recover after 1 minute
guard.getProviderHealth("openai");  // { healthy, lastFailure, unhealthyUntil }

Features

• ✅ Cost limits: per-request and per-session spend caps (checked before network calls)
• ✅ Spend tracking: confirmed vs potential buckets (timeouts recorded as potential, not ignored)
• ✅ Intelligent retries: 429, 408, 5xx with Retry-After and exponential backoff
• ✅ Provider fallback: automatic failover to backup providers in order
• ✅ Provider health: tracks unhealthy providers, auto-recovers after interval
• ✅ Timeout enforcement: Promise.race + AbortSignal (guarantees client stops waiting)
• ✅ Event hooks: onRetry, onFallback, onLimitHit for observability
• ✅ Concurrent-safe caps: atomic reservation prevents concurrent calls from exceeding limits

Installation

npm install ai-prod-guard

API

createRuntimeGuard(options)

interface RuntimeGuardOptions {
  // Cost limits
  maxCostPerRequest?: number;             // Block requests over this USD amount
  maxCostPerSession?: number;             // Block if session would exceed this USD amount

  // Retry behavior
  maxRetries?: number;                    // default: 2
  cooldownMs?: number;                    // default: 500 (exponential backoff)
  maxAttemptsPerRequest?: number;         // default: (maxRetries+1) * providerCount
  timeoutMs?: number;                     // default: 30_000

  // Providers
  primaryProvider: ProviderConfig;
  fallbackProviders?: ProviderConfig[];

  // Health
  healthCheckIntervalMs?: number;         // default: 60_000

  // Hooks
  onRetry?: (event: RetryEvent) => void;
  onFallback?: (event: FallbackEvent) => void;
  onLimitHit?: (event: LimitEvent) => void;
}

interface ProviderConfig {
  name: string;
  call: (request: unknown, signal?: AbortSignal) => Promise<unknown>;
  estimateCost?: (attempt: number) => number;
}

guard.call(fn)

Execute a guarded call.

const result = await guard.call(async (provider, signal) => {
  return await provider.call({ /* request */ }, signal);
});

guard.getSpendSummary()

Get spend breakdown. Do not merge these numbers—they mean different things.

const { confirmedUsd, potentialUsd, totalExposureUsd } = guard.getSpendSummary();
// confirmedUsd: successfully completed calls (definitely billed)
// potentialUsd: timed out or disconnected calls (may have billed)
// totalExposureUsd: confirmed + potential + in-flight

guard.getSessionSpentUsd()

Conservative total: confirmedUsd + potentialUsd.

const spent = guard.getSessionSpentUsd();

guard.resetSession()

Reset session spend and provider health.

guard.resetSession();

guard.getProviderHealth(name) / guard.getAllProviderHealth()

Check provider health.

const health = guard.getProviderHealth("openai");
// { name, healthy, lastFailure?, unhealthyUntil? }

Spend Accounting

Three buckets (never merge):

| Bucket | Meaning | |--------|---------| | confirmedUsd | Calls that returned successfully. You are billed. | | potentialUsd | Calls that timed out or disconnected. Provider may have kept running and billed. Upper bound on surprise charges. | | reservedUsd | In-flight concurrent calls. Included in totalExposureUsd. |

429 and other rejections are not recorded (provider rejected the request, no charge).

Retry Behavior

| Error | Action | |---|---| | 429 (rate limit) | Retry with Retry-After backoff | | 408 (timeout) | Retry immediately | | 5xx (server error) | Retry with exponential backoff | | 4xx (except 408/429) | No retry, mark provider unhealthy, try fallback | | Network error / timeout | Retry, then fallback |

Provider Health

Providers are marked unhealthy after non-retryable errors (401, 403, etc.). Unhealthy providers are skipped on retries.

Health is reset after healthCheckIntervalMs (default 60 seconds). On the next request, the provider is tried again.

Event Hooks

onRetry

Fired when a request is retried after a transient error.

onRetry: (event) => {
  // event.requestId, attempt, error, retryAfter, provider
  metrics.increment("ai.retry", { provider: event.provider });
}

onFallback

Fired when switching to a fallback provider.

onFallback: (event) => {
  // event.requestId, failedProvider, nextProvider, reason
  logger.warn(`Fallback: ${event.failedProvider} → ${event.nextProvider}`);
}

onLimitHit

Fired when a cost or attempt limit is exceeded.

onLimitHit: (event) => {
  // event.requestId, limitType, current, limit
  if (event.limitType === "session_cost") {
    logger.error(`Session limit: $${event.current} > $${event.limit}`);
  }
}

Limitations

⚠️ Per-instance only — Spend caps and provider health are local to one RuntimeGuard instance. For multi-service limits, use a backend system.

⚠️ Cost estimates required — estimateCost() is called before each request. Actual provider costs may differ.

⚠️ Timeout does not guarantee upstream cancellation — If the provider SDK does not support AbortSignal, the provider may keep running after the client times out.

⚠️ Provider health not persisted — Health state is lost on process restart.

Local Development vs Production

• Local dev: Use badgr-auto — a CLI tool that helps you debug retry issues, test providers, and understand AI patterns before shipping.

• Production: Use ai-prod-guard — a runtime guard that protects your deployed services with cost caps, retry logic, failover, and health tracking.

Requirements

• Node.js 18+
• Optional: Provider SDKs that support AbortSignal (OpenAI, Anthropic, etc.)

License

MIT