@zhivex-ai/gateway

Vertex partner routing

Register one Vertex adapter with Google Cloud credentials. The publisher stays inside modelId; the gateway provider remains vertex for Google-hosted models.

import { createGateway } from "@zhivex-ai/gateway";
import { createVertex } from "@zhivex-ai/vertex";

const gateway = createGateway({
  adapters: { vertex: createVertex({
    projectId: process.env.GOOGLE_CLOUD_PROJECT,
    location: "global"
  }) },
  maxRetries: 0
});

const result = await gateway.generate({
  primary: { provider: "vertex", modelId: "openai/gpt-oss-120b-maas" },
  fallbacks: [{ provider: "vertex", modelId: "gemini-3.7-flash" }],
  messages: [{ role: "user", content: "Explain a binary search." }]
});

This requires working ADC or another bearer credential resolved by Vertex. Model permissions, regional availability and quotas apply separately to each target. See the Vertex provider for route-specific limits.

Routing and fallback package for Zhivex AI SDK.

The gateway now supports:

• generate()
• streamText()
• generateObject()
• streamObject()
• runAgent()
• streamAgent()

Tool loops run in one Core execution; later model steps can fail over without replaying completed tools. Streaming fallbacks are resolved before the first provider event is emitted.

For agent routing, the gateway can also filter by agentCapabilities, such as provider support tier or approval-capable MCP support, before selecting the final target.

Install

bun add @zhivex-ai/gateway @zhivex-ai/core @zhivex-ai/anthropic @zhivex-ai/openai @zhivex-ai/ollama @zhivex-ai/vertex

Install the provider packages used by your own adapter map; these examples use OpenAI, Ollama, Anthropic and Vertex.

Usage

import { createGateway } from "@zhivex-ai/gateway";
import { createOpenAI } from "@zhivex-ai/openai";
import { createOllama } from "@zhivex-ai/ollama";

const gateway = createGateway({
  adapters: {
    openai: createOpenAI({ apiKey: process.env.OPENAI_API_KEY }),
    ollama: createOllama()
  },
  maxFallbacks: 8,
  maxRetries: 1,
  maxTotalAttempts: 16,
  attemptTimeoutMs: 15_000,
  streamIdleTimeoutMs: 60_000,
  observerTimeoutMs: 1_000,
  unknownCostPolicy: "reject"
});

const abortController = new AbortController();

const result = await gateway.generate({
  primary: { provider: "openai", modelId: "gpt-4o-mini" },
  fallbacks: [{ provider: "ollama", modelId: "llama3.2" }],
  messages: [{ role: "user", content: "Summarize the benefits of fallback routing." }],
  routingMode: "balanced",
  maxCostPer1kTokens: 0.01,
  abortSignal: abortController.signal
});

console.log(result.text);
console.log(result.providerUsed);
console.log(result.attempts);

The gateway also supports streamText(), generateObject(), and streamObject() through one Core generation loop. A provider is fixed once its stream emits, while a later model step in the same tool loop can still fail over before emitting provider output. Object routes skip incompatible targets before making a provider call: native mode requires structuredOutput, prompted mode requires jsonMode, and auto mode accepts either capability. In auto mode, the gateway prepares native output or a schema prompt for each destination independently, and objectMode reports the mode used by the final provider.

Routing guarantees

• Text, object, and agent operations retry eligible failures on the current target and then continue through the ordered fallback targets. Agent routing happens inside one runAgent() or streamAgent() execution, so a fallback does not restart the agent, duplicate its run, or replay completed tools.
• Text and object streaming fallback is resolved before the first event is exposed. Agent streams may expose lifecycle events such as agent-run-start first, but provider fallback is resolved before the first provider event. Once a provider stream emits an event, an error from that stream is propagated without mixing in another provider's transcript.
• attemptTimeoutMs and the per-provider attemptTimeoutsMs do more than reject the gateway promise: they abort a non-streaming provider call or a streaming call that has not produced its first event. After the first event, streamIdleTimeoutMs and streamIdleTimeoutsMs abort a provider that stops producing events; the default is 60 seconds and false explicitly disables it. A request-level abortSignal remains active for the full operation and stops pending retries, backoff, fallback routing, observers, and active streams.
• ProviderHTTPError is classified by its typed HTTP status. Status 408, 429, and 5xx errors are retryable on the same target. Other 4xx errors are not retried on that target, but an eligible fallback can still handle a provider- or model-specific rejection.
• Stream attempts record provider-success only after the provider iterator completes; latencyMs covers the full attempt. Failures after the first event, including provider error events and cancellation, produce a failed attempt without retry/fallback. onAgentRoute still fires on selection after the first provider event, before the terminal attempt exists. Iterator cleanup is best-effort and never delays timeout/cancellation or replaces the original failure.
• Retry waits honor finite non-negative ProviderHTTPError.retryAfterMs, taking the greater of that value and the configured linear backoff, capped at 60 seconds per wait. Caller cancellation interrupts the wait.
• Attempt diagnostics redact credential-like URL parameters and bearer tokens before they are exposed through attempts[].errorMessage or observers.
• When maxCostPer1kTokens is set, a target without configured or catalog pricing is rejected by default. Set unknownCostPolicy: "allow" on createGateway() only when routing to models with unknown cost is acceptable.
• Requests containing image attachments only route to models that declare capabilities.vision: true. The gateway never removes images to make a target appear compatible; if one target cannot accept the original request, it is skipped in favor of a compatible fallback.
• scoreTarget(context) can replace the built-in name-based heuristic with application metrics. It must return a finite number; higher scores route first.
• Routing amplification is bounded even if a request is assembled from external input: requests accept at most maxFallbacks targets (default 8, hard maximum 32), maxRetries cannot exceed 5, and maxTotalAttempts caps provider calls across the whole routed operation including later agent steps (default 32, hard maximum 128). Model IDs are non-empty, limited to 256 characters, and cannot contain control characters. maxCostPer1kTokens and configured/catalog costs must be finite and non-negative.
• onAttempt and onAgentRoute are best-effort observers. Legacy callbacks run detached; observerMode: "await" explicitly awaits attempt callbacks and "background" queues them with bounded capacity. They receive an abortSignal and are allowed observerTimeoutMs to finish (default 1 second); rejection, timeout, or request cancellation cannot retry successful provider work or block routing indefinitely.

Keep primary/fallback selection and these ceilings under application control when mapping an HTTP request into GatewayRequest. maxCostPer1kTokens limits model price, not the final invoice; use maxTotalAttempts, provider-side spend limits, authentication, and rate limiting for a complete cost boundary.

For agent workloads, use runAgent() or streamAgent() to route by both regular model capabilities and agent-specific capabilities such as supportTier, approvalRequests, or remoteMcp.

const agentResult = await gateway.runAgent({
  primary: { provider: "kimi", modelId: "kimi-k3" },
  fallbacks: [{ provider: "qwen", modelId: "qwen-plus" }],
  prompt: "Use hosted retrieval and an MCP map server.",
  requiredAgentCapabilities: {
    supportTier: "tier-b",
    hostedFileSearch: true,
    remoteMcp: true
  }
});

console.log(agentResult.providerUsed);
console.log(agentResult.attempts);
console.log(agentResult.routeDecision);

Agent requests also forward the durable Core controls runId, scope, idempotencyKey, parentRunId, policy, and toolApprovalPolicy. Use a durable store plus idempotencyKey for effectful production runs.

Migration note

GatewayConfig.groundedAdapters has been removed because it was not used by any gateway operation. Register provider adapters through adapters; @zhivex-ai/gateway does not currently expose a grounded-generation route.

This package is the SDK-local routing layer. It is not the Zhivex-hosted Gateway API and it is not re-exported from @zhivex-ai/sdk.

Repository and full documentation:

• https://github.com/Zhivex/zhivex-ai-sdk

Continuing canonical tool history

GatewayRequest.messages accepts GatewayInputMessage[], a union of the existing GatewayMessage (role/content/images) and core ModelMessage (role/parts). Existing text/image requests remain valid. Each message must use exactly one representation; content or images combined with parts is rejected.

import type { ModelMessage } from "@zhivex-ai/core";
import { createAnthropic } from "@zhivex-ai/anthropic";
import { createGateway, type GatewayRequest } from "@zhivex-ai/gateway";

const gateway = createGateway({ adapters: { anthropic: createAnthropic() } });
const messages: ModelMessage[] = [
  { role: "user", parts: [{ type: "text", text: "What is the temperature in Buenos Aires?" }] },
  { role: "assistant", parts: [{
    type: "tool-call",
    toolCall: { id: "call_weather_1", name: "weather", input: { city: "Buenos Aires" } }
  }] },
  { role: "tool", parts: [{
    type: "tool-result",
    toolResult: {
      toolCallId: "call_weather_1", toolName: "weather",
      output: { temperatureC: 18 }, isError: false
    }
  }] }
];
const request: GatewayRequest = {
  messages,
  primary: { provider: "anthropic", modelId: "claude-sonnet-4-6" },
  fallbacks: [{ provider: "anthropic", modelId: "claude-opus-4-6" }]
};
const answer = await gateway.generate(request);
const streamedAnswer = await gateway.streamText(request).collect();

Historical results are input to the model and never invoke a tool executor. New tool calls retain the existing tools, toolExecution, and maxSteps policy. For an error result, set isError: true and error: { message: "..." } instead of output. Successful results require a JSON output, including strings or null. Tool inputs must be JSON objects. IDs must be unique across the request; results must match both ID and tool name, occur after their calls, and resolve every call before another conversation turn. Multiple calls and results retain input order; results can arrive in a different ID order, grouped in one or consecutive tool messages. System messages must precede conversation turns.

| Surface / destination | Canonical history support | | --- | --- | | generate, streamText | Supported, including usage estimates, attempts, finish reasons, cancellation and routing | | generateObject, streamObject | Same input validation and routing; destination must also support the requested object mode | | runAgent, streamAgent | Fresh canonical history imports; durable resumes use state/runId without resupplying messages | | Anthropic Messages | Text, user images, assistant tool calls, tool results and native is_error | | OpenAI Chat / Responses | Callable tool history, including multiple results; explicit JSON success/error envelopes | | DeepSeek Chat | Same JSON envelope support; portable replay defaults to non-thinking; reasoning-only models are excluded | | Qwen Chat / Responses | Same JSON envelope support; portable replay defaults to non-thinking; thinking-only models are excluded | | Other / older provider adapters | Must explicitly advertise capabilities.toolHistory; otherwise skipped with model-capabilities, even if tools is true |

The gateway is extensible through ModelCapabilities.toolHistory: "native" means the adapter preserves error state natively, and "json" means it honors ModelGenerateInput.toolResultFormat: "envelope". Anthropic is also accepted for backward compatibility with its existing native mapping. An older OpenAI, DeepSeek or Qwen adapter without the declaration is skipped safely. Custom adapters can declare support after implementing and testing the same contract; there is no provider-name allowlist for opting in.

For JSON transports, each tool result is a separate Chat tool message or Responses function_call_output. Its content/output string contains { "output": <JSON> } on success or { "error": { "message": "..." } } on failure. This discriminant keeps an error-shaped successful value distinguishable from an actual failure. Calls and results retain their IDs and order. The original canonical messages and isError remain unchanged. Direct SDK calls keep their prior raw result format unless they explicitly request the new low-level format. Anthropic keeps native is_error; it does not receive a JSON envelope. Hosted/provider-native tools are not part of this portable callable-history contract.

DeepSeek/Qwen portable replay has no private thinking state: it defaults to thinking.type: "disabled" / enable_thinking: false on those destinations only. Explicit reasoning/thinking requests are skipped before HTTP; thinking-only models do not advertise this capability. Text before assistant tool calls is supported. JSON destinations reject text interleaved after calls instead of moving it to the beginning. Anthropic can preserve that interleaving natively.

Tool-free canonical text/image messages continue to use normal capability checks. Audio, files, provider-data, provider metadata, unsupported roles/parts and unknown fields are rejected before provider calls. Validation errors contain no message content. Provider failures and caller abort reasons for tool-history requests use fixed, sanitized diagnostics. Once a stream emits an event, a failure terminates it without retry/fallback; collect() rejects on both thrown errors and provider error events.

Run bun run test packages/gateway/tests for regressions. After bun run build, run bun run scripts/gateway-history-package-smoke.ts for a tarball consumer with TypeScript compilation and generation/streaming on all six transport variants (Anthropic; OpenAI Chat/Responses; DeepSeek Chat; Qwen Chat/Responses). It installs candidate core/gateway/provider tarballs without checkout links. Add --live --provider=openai (or deepseek, qwen, anthropic) to require an authorized credential and perform two bounded live calls for one provider. A missing credential is a blocked live check, not a passing certification. Publication evidence and the actual version are recorded in the delivery record.

Usage accounting

generate, generateObject, streamText().collect() and streamObject().collect() preserve all reported TokenUsage fields, including cachedInputTokens, cacheWriteTokens, reasoningTokens and speed. Reported zeros remain zeros; missing optional details remain absent. Only missing inputTokens/outputTokens are estimated from text and a missing totalTokens is derived from those base counters. estimated is true if any of those three base counters was missing, including a derived total. Reasoning/cache details are never added again to the total. Agent results retain their existing usage aggregation without an estimated flag.

Detailed cost valuation

Opt in with costAccounting: { unknownCostPolicy: "allow" } and an immutable modelCatalog carrying pricing metadata. routeDecision.estimatedCosts contains cold-cache preflight estimates from input length and expectedOutputTokens (or request maxTokens). They are estimates, not spending limits. With unknownCostPolicy: "reject", destinations with unknown quotes are excluded before a call. The existing maxCostPer1kTokens budget remains independent.

Each actual attempt carries its reported usage and cost, including provider/model, currency, catalog/pricing versions, optional provenance source, components, assumptions and unknown reasons. A failed attempt without usage has amount: null; a successful fallback does not make that failure free. Sum amounts only when every actual attempt is known or explicitly estimated; otherwise total cost remains unknown. Streams use terminal usage, and partial stream failures preserve any usage already received.

calculateModelCost() is also exported from Core and SDK for standalone valuation. Input tokens must include cached reads and cache writes; these are subtracted from ordinary input before applying their separate prices. Missing cache counters remain unknown unless cacheAssumption: "none" explicitly assumes zeros. Missing relevant prices or base usage never become a free estimate. Long-context multipliers apply above the catalog threshold to all input classes and output. Zero tokens incur zero cost even when that unused category has no rate; its rate remains null.

Reasoning semantics vary between adapters. Set reasoningAccounting: { openai: "included", gemini: "additional" } only for the provider protocols you have verified: included leaves the output count unchanged, additional adds reasoning once. Without an explicit setting, a positive reasoning counter produces an unknown valuation. For standalone valuation this setting is a single string. The catalog does not describe fast-tier pricing, so speed: "fast" remains unknown. Invalid reported counters produce unknown accounting without retrying a successful model call. This API is neither billing nor a strict monetary budget.

Local destination metrics

Pass metrics: createGatewayMetrics({ windowMs: 60_000, maxTargets: 128, maxSamplesPerTarget: 256 }) to opt in. Query metrics.snapshot({ provider, modelId }) for in-flight count, recent successes/errors/cancellations, full-attempt p50/p95 and first-text p95. Cold destinations return undefined; expired samples have no latency estimates. Client cancellation is excluded from provider error and latency samples. The clock is injectable through now for deterministic tests.

Storage is bounded by targets and samples. The least recently touched idle destination is evicted when full; if all target slots are active, a new destination remains untracked until capacity is available. No prompts, outputs, error payloads or credentials are retained. The default store is local to the process; applications may inject the synchronous GatewayMetricsStore contract. Hook failures are best effort. Do not put expensive synchronous work in a metrics hook.

With metrics enabled, explicitly returning an event/text/object stream iterator cancels that routed operation and releases the in-flight slot. This applies to all consumers of that operation, including collect. Consume the stream fully if you also need its final result. Caller abort and timeout also release slots even when a provider ignores its signal. An operation with no explicit return/abort continues until completion or timeout; the SDK cannot detect garbage collection as abandonment.

Circuit breaker and adaptive routing

Pass circuitBreaker: createGatewayCircuitBreaker({ failureThreshold: 5, cooldownMs: 30_000 }) to opt in to local closed/open/half-open circuits. Retryable errors (including 429/5xx and attempt timeouts) accumulate; validation, authentication and client cancellation do not. Open circuits receive no calls. Recovery allows one concurrent probe by default; successful probes close the circuit, failed probes reopen it. Late results from an older circuit epoch cannot close a newly opened circuit. When all eligible circuits are unavailable the operation raises GatewayCircuitOpenError.

Cooldown is min(maxCooldownMs, max(cooldownMs, Retry-After)), with defaults of 60 seconds maximum and 30 seconds minimum. The circuit never sleeps until recovery: it skips the target. Existing bounded retry backoff applies only while the circuit remains closed. Target storage is bounded; closed idle entries may be evicted, while capacity exhausted by active/open entries refuses admission. State observers receive only target IDs, state and timestamps, and run best effort. Metrics/circuit-enabled stream iterator return cancels the operation. Neither feature coordinates across processes.

adaptiveRouting is an explicit alternative to legacy scoring. Supply a policy version, nonnegative weights for latency, cost, quality, load, errorRate, positive latencyScaleMs/costScale, and explicit coldStart, unknownCost, missingQuality policies (allow/reject). Quality profiles carry a target, task intent, score in [0,1] and evaluation version. There are no model-name quality heuristics in this mode.

The score is quality reward minus normalized p95 latency, estimated request cost, in-flight load and observed error rate penalties. Enable metrics and costAccounting for those signals. Missing penalty signals under allow receive a conservative normalized penalty (default 1), missing quality/throughput receive no reward, and all remain listed as missing; reject excludes the destination. Expired samples become cold start. Capabilities and circuits are filtered before ordering, and circuits are rechecked atomically before each call. Ties preserve primary/fallback input order. routeDecision.adaptive records policy version, signals, profile versions and exclusions. Do not configure both scoreTarget and adaptiveRouting. These rules are transparent heuristics, not a claim of globally optimal routing.

Composing configured agents

Pass agent: configuredAgent to runAgent or streamAgent. The gateway substitutes a routed LanguageModel and uses the existing Core runtime. Context schema, guardrails, output schema, compactor, approval signer, subagents, harness binding, environment, hooks and store remain on the definition. Supply ephemeral context on each invocation. Defined request options override definition defaults; policy and metadata merge shallowly. compaction: false explicitly disables the default for an invocation. The original definition is not mutated.

Configured gateway runs persist a gatewayAgentRouteBinding metadata value for agent ID, primary/fallback targets, routing policy version and harness fingerprint. Resume must retain that binding; changing it fails before provider work. An older direct run without this binding requires a separately designed migration instead of silent adoption. Core still validates harness/environment fingerprints. Reserve gatewayAgentRouteBinding and gatewayPortableHistory for runtime use.

For canonical history, import a fresh run with messages only. Do not combine import with prompt, state, runId, handoff, approvals or idempotencyKey. Resolved historical tools are input, never effects to replay. Then resume with store/runId or state and approvals as needed, without messages. Portable-history capability checks survive store reloads and compaction. Legacy inputs remain supported. Provider-specific approval data stays in durable state; it is not part of the portable history import format.

Production controls

All controls below are optional and compose with text, object and agent routing. They do not require provider-specific changes. Existing default adapter maps and legacy scoring remain supported. Adaptive missing-data penalties are intentionally more conservative than the previous zero-penalty behavior.

import {
  createGateway, createGatewayAdmissionController, createGatewayBudgetStore,
  createGatewayMetrics, createGatewayRoutingPolicy
} from "@zhivex-ai/gateway";
import { createInMemoryGenerateCache, createModelCatalog } from "@zhivex-ai/core";
import { createOpenAI } from "@zhivex-ai/openai";

// Supply a verified catalog snapshot for your actual models before enabling cost routing.
const catalog = createModelCatalog([{
  provider: "openai", modelId: "your-model",
  inputCostPer1kTokens: 0.001, outputCostPer1kTokens: 0.003
}], { snapshotVersion: "example-only", pricing: {
  version: "example-only", currency: "USD", unit: "per_1k_tokens"
} });
const budgets = createGatewayBudgetStore({ limit: 20, currency: "USD" });
const gateway = createGateway({
  adapters: { openai: createOpenAI() },
  modelCatalog: catalog,
  costAccounting: { unknownCostPolicy: "reject", cacheAssumption: "none" },
  timeoutMs: 20_000,
  metrics: createGatewayMetrics(),
  adaptiveRouting: createGatewayRoutingPolicy("interactive"),
  admission: createGatewayAdmissionController({
    maxConcurrent: 16, requestsPerMinute: 600, tokensPerMinute: 100_000,
    maxQueue: 32, queueTimeoutMs: 500
  }),
  budget: { store: budgets, currency: "USD", reserveAmount: 0.10 },
  cache: { store: createInMemoryGenerateCache(), scope: "credential-revision-1" },
  affinity: { ttlMs: 300_000, maxEntries: 1000, maxScoreLoss: 0.1 },
  observerMode: "background", observerQueueCapacity: 256
});
const result = await gateway.generate({
  primary: { provider: "openai", modelId: "your-model" },
  messages: [{ role: "user", content: "Explain binary search." }],
  maxTokens: 200, budgetScope: "tenant-123:2026-09", cacheScope: "tenant-123",
  affinityKey: "conversation-456"
});
await gateway.flushObservers();
await gateway.flushControls();
console.log(result.text, gateway.diagnostics(), budgets.snapshot("tenant-123:2026-09"));

Deadline and admission

timeoutMs covers one invocation, including queueing, retries, backoff, tools, observers and streamed output. A request may shorten the configured deadline but cannot extend it. Expiration raises GatewayDeadlineError and aborts downstream signals. Providers/tools must cooperate to stop external effects; ignoring abort cannot keep the gateway response pending indefinitely. Durable resumes start a new invocation deadline. Normal completion removes timers and signal listeners.

Admission is per provider/model/deployment and happens before every upstream call, including retries and later agent steps. Concurrency slots are released on completion, abort or stream iterator return. The global queue is bounded and waits at most queueTimeoutMs; the local implementation polls at up to 10 ms intervals and does not promise strict FIFO fairness. Capacity denial skips to an eligible fallback without marking the destination unhealthy.

RPM and TPM use fixed 60-second windows. TPM reserves serialized input length / 4 plus maxTokens, and requires maxTokens; this is an estimate, not provider token parity. Reported total tokens reconcile reservations in the same window; unknown usage keeps the reservation. RPM is not refunded after admission. Capacity pressure never evicts an unexpired quota window. Set upstream provider limits as the final boundary for usage the provider may continue after cancellation.

Monetary reservations

Each dispatched model attempt reserves budget.reserveAmount in the trusted budgetScope. Successful known usage reconciles that amount against catalog pricing; failed, cancelled, partial or unpriceable usage retains the reservation as uncertain spend. No reservation is made for an exact cache hit. Budget denial stops the operation without retry or fallback. The configured currency must match the catalog valuation to release any unused reservation.

This limits admitted reservations, not the exact provider invoice. Choose a conservative per-attempt reservation alongside maxTokens; actual reported cost can exceed the reservation and is recorded fully, blocking subsequent work when the scope is exhausted. Unknown spend remains blocked until the application reconciles its accounting externally. Local scopes never evict or automatically reset; use explicit period scopes and bounded maxScopes. Do not use arbitrary client-supplied scope values. For authoritative billing, implement the store against the service's durable ledger.

GatewayBudgetStore and GatewayAdmissionController are asynchronous injection contracts for shared backends. The supplied factories are process-local; they do not implement Redis or cross-process coordination. Remote implementations must reserve atomically, expire abandoned concurrency leases, honor abort and make settlement/release idempotent. Settlement runs outside the provider response path. Failed or timed-out settlement cannot retry successful provider work; diagnostics report it and further budgeted dispatches fail closed on that gateway instance. Recover the backend/ledger before replacing the instance.

Adaptive policies and destination identity

createGatewayRoutingPolicy("interactive" | "economy" | "quality") returns an explicit versioned starting policy. Configure quality profiles for economy and quality presets, which reject missing quality. Calibrate costs and latency scales on your workload; presets are not measured service-level guarantees.

latencyMetric: "ttft" uses first-text latency; "total" retains full-attempt latency. Optional weights.throughput uses measured median output tokens/second with throughputScale (default 100). Throughput is an approximation using reported output tokens minus the first token and elapsed time after first text. Cache hits and cancellations do not supply provider latency evidence. minSamples prevents single samples from being treated as mature health evidence, and minQuality enforces an evaluated quality floor before ranking.

Allowed missing latency/cost/load/error signals incur missingSignalPenalty (default 1, minimum 1). They are never represented as measured zero cost/latency. explorationEvery optionally probes one eligible cold destination every N operations; presets use 20. Exploration rotates candidates and cannot bypass capability, cost, quality-floor or circuit exclusions. It is local and deterministic, not a learned optimal policy.

A target may supply deploymentId. Register it in deployments: { "region-a": { provider: "openai", adapter } } with an adapter configured for that endpoint, region and credential. Unknown or mismatched IDs are skipped without silently using the default adapter. Deduplication, metrics, circuits, quotas, quality profiles, cache partitions and attempt diagnostics include deployment identity. Catalog pricing still uses provider/model identity; deployment-specific negotiated tariffs require a separate pricing design.

Optional affinity requires adaptive routing and both cacheScope and affinityKey. After a successful attempt, subsequent invocations may reuse that destination only if it remains eligible and within maxScoreLoss of the best score. Expired entries are ignored; capacity evicts the oldest stored affinity. Exploration takes precedence. This promotes provider prompt-cache locality; it does not prove an upstream cache hit or change provider caching parameters.

Exact cache and observer lifecycle

Gateway caching reuses Core's canonical cache key middleware and configured store. It requires both a configured authentication scope and a per-request cacheScope; keys additionally include budget scope and deployment. Rotate the configured scope when credentials/endpoints change. Store TTL/retention belongs to the selected Core cache implementation. No semantic similarity cache is enabled.

Only non-streaming text-only model steps without tools or providerOptions are eligible. Tool effects, portable tool history, images and provider state bypass this cache. Identical simultaneous misses share one upstream call (up to 1024 active keys). One caller cancelling does not cancel the other subscribers; all subscribers leaving aborts the shared call. Cache reads time out after cache.timeoutMs (default 50 ms); writes are detached, bounded to 256 pending operations and use the same timeout. Cache failures do not retry successful provider work. flushControls() waits for the current cache-write and resource-settlement batch; resource settlements have resourceTimeoutMs (default 1000 ms). Pending counts and dropped cache writes are available in diagnostics. attempt.cacheHit identifies reuse, and detailed attempt valuation uses zero new upstream tokens; response usage still describes the reused output. A coalesced follower is also a reuse; correlate requests with your application ledger when attributing the originating call's cost.

The legacy observer behavior stays detached. observerMode: "background" adds a bounded FIFO attempt queue, diagnostics().droppedObservers, and flushObservers() for the currently queued batch. Every callback remains bounded by observerTimeoutMs. "await" explicitly awaits attempt observers. None of these modes can preempt synchronous CPU work inside a callback. Route-selection observers keep their existing detached behavior.

Verification

Run bun run test packages/gateway/tests for routing and resource-lifetime regressions. After bun run build, run bun scripts/benchmarks/gateway-controls.mjs /tmp/gateway-controls.json to compare local default/control overhead and verify exact-cache/shared-miss provider call counts. The benchmark asserts correct answers and actual fixture calls, includes runtime/source/artifact fingerprints, and explicitly excludes live provider performance or competitive claims. Its five measured trials are a smoke baseline, not a statistically reliable production tail-latency estimate.