@flashyos/llm-gateway

A provider-agnostic inference seam: adapters, automatic failover, and an empty-success guard.

Why this exists

This code lived twice — once in ClaimYour.Gold, once in flashyos/apps/ops — as byte-identical copies. Two copies of anything drift; two copies of the thing every AI call passes through drift expensively.

Usage

The package takes configuration and an error factory. It does not read your environment or import your error class — that boundary is what makes it a package rather than a copy with extra steps.

import { createGateway } from '@flashyos/llm-gateway';

const gateway = createGateway(
  {
    provider: process.env.LLM_PROVIDER,        // 'anthropic' | 'gatewayz'
    anthropicApiKey: process.env.ANTHROPIC_API_KEY,
    gatewayzApiKey: process.env.GATEWAYZ_API_KEY,
    gatewayzBaseUrl: process.env.GATEWAYZ_BASE_URL,
  },
  (code, httpStatus, message) => new AppError(code, httpStatus, message),
);

const { text, inputTokens, outputTokens, requestId, provider } =
  await gateway.call({ model, systemPrompt, userPrompt, maxTokens: 4096 });

Attribution is opt-in. Pass an initiativeId on a call and it is echoed back on the result (null when unset):

const r = await gateway.call({ model, systemPrompt, userPrompt, maxTokens: 4096, initiativeId: 'init/orbital-refi-q3' });
// r.initiativeId === 'init/orbital-refi-q3', r.requestId, r.inputTokens, r.outputTokens

It exists so a per-initiative compute history can be built from the calls themselves — the caller pairs the tag and the token counts to attribute cost to the work that spent it. Existing callers pass nothing and get initiativeId: null.

The tag is sent to Gatewayz — since 0.2.1. The Gatewayz adapter puts the initiativeId on the wire as the x-gatewayz-tag request header (GATEWAYZ_TAG_HEADER), which Gatewayz echoes in its own usage record — so attribution no longer depends on this process capturing every response, and a per-initiative total can be reconciled against the gateway's count rather than only against our sum (asks J and K in docs/gatewayz-asks.md). It is a header, never part of the model's input, and Anthropic direct never receives it: a header a provider did not ask for is noise on the wire, not attribution. ATTRIBUTION_TAG_SENT_SINCE is the exported figure to pin on.

What it guarantees

Cutover is configuration. provider selects who serves inference; the other configured provider becomes the automatic fallback. Changing providers is an environment change, never a deploy.

Degradation never costs availability. An unknown provider id, or a primary without credentials, falls back to Anthropic direct rather than failing.

Failover is automatic. A feature flag needs a human awake; the breaker does not. After three consecutive failures the primary is skipped outright for 60 seconds, healing on read so nothing has to be running for it to recover.

A caller's mistake is not a provider failure — since 0.2.0. Every error the primary raised used to count towards the breaker, so three malformed requests in a row opened the circuit and moved all traffic to the fallback for a minute: a typo could fail a provider over, and the operator read an outage where there was a bad model id. A 4xx no longer counts, with one exception — 429 is the provider saying "not now", which is a real condition of the provider. An error carrying no status at all still counts, because a transport failure, a DNS error and an aborted socket all arrive that way and they are what the breaker is for. CLIENT_ERROR_NOT_PROVIDER_FAILURE_SINCE is the exported figure to pin on. Failover itself is unchanged: losing a gateway must never cost availability, and a model id wrong for one provider can be right for the other.

It does not cover a provider that returns a caller error under a 5xx status. Gatewayz answers an unknown or unpriced model id with 503 service_unavailable, which is a client error wearing a server status; it still counts, and still trips the breaker.

An empty success is a failure. A response with no content and zero output tokens — how some gateways report a failed upstream call — is raised as an error and falls over, rather than being returned as a free, silent non-answer.

Cost is the caller's. The gateway reports tokens and a requestId; it does not price the call. A provider's aggregate usage feed cannot attribute a figure to one request, so per-call cost is computed by the host against its own model registry, and a provider's totals are a reconciliation input rather than a source.

Not yet verified through a gateway

Tool use and prompt caching (cache_control). Nothing in the estate sends either today. Do not route tool-using or cache-annotated traffic through a gateway until its semantics matrix covers both — a gateway that silently drops cache directives does not fail, it multiplies the bill.