FlexInference (TypeScript)

The official TypeScript SDK for FlexInference, an inference router that works with OpenAI, Google Gemini, and Anthropic. Send the OpenAI requests you already send and bring your own provider key. Set one required field, start_within, to tell us how long you will wait, and we trade a little latency for a smaller bill. You can call it four ways. Use responses, chat.completions, interactions (the Gemini shape), or messages (the Anthropic shape). Any of them reaches any provider.

npm install flexinference

Quickstart

import { FlexInference, outputText } from "flexinference";

const client = new FlexInference({ apiKey: "flex_live_..." });

const res = await client.responses.create({
  model: "gpt-5.5",
  input: "Write a haiku about cheap GPUs.",
  start_within: "00h-00m-30s", // typed - no cast needed
});

console.log(outputText(res));

Responses come back as the raw OpenAI JSON (we never reshape the body), so there is no output_text field on the wire. OpenAI's own SDKs compute that. outputText(res) pulls the assistant's text out of either a response or a chat completion for you.

start_within is required on every request. It takes "default", "priority", "auto", or a duration in the form "HHh-MMm-SSs" from 5 seconds to 10 minutes. A duration tells us how long you will wait for the request to start. We try the cheaper flex tier first and give it that long to begin. If flex cannot start in time, we move your request up to your standard tier so it still finishes. The words "default", "priority", and "auto" map to those OpenAI service tiers and proxy any model. Types catch most mistakes for you. A missing value, or a value that does not fit the allowed shapes, fails at compile time. A duration that fits the shape but is malformed or out of range fails at runtime, the same check plain-JavaScript callers get. See the docs for the full rules.

This runs the opposite way from the usual fallback. Most setups start on the strong model and drop to a cheaper one when they need to. We start on the cheaper flex tier and escalate up to your standard model if flex cannot start in time, so a slow flex tier never costs you the answer.

Providers (OpenAI, Gemini, and Anthropic)

FlexInference routes to OpenAI, Google Gemini, and Anthropic. Send the same OpenAI-shaped request and pass whichever model id you want. Use gpt-5.5, o4-mini, gemini-3.5-flash, claude-opus-4-8, and so on. We translate Gemini and Anthropic to and from the OpenAI shape, so your code is identical for all three.

• OpenAI: default (the standard tier), priority, auto, and flex on flex-capable models. You reach flex by passing a duration.
• Gemini: default maps to Gemini's standard tier. You also get priority and flex on the Gemini flex models (gemini-3.5-flash, gemini-3.1-flash-lite, gemini-3.1-pro-preview, gemini-3-flash-preview, gemini-2.5-pro, gemini-2.5-flash, gemini-2.5-flash-lite). Gemini has no auto tier, so start_within: "auto" on a Gemini model returns 400.
• Anthropic (Claude): proxy-only. default, priority, and auto work; there is no flex tier, so a duration start_within on a claude-* model returns 400 flex_unsupported_for_anthropic. Anthropic requires an output-token field upstream. FlexInference forwards max_output_tokens (max_completion_tokens on Chat, max_tokens on Messages) when set and does not synthesize a default, so omitting it returns Anthropic's own request error. You keep the unified API and tier control, and draw down your own Anthropic credits.

Add the provider key you'll use (OpenAI, Gemini, and/or Anthropic) in the dashboard. Text, streaming, structured outputs, function calling, image input, and web search work across providers (send a Responses web_search tool; we map it to Gemini's google_search).

Do not send service_tier. The router sets the tier from start_within, and it rejects a caller-supplied service_tier with 400 service_tier_not_allowed.

Streaming

const stream = await client.responses.create({
  model: "gpt-5-nano",
  input: "Count to ten.",
  stream: true,
  start_within: "00h-00m-20s",
});

for await (const event of stream) {
  if (event.type === "response.output_text.delta") process.stdout.write(event.delta);
}

Chat Completions

The Chat Completions endpoint works identically:

const res = await client.chat.completions.create({
  model: "gpt-5.5",
  messages: [{ role: "user", content: "Hello!" }],
  start_within: "default",
});

Interactions (Gemini shape)

Speak Google's Interactions shape and reach any model. interactionText(res) pulls the assistant text out of the interaction's steps.

import { interactionText } from "flexinference";

const res = await client.interactions.create({
  model: "gemini-3.5-flash",
  input: "Summarize this contract.",
  start_within: "00h-01m-00s",
});

console.log(interactionText(res));

Messages (Anthropic shape)

Speak Anthropic's Messages shape and reach any model. Anthropic requires max_tokens upstream. FlexInference forwards it when set and does not synthesize a default, so omitting it returns Anthropic's own request error. messageText(res) pulls the assistant text out of the message content.

import { messageText } from "flexinference";

const res = await client.messages.create({
  model: "claude-opus-4-8",
  max_tokens: 1024,
  messages: [{ role: "user", content: "Summarize this contract." }],
  start_within: "default",
});

console.log(messageText(res));

Timeouts and cancellation

Streaming and non-streaming requests are timed differently, so a long generation is never cut off yet a hung request never hangs forever:

• Non-streaming requests have a body-read budget after response headers arrive: timeout, default 600000 ms (10 min). The worst case is roughly firstByteTimeout + timeout.
• Streaming requests have no total cap (a healthy stream runs as long as tokens keep arriving). They are bounded by two clocks: firstByteTimeout (wait for the first response, default 60000 ms) and idleTimeout (max silence between chunks before the stream is treated as hung, default 60000 ms).

The first-byte wait is auto-raised for a flex start_within (the router withholds response headers until the flex race resolves), so a long deadline just works. Raise firstByteTimeout yourself only for very large non-flex contexts whose first token is slow.

const client = new FlexInference({
  apiKey: "flex_live_...",
  timeout: 120_000, // non-streaming total budget
  firstByteTimeout: 90_000, // wait for the first response
  idleTimeout: 30_000, // max silence between streamed chunks
});

Any of these can be overridden per request, alongside an AbortSignal to cancel yourself. Cancelling a stream stops it mid-flight:

const controller = new AbortController();
const res = await client.responses.create(
  { model: "gpt-5.5", input: "Write a haiku about cheap GPUs.", start_within: "priority" },
  { signal: controller.signal, idleTimeout: 15_000 },
);
// controller.abort() from elsewhere to cancel
console.log(outputText(res));

Errors

Non-2xx responses throw FlexInferenceError, carrying status, type, code, and param. The router shapes error bodies to match the endpoint you called (OpenAI on responses/chat, Anthropic on messages, Google on interactions) so the SDK you would use for that surface parses them; FlexInferenceError reads all three. message and status are always set; code, param, and docUrl are populated on the OpenAI surface.

import { FlexInferenceError } from "flexinference";

try {
  await client.responses.create({ model: "gpt-5.5", input: "hi", start_within: "priority" });
} catch (err) {
  if (err instanceof FlexInferenceError && err.code === "no_byok_key") {
    console.log("Add your OpenAI key in the dashboard.");
  } else {
    throw err;
  }
}

Every FlexInference error tells you four things. It says what went wrong, why it went wrong, how to fix it, and it shows an example of a request that works. The message reads like a note from a person, so an agent can act on it instead of guessing. Provider errors are reshaped into the same surface envelope (and normalized into FlexInferenceError), so you get one consistent error type no matter which model ran. For instance, a duration on a claude-* model returns 400 flex_unsupported_for_anthropic with a message that tells you to drop the duration or switch to default, priority, or auto.

Billing / 402

If your account's billing is past due, the router pauses paid flex requests and returns 402 Payment Required on them. Free routing keeps working, so your standard-tier calls still go through. Flex is the only part you ever pay for, and you only pay it when it saves you money. The SDK throws a typed PaymentRequiredError (a subclass of FlexInferenceError) for HTTP 402, so you can catch it on its own and prompt the user to update payment while letting other errors propagate:

import { PaymentRequiredError } from "flexinference";

try {
  await client.responses.create({ model: "gpt-5.5", input: "hi", start_within: "00h-00m-30s" });
} catch (err) {
  if (err instanceof PaymentRequiredError) {
    console.log("Billing is past due - update payment in the dashboard to resume flex.");
  } else {
    throw err;
  }
}

Because PaymentRequiredError extends FlexInferenceError, existing catch (err instanceof FlexInferenceError) handlers keep catching 402s too.

Configuration

| Option | Default | Description | | ------------------ | ---------------------------------- | ------------------------------------------------------------------------------- | | apiKey | - | Your flex_live_ key (required). | | baseURL | https://api.flexinference.com/v1 | Override the router endpoint. | | fetch | global fetch | Provide a custom fetch implementation. | | timeout | 600000 | Non-streaming total budget in ms (streaming has no total). | | firstByteTimeout | 60000 | Wait for the first response in ms; auto-raised for a flex start_within. | | idleTimeout | 60000 | Max silence between streamed chunks in ms before the stream is treated as hung. |

All three timeouts can also be passed per request (with signal) in the second argument.

License

MIT