@afterquery/inference

TypeScript SDK for the Inference V1 gateway.

The package intentionally talks to your Inference backend instead of provider APIs directly. Provider secrets, OpenRouter-first routing, direct-provider fallback, retries, logging, and policy live in the backend.

Two integration modes

1) Gateway-native SDK

Use @afterquery/inference when you can replace direct provider calls in app code and want the gateway's normalized response helpers (content, route, cost, etc.).

2) OpenAI-compatible ingress

Use a DB-managed runtime key provisioned with:

• clientShape = openai_compat
• a bound projectSlug
• defaultFeature

This is the path for eval harnesses, LiteLLM, or existing OpenAI-client code that should only swap base_url / api_key.

Install

npm install @afterquery/inference

Gateway-native SDK: create client

import { Inference } from '@afterquery/inference';

const inference = new Inference({
  baseURL: process.env.INFERENCE_BASE_URL!,
  apiKey: process.env.INFERENCE_API_KEY!,
});

createInferenceClient() is still available:

import { createInferenceClient } from '@afterquery/inference';

const inference = createInferenceClient({
  baseUrl: 'http://localhost:8787',
  apiKey: process.env.INFERENCE_API_KEY,
});

Responses

const response = await inference.responses.create({
  model: 'gpt-4.1-mini',
  input: 'Write a concise product summary.',
  metadata: {
    project: 'demo',
    feature: 'product-copy',
  },
});

console.log(response.content);
console.log(response.route?.fallbackUsed);

Background responses

Use background responses for long-running jobs that should return a pollable id quickly.

let response = await inference.responses.create({
  model: 'perplexity/sonar-deep-research',
  input: 'Write a concise research brief about async deep research APIs.',
  background: true,
  metadata: {
    project: 'demo',
    feature: 'deep-research',
  },
});

while (response.status === 'queued' || response.status === 'in_progress') {
  await new Promise((resolve) => setTimeout(resolve, 10_000));
  response = await inference.responses.retrieve(response.id);
}

console.log(response.output_text);
console.log(response.inference?.route?.provider);

Foreground responses.create calls keep returning the normalized gateway response shape. With background: true, the SDK returns the OpenAI-style background response object used by GET /v1/responses/{id}.

Chat completions

const result = await inference.chat.completions.create({
  model: 'claude-sonnet-4-6',
  messages: [{ role: 'user', content: 'Say hello in one sentence.' }],
  metadata: {
    project: 'demo',
    feature: 'inline-chat',
  },
});

Compatibility alias:

await inference.chat({
  model: 'gpt-4.1-mini',
  messages: [{ role: 'user', content: 'Hello' }],
  metadata: { project: 'demo', feature: 'chat' },
});

Provider override

Omit provider for the gateway default route plan. Most models route OpenRouter-first; configured Claude Bedrock models route AWS Bedrock-first with OpenRouter fallback. Set provider when a request should force one gateway provider and skip gateway fallback:

await inference.chat.completions.create({
  provider: 'anthropic',
  model: 'claude-sonnet-4-6',
  messages: [{ role: 'user', content: 'Use Claude directly.' }],
  metadata: { project: 'demo', feature: 'direct-anthropic' },
});

Valid chat/responses providers are openrouter, openai, anthropic, and google.

AWS Bedrock Anthropic routing

When the gateway has AWS_BEARER_TOKEN_BEDROCK configured (AWS_BEARER_TOKEN is also accepted), select Claude models route through org-owned AWS Bedrock first and fall back to regular OpenRouter if Bedrock errors. Callers do not pass a BYOK flag; they request the normal Anthropic/OpenRouter model id:

await inference.chat.completions.create({
  model: 'anthropic/claude-opus-4.8',
  messages: [{ role: 'user', content: 'Use the gateway routing policy.' }],
  metadata: { project: 'demo', feature: 'bedrock-anthropic' },
});

The current Bedrock-first set is Opus 4.8, Opus 4.7, and Sonnet 4.6. Request logs show Bedrock traffic as providerConnector=aws_bedrock / provider_connector='aws_bedrock', provider=anthropic, and a Bedrock inference-profile model id such as global.anthropic.claude-opus-4-8; fallback attempts remain normal OpenRouter attempts.

Tool calling

Chat completions support OpenAI-style function tools through both the native SDK and OpenAI-compatible ingress.

const result = await inference.chat.completions.create({
  model: 'gpt-4.1-mini',
  messages: [{ role: 'user', content: 'What is the weather in Oakland?' }],
  tools: [
    {
      type: 'function',
      function: {
        name: 'get_weather',
        description: 'Lookup the current weather by city.',
        parameters: {
          type: 'object',
          properties: { city: { type: 'string' } },
          required: ['city'],
        },
      },
    },
  ],
  toolChoice: 'auto',
  metadata: { project: 'demo', feature: 'assistant-tools' },
});

for (const toolCall of result.toolCalls ?? []) {
  console.log(toolCall.id, toolCall.function.name, toolCall.function.arguments);
}

Vision input

await inference.responses.create({
  model: 'gpt-4o-mini',
  input: [
    {
      role: 'user',
      content: [
        { type: 'input_text', text: 'What is in this image?' },
        { type: 'input_image', imageUrl: 'https://example.com/image.png' },
      ],
    },
  ],
  metadata: {
    project: 'demo',
    feature: 'vision',
  },
});

Prompt caching

For Anthropic direct or OpenRouter Claude routes, content blocks can carry Anthropic/OpenRouter prompt-cache controls. The SDK accepts both camel-case cacheControl and provider-native cache_control. Responses include upstream cache accounting under usage.cacheStatus, usage.cacheReadTokens, and usage.cacheWriteTokens when the provider reports it. Direct-provider cost estimates are cache-adjusted only for provider/model policies we have explicitly verified; otherwise cache-affected cost is marked unavailable instead of silently overcharging or undercharging.

await inference.chat.completions.create({
  provider: 'anthropic',
  model: 'claude-sonnet-4-6',
  messages: [
    {
      role: 'system',
      content: [
        {
          type: 'text',
          text: largeStableContext,
          cacheControl: { type: 'ephemeral', ttl: '1h' },
        },
      ],
    },
    { role: 'user', content: 'Use the cached context to answer.' },
  ],
  metadata: { project: 'demo', feature: 'prompt-cache' },
});

Embeddings

await inference.embeddings.create({
  model: 'text-embedding-3-small',
  input: 'Document text',
  metadata: {
    project: 'demo',
    feature: 'search-index',
  },
});

Compatibility alias:

await inference.embed({
  model: 'text-embedding-3-small',
  input: 'Document text',
  metadata: { project: 'demo', feature: 'search-index' },
});

Image generation

await inference.images.generate({
  model: 'gpt-image-1',
  prompt: 'A clean diagram of an AI gateway.',
  size: '1024x1024',
  metadata: {
    project: 'demo',
    feature: 'image-generation',
  },
});

OpenAI-compatible ingress examples

Python OpenAI client

from openai import OpenAI

client = OpenAI(
    base_url="https://your-gateway.example.com/v1",
    api_key="inf_live_...",
)

response = client.chat.completions.create(
    model="gpt-4.1-mini",
    messages=[{"role": "user", "content": "Reply with a one-line summary."}],
)

print(response.choices[0].message.content)

TypeScript OpenAI client

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://your-gateway.example.com/v1',
  apiKey: process.env.INFERENCE_API_KEY!,
});

const response = await client.chat.completions.create({
  model: 'gpt-4.1-mini',
  messages: [{ role: 'user', content: 'Reply with a one-line summary.' }],
});

console.log(response.choices[0]?.message?.content);

The same OpenAI client can call background Responses through the gateway:

let response = await client.responses.create({
  model: 'perplexity/sonar-deep-research',
  input: 'Write a concise research brief about async deep research APIs.',
  background: true,
  metadata: {
    project: 'demo',
    feature: 'deep-research',
  },
});

response = await client.responses.retrieve(response.id);
console.log(response.output_text);

LiteLLM

from litellm import completion

response = completion(
    model="gpt-4.1-mini",
    custom_llm_provider="openai",
    api_base="https://your-gateway.example.com/v1",
    api_key="inf_live_...",
    messages=[{"role": "user", "content": "Grade this answer in one sentence."}],
)

print(response.choices[0].message.content)

For chat completions with compat keys, the backend derives project and feature from the provisioned key defaults, so the harness does not need to know the gateway-native metadata contract. Background Responses include those metadata fields explicitly so the job can be authorized and logged. OpenAI-compatible tools, tool_choice, tool_calls, and role: "tool" messages are forwarded for OpenAI/OpenRouter-backed chat routes.

OpenAI-compatible callers can force a gateway provider with the nonstandard top-level provider field. Depending on the client library, use an escape hatch such as extra_body or a typed cast. Prompt-caching content blocks can carry provider-native cache_control when the client allows extra content-block fields.

response = client.chat.completions.create(
    model="anthropic/claude-sonnet-4.6",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": large_stable_context,
                    "cache_control": {"type": "ephemeral", "ttl": "1h"},
                },
                {"type": "text", "text": "Answer the question."},
            ],
        }
    ],
    extra_body={"provider": "openrouter"},
)

Gateway-level retries

The SDK retries only network/gateway failures. Provider retries and OpenRouter -> direct provider fallback happen inside the backend.

const inference = new Inference({
  baseURL: process.env.INFERENCE_BASE_URL!,
  apiKey: process.env.INFERENCE_API_KEY!,
  timeoutMs: 60_000,
  retry: {
    maxRetries: 2,
    initialDelayMs: 250,
    maxDelayMs: 3_000,
  },
});

Per request:

await inference.responses.create(payload, {
  timeoutMs: 30_000,
  idempotencyKey: 'request-123',
});

Route metadata

Responses may include route information from the backend:

response.route?.attempts
response.route?.fallbackUsed
response.route?.provider
response.route?.model

This is how callers can inspect whether OpenRouter was used or whether the backend fell back to OpenAI, Anthropic, or Google directly.