@inference/sdk

@inference/sdk is a thin adapter around the official openai, @anthropic-ai/sdk, and Vercel AI SDK provider factories for calling the Inference proxy with provider-native APIs.

The SDK does not implement transport, streaming, retries, or provider resource internals. It instantiates the caller's official SDK class or AI SDK provider factory with Inference defaults, adds Inference routing headers, patches the methods that benefit from Inference-specific request options, and returns the configured official client or AI SDK provider instance.

Current scope

• OpenAI and Anthropic adapters
• Returns the configured official client directly
• Patched OpenAI methods:
  • chat.completions.create
  • responses.create
  • completions.create
  • embeddings.create
• Patched Anthropic methods:
  • messages.create
  • messages.stream
  • messages.parse
  • messages.countTokens
• AI SDK v6 provider adapters:
  • OpenAI-compatible language models and embeddings
  • Anthropic language models
  • per-call providerOptions.inference support for task, metadata, and environment
• All other official client methods remain available directly on the returned client

Usage

For proxy ingestion clients:

• apiKey is sent to the proxy as Authorization: Bearer <key>.
• OpenAI-compatible clients can omit provider and use the proxy's default downstream route.
• When supplied, provider.apiKey and provider.baseUrl are sent as downstream override headers.
• Anthropic clients still require provider until the proxy supports Anthropic default routing.

OpenAI-compatible providers

import OpenAI from "openai";
import { createInferenceClient } from "@inference/sdk";

const inference = createInferenceClient({
  openai: OpenAI,
  apiKey: process.env.INFERENCE_API_KEY!,
  projectId: process.env.INFERENCE_PROJECT_ID,
  environment: process.env.NODE_ENV,
  options: {
    maxRetries: 0,
  },
});

const response = await inference.chat.completions.create(
  {
    model: "z-ai/glm-5-turbo",
    messages: [{ role: "user", content: "Hello" }],
  },
  {
    task: "contact-research",
    metadata: {
      userId: "123",
      workflow: "lead-enrichment",
    },
  },
);

const models = await inference.models.list();

Use provider when you want to override the downstream base URL and API key:

const openrouter = createInferenceClient({
  openai: OpenAI,
  apiKey: process.env.INFERENCE_API_KEY!,
  provider: {
    baseUrl: "https://openrouter.ai/api",
    apiKey: process.env.OPENROUTER_API_KEY!,
  },
});

Auto-train

Auto-train distills a task onto a smaller model automatically. Requests are served by the teacherModel while Inference collects successful samples; once config.minSamples samples accumulate, a distilled model is trained (and deployed when autoDeploy is enabled) and traffic cuts over transparently — no code change required.

The highest-level entry point is createAutoTrainClient on an OpenAI-compatible client. It scopes a task, system prompt, and (optionally) a Zod output schema:

import OpenAI from "openai";
import { z } from "zod";

import { createInferenceClient } from "@inference/sdk";

const client = createInferenceClient({
  openai: OpenAI,
  apiKey: process.env.INFERENCE_API_KEY!,
});

const personExtractor = client.createAutoTrainClient({
  task: "person-extraction",
  teacherModel: "glm-5.2",
  systemPrompt: "Extract information about people from text.",
  schema: z.object({
    name: z.string(),
    age: z.number(),
  }),
  config: {
    minSamples: 100,
    autoTrain: true,
    autoDeploy: true,
  },
});

// Typed as { name: string; age: number }
const person = await personExtractor.run({
  input: "Hello, my name is John and I am 30 years old.",
});

Omit schema for free-form text tasks; run() then resolves to the raw response string.

For full control over the request body, pass autoTrain per call instead (the request model is ignored for routing — the teacher or the trained model is chosen server-side):

const response = await client.chat.completions.create(
  {
    model: "auto-train",
    messages: [
      { role: "system", content: "Extract information about people." },
      { role: "user", content: "John is 30 years old." },
    ],
  },
  {
    task: "person-extraction",
    autoTrain: {
      teacherModel: "glm-5.2",
      config: {
        minSamples: 100,
        autoTrain: true,
        autoDeploy: true,
      },
    },
  },
);

The InferenceAutoTrainConfig and InferenceAutoTrainRequestOptions types are exported for callers that build these options programmatically.

Changing the prompt or schema

Each distinct combination of system prompt, output schema, and teacher model is a separate auto-train identity, and a task serves at most one trained identity at a time:

• Changing the prompt or schema starts a new identity. Its requests are served by the teacher model while samples accumulate and the replacement model trains. The previous model keeps serving requests that still use the old prompt/schema during this time.
• When the replacement finishes training, the old identity is retired and the task's single deployment is swapped to the new model. The whole task briefly serves from the teacher during the swap (routing-cache drain, model load, and a smoke test — typically a few minutes), then traffic cuts over to the new model.
• Each retrain is recorded as a new version of the same deployment, so the deployment's version history is the task's retrain audit trail.
• Reverting to a previously used prompt/schema does not resurrect its old model: a retired identity routes to the teacher model permanently.

Anthropic

import Anthropic from "@anthropic-ai/sdk";
import { createInferenceClient } from "@inference/sdk";

const inference = createInferenceClient({
  anthropic: Anthropic,
  apiKey: process.env.INFERENCE_API_KEY!,
  projectId: process.env.INFERENCE_PROJECT_ID,
  environment: process.env.NODE_ENV,
  options: {
    maxRetries: 0,
  },
  provider: {
    baseUrl: "https://api.anthropic.com",
    apiKey: process.env.ANTHROPIC_API_KEY!,
  },
});

const response = await inference.messages.create(
  {
    model: "claude-sonnet-4-6",
    max_tokens: 1024,
    messages: [{ role: "user", content: "Hello" }],
  },
  {
    task: "contact-research",
    metadata: {
      userId: "123",
      workflow: "lead-enrichment",
    },
  },
);

const tokenCount = await inference.messages.countTokens(
  {
    model: "claude-sonnet-4-6",
    messages: [{ role: "user", content: "Hello" }],
  },
  {
    task: "contact-research",
  },
);

Inference query client

The query client talks to the observability API and lets you list, filter, paginate, and hydrate recorded inferences for the project scoped to your platform API key.

Basic setup

import { createQueryClient } from "@inference/sdk";

const inferenceQuery = createQueryClient({
  apiKey: process.env.INFERENCE_API_KEY!,
  projectId: process.env.INFERENCE_PROJECT_ID,
});

If you are pointing at a non-production observability API, override baseUrl:

import { createQueryClient } from "@inference/sdk";

const inferenceQuery = createQueryClient({
  baseUrl: "http://localhost:8888",
  apiKey: process.env.INFERENCE_API_KEY!,
  projectId: process.env.INFERENCE_PROJECT_ID,
});

Query the first page

import { createQueryClient } from "@inference/sdk";

const inferenceQuery = createQueryClient({
  apiKey: process.env.INFERENCE_API_KEY!,
});

const firstPage = await inferenceQuery.query({
  page: {
    limit: 25,
  },
  sort: {
    by: "sentAt",
    order: "desc",
  },
});

console.log(firstPage.totalCount);
console.log(firstPage.hasMore);
console.log(firstPage.nextCursor);
console.log(firstPage.items[0]?.id);

Query with filters

import { createQueryClient } from "@inference/sdk";

const inferenceQuery = createQueryClient({
  apiKey: process.env.INFERENCE_API_KEY!,
});

const page = await inferenceQuery.query({
  filters: {
    dateRange: {
      start: new Date("2026-03-01T00:00:00Z"),
      end: new Date("2026-03-31T23:59:59Z"),
    },
    tasks: [null, "contact-research"],
    providers: ["openai"],
    environments: [null, "production"],
    statuses: ["success", "429"],
    numeric: [{ field: "totalCost", op: "gt", value: 0.05 }],
  },
  page: {
    limit: 25,
  },
  sort: {
    by: "totalCost",
    order: "desc",
  },
});

for (const item of page.items) {
  console.log(item.id, item.model, item.totalCost, item.taskId);
}

Fetch the next page with a cursor

import { createQueryClient } from "@inference/sdk";

const inferenceQuery = createQueryClient({
  apiKey: process.env.INFERENCE_API_KEY!,
});

const firstPage = await inferenceQuery.query({
  page: { limit: 25 },
});

if (firstPage.nextCursor) {
  const secondPage = await inferenceQuery.query({
    page: {
      cursor: firstPage.nextCursor,
      limit: 25,
    },
  });

  console.log(secondPage.items.length);
}

Stream through all pages with the paginator

import { createQueryClient } from "@inference/sdk";

const inferenceQuery = createQueryClient({
  apiKey: process.env.INFERENCE_API_KEY!,
});

for await (const page of inferenceQuery.queryPages({
  filters: {
    dateRange: {
      start: "2026-03-01T00:00:00Z",
      end: "2026-03-31T23:59:59Z",
    },
  },
  page: {
    limit: 100,
  },
})) {
  for (const item of page.items) {
    console.log(item.id, item.sentAt, item.statusCode);
  }
}

Include request and response payloads on the page items

import { createQueryClient } from "@inference/sdk";

const inferenceQuery = createQueryClient({
  apiKey: process.env.INFERENCE_API_KEY!,
});

const page = await inferenceQuery.query({
  filters: {
    dateRange: {
      start: new Date(Date.now() - 24 * 60 * 60 * 1000),
      end: new Date(),
    },
  },
  includeContent: true,
  page: {
    limit: 10,
  },
});

for (const item of page.items) {
  if (item.contentError) {
    console.error(item.id, item.contentError.message);
    continue;
  }

  console.log(item.id, item.content?.request?.path, item.content?.response);
}

Fetch request and response payloads for one inference

import { createQueryClient } from "@inference/sdk";

const inferenceQuery = createQueryClient({
  apiKey: process.env.INFERENCE_API_KEY!,
});

const content = await inferenceQuery.getInferenceContent({
  inferenceId: "inf_123",
});

console.log(content.request?.method, content.request?.path);
console.log(content.request?.body);
console.log(content.response?.body);

Discover available filter values

import { createQueryClient } from "@inference/sdk";

const inferenceQuery = createQueryClient({
  apiKey: process.env.INFERENCE_API_KEY!,
});

const filterOptions = await inferenceQuery.getFilterOptions({
  includeUploads: true,
});

console.log(filterOptions.models);
console.log(filterOptions.providers);
console.log(filterOptions.environments);
console.log(filterOptions.tasks);
console.log(filterOptions.uploads);

List uploads directly

import { createQueryClient } from "@inference/sdk";

const inferenceQuery = createQueryClient({
  apiKey: process.env.INFERENCE_API_KEY!,
});

const uploads = await inferenceQuery.listUploads({
  includeArchived: false,
  limit: 50,
  offset: 0,
});

console.log(uploads.totalCount, uploads.nextOffset);
console.log(uploads.items.map((upload) => upload.name));

Optionally pass projectId

Most callers do not need this, but you can still pass projectId in the client config or per call when you want to be explicit.

import { createQueryClient } from "@inference/sdk";

const inferenceQuery = createQueryClient({
  apiKey: process.env.INFERENCE_API_KEY!,
  projectId: "project_123",
});

const page = await inferenceQuery.query({
  page: { limit: 10 },
});

const sameProjectContent = await inferenceQuery.getInferenceContent({
  inferenceId: page.items[0]!.id,
});

Notes:

• The query client talks to the observability API, not the provider proxy API.
• apiKey is sufficient when the key is scoped to one project or has a default project.
• Set projectId when you use a key scoped to multiple projects and want to select a project explicitly.
• The query client uses cursor pagination. Use page.cursor for manual paging or queryPages() for an async iterator.
• includeContent hydrates request and response payloads for the current page only and reports per-item fetch failures in contentError.
• Upload-backed inference filters are fetched only when you pass includeUploads: true to getFilterOptions().
• By default, inference queries return API-recorded inferences. To focus on upload-backed inferences, filter by upload IDs.

Vercel AI SDK v6

import { createAnthropic } from "@ai-sdk/anthropic";
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { embed, generateObject, generateText, streamText } from "ai";
import { z } from "zod";

import {
  createInferenceProvider,
} from "@inference/sdk";

const inferenceOpenAI = createInferenceProvider({
  createOpenAICompatible,
  apiKey: process.env.INFERENCE_API_KEY!,
  environment: process.env.NODE_ENV,
});

const inferenceAnthropic = createInferenceProvider({
  createAnthropic,
  apiKey: process.env.INFERENCE_API_KEY!,
  environment: process.env.NODE_ENV,
  options: {},
  provider: {
    baseUrl: "https://api.anthropic.com",
    apiKey: process.env.ANTHROPIC_API_KEY!,
  },
});

const textResult = await generateText({
  model: inferenceOpenAI.chatModel("gpt-4o-mini"),
  prompt: "Reply with exactly OK.",
  providerOptions: {
    inference: {
      task: "contact-research",
      metadata: {
        userId: "123",
      },
    },
  },
});

const embeddingResult = await embed({
  model: inferenceOpenAI.embeddingModel("text-embedding-3-small"),
  value: "sunny day at the beach",
  providerOptions: {
    inference: {
      task: "semantic-indexing",
    },
    openai: {
      dimensions: 512,
    },
  },
});

const objectResult = await generateObject({
  model: inferenceAnthropic.messages("claude-sonnet-4-6"),
  prompt: "Return an object with status set to OK.",
  providerOptions: {
    inference: {
      task: "structured-extraction",
      metadata: {
        userId: "user_123",
      },
    },
  },
  schema: z.object({
    status: z.string(),
  }),
});

const streamed = streamText({
  model: inferenceAnthropic("claude-sonnet-4-6"),
  prompt: "Reply with exactly OK.",
  providerOptions: {
    inference: {
      task: "streaming-chat",
    },
  },
});

for await (const chunk of streamed.textStream) {
  process.stdout.write(chunk);
}

console.log(textResult.text, embeddingResult.embedding.length, objectResult.object.status);

Use providerOptions.inference.metadata for Inference observability metadata. Use providerOptions.openai or providerOptions.anthropic only for provider-native request options that should be forwarded to that upstream SDK. OpenAI-compatible AI SDK providers default includeUsage to true so streamed responses include usage information for observability. Set options: { includeUsage: false } if a provider rejects stream_options.

Vercel AI SDK provider registry

import { createAnthropic } from "@ai-sdk/anthropic";
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { createProviderRegistry, embedMany, streamObject } from "ai";
import { z } from "zod";

import {
  createInferenceProvider,
} from "@inference/sdk";

const registry = createProviderRegistry({
  anthropic: createInferenceProvider({
    createAnthropic,
    apiKey: process.env.INFERENCE_API_KEY!,
    provider: {
      baseUrl: "https://api.anthropic.com",
      apiKey: process.env.ANTHROPIC_API_KEY!,
    },
  }),
  openai: createInferenceProvider({
    createOpenAICompatible,
    apiKey: process.env.INFERENCE_API_KEY!,
    provider: {
      baseUrl: "https://api.openai.com",
      apiKey: process.env.OPENAI_API_KEY!,
    },
  }),
});

const embeddings = await embedMany({
  model: registry.embeddingModel("openai:text-embedding-3-small"),
  values: ["sunny day at the beach", "rainy afternoon in the city"],
  providerOptions: {
    inference: {
      task: "semantic-indexing",
      metadata: {
        batchId: "embeddings-123",
      },
    },
  },
});

const streamedObject = streamObject({
  model: registry.languageModel("anthropic:claude-sonnet-4-6"),
  prompt: "Return an object with status set to OK.",
  schema: z.object({
    status: z.string(),
  }),
  providerOptions: {
    inference: {
      task: "structured-extraction",
      metadata: {
        requestId: "req_123",
      },
    },
  },
});

for await (const partialObject of streamedObject.partialObjectStream) {
  console.log(partialObject);
}

console.log(embeddings.embeddings.length);

Provider cookbook

The SDK currently documents and tests these provider routes:

• OpenAI
• OpenRouter
• Anthropic
• Inference.net
• Cerebras
• Gemini (via the OpenAI-compatible path)

Any other OpenAI-compatible provider can use the same OpenAI/OpenAI-compatible pattern by swapping provider.baseUrl and the provider API key.

Official SDK adapters

import Anthropic from "@anthropic-ai/sdk";
import OpenAI from "openai";

import { createInferenceClient } from "@inference/sdk";

const common = {
  environment: process.env.NODE_ENV,
  apiKey: process.env.INFERENCE_API_KEY!,
};

const providers = {
  anthropic: createInferenceClient({
    ...common,
    anthropic: Anthropic,
    provider: {
      apiKey: process.env.ANTHROPIC_API_KEY!,
      baseUrl: "https://api.anthropic.com",
    },
  }),
  cerebras: createInferenceClient({
    ...common,
    openai: OpenAI,
    provider: {
      apiKey: process.env.CEREBRAS_API_KEY!,
      baseUrl: "https://api.cerebras.ai",
    },
  }),
  gemini: createInferenceClient({
    ...common,
    openai: OpenAI,
    provider: {
      apiKey: process.env.GEMINI_API_KEY!,
      baseUrl: "https://generativelanguage.googleapis.com/v1beta/openai",
    },
  }),
  inferenceNet: createInferenceClient({
    ...common,
    openai: OpenAI,
  }),
  openai: createInferenceClient({
    ...common,
    openai: OpenAI,
    provider: {
      apiKey: process.env.OPENAI_API_KEY!,
      baseUrl: "https://api.openai.com",
    },
  }),
  openrouter: createInferenceClient({
    ...common,
    openai: OpenAI,
    provider: {
      apiKey: process.env.OPENROUTER_API_KEY!,
      baseUrl: "https://openrouter.ai/api",
    },
  }),
};

const task = "provider-cookbook";

const openaiResult = await providers.openai.chat.completions.create(
  {
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: "Reply with exactly OK." }],
  },
  { task },
);

const openrouterResult = await providers.openrouter.responses.create(
  {
    input: "Reply with exactly OK.",
    model: "z-ai/glm-5-turbo",
  },
  { task },
);

const anthropicResult = await providers.anthropic.messages.create(
  {
    model: "claude-sonnet-4-6",
    max_tokens: 256,
    messages: [{ role: "user", content: "Reply with exactly OK." }],
  },
  { task },
);

const inferenceNetResult = await providers.inferenceNet.chat.completions.create(
  {
    model: "google/gemma-3-27b-instruct/bf-16",
    messages: [{ role: "user", content: "Reply with exactly OK." }],
  },
  { task },
);

const cerebrasResult = await providers.cerebras.chat.completions.create(
  {
    model: "llama-4-scout-17b-16e-instruct",
    messages: [{ role: "user", content: "Reply with exactly OK." }],
  },
  { task },
);

const geminiResult = await providers.gemini.chat.completions.create(
  {
    model: "gemini-2.5-flash",
    messages: [{ role: "user", content: "Reply with exactly OK." }],
  },
  { task },
);

console.log({
  anthropic: anthropicResult.content[0],
  cerebras: cerebrasResult.choices[0]?.message.content,
  gemini: geminiResult.choices[0]?.message.content,
  inferenceNet: inferenceNetResult.choices[0]?.message.content,
  openai: openaiResult.choices[0]?.message.content,
  openrouter: openrouterResult.output_text,
});

When routing to Inference.net itself, omit provider and authenticate only with your Inference API key. Batch and file upload routes must be called directly against the REST/upload endpoints, not through the proxy.

Vercel AI SDK adapters

import { createAnthropic } from "@ai-sdk/anthropic";
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { generateText } from "ai";

import {
  createInferenceProvider,
} from "@inference/sdk";

const common = {
  environment: process.env.NODE_ENV,
  apiKey: process.env.INFERENCE_API_KEY!,
};

const aiProviders = {
  anthropic: createInferenceProvider({
    ...common,
    createAnthropic,
    provider: {
      apiKey: process.env.ANTHROPIC_API_KEY!,
      baseUrl: "https://api.anthropic.com",
    },
  }),
  cerebras: createInferenceProvider({
    ...common,
    createOpenAICompatible,
    provider: {
      apiKey: process.env.CEREBRAS_API_KEY!,
      baseUrl: "https://api.cerebras.ai",
    },
  }),
  gemini: createInferenceProvider({
    ...common,
    createOpenAICompatible,
    provider: {
      apiKey: process.env.GEMINI_API_KEY!,
      baseUrl: "https://generativelanguage.googleapis.com/v1beta/openai",
    },
  }),
  inferenceNet: createInferenceProvider({
    ...common,
    createOpenAICompatible,
  }),
  openai: createInferenceProvider({
    ...common,
    createOpenAICompatible,
  }),
  openrouter: createInferenceProvider({
    ...common,
    createOpenAICompatible,
    provider: {
      apiKey: process.env.OPENROUTER_API_KEY!,
      baseUrl: "https://openrouter.ai/api",
    },
  }),
};

const task = "provider-cookbook";

const openaiText = await generateText({
  model: aiProviders.openai.chatModel("gpt-4o-mini"),
  prompt: "Reply with exactly OK.",
  providerOptions: { inference: { task } },
});

const openrouterText = await generateText({
  model: aiProviders.openrouter.chatModel("z-ai/glm-5-turbo"),
  prompt: "Reply with exactly OK.",
  providerOptions: { inference: { task } },
});

const anthropicText = await generateText({
  model: aiProviders.anthropic.messages("claude-sonnet-4-6"),
  prompt: "Reply with exactly OK.",
  providerOptions: { inference: { task } },
});

const inferenceNetText = await generateText({
  model: aiProviders.inferenceNet.chatModel("google/gemma-3-27b-instruct/bf-16"),
  prompt: "Reply with exactly OK.",
  providerOptions: { inference: { task } },
});

const cerebrasText = await generateText({
  model: aiProviders.cerebras.chatModel("llama-4-scout-17b-16e-instruct"),
  prompt: "Reply with exactly OK.",
  providerOptions: { inference: { task } },
});

const geminiText = await generateText({
  model: aiProviders.gemini.chatModel("gemini-2.5-flash"),
  prompt: "Reply with exactly OK.",
  providerOptions: { inference: { task } },
});

console.log({
  anthropic: anthropicText.text,
  cerebras: cerebrasText.text,
  gemini: geminiText.text,
  inferenceNet: inferenceNetText.text,
  openai: openaiText.text,
  openrouter: openrouterText.text,
});

Design constraints

• Uses the caller's installed openai, @anthropic-ai/sdk, AI SDK provider factories, and AI SDK Core package
• Treats Inference routing headers as adapter-owned
• Keeps unsupported endpoints available through the returned official client
• Keeps unsupported AI SDK model methods available through the returned provider
• Defaults to https://api.inference.net
• Expects upstream provider URLs in provider.baseUrl root form (for example, https://api.openai.com or https://api.anthropic.com); a trailing /v1 is normalized away for OpenAI-compatible and Anthropic overrides
• AI SDK adapters use providerOptions.inference for per-call task, metadata, and environment
• AI SDK image models are not wrapped in this release

Installation

• Install @inference/sdk
• Install the provider SDK you plan to use:
  • openai
  • @anthropic-ai/sdk
• Install the AI SDK packages you plan to use:
  • ai
  • @ai-sdk/openai-compatible
  • @ai-sdk/anthropic
• Both provider SDKs are optional peer dependencies; you only need the one you initialize the adapter with

Next milestones

• Add more wrapped OpenAI and Anthropic resource methods as backend support grows
• Add image-model wrapping once local and hosted image support is verified for the AI SDK adapters
• Add a CI matrix that validates compatibility across supported provider SDK versions