@inference/sdk

@inference/sdk is a thin adapter around the official openai, @anthropic-ai/sdk, and Vercel AI SDK provider factories for calling the Inference proxy with provider-native APIs.

The SDK does not implement transport, streaming, retries, or provider resource internals. It instantiates the caller's official SDK class or AI SDK provider factory with Inference defaults, adds Inference routing headers, patches the methods that benefit from Inference-specific request options, and returns the configured official client or AI SDK provider instance.

Current scope

• OpenAI and Anthropic adapters
• Returns the configured official client directly
• Patched OpenAI methods:
  • chat.completions.create
  • responses.create
  • completions.create
  • embeddings.create
• Patched Anthropic methods:
  • messages.create
  • messages.stream
  • messages.parse
  • messages.countTokens
• AI SDK v6 provider adapters:
  • OpenAI-compatible language models and embeddings
  • Anthropic language models
  • per-call providerOptions.inference support for task, metadata, and environment
• All other official client methods remain available directly on the returned client

Usage

OpenAI-compatible providers

import OpenAI from "openai";
import { createInferenceClient } from "@inference/sdk";

const inference = createInferenceClient({
  openai: OpenAI,
  observabilityApiKey: process.env.INFERENCE_API_KEY!,
  environment: process.env.NODE_ENV,
  options: {
    maxRetries: 0,
  },
  proxy: {
    baseUrl: "https://openrouter.ai/api",
    apiKey: process.env.OPENROUTER_API_KEY!,
  },
});

const response = await inference.chat.completions.create(
  {
    model: "z-ai/glm-5-turbo",
    messages: [{ role: "user", content: "Hello" }],
  },
  {
    task: "contact-research",
    metadata: {
      userId: "123",
      workflow: "lead-enrichment",
    },
  },
);

const models = await inference.models.list();

Anthropic

import Anthropic from "@anthropic-ai/sdk";
import { createInferenceClient } from "@inference/sdk";

const inference = createInferenceClient({
  anthropic: Anthropic,
  observabilityApiKey: process.env.INFERENCE_API_KEY!,
  environment: process.env.NODE_ENV,
  options: {
    maxRetries: 0,
  },
  proxy: {
    baseUrl: "https://api.anthropic.com",
    apiKey: process.env.ANTHROPIC_API_KEY!,
  },
});

const response = await inference.messages.create(
  {
    model: "claude-sonnet-4-6",
    max_tokens: 1024,
    messages: [{ role: "user", content: "Hello" }],
  },
  {
    task: "contact-research",
    metadata: {
      userId: "123",
      workflow: "lead-enrichment",
    },
  },
);

const tokenCount = await inference.messages.countTokens(
  {
    model: "claude-sonnet-4-6",
    messages: [{ role: "user", content: "Hello" }],
  },
  {
    task: "contact-research",
  },
);

Vercel AI SDK v6

import { createAnthropic } from "@ai-sdk/anthropic";
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { embed, generateObject, generateText, streamText } from "ai";
import { z } from "zod";

import {
  createInferenceProvider,
} from "@inference/sdk";

const inferenceOpenAI = createInferenceProvider({
  createOpenAICompatible,
  observabilityApiKey: process.env.INFERENCE_API_KEY!,
  environment: process.env.NODE_ENV,
  proxy: {
    baseUrl: "https://api.openai.com",
    apiKey: process.env.OPENAI_API_KEY!,
  },
});

const inferenceAnthropic = createInferenceProvider({
  createAnthropic,
  observabilityApiKey: process.env.INFERENCE_API_KEY!,
  environment: process.env.NODE_ENV,
  options: {},
  proxy: {
    baseUrl: "https://api.anthropic.com",
    apiKey: process.env.ANTHROPIC_API_KEY!,
  },
});

const textResult = await generateText({
  model: inferenceOpenAI.chatModel("gpt-4o-mini"),
  prompt: "Reply with exactly OK.",
  providerOptions: {
    inference: {
      task: "contact-research",
      metadata: {
        userId: "123",
      },
    },
  },
});

const embeddingResult = await embed({
  model: inferenceOpenAI.embeddingModel("text-embedding-3-small"),
  value: "sunny day at the beach",
  providerOptions: {
    inference: {
      task: "semantic-indexing",
    },
    openai: {
      dimensions: 512,
    },
  },
});

const objectResult = await generateObject({
  model: inferenceAnthropic.messages("claude-sonnet-4-6"),
  prompt: "Return an object with status set to OK.",
  providerOptions: {
    inference: {
      task: "structured-extraction",
      metadata: {
        userId: "user_123",
      },
    },
  },
  schema: z.object({
    status: z.string(),
  }),
});

const streamed = streamText({
  model: inferenceAnthropic("claude-sonnet-4-6"),
  prompt: "Reply with exactly OK.",
  providerOptions: {
    inference: {
      task: "streaming-chat",
    },
  },
});

for await (const chunk of streamed.textStream) {
  process.stdout.write(chunk);
}

console.log(textResult.text, embeddingResult.embedding.length, objectResult.object.status);

Use providerOptions.inference.metadata for Inference observability metadata. Use providerOptions.openai or providerOptions.anthropic only for provider-native request options that should be forwarded to that upstream SDK. OpenAI-compatible AI SDK providers default includeUsage to true so streamed responses include usage information for observability. Set options: { includeUsage: false } if a provider rejects stream_options.

Vercel AI SDK provider registry

import { createAnthropic } from "@ai-sdk/anthropic";
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { createProviderRegistry, embedMany, streamObject } from "ai";
import { z } from "zod";

import {
  createInferenceProvider,
} from "@inference/sdk";

const registry = createProviderRegistry({
  anthropic: createInferenceProvider({
    createAnthropic,
    observabilityApiKey: process.env.INFERENCE_API_KEY!,
    proxy: {
      baseUrl: "https://api.anthropic.com",
      apiKey: process.env.ANTHROPIC_API_KEY!,
    },
  }),
  openai: createInferenceProvider({
    createOpenAICompatible,
    observabilityApiKey: process.env.INFERENCE_API_KEY!,
    proxy: {
      baseUrl: "https://api.openai.com",
      apiKey: process.env.OPENAI_API_KEY!,
    },
  }),
});

const embeddings = await embedMany({
  model: registry.embeddingModel("openai:text-embedding-3-small"),
  values: ["sunny day at the beach", "rainy afternoon in the city"],
  providerOptions: {
    inference: {
      task: "semantic-indexing",
      metadata: {
        batchId: "embeddings-123",
      },
    },
  },
});

const streamedObject = streamObject({
  model: registry.languageModel("anthropic:claude-sonnet-4-6"),
  prompt: "Return an object with status set to OK.",
  schema: z.object({
    status: z.string(),
  }),
  providerOptions: {
    inference: {
      task: "structured-extraction",
      metadata: {
        requestId: "req_123",
      },
    },
  },
});

for await (const partialObject of streamedObject.partialObjectStream) {
  console.log(partialObject);
}

console.log(embeddings.embeddings.length);

Provider cookbook

The SDK currently documents and tests these provider routes:

• OpenAI
• OpenRouter
• Anthropic
• Inference.net
• Cerebras
• Gemini (via the OpenAI-compatible path)

Any other OpenAI-compatible provider can use the same OpenAI/OpenAI-compatible pattern by swapping proxy.baseUrl and the provider API key.

Official SDK adapters

import Anthropic from "@anthropic-ai/sdk";
import OpenAI from "openai";

import { createInferenceClient } from "@inference/sdk";

const common = {
  environment: process.env.NODE_ENV,
  observabilityApiKey: process.env.INFERENCE_API_KEY!,
};

const providers = {
  anthropic: createInferenceClient({
    ...common,
    anthropic: Anthropic,
    proxy: {
      apiKey: process.env.ANTHROPIC_API_KEY!,
      baseUrl: "https://api.anthropic.com",
    },
  }),
  cerebras: createInferenceClient({
    ...common,
    openai: OpenAI,
    proxy: {
      apiKey: process.env.CEREBRAS_API_KEY!,
      baseUrl: "https://api.cerebras.ai",
    },
  }),
  gemini: createInferenceClient({
    ...common,
    openai: OpenAI,
    proxy: {
      apiKey: process.env.GEMINI_API_KEY!,
      baseUrl: "https://generativelanguage.googleapis.com/v1beta/openai",
    },
  }),
  inferenceNet: createInferenceClient({
    ...common,
    openai: OpenAI,
    proxy: {
      apiKey: process.env.INFERENCE_NET_API_KEY!,
      baseUrl: "https://api.inference.net",
    },
  }),
  openai: createInferenceClient({
    ...common,
    openai: OpenAI,
    proxy: {
      apiKey: process.env.OPENAI_API_KEY!,
      baseUrl: "https://api.openai.com",
    },
  }),
  openrouter: createInferenceClient({
    ...common,
    openai: OpenAI,
    proxy: {
      apiKey: process.env.OPENROUTER_API_KEY!,
      baseUrl: "https://openrouter.ai/api",
    },
  }),
};

const task = "provider-cookbook";

const openaiResult = await providers.openai.chat.completions.create(
  {
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: "Reply with exactly OK." }],
  },
  { task },
);

const openrouterResult = await providers.openrouter.responses.create(
  {
    input: "Reply with exactly OK.",
    model: "z-ai/glm-5-turbo",
  },
  { task },
);

const anthropicResult = await providers.anthropic.messages.create(
  {
    model: "claude-sonnet-4-6",
    max_tokens: 256,
    messages: [{ role: "user", content: "Reply with exactly OK." }],
  },
  { task },
);

const inferenceNetResult = await providers.inferenceNet.chat.completions.create(
  {
    model: "google/gemma-3-27b-instruct/bf-16",
    messages: [{ role: "user", content: "Reply with exactly OK." }],
  },
  { task },
);

const cerebrasResult = await providers.cerebras.chat.completions.create(
  {
    model: "llama-4-scout-17b-16e-instruct",
    messages: [{ role: "user", content: "Reply with exactly OK." }],
  },
  { task },
);

const geminiResult = await providers.gemini.chat.completions.create(
  {
    model: "gemini-2.5-flash",
    messages: [{ role: "user", content: "Reply with exactly OK." }],
  },
  { task },
);

console.log({
  anthropic: anthropicResult.content[0],
  cerebras: cerebrasResult.choices[0]?.message.content,
  gemini: geminiResult.choices[0]?.message.content,
  inferenceNet: inferenceNetResult.choices[0]?.message.content,
  openai: openaiResult.choices[0]?.message.content,
  openrouter: openrouterResult.output_text,
});

Vercel AI SDK adapters

import { createAnthropic } from "@ai-sdk/anthropic";
import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { generateText } from "ai";

import {
  createInferenceProvider,
} from "@inference/sdk";

const common = {
  environment: process.env.NODE_ENV,
  observabilityApiKey: process.env.INFERENCE_API_KEY!,
};

const aiProviders = {
  anthropic: createInferenceProvider({
    ...common,
    createAnthropic,
    proxy: {
      apiKey: process.env.ANTHROPIC_API_KEY!,
      baseUrl: "https://api.anthropic.com",
    },
  }),
  cerebras: createInferenceProvider({
    ...common,
    createOpenAICompatible,
    proxy: {
      apiKey: process.env.CEREBRAS_API_KEY!,
      baseUrl: "https://api.cerebras.ai",
    },
  }),
  gemini: createInferenceProvider({
    ...common,
    createOpenAICompatible,
    proxy: {
      apiKey: process.env.GEMINI_API_KEY!,
      baseUrl: "https://generativelanguage.googleapis.com/v1beta/openai",
    },
  }),
  inferenceNet: createInferenceProvider({
    ...common,
    createOpenAICompatible,
    proxy: {
      apiKey: process.env.INFERENCE_NET_API_KEY!,
      baseUrl: "https://api.inference.net",
    },
  }),
  openai: createInferenceProvider({
    ...common,
    createOpenAICompatible,
    proxy: {
      apiKey: process.env.OPENAI_API_KEY!,
      baseUrl: "https://api.openai.com",
    },
  }),
  openrouter: createInferenceProvider({
    ...common,
    createOpenAICompatible,
    proxy: {
      apiKey: process.env.OPENROUTER_API_KEY!,
      baseUrl: "https://openrouter.ai/api",
    },
  }),
};

const task = "provider-cookbook";

const openaiText = await generateText({
  model: aiProviders.openai.chatModel("gpt-4o-mini"),
  prompt: "Reply with exactly OK.",
  providerOptions: { inference: { task } },
});

const openrouterText = await generateText({
  model: aiProviders.openrouter.chatModel("z-ai/glm-5-turbo"),
  prompt: "Reply with exactly OK.",
  providerOptions: { inference: { task } },
});

const anthropicText = await generateText({
  model: aiProviders.anthropic.messages("claude-sonnet-4-6"),
  prompt: "Reply with exactly OK.",
  providerOptions: { inference: { task } },
});

const inferenceNetText = await generateText({
  model: aiProviders.inferenceNet.chatModel("google/gemma-3-27b-instruct/bf-16"),
  prompt: "Reply with exactly OK.",
  providerOptions: { inference: { task } },
});

const cerebrasText = await generateText({
  model: aiProviders.cerebras.chatModel("llama-4-scout-17b-16e-instruct"),
  prompt: "Reply with exactly OK.",
  providerOptions: { inference: { task } },
});

const geminiText = await generateText({
  model: aiProviders.gemini.chatModel("gemini-2.5-flash"),
  prompt: "Reply with exactly OK.",
  providerOptions: { inference: { task } },
});

console.log({
  anthropic: anthropicText.text,
  cerebras: cerebrasText.text,
  gemini: geminiText.text,
  inferenceNet: inferenceNetText.text,
  openai: openaiText.text,
  openrouter: openrouterText.text,
});

Design constraints

• Uses the caller's installed openai, @anthropic-ai/sdk, AI SDK provider factories, and AI SDK Core package
• Treats Inference routing headers as adapter-owned
• Keeps unsupported endpoints available through the returned official client
• Keeps unsupported AI SDK model methods available through the returned provider
• Defaults to https://api.inference.net
• Expects upstream provider URLs in proxy.baseUrl root form (for example, https://api.openai.com or https://api.anthropic.com); a trailing /v1 is normalized away for OpenAI-compatible and Anthropic overrides
• AI SDK adapters use providerOptions.inference for per-call task, metadata, and environment
• AI SDK image models are not wrapped in this release

Installation

• Install @inference/sdk
• Install the provider SDK you plan to use:
  • openai
  • @anthropic-ai/sdk
• Install the AI SDK packages you plan to use:
  • ai
  • @ai-sdk/openai-compatible
  • @ai-sdk/anthropic
• Both provider SDKs are optional peer dependencies; you only need the one you initialize the adapter with

Next milestones

• Add more wrapped OpenAI and Anthropic resource methods as backend support grows
• Add image-model wrapping once local and hosted image support is verified for the AI SDK adapters
• Add a CI matrix that validates compatibility across supported provider SDK versions