@happyvertical/ai

Unified interface for AI model interactions across multiple providers. Supports OpenAI, LiteLLM, Bifrost, Ollama, Anthropic Claude, Google Gemini, AWS Bedrock, Hugging Face, Claude CLI, in-browser WebLLM (WebGPU), Qwen3-TTS, and video-generation providers (Gemini Veo, BytePlus ModelArk/Seedance, Seevio/Seedance 2.5, OpenAI-compatible /v1/videos gateways) with a consistent API for chat, completions, embeddings, streaming, function calling, image operations, asynchronous video generation, text-to-speech, and gateway admin provisioning where available.

Installation

pnpm add @happyvertical/ai

Requires @happyvertical/utils as a peer dependency.

Quick Start

import { getAI } from '@happyvertical/ai';

const ai = await getAI({
  type: 'openai',
  apiKey: process.env.OPENAI_API_KEY!,
  defaultModel: 'gpt-4o'
});

// Chat completion
const response = await ai.chat([
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'user', content: 'What is TypeScript?' }
]);
console.log(response.content);

// Simple message (convenience wrapper around chat)
const reply = await ai.message('Explain generics in one sentence');

// Streaming
for await (const chunk of ai.stream([
  { role: 'user', content: 'Write a haiku' }
])) {
  process.stdout.write(chunk);
}

Providers

// OpenAI (default when type is omitted)
const openai = await getAI({ apiKey: 'sk-...' });

// ChatGPT plan usage for an explicitly authorized local OSS app.
// LocalChatGPTSessionManager owns refreshable credentials; this provider asks
// it for a usable access token immediately before each request.
import { LocalChatGPTSessionManager } from '@happyvertical/auth/local';

const sessions = new LocalChatGPTSessionManager({
  appName: 'my-local-app',
  path: '/user/config/my-local-app/chatgpt.json',
});
const chatgpt = await getAI({
  type: 'chatgpt-plan',
  clientId: selectedAccount.clientId,
  sessionManager: sessions,
  defaultModel: selectedModel.slug,
});

// LiteLLM (OpenAI-compatible gateway)
const litellm = await getAI({
  type: 'litellm',
  apiKey: process.env.LITELLM_API_KEY!,
  baseUrl: process.env.LITELLM_BASE_URL || 'https://llm.happyvertical.com/v1',
  defaultModel: process.env.LITELLM_MODEL, // Use a model id returned by /v1/models
});

// Bifrost (OpenAI-compatible gateway with governance admin APIs)
const bifrost = await getAI({
  type: 'bifrost',
  apiKey: process.env.BIFROST_API_KEY!,
  adminUser: process.env.BIFROST_ADMIN_USER,
  adminPassword: process.env.BIFROST_ADMIN_PASSWORD,
  adminUrl: process.env.BIFROST_ADMIN_URL,
  baseUrl: process.env.BIFROST_BASE_URL || 'http://localhost:8080',
  defaultModel: process.env.BIFROST_MODEL,
});

// LiteLLM and Bifrost getModels() list gateway audio models with their own
// capabilities: transcription models (whisper, transcribe, speech-to-text
// tokens) report ['transcription'] and TTS models (tts, speech tokens) report
// ['speech']. Markers match whole model-ID tokens, not substrings. They are
// never auto-selected for chat, vision, embeddings, or image generation.
// Moderation and rerank models stay filtered out of the listing.
const transcribers = (await bifrost.getModels()).filter((model) =>
  model.capabilities.includes('transcription'),
);

// Ollama (local by default)
const ollama = await getAI({
  type: 'ollama',
  baseUrl: process.env.OLLAMA_BASE_URL || process.env.OLLAMA_HOST || 'http://localhost:11434',
  apiKey: process.env.OLLAMA_API_KEY, // Optional, only needed for remote/cloud hosts
  defaultModel: process.env.OLLAMA_MODEL, // Optional; otherwise the first compatible local model is selected
});

// Bare host:port values are also accepted and normalized to http://
const ollamaNode = await getAI({
  type: 'ollama',
  baseUrl: 'warthog:11434',
});

// Anthropic Claude
const claude = await getAI({ type: 'anthropic', apiKey: process.env.ANTHROPIC_API_KEY! });

// Google Gemini
const gemini = await getAI({ type: 'gemini', apiKey: process.env.GEMINI_API_KEY! });

// AWS Bedrock
const bedrock = await getAI({
  type: 'bedrock',
  region: 'us-east-1',
  credentials: { accessKeyId: '...', secretAccessKey: '...' }
});

// Hugging Face
const hf = await getAI({ type: 'huggingface', apiToken: process.env.HF_TOKEN! });

// Claude CLI (uses Claude Max subscription, no API key needed)
const cli = await getAI({ type: 'claude-cli', defaultModel: 'sonnet' });

// Qwen3-TTS (text-to-speech only)
const tts = await getAI({ type: 'qwen3-tts', endpoint: 'http://localhost:8880' });

// BytePlus ModelArk / Seedance (video generation only)
const modelark = await getAI({
  type: 'byteplus-modelark',
  apiKey: process.env.MODELARK_API_KEY!, // or ARK_API_KEY
});

// Seevio / Seedance 2.5 (video generation only)
const seevio = await getAI({
  type: 'seevio',
  apiKey: process.env.SEEVIO_API_KEY!,
  // Default model is pinned to seedance-2-5.
});

// OpenAI-compatible /v1/videos gateway (video generation only)
const video = await getAI({
  type: 'openai-compat-video',
  baseUrl: process.env.OPENAI_COMPAT_VIDEO_BASE_URL || 'https://llm.happyvertical.com/v1',
  apiKey: process.env.OPENAI_COMPAT_VIDEO_API_KEY!,
});

// TypeSafe / Jev (typed decisions only)
const typesafe = await getAI({
  type: 'typesafe',
  apiKey: process.env.TYPESAFE_API_KEY!,
  defaultModel: 'jev-latest',
});

// Laya (self-hosted typed decisions only; baseUrl is the laya-serve root)
const laya = await getAI({
  type: 'laya',
  baseUrl: process.env.LAYA_BASE_URL!, // e.g. http://localhost:8000
  apiKey: process.env.LAYA_API_KEY,    // only if the server sets LAYA_API_KEY
  defaultModel: 'typed-decisions',     // checkpoint; omit to let the server route
});

// WebLLM (in-browser, WebGPU, no server or API key; see "In-Browser Chat")
const local = await getAI({ type: 'webllm', model: 'Llama-3.2-1B-Instruct-q4f16_1-MLC' });

ChatGPT plan usage

type: 'chatgpt-plan' is a separate, explicit provider for local open-source apps using Sign in with ChatGPT plan usage. It does not use an OpenAI API key and is never selected automatically or used as a fallback. Pass the selected registration's clientId and a protected sessionManager compatible with LocalChatGPTSessionManager from @happyvertical/auth/local.

getModels() reads the selected account's current model catalog. Use a returned model ID as defaultModel or the per-call model. Chat and stream requests follow the preview's models and inference contract: they always call the public Responses API with store: false, stream: true, and complete only after response.completed.

The provider rejects request options that the preview limitations do not allow, including maxTokens, temperature, topP, user, hosted tools, and prior-response state. System messages are sent as developer messages. It supports text and image input; embeddings, hosted image generation, file search, Code Interpreter, native computer use, hosted MCP/connectors, and audio are unavailable on this provider.

Handle plan-specific failures without changing providers implicitly:

import {
  ChatGPTPlanSessionRevokedError,
  ChatGPTPlanUsageLimitError,
  ChatGPTPlanUsageUnavailableError,
  ChatGPTPlanWorkspaceError,
} from '@happyvertical/ai';

try {
  await chatgpt.message('Draft the implementation plan.');
} catch (error) {
  if (error instanceof ChatGPTPlanUsageLimitError) {
    // Pause plan-backed requests and link to ChatGPT Settings → Usage.
  } else if (error instanceof ChatGPTPlanWorkspaceError) {
    // Explain the selected account/workspace restriction.
  } else if (error instanceof ChatGPTPlanSessionRevokedError) {
    // Ask the user to sign in again.
  } else if (error instanceof ChatGPTPlanUsageUnavailableError) {
    // Preserve the session and retry later with bounded backoff.
  }
}

The provider preserves structured status, code, parameter, and request-ID details where OpenAI supplies them. Bearer credentials are sent only in the Authorization header and are redacted from provider-generated diagnostics. Callers can still offer another configured provider, but must make that billing choice explicit.

In-Browser Chat

type: 'webllm' runs a chat model entirely on the visitor's GPU through @mlc-ai/web-llm (WebGPU). A static site can use getAI() with no server and no API key.

pnpm add @happyvertical/ai @mlc-ai/web-llm   # web-llm is an optional peer

@mlc-ai/web-llm is imported lazily on the first request, so the root entry and every other provider never load it. getAI({ type: 'webllm' }) and the @happyvertical/ai/local subpath reach the same provider; the subpath also exports WebLLMProvider, isWebGPUAvailable(), disposeWebLLMEngines(), and the error classes.

Structured output

responseSchema takes a JSON Schema (object or string) and maps to WebLLM's grammar-constrained response_format. Decoding is masked to the schema, so a 1-2B model cannot emit a value outside an enum:

import { getAI, WebGPUUnavailableError } from '@happyvertical/ai';

try {
  const ai = await getAI({
    type: 'webllm',
    model: 'Llama-3.2-1B-Instruct-q4f16_1-MLC', // default
    onLoadProgress: ({ progress, text }) => {
      progressBar.value = progress; // 0..1 while the model downloads and compiles
      status.textContent = text;
    },
  });

  const controller = new AbortController(); // stop button -> controller.abort()
  const reply = await ai.message('I want to run a small bakery with online orders.', {
    signal: controller.signal,
    temperature: 0,
    history: [
      {
        role: 'system',
        content: 'Choose the one package that fits. Reply as JSON.',
      },
    ],
    responseSchema: {
      type: 'object',
      properties: {
        package: { enum: ['smrt-commerce', 'smrt-content', 'smrt-events'] },
        reason: { type: 'string' },
      },
      required: ['package', 'reason'],
    },
  });
  const { package: chosen } = JSON.parse(reply); // always one of the three
} catch (error) {
  if (error instanceof WebGPUUnavailableError) showFallback(); // no navigator.gpu
  else throw error;
}

responseSchema is honored only by providers that constrain decoding with a grammar (today, webllm); other providers ignore it. It implies JSON output and turns off continueOnLength. Pass responseFormat: { type: 'json_object' } for unconstrained JSON mode.

Model lifecycle

• Progress: onLoadProgress receives { progress, text, timeElapsed } while the engine downloads and compiles the model. Loading happens before the request timeout starts, so a multi-GB download does not eat the generation budget.

• Reuse: one engine is cached per model id and shared by every provider instance, including loads still in flight. A failed load is retried on the next call. disposeWebLLMEngines() unloads them.

• Bring your own engine: pass engine (an instance, or an async factory) to keep inference off the UI thread, for example with a Web Worker:

  import { CreateWebWorkerMLCEngine } from '@mlc-ai/web-llm';
  
  const ai = await getAI({
    type: 'webllm',
    engine: () =>
      CreateWebWorkerMLCEngine(
        new Worker(new URL('./llm-worker.ts', import.meta.url), { type: 'module' }),
        'Llama-3.2-1B-Instruct-q4f16_1-MLC',
      ),
  });

  A supplied engine is never loaded, cached, or unloaded by this package and the WebGPU check is skipped; model is sent only when you set it.

• Cancel: signal and timeout call engine.interruptGenerate() and throw AIError with code AI_ABORTED / AI_TIMEOUT. WebLLM generates one request at a time per engine and interruptGenerate() stops whatever is running, so requests to one engine are queued by this package: timeout and signal cover the wait too, and an aborted or timed-out request never interrupts another's generation. If something outside this package interrupts a supplied engine mid-request, the call throws AI_INTERRUPTED instead of returning a truncated reply as a normal stop.

Capabilities and limits

chat, stream, complete, message, tool calls (only on models WebLLM lists for function calling, e.g. Hermes; see getModels() and supportsFunctions), usage events through onUsage, and the shared request controls are supported. getModels() returns WebLLM's prebuilt chat model list without network calls. Embeddings, image embedding/description/generation, video, and TTS/voice methods throw AIError with code NOT_SUPPORTED, and getCapabilities() reports them as false. Without WebGPU, requests throw WebGPUUnavailableError (code: 'WEBGPU_UNAVAILABLE'); a missing peer throws WebLLMPeerMissingError (code: 'WEBLLM_PEER_MISSING').

The root entry still statically imports Node-oriented providers (for example the Gemini and Claude CLI modules), so bundling it for a browser may need those excluded; a browser-safe root entry is tracked separately.

Typed Decisions

The optional decision providers (typesafe, for TypeSafe's hosted Jev, and laya, for a self-hosted Laya server; see Laya) evaluate a shared state against a batch of typed predicates, choices, and ordered scores. They do not provide chat or text generation. Check both optional contracts before routing a request:

const ai = await getAI({ type: 'typesafe', apiKey: process.env.TYPESAFE_API_KEY! });
const capabilities = await ai.getCapabilities();
if (capabilities.decisions === true && ai.decide) {
  const result = await ai.decide({
    state: { message: 'Please refund my duplicate charge.' },
    questions: {
      refund: { type: 'predicate', instructions: 'Does `message` request a refund?' },
      route: { type: 'choice', instructions: 'Which team?', criteria: { billing: 'Payments', support: 'General help' } },
      urgency: { type: 'score', instructions: 'How urgent?', criteria: ['Low', 'Medium', 'High'] },
    },
  });
  console.log(result.answers.refund); // { type: 'predicate', probability: ... }
}

defaultModel and request model select a Jev model. Results preserve the actual provider model, token usage, complete distributions, and provenance. Probabilities are provider output and are not calibrated for comparison with another provider (Laya and Jev in particular are not interchangeable). The optional capability and method preserve compatibility with existing provider implementations.

TypeSafe documents normalized choice and score distributions. For measured wire responses that are demonstrably rounded to hundredths, the adapter also accepts a positive distribution within one percentage point of unit mass and normalizes it by its actual sum. It continues to reject non-finite, out-of-range, missing, extra, all-zero, and non-hundredth distributions.

Laya (self-hosted)

type: 'laya' is a second decision provider, for a Laya server (pip install "laya[serve]", then laya-serve; Apache-2.0 code and weights at convaiinnovations/laya). State and questions are sent to the configured server, which you can host inside your network; there is no hosted dependency. HTTP redirects are rejected to prevent forwarding state or credentials to another endpoint. Configure the final server URL directly. It implements the same decide() contract and, like TypeSafe, is decision-only: chat(), complete(), embed(), streaming and every other operation throw an AIError with code NOT_IMPLEMENTED.

laya-serve answers the Jev POST /v1/systemone wire shape, but the TypeSafe adapter is not a drop-in client for it: Laya rounds each probability to four decimals independently, picks its checkpoint with its own router, takes a max_len token window, and reports truncation. The Laya adapter handles those directly.

const ai = await getAI({
  type: 'laya',
  baseUrl: 'http://localhost:8000',
  defaultModel: 'typed-decisions',
  maxLen: 1024,
  timeout: 10_000,
});

const result = await ai.decide!({
  state: { message: 'Please refund my duplicate charge.' },
  questions: {
    route: { type: 'choice', instructions: 'Which team?', criteria: { billing: 'Payments', support: 'General help' } },
  },
});
result.model;                      // the checkpoint that answered, e.g. 'typed-decisions'
result.provenance.details;         // requested model, repo, token and truncation facts, raw values

// Per-request checkpoint and token window. maxLen is Laya-only, so type the
// options as LayaDecisionOptions (a subtype of DecisionOptions).
const options: LayaDecisionOptions = { model: 'english', maxLen: 512 }; // import type from '@happyvertical/ai'
await ai.decide!(request, options);

| Option | Meaning | | --- | --- | | baseUrl | Required. The server root, e.g. http://localhost:8000. A trailing /v1 or /v1/systemone is accepted, so a Jev-style base URL works. A path prefix behind a reverse proxy is kept. Env (Node): LAYA_BASE_URL. | | apiKey | Optional. Sent as Authorization: Bearer ...; needed only when the server sets LAYA_API_KEY. A 401 becomes an AuthenticationError. Env (Node): LAYA_API_KEY. | | defaultModel | A checkpoint: typed-decisions, english, multilingual, an alias, or a name the server registered. Omit it and nothing is sent, so the server routes by language and question IDs. A request model overrides it. | | maxLen | The total token window (max_len) sent with each request. Omit it for the checkpoint's own window (512 for english, 1024 for typed-decisions and multilingual). The server rejects values above its LAYA_MAX_TOKEN_BUDGET (8192 by default). A request maxLen overrides it. | | timeout, rateLimit, onUsage, onRequest, headers | The shared conventions, as for every provider. The adapter does not retry by itself; configure rateLimit to retry a busy server. |

How requests map. One decide() call is one POST {baseUrl}/v1/systemone carrying every question, so Laya answers them in one pass over the shared state. predicate is sent as Laya's noul; choice and score keep their shape. Only state, questions, model and max_len are sent: generation controls (maxTokens, temperature, reasoning) and unknown properties on a question are never forwarded. Laya limits a request to 64 questions, 100 options per choice, 32 levels per score, 512 options in total and 50,000 characters of state; the server answers anything larger with an HTTP 413 and the adapter reports it as a non-retryable API_ERROR (it never splits a request, because Laya's routing and usage depend on the whole question set). /v1/systemone/batch, which answers one question set over many states, is not used: decide() takes one state.

Results and provenance. result.model and provenance.model are the checkpoint that answered, read from the response's routing.model. The response's own model field is the constant laya-rl-agent (Laya's decision head), kept as provenance.details.serverModel. A server running with LAYA_JEV_STRICT=1 omits routing; the adapter then reports laya-rl-agent and leaves details.checkpoint unset rather than guess. provenance.details is a JSON map with serverModel, and where known requestedModel, checkpoint, checkpointRepo, routeReason, maxLen, stateTokens, stateTokensDropped, truncated and truncatedQuestions. Check truncated: a state longer than the window is cut, Laya still answers, and nothing else in the result says so. usage is the input token count (completionTokens is always 0).

Rounded distributions. laya-serve rounds each probability to four decimals on its own, so a choice or score distribution routinely sums to 1.0001 or 0.9999 (about half the responses in our tests). The adapter accepts a distribution whose mass is within n * 0.00005 of 1 (n values, capped at 0.01), divides it by its actual sum, and records the values as received in provenance.details.rawProbabilities (keyed by question ID, with the IDs in details.renormalized). Ranking and the chosen option are unchanged. Anything else is rejected as INVALID_RESPONSE: values that are missing, extra, not finite, outside [0, 1], summing to zero, or off by more than the rounding bound; a choice that is not a requested option; a score outside the rubric or whose legend does not match the requested levels (a non-string level may come back as Python-style JSON text and is compared as parsed JSON); answers whose IDs or types differ from the request; and a malformed usage or routing.

Errors. 401 is AUTH_ERROR. 503 (Laya's LAYA_MAX_CONCURRENT admission limit, with Retry-After) and 429 are RateLimitErrors, which the shared rateLimit option retries. Other HTTP failures are API_ERROR carrying the server's detail, retryable only for 5xx. A timeout is REQUEST_TIMEOUT, a caller abort REQUEST_ABORTED, and a connection failure NETWORK_ERROR. The first request for a checkpoint the server has not preloaded can take a minute while it loads (Laya preloads only LAYA_MODELS), so size timeout for a cold start or preload what you use. Run the stock laya-serve; no wrapper is needed.

Not interchangeable with Jev. Laya's probabilities and its confidence are not Jev's. The confidence on a choice or score answer is normalized entropy (1 - H(p)/log k), not Jev's (n*p_max - 1)/(n - 1), so a threshold tuned for one gates differently on the other. Laya's typed-decisions checkpoint was found under-confident in a small shop evaluation (59 hand-written items, so wide error bars): its numbers rank answers usefully but are not literal probabilities. Measure a cut-off on your own data and do not swap providers behind one threshold. Laya's abstention gate (min_confidence) and its task and lang routing hints are not exposed.

Video Generation

Video generation is asynchronous: submitVideoGenerationJob returns a JSON-serializable handle immediately, and the render itself runs as a provider-side job that you poll with getVideoGenerationJob. Persist the handle (it has no closures or client instances) to resume polling after a restart, and always call cancelVideoGenerationJob on abort so you are not billed for orphaned renders.

const ai = await getAI({ type: 'gemini', apiKey: process.env.GEMINI_API_KEY! });

const job = await ai.submitVideoGenerationJob({
  prompt: 'A drone shot flying over a coastal cliff at sunrise',
  durationSeconds: 8,
  resolution: '1080p',
  aspectRatio: '16:9',
});

// Persist `job` (plain JSON) and resume polling later, even after a process restart.
let status = await ai.getVideoGenerationJob(job);
while (status.status === 'queued' || status.status === 'running') {
  await new Promise((resolve) => setTimeout(resolve, 5000));
  status = await ai.getVideoGenerationJob(job);
}

if (status.status === 'succeeded') {
  const result = await ai.fetchVideoGenerationResult(job);
  if (result.data) fs.writeFileSync('output.mp4', result.data);
}

// On abort/step cancellation (best-effort — see below):
await ai.cancelVideoGenerationJob(job).catch((error) => {
  console.warn('Could not cancel video job', job.jobId, error);
});

Supported providers: gemini (Veo, via @google/genai long-running operations), byteplus-modelark (Seedance, raw-HTTP ModelArk task API), seevio (native Seedance 2.5 task API), and openai-compat-video (thin adapter over a /v1/videos-shaped REST surface — LiteLLM's Veo passthrough, Sora-shaped gateways). Providers without video support throw NOT_IMPLEMENTED, and ai.getCapabilities() reports videoGeneration: false for them.

Cancellation is best-effort, not a guarantee. Gemini has no cancel endpoint for video-generation operations at all and always throws; ModelArk can only cancel a task that hasn't started rendering yet (queued) and rejects cancellation of a running one. Callers must catch and tolerate a cancelVideoGenerationJob failure rather than treating it as fatal.

byteplus-modelark locally throttles submitVideoGenerationJob to approximate Seedance's documented account limits (QPS 2, small burst / 3 concurrent submissions — pass rateLimit: { requestsPerMinute, maxConcurrent } to override the defaults). This limiter is shared across provider instances constructed with the same apiKey, since a fresh instance is often constructed per pipeline step. It bounds concurrent submit HTTP requests, not the lifetime of the render tasks those submissions create — ModelArk doesn't expose a way to observe when a task actually finishes rendering, so task-lifetime concurrency isn't tracked. submitVideoGenerationJob is also registered with the shared rate-limit pacing wrapper, so rateLimit.enabled works the same way it does for chat.

Every provider exposes a cheap validateVideoGenerationAccess() auth-shaped check (a low-cost call — the exact shape is provider-specific: a model/task list for Gemini/ModelArk, a single-resource probe for openai-compat-video since LiteLLM's list route requires a parameter this generic adapter can't supply). It is not free — callers on a hot path must cache the result themselves rather than calling it on every iteration.

Seevio / Seedance 2.5

seevio is an independent adapter, not a ModelArk compatibility mode. It pins generation to seedance-2-5, which supports 4–30 second 480p/720p video, native audio, adaptive aspect ratio, first/last-frame image-to-video, and up to 50 multimodal reference assets. It rejects floating or alternate model IDs so an application does not silently move to a different billed model.

Use the generic referenceMedia extension for public HTTPS image, video, and audio URLs. Existing referenceImages remains available; Seevio accepts it only when every value is a public HTTPS URL because its API retrieves media directly and has no upload endpoint. Buffers and data URLs are intentionally rejected. One or two image-only references select first/last-frame image-to-video; otherwise media references select reference-to-video.

const job = await seevio.submitVideoGenerationJob({
  prompt: 'Use the product image and camera movement reference',
  durationSeconds: 8,
  resolution: '720p',
  aspectRatio: 'adaptive',
  generateAudio: true,
  referenceMedia: [
    { type: 'image', url: 'https://assets.example.com/product.jpg' },
    { type: 'video', url: 'https://assets.example.com/motion.mp4', durationSeconds: 4 },
  ],
});

Seevio reserves credits on submission. The serialized job handle preserves the reserved credit metadata; status results expose normalized billing data and result URL expiry metadata. Submit transport failures are never retried because Seevio documents no idempotency key. Polling is enforced at no more than once per task every 10 seconds. Seevio does not document task cancellation, so cancelVideoGenerationJob explicitly throws NOT_IMPLEMENTED.

Generated URLs are accepted only from https://cdn.seevio.ai by default (or from resultUrlOrigins you explicitly review); fetchVideoGenerationResult checks every redirect against that allow-list before downloading bytes. The unbilled access check is a random GET /v1/tasks/:id: an authenticated 404 is considered valid access, while 401, 403, 402, and 429 retain distinct errors.

fetchVideoGenerationResult limits downloads to 200 MiB by default (override with maxResultBytes), verifies a video/* response type, and enforces the provider timeout while streaming—before buffering the result. For a Node deployment behind an HTTPS proxy, enable Node's environment-proxy support before startup (for example NODE_USE_ENV_PROXY=1 or node --use-env-proxy on supported Node releases); this adapter uses standard fetch and does not silently configure a proxy dispatcher.

Gateway Admin

Gateway providers that support provisioning expose ai.admin.

const ai = await getAI({
  type: 'bifrost',
  apiKey: process.env.BIFROST_API_KEY!,
  adminUrl: process.env.BIFROST_ADMIN_URL || 'http://localhost:8080',
  adminUser: process.env.BIFROST_ADMIN_USER!,
  adminPassword: process.env.BIFROST_ADMIN_PASSWORD!,
  baseUrl: 'http://localhost:8080',
});

const project = await ai.admin!.createProject({
  name: 'Tenant A Production',
  tenantId: 'customer-tenant-a',
  budget: { maxLimit: 100, resetDuration: '1M' },
});

const key = await ai.admin!.createVirtualKey({
  name: 'Tenant A API Key',
  projectId: project.id,
  providerConfigs: [
    {
      provider: 'openai',
      weight: 1,
      allowedModels: ['gpt-4o-mini'],
    },
  ],
  keyIds: ['*'],
  budget: { maxLimit: 25, resetDuration: '1M' },
  rateLimit: {
    tokenMaxLimit: 10000,
    tokenResetDuration: '1h',
    requestMaxLimit: 100,
    requestResetDuration: '1m',
  },
});

console.log(key.key);

LiteLLM uses the same SDK surface, mapping projects to LiteLLM teams and virtual keys to /key/generate.

Generation Safety

Every generative provider constructor applies the same safe defaults, including instances created directly instead of through getAI():

• 4,096 output tokens when the caller passes no maxTokens (defaultOutputTokens)
• a 131,072-token output ceiling (maxOutputTokens), so an explicit maxTokens is bounded by the model, not this package
• 1,024 reasoning tokens per request
• one generated image per request
• a 120-second provider timeout
• zero provider retries (one total upstream attempt)
• local rejection with AI_LIMIT_EXCEEDED before transport when a ceiling is exceeded

Reasoning is opt-in. The 1,024-token default is a ceiling for explicitly requested reasoning, not an automatically enabled thinking budget.

Explicit reasoning.effort is forwarded as top-level reasoning_effort for direct OpenAI and LiteLLM Chat Completions, including streaming and none. Bifrost retains its nested reasoning envelope and budget/thoughts fields. Omitted effort stays omitted; the upstream provider validates model support.

LiteLLM rejects caller-supplied reasoning.maxTokens, reasoning.includeThoughts, and legacy includeThoughts before model discovery or transport. Its Chat Completions protocol has no generic separate reasoning budget/thoughts envelope: forwarding it as reasoning can reach an OpenAI route as an unsupported parameter. Use effort plus maxTokens (the combined completion limit on OpenAI routes); this is not a separately enforced reasoning-token cap. The SDK's inferred local normalization budget does not trigger this rejection. See LiteLLM's Chat Completions examples.

An approved workload can raise a client-level ceiling explicitly. Prefer a workload-specific client so the wider limit is not shared accidentally.

const ai = await getAI({
  type: 'bifrost',
  apiKey: process.env.BIFROST_API_KEY!,
  baseUrl: process.env.BIFROST_BASE_URL!,
  defaultModel: 'gemini-2.5-flash',
  timeout: 105_000,
  maxRetries: 0,
  generationLimits: {
    maxOutputTokens: 8192,
    maxReasoningTokens: 1024,
  },
  usageTags: {
    app: 'writer',
    environment: 'production',
    feature: 'long-form',
  },
  onRequest: (event) => telemetry.record(event),
});

const controller = new AbortController();
const result = await ai.message('Write the approved long-form response', {
  maxTokens: 8192,
  timeout: 105_000,
  signal: controller.signal,
  reasoning: { effort: 'low', maxTokens: 1024 },
});

signal and the SDK timeout are composed and propagated to the provider. A client abort is useful for correctness, but it does not guarantee that a remote provider stops work or billing. Token ceilings and single attempts are the primary cost controls.

onRequest receives one prompt-free terminal event for success, failure, timeout, caller abort, or local rejection. Events include requested and effective token ceilings plus sanitized usageTags; prompts, responses, and credentials are never included. Legacy thinkingLevel options remain available as deprecated aliases and are normalized through the reasoning ceiling.

Stale-connection retry (OpenAI-compatible providers)

maxRetries stays 0 by default, but the openai, litellm, and bifrost providers resend a chat() or stream() request once, immediately when it fails on a stale keep-alive socket: the OpenAI SDK's APIConnectionError (not its timeout subclass) or a cause chain carrying ECONNRESET, EPIPE, UND_ERR_SOCKET, or "socket hang up".

• chat() retries only when the failure happens before the response status and headers arrive. A reset while reading the body after headers surfaces unchanged. Most servers send a non-streamed reply's headers only once the generation is done, so a reset before headers is usually a stale socket that never reached the server; but if a proxy or load balancer cuts a long request mid-generation, the resend can re-run (and re-bill) a generation the server had begun.
• stream() also retries after headers, as long as no chunk has been yielded to the caller. No content has been delivered, but the server may already have started generating, so this can re-run (and re-bill) a generation the server had begun.

It never retries after an HTTP status response (including 5xx), a timeout, a caller abort, or once a streamed chunk has been yielded. The retry shares the call's timeout and signal; if it also fails, the same error surfaces as without it. A retry adds one to the onRequest event's attempts. It is independent of maxRetries; opt out per client with retryConnectionErrors: false.

Reasoning Text, Finish Reasons, and Chat-Template Arguments

Servers that run a reasoning parser return the model's thinking separately from its answer. The SDK surfaces it as AIResponse.reasoning from chat() and through ChatOptions.onReasoning from stream(). It is absent, and onReasoning is not called, when the server sends none.

onReasoning is called once with the accumulated reasoning deltas when the stream ends, including when it fails, is aborted, or the consumer stops iterating early (it then gets whatever arrived first), so treat it as a diagnostic rather than a completion signal. An error it throws is swallowed and cannot break the stream or replace the stream's own error.

Aborting a stream through signal, or hitting its timeout, while you keep iterating now throws AI_ABORTED or AI_TIMEOUT on the OpenAI-compatible providers instead of ending quietly as if the reply were complete; in that case onFinishReason is not called. Breaking out of the loop still ends it without an error.

const ai = await getAI({ type: 'openai', baseUrl: 'http://vllm:8000/v1', apiKey: 'local' });
const reply = await ai.chat(messages, {
  // Forwarded as chat_template_kwargs, e.g. to turn Gemma/Qwen thinking off
  // for this request without restarting the server.
  chatTemplateKwargs: { enable_thinking: false },
});
if (!reply.content.trim() && reply.reasoning) {
  // The answer landed in the reasoning channel: retry or fall back.
}

for await (const text of ai.stream(messages, {
  onReasoning: (reasoning) => log.debug({ reasoningChars: reasoning.length }),
  onFinishReason: (reason) => log.debug({ reason }),
})) process.stdout.write(text);

| Provider | reasoning source | chatTemplateKwargs | | --- | --- | --- | | openai, litellm, bifrost (OpenAI-compatible, incl. vLLM/SGLang) | message/delta reasoning_content, else a string reasoning | sent as chat_template_kwargs | | ollama | message.thinking (/api/generate thinking for complete) | ignored (use reasoning / think) | | anthropic | not populated; signed thinking is replayed through toolCalls[0].thinkingBlocks | ignored | | gemini | not populated; thought summaries are not split out of content | ignored | | others (incl. bedrock) | not populated | ignored |

• The OpenAI-compatible providers always set finishReason: stop, length, tool_calls (also for a reply that carries tool calls but reports stop), or content_filter, kept as the server reported them. An absent value is stop; any other value (e.g. vLLM abort, or legacy function_call, which carries no toolCalls) is other, with the server's value in rawFinishReason, so a partial reply is not mistaken for a complete one. Streams report the same normalised value through onFinishReason.
• chatTemplateKwargs must be a plain JSON object: at most 8,192 bytes serialized, 4 levels deep, and 32 keys per object or items per array, with finite numbers and no __proto__/constructor/prototype keys. Anything else throws AI_INVALID_CHAT_TEMPLATE_KWARGS before a request is sent. The error locates the entry by position (chatTemplateKwargs{#2}[0] is the first item under the third key) and never includes a key name or a value. Null-prototype objects (Object.create(null)) are accepted; the copy sent is always an ordinary object. Oversized input is rejected while it is walked. Each property is read once into the copy that is sent, so getters cannot answer differently between validation and transport, and later mutation of the caller's object has no effect. An empty object is not sent.
• The openai provider throws AI_UNSUPPORTED_OPTION for a non-empty chatTemplateKwargs when its resolved base URL is api.openai.com (the default when neither baseUrl nor OPENAI_BASE_URL is set): the hosted API does not accept the field, and dropping it silently would lose a template switch the caller asked for. LiteLLM and Bifrost forward it to their gateway.
• With continueOnLength, reasoning from every part is joined and reported once.

Continuing Truncated Replies

When a reply stops because it hit its output limit (finishReason: 'length': OpenAI-compatible length, Anthropic/Bedrock max_tokens, Gemini MAX_TOKENS, Ollama length), getAI() clients can ask the model to carry on and return one stitched reply. It is off by default: each continuation is a full extra request (up to 1 + maxContinuations times the spend), which should be a deliberate choice, and the generation guardrails above apply to every part.

const result = await ai.chat(messages, { continueOnLength: true });
// or { continueOnLength: { maxContinuations: 2 } }  (default 3)
// or set it as the client default: getAI({ ..., continueOnLength: true })

result.content;   // stitched text
result.parts;     // requests used (set when continuation ran)
result.truncated; // true if the reply still ended on the limit

• Each continuation re-sends the conversation with the partial reply as an assistant turn plus a "continue exactly where you stopped" user turn, and any text the model repeats at the seam (12+ characters) is trimmed. When a part ends a sentence and the next starts a new one without a space, the space is restored.
• chat, complete, message, and stream all support it (Ollama's complete switches from /generate to a one-message chat when it is on); a stream stays one continuous stream (the first characters of each continuation are held briefly so a repeated seam can be trimmed). onProgress receives only the stitched text.
• usage on the result is the sum across parts. Provider onUsage hooks still fire once per upstream request, so their sum equals the total.
• Tool calls and responseFormat: { type: 'json_object' } are never continued (stitching them is unsafe); the reply is returned as-is with truncated: true.
• With rateLimit pacing, every part is its own paced request: a rate-limit retry repeats only the part that was rejected, never the parts already paid for.
• A reply that hits the limit before producing any text (reasoning used the whole budget) is not continued; it is returned with truncated: true.
• Aborting through signal stops before the next part is requested. A part that fails throws the error; the partial text is not returned.
• truncated: true is also set when continuation is off and the reply hit the limit.
• Streaming adapters report the finish reason through ChatOptions.onFinishReason; a continued stream reports it once, with the last part's reason.

Opt-In Rate-Limit Pacing

Use rateLimit when multiple calls share the same provider budget and you want getAI() to serialize requests, honor Retry-After hints, and retry only rate-limit failures.

Pacing is enabled when:

• you set enabled: true, or
• you omit enabled and set any pacing field such as key, cooldownMs, initialDelayMs, or maxAttempts

const ai = await getAI({
  type: 'gemini',
  apiKey: process.env.GEMINI_API_KEY!,
  defaultModel: 'gemini-2.5-flash',
  rateLimit: {
    enabled: true,
    key: 'gemini:shared-batch-key',
    cooldownMs: 2000,
    initialDelayMs: 15000,
    maxAttempts: 2,
  },
});

• key coordinates pacing across multiple clients in the same process
• cooldownMs spaces successful calls that share the same budget
• initialDelayMs is the fallback retry delay when the provider omits Retry-After
• maxAttempts counts the first call plus any rate-limit retries

When rateLimit is omitted, or enabled: false is set explicitly, no in-process retry wrapper is applied. Provider retries still default to zero.

rateLimit Options

| Field | Type | Default | Notes | |------|------|---------|------| | enabled | boolean | unset | Set to true for explicit opt-in, or false to force pacing off even if other pacing fields are present | | key | string | derived | Shared budget key; clients with the same key coordinate with each other | | cooldownMs | number | 0 | Minimum delay after a successful call before the next call with the same key | | initialDelayMs | number | 5000 | Fallback retry delay when the provider does not return Retry-After | | maxAttempts | number | 1 | Total attempts, including the initial call; increase explicitly only for an approved workload | | maxRetryDelayMs | number | 60000 | Longest reset wait still retried; a limit that resets further away is not retried and its error has retryable: false | | requestsPerMinute | number | provider-specific | Used by qwen3-tts local token-bucket limiting | | maxConcurrent | number | provider-specific | Used by qwen3-tts local concurrency limiting |

• If key is omitted, @happyvertical/ai derives a provider-scoped key from the configured credentials
• Setting any of key, cooldownMs, initialDelayMs, or maxAttempts also opts in when enabled is omitted
• Only normalized rate-limit failures are retried
• The request timeout covers the whole call from the caller's side: time queued behind other calls on the same key, cooldown and retry waits, and the request itself, which gets only what is left. A call still queued at its deadline fails with AIError code AI_TIMEOUT and never reaches the provider. When a rate limit has closed the key past the deadline, the call fails at once with a RateLimitError carrying the original reason and the remaining retryAfterMs. A retry that cannot finish before the deadline is not started. Aborting signal releases a queued call (AI_ABORTED). Methods without request controls (embed, getModels, speech) use the client timeout for their queue wait when one is set. With continueOnLength, each part is its own request with its own timeout.
• stream() is left unchanged; pacing is applied to the promise-returning request methods

Example quota-sensitive batch workload:

const ai = await getAI({
  type: 'gemini',
  apiKey: process.env.GEMINI_API_KEY!,
  defaultModel: 'gemini-2.5-flash',
  rateLimit: {
    enabled: true,
    key: 'praeco:multi-site-analysis',
    cooldownMs: 2000,
    initialDelayMs: 15000,
    maxAttempts: 2,
  },
});

for (const site of sites) {
  const summary = await ai.message(`Summarize anomalies for ${site.name}`);
  console.log(site.name, summary);
}

Environment Variables

getAI() reads HAVE_AI_* variables. Explicit options passed to getAI() take precedence over those env vars.

| Variable | Purpose | |----------|---------| | HAVE_AI_PROVIDER / HAVE_AI_TYPE | Provider type | | HAVE_AI_MODEL / HAVE_AI_DEFAULT_MODEL | Default model | | HAVE_AI_API_KEY | API key (fallback) | | HAVE_AI_BASE_URL | Custom base URL | | HAVE_AI_TIMEOUT | Request timeout (ms) | | HAVE_AI_MAX_RETRIES | Max retry attempts |

Node Auto-Detection Env Vars

getAIAuto() also checks provider-specific Node.js environment variables:

• LITELLM_BASE_URL, LITELLM_API_KEY
• OLLAMA_HOST, OLLAMA_BASE_URL, OLLAMA_API_KEY
• OPENAI_API_KEY
• ANTHROPIC_API_KEY
• GEMINI_API_KEY, GOOGLE_API_KEY
• HF_TOKEN
• AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION
• MODELARK_API_KEY / ARK_API_KEY, MODELARK_BASE_URL
• OPENAI_COMPAT_VIDEO_BASE_URL, OPENAI_COMPAT_VIDEO_API_KEY
• TYPESAFE_API_KEY, TYPESAFE_BASE_URL (last-resort decision-only auto-detection)
• LAYA_BASE_URL, LAYA_API_KEY (decision-only; LAYA_BASE_URL alone selects Laya as the last resort, after TypeSafe)

API Overview

Factory Functions

• getAI(options) — Creates a provider instance by type
• getAIAuto(options) — Auto-detects provider from credentials

AIInterface Methods

All providers implement AIInterface:

| Method | Description | |--------|-------------| | chat(messages, options?) | Chat completion returning AIResponse | | decide?(request, options?) | Optional typed predicate, choice, and score batch evaluation | | message(text, options?) | Simple single-turn convenience method | | complete(prompt, options?) | Text completion | | stream(messages, options?) | Streaming chat (async iterable) | | embed(text, options?) | Text embeddings | | embedImage(image, options?) | Image embeddings (Gemini and Bedrock native, OpenAI and Ollama via describe-then-embed) | | describeImage(image, prompt?, options?) | Image description via vision models | | generateImage(prompt, options?) | Image generation (DALL-E, Imagen, Titan Image Generator, Ollama-compatible image models) | | submitVideoGenerationJob(options) | Submit an async video-generation job, returning a serializable handle | | getVideoGenerationJob(handle) | Poll job status (queued|running|succeeded|failed|cancelled) | | fetchVideoGenerationResult(handle) | Fetch the generated video once the job has succeeded | | cancelVideoGenerationJob(handle) | Cancel an in-flight video-generation job | | validateVideoGenerationAccess() | Cheap auth-shaped check for video-generation access (cache the result) | | countTokens(text) | Token count estimation | | getModels() | List available models | | getCapabilities() | Query provider capabilities | | synthesizeSpeech(text, options?) | Text-to-speech synthesis | | streamSpeech(text, options?) | Streaming TTS | | cloneVoice(options) | Clone a voice from audio sample | | designVoice(options) | Design a voice via text description | | getVoices(options?) | List available voices |

Error Types

All extend AIError: AuthenticationError, RateLimitError, ModelNotFoundError, ContextLengthError, ContentFilterError, WebGPUUnavailableError, WebLLMPeerMissingError.

OpenAI and compatible providers (including Bifrost) return ContextLengthError for HTTP 413, the structured context_length_exceeded code, and HTTP 400 responses identifying an exceeded input token or context window limit. Other HTTP 400 failures remain API errors; callers can use the error type to skip or shorten oversized inputs without treating unrelated provider failures as overflow.

• AIError.retryable distinguishes retryable failures from terminal ones
• RateLimitError.retryAfter exposes provider retry hints in seconds when available
• RateLimitError.reason keeps the provider's own text (for example token limit exceeded (277867/250000, resets every 1h)), and the message includes it, so callers can say why a call was refused
• RateLimitError.retryAfterMs is the time until the limit resets when the provider said so: Retry-After, a reset header (x-ratelimit-reset-*, anthropic-ratelimit-*-reset), or text such as "try again in 20s"
• RateLimitError.limitWindowMs is set when the provider names a limit window but no reset time ("resets every 1h"); the limit lifts within that time
• error.cause is the provider error it was mapped from
• RateLimitError.retryable is false when the limit resets more than a minute away (retryAfterMs, else limitWindowMs; rateLimit.maxRetryDelayMs changes the threshold). Job runners should not re-run such a job soon; they can reschedule it after retryAfterMs ?? limitWindowMs

try {
  await ai.chat(messages);
} catch (error) {
  if (error instanceof RateLimitError) {
    console.log('why:', error.reason);
    if (error.retryable) console.log('retry after seconds:', error.retryAfter);
    else console.log('lifts in ms:', error.retryAfterMs ?? error.limitWindowMs);
  }
}

Legacy Classes

AIClient, OpenAIClient, AIThread, and AIMessageClass are exported for backward compatibility. New code should use getAI() and the AIInterface methods.

Function Calling

const response = await ai.chat([
  { role: 'user', content: 'What is the weather in Tokyo?' }
], {
  tools: [{
    type: 'function',
    function: {
      name: 'get_weather',
      description: 'Get weather for a location',
      parameters: {
        type: 'object',
        properties: { location: { type: 'string' } },
        required: ['location']
      }
    }
  }]
});

if (response.toolCalls) {
  console.log(response.toolCalls[0].function.name);
}

To continue the loop, replay the assistant turn with tool_calls unchanged and answer each call with a role: 'tool' message carrying its tool_call_id. Every chat provider maps this to its native pairing (tool_call_id for OpenAI-compatible APIs including Bifrost and LiteLLM, tool_use_id for Anthropic, toolUseId for Bedrock, functionResponse for Gemini, tool_name for Ollama). name on the tool message is optional; it is resolved from the matching call when a provider needs it.

const call = response.toolCalls[0];
const followUp = await ai.chat([
  { role: 'user', content: 'What is the weather in Tokyo?' },
  { role: 'assistant', content: response.content, tool_calls: response.toolCalls },
  { role: 'tool', tool_call_id: call.id, content: JSON.stringify({ tempC: 21 }) },
], { tools });

Replay response.toolCalls as returned, and keep every field if you persist the conversation: provider-specific replay data rides on the tool calls. Gemini 3 needs thoughtSignature, and Anthropic with extended thinking (reasoning.maxTokens) needs thinkingBlocks (the signed thinking that opened the tool-calling turn). If an Anthropic tool loop's latest tool_use turn has no thinkingBlocks, that request is sent without extended thinking instead of failing. Gemini replays only calls that have a result, and when a Gemini 3 step has no thoughtSignature it sends Google's documented placeholder signature, which avoids the 400 but lowers reasoning quality, so persist the real one.

A loop's final round usually asks for a plain answer (toolChoice: 'none', often with no tools) while replaying the tool history. Anthropic and Bedrock reject tool history without declared tools, so both declare a minimal definition for each tool the history references. Anthropic sends tool_choice: { type: 'none' }; Bedrock Converse has no "none" choice, so it keeps the tools, omits toolChoice, and adds a system instruction not to call tools (an instruction, not a guarantee; check finishReason).

stream() yields text only and never returns tool calls, so run tool-calling rounds with chat(). Anthropic streams ignore tools/toolChoice and declare tools only when the replayed history needs them, with tool_choice none.

Usage Tracking

Track token usage, costs, and performance across all providers with the onUsage callback:

const ai = await getAI({
  type: 'openai',
  apiKey: process.env.OPENAI_API_KEY!,
  onUsage: (event) => {
    console.log(`[${event.provider}/${event.model}] ${event.operation}: ${event.usage?.totalTokens} tokens in ${event.duration}ms`);
    // Or: save to database, send to analytics, aggregate in-memory, etc.
  },
});

The UsageEvent payload:

| Field | Type | Description | |-------|------|-------------| | provider | string | Provider name ('openai', 'anthropic', 'gemini', etc.) | | model | string | Model used (e.g. 'gpt-4o', 'claude-3-5-sonnet-20241022') | | operation | string | 'chat' | 'complete' | 'message' | 'embed' | 'stream' | ... | | usage? | TokenUsage | { promptTokens, completionTokens, totalTokens } (if available) | | duration | number | Wall-clock time in milliseconds | | timestamp | Date | When the call completed | | tags? | Record<string, string> | Merged from global + per-call usageTags |

• Works with all providers and methods (chat, complete, message, embed, stream)
• complete() and message() report through their underlying chat() call
• Errors thrown inside onUsage are silently caught and will not affect API results

Tagging Usage Events

Attach custom tags to correlate usage with features, users, or workflows:

// Global tags applied to every call
const ai = await getAI({
  type: 'openai',
  apiKey: process.env.OPENAI_API_KEY!,
  usageTags: { app: 'indagator', team: 'news' },
  onUsage: (event) => {
    console.log(event.tags); // { app: 'indagator', team: 'news', feature: 'summarize' }
  },
});

// Per-call tags merge over global tags
await ai.chat(messages, {
  usageTags: { feature: 'summarize', userId: 'u_123' },
});

Claude Code Context

Install context files for AI-assisted development:

npx have-ai-context

License

MIT

JSON schema response formats

AIResponseFormat is exported for chat, completion, legacy OpenAI completion, and message options. OpenAI-compatible providers pass responseFormat: { type: 'json_schema', json_schema: { name, schema, description?, strict? } } to the API unchanged. Schema enforcement depends on the selected provider; this type does not add support to providers that do not implement that format. Automatic length continuation is disabled for JSON schema output, as it is for JSON-object output, so truncated structured responses are never concatenated.