@ljoukov/llm

Unified TypeScript wrapper over:

• OpenAI Responses API (openai)
• Anthropic Claude API via short haiku-*, sonnet-*, opus-*, and fable-* model ids
• Claude Code subscription models via claude-* model ids and the locally authenticated Claude Code runtime
• Google Gemini via Vertex AI or the Gemini Developer API (@google/genai)
• Fireworks direct serverless models (DeepSeek V4.1 Flash, GLM-5.3, and Kimi K3)
• ChatGPT subscription models via chatgpt-* model ids (reuses Codex auth store, or a token provider)
• Agentic orchestration with subagents via runAgentLoop() + built-in delegation control tools

Designed around a single streaming API that yields:

• response text deltas
• thought (reasoning summary) deltas
• usage token counts + estimated USD cost

Install

npm i @ljoukov/llm

Requires Node.js 22 or newer.

Environment variables

This package optionally loads a .env.local file from process.cwd() (Node.js) on first use (dotenv-style KEY=value syntax) and does not override already-set process.env values. It always falls back to plain environment variables.

See Node.js docs on environment variables and dotenv files: https://nodejs.org/api/environment_variables.html#dotenv

OpenAI

• OPENAI_API_KEY
• OPENAI_RESPONSES_WEBSOCKET_MODE (auto | off | only, default: auto)
• OPENAI_BASE_URL (optional; defaults to https://api.openai.com/v1)

Gemini

Use one backend:

• GEMINI_API_KEY or GOOGLE_API_KEY for the Gemini Developer API
• GOOGLE_SERVICE_ACCOUNT_JSON for Vertex AI (the contents of a service account JSON key file, not a file path)
• LLM_FILES_GCS_BUCKET for canonical file storage used by files.create() and automatic large-attachment offload
• LLM_FILES_GCS_PREFIX (optional object-name prefix inside LLM_FILES_GCS_BUCKET)
• VERTEX_GCS_BUCKET for Vertex-backed Gemini file attachments / file_id inputs
• VERTEX_GCS_PREFIX (optional object-name prefix inside VERTEX_GCS_BUCKET)

GOOGLE_SERVICE_ACCOUNT_JSON selects Vertex AI when set, including when an API key is also present. Otherwise, GEMINI_API_KEY or GOOGLE_API_KEY selects the Gemini Developer API. Unset the service account variable to use an API key.

Canonical files are stored in GCS with a default 48h TTL. OpenAI and ChatGPT consume those files via signed HTTPS URLs. Gemini still mirrors canonical files lazily into provider-native storage when needed:

• Gemini Developer API mirrors into Gemini Files
• Vertex-backed Gemini mirrors into VERTEX_GCS_BUCKET and uses gs://... URIs

Configure lifecycle rules on those buckets if you want hard 48-hour cleanup for mirrored objects.

Vertex AI service account setup

You need a Google service account key JSON for your Firebase / GCP project (this is what you put into GOOGLE_SERVICE_ACCOUNT_JSON).

• Firebase Console: your project -> Project settings -> Service accounts -> Generate new private key
• Google Cloud Console: IAM & Admin -> Service Accounts -> select/create an account -> Keys -> Add key -> Create new key -> JSON

Either path is enough. Both produce the same kind of service account key .json file.

Official docs: https://docs.cloud.google.com/iam/docs/keys-create-delete

Store the JSON on one line (recommended):

jq -c . < path/to/service-account.json

Set it for local dev:

export GOOGLE_SERVICE_ACCOUNT_JSON="$(jq -c . < path/to/service-account.json)"

If deploying to Cloudflare Workers/Pages:

jq -c . < path/to/service-account.json | wrangler secret put GOOGLE_SERVICE_ACCOUNT_JSON

Anthropic Claude API

• ANTHROPIC_API_KEY

Keep provider keys in server-side environment variables or a git-ignored .env.local file.

Claude Code subscription models

Install Claude Code and sign in with your Claude subscription using claude auth login. The claude-* models reuse that local login through the Claude Agent SDK, which launches Claude Code as a subprocess. They require a local Node.js environment where Claude Code can run; the subscription route is unsuitable for edge runtimes. The executable defaults to claude on PATH; set LLM_CLAUDE_EXECUTABLE to use a specific installation.

The library keeps API-key authentication separate from this route so a parent process's ANTHROPIC_API_KEY cannot silently turn a subscription call into an API-key call. It does not read or copy raw OAuth credentials into the library. Claude subscription limits and any enabled Claude Code extra usage apply.

Fireworks

• FIREWORKS_TOKEN (or FIREWORKS_API_KEY)

ChatGPT subscription models

By default, chatgpt-* models reuse the ChatGPT OAuth tokens stored by the Codex CLI:

• ${CODEX_HOME:-~/.codex}/auth.json

If you deploy to multiple environments (Vercel, GCP, local dev, etc.), use a centralized HTTPS token provider that owns refresh-token rotation and serves short-lived access tokens.

• CHATGPT_AUTH_TOKEN_PROVIDER_URL (example: https://chatgpt-auth.<your-domain>)
• CHATGPT_AUTH_TOKEN_PROVIDER_API_KEY (prefer a named caller token created by the provider CLI)
• CHATGPT_AUTH_API_KEY (backward-compatible fallback; the Worker uses this as its admin secret)
• CHATGPT_AUTH_TOKEN_PROVIDER_STORE (deprecated compatibility setting; the current Worker always uses D1)
• CHATGPT_CODEX_PROXY_URL (optional Vercel proxy endpoint; accepts either https://<project>.vercel.app or https://<project>.vercel.app/api/codex/responses)
• CHATGPT_CODEX_PROXY_API_KEY (bearer token for CHATGPT_CODEX_PROXY_URL)
• CHATGPT_CODEX_ENDPOINT (optional direct endpoint override; defaults to https://chatgpt.com/backend-api/codex/responses)
• CHATGPT_CODEX_IMAGES_ENDPOINT (optional direct Images endpoint override; generation/edit paths are derived from it)
• CHATGPT_RESPONSES_WEBSOCKET_MODE (auto | off | only, default: auto)

This repo includes a Cloudflare Workers token provider implementation in workers/chatgpt-auth/. It also includes a minimal Vercel streaming proxy app in vercel/codex-proxy/.

If CHATGPT_AUTH_TOKEN_PROVIDER_URL plus CHATGPT_AUTH_TOKEN_PROVIDER_API_KEY (or the legacy CHATGPT_AUTH_API_KEY) are set, chatgpt-* models fetch a newly selected account from the token provider for each top-level request and do not read the local Codex auth store. The included Worker rotates atomically across enabled subscription accounts and records which named caller received which account. See workers/chatgpt-auth/README.md for browser-assisted account login and caller-management commands.

If CHATGPT_CODEX_PROXY_URL + CHATGPT_CODEX_PROXY_API_KEY are set, chatgpt-* text and image requests are sent through that proxy and the local process does not need access to the Codex auth store or token provider. The Vercel proxy fetches short-lived ChatGPT access tokens from workers/chatgpt-auth; it streams Responses traffic and forwards Images JSON. These proxy bearer tokens should be server-side environment variables; do not expose them in browser bundles.

Responses transport

For OpenAI and chatgpt-* model paths, this library now tries Responses WebSocket transport first and falls back to HTTP/SSE automatically when needed.

• auto (default): try WebSocket first, then fall back to SSE
• off: use SSE only
• only: require WebSocket (no fallback)

When fallback is triggered by an unsupported WebSocket upgrade response (for example 426), the library keeps using SSE for the rest of the process to avoid repeated failing upgrade attempts.

Adaptive per-model concurrency

Provider calls use adaptive, overload-aware concurrency (with retry/backoff where supported). Configure hard caps in code (clamped to 1..64):

import { configureModelConcurrency } from "@ljoukov/llm";

configureModelConcurrency({
  globalCap: 8,
  providerCaps: {
    openai: 16,
    google: 3,
    fireworks: 8,
    anthropic: 4,
    claude: 2,
  },
  modelCaps: {
    "gpt-6-luna": 24,
  },
  providerModelCaps: {
    google: {
      "gemini-3.1-pro-preview": 2,
    },
  },
});

Default caps (without configuration):

• OpenAI: 12
• Google preview models (*preview*): 2
• Other Google models: 4
• Fireworks: 6
• Anthropic: 4
• Claude Code subscription: 2

Usage

Use OpenAI-style request fields:

• input: string or message array
• instructions: optional top-level system instructions
• message roles: developer, system, user, assistant

Basic (non-streaming)

import { generateText } from "@ljoukov/llm";

const result = await generateText({
  model: "gpt-6-luna",
  input: "Write one sentence about TypeScript.",
});

console.log(result.text);
console.log(result.usage, result.costUsd);

Image Generation

generateImages() is the low-level, direct image API. Its model selects both the credential route and the image endpoint; no GPT text model is involved:

| Library model | Authentication and billing | Endpoint | | --- | --- | --- | | gpt-image-2 | OPENAI_API_KEY / public API billing | OpenAI Images API /v1/images/generations or /v1/images/edits | | chatgpt-gpt-image-2 | ChatGPT/Codex login / subscription allowance | ChatGPT Codex /backend-api/codex/images/generations or /backend-api/codex/images/edits |

Providing styleImages selects the corresponding edit endpoint. Otherwise, the generation endpoint is used. The subscription path calls GPT Image 2 directly; it does not send the prompt through Responses or an orchestrating text model.

Direct public API

import {
  generateImages,
  type LlmOpenAiImageResolution,
  OPENAI_GPT_IMAGE_2_QUALITY_LEVELS,
  OPENAI_GPT_IMAGE_2_RESOLUTIONS,
  OPENAI_GPT_IMAGE_2_SEMANTIC_SIZE_RESOLUTIONS,
  OPENAI_GPT_IMAGE_2_SIZE_CONSTRAINTS,
} from "@ljoukov/llm";

const customResolution = "1440x960" satisfies LlmOpenAiImageResolution;

const images = await generateImages({
  model: "gpt-image-2",
  stylePrompt: "Warm amber desk light, deep blue night, cinematic laboratory mood.",
  imagePrompts: ["A compact lab bench still life with glassware and an open notebook"],
  imageSize: "landscape",
  imageQuality: "low",
  numImages: 1,
});

console.log(OPENAI_GPT_IMAGE_2_RESOLUTIONS, OPENAI_GPT_IMAGE_2_QUALITY_LEVELS);
console.log(OPENAI_GPT_IMAGE_2_SEMANTIC_SIZE_RESOLUTIONS);
console.log(OPENAI_GPT_IMAGE_2_SIZE_CONSTRAINTS);
console.log(images[0]?.mimeType, images[0]?.data.byteLength);

// Use imageResolution instead when exact/custom dimensions are required.
const exactImages = await generateImages({
  model: "gpt-image-2",
  stylePrompt: "Technical product illustration.",
  imagePrompts: ["An exploded view of a mechanical keyboard switch"],
  imageResolution: customResolution,
  imageQuality: "medium",
  outputFormat: "webp",
  outputCompression: 80,
});

GPT Image 2 has two mutually exclusive size controls:

• imageSize: "square" → 1024x1024, "landscape" → 1536x1024, "portrait" → 1024x1536, or "auto".
• imageResolution: "auto" or an exact WIDTHxHEIGHT string, including custom dimensions.

Do not pass both. OPENAI_GPT_IMAGE_2_RESOLUTIONS exposes the popular exact presets plus "auto". Custom resolutions must satisfy OPENAI_GPT_IMAGE_2_SIZE_CONSTRAINTS: each edge is at most 3840px and a multiple of 16px, the aspect ratio is at most 3:1, and total pixels are between 655,360 and 8,294,400. Resolutions above 3,686,400 pixels are experimental. Gemini image requests also use a property named imageSize, but their model-discriminated values are "1K", "2K", or "4K".

The direct public route supports imageSize or imageResolution, imageQuality, numImages, background, outputFormat, outputCompression, and moderation. stylePrompt, styleImages, and imagePrompts compose the generation/edit prompt. imagePrompts produces one request per entry, while numImages controls n within each request. partialImages is reserved for a streaming image API and is rejected by the non-streaming generateImages() call.

The official GPT Image 2 output estimates for one image, checked October 22, 2026, are:

| Quality | Square 1024x1024 | Portrait 1024x1536 | Landscape 1536x1024 | | --- | ---: | ---: | ---: | | low | $0.006 | $0.005 | $0.005 | | medium | $0.053 | $0.041 | $0.041 | | high | $0.211 | $0.165 | $0.165 |

When usage is reported, the library prices prompt text at $5, input images at $8, and image output at $30 per million tokens. It uses reported output tokens for any generated resolution and avoids adding the per-image estimate again. Direct Images API calls have no cached-input discount. When output usage is missing, the table supplies an approximate output charge; custom resolutions use the nearest listed orientation, and auto uses the medium square default. Known input tokens are still added. Consult the current OpenAI image generation cost calculator before relying on a snapshot.

Direct ChatGPT subscription API

To use ChatGPT/Codex subscription-backed image generation instead of the public Images API, use chatgpt-gpt-image-2:

const images = await generateImages({
  model: "chatgpt-gpt-image-2",
  stylePrompt: "Warm amber desk light, deep blue night, cinematic laboratory mood.",
  imagePrompts: [
    "Create a tall upright 2:3 portrait canvas showing a compact lab bench with glassware and an open notebook. The final image must be taller than it is wide.",
  ],
  background: "auto",
});

This route reuses the same ChatGPT auth setup as other chatgpt-* models. It supports background; up to five styleImages can be supplied for an edit. It returns one image per imagePrompts entry, so repeat or vary entries when multiple images are needed. The library always sends n: 1, size: "auto", and quality: "auto", and therefore does not expose structured image count, size, resolution, or quality properties on LlmChatGptGenerateImagesRequest. The current subscription endpoint returns PNG and does not expose request controls for outputFormat, outputCompression, moderation, or action. Telemetry records an API-equivalent costUsd estimate from returned usage, or the service's returned quality/size when usage is absent. These estimates are for comparison; they are not subscription charges or remaining plan allowance.

Put the desired canvas orientation or aspect ratio directly in imagePrompts. In a live check on July 13, 2026, prompt instructions selected all three canvas shapes while the request kept size, quality, and background on auto:

| Prompt intent | Observed service quality | Observed PNG dimensions | | --- | --- | --- | | Very wide panoramic 3:1 landscape canvas | medium | 2172x724 | | Tall upright 2:3 portrait canvas | low | 1024x1536 | | Exact 1:1 square icon canvas | low | 1254x1254 |

Structured non-auto size requests were normalized by the private service and are intentionally not part of the subscription API. Prompt-driven aspect ratios work, but exact pixel dimensions remain service-selected. Use public gpt-image-2 with imageResolution when pixel dimensions are a hard requirement.

Image generation in agent loops

For the common agent case, enable imageGenerationTool on runAgentLoop(). The outer model is always the text/reasoning model that decides whether to invoke image generation:

import { runAgentLoop } from "@ljoukov/llm";

const result = await runAgentLoop({
  model: "chatgpt-gpt-5.6-sol",
  input:
    "Create a very wide panoramic 3:1 landscape canvas illustrating a moonlit paper-cut forest. The final image must be much wider than it is tall.",
  imageGenerationTool: true,
});

console.log(result.steps.flatMap((step) => step.toolCalls));

By default, the convenience option follows the outer model's credential route:

• An OpenAI API text model such as gpt-5.6 receives the provider-hosted Responses image_generation tool with nested model: "gpt-image-2".
• A ChatGPT-authenticated text model receives a runtime image_generation function tool that directly calls the subscription Images endpoint with model: "gpt-image-2".

Set imageGenerationTool.model to override that default and choose the other credential route. Structured size and quality options require an explicit public model: "gpt-image-2"; subscription tools derive aspect ratio from the prompt and let the service select quality automatically.

For advanced routing, wire the tool explicitly. This example deliberately uses a public OpenAI text model for orchestration and the ChatGPT subscription for rendering:

import { createImageGenerationTool, runToolLoop } from "@ljoukov/llm";

const result = await runToolLoop({
  model: "gpt-5.6",
  input:
    "Generate a tall upright 2:3 portrait poster canvas of a glass observatory above the clouds. The final image must be taller than it is wide.",
  tools: {
    image_generation: createImageGenerationTool({
      model: "chatgpt-gpt-image-2",
    }),
  },
});

To expose the public hosted Responses tool directly, use modelTools. Hosted image output is returned as an inlineData part in result.content:

const result = await runToolLoop({
  model: "gpt-5.6",
  input: "Generate a wide watercolor panorama of a coastal village.",
  tools: {},
  modelTools: [
    {
      type: "image-generation",
      model: "gpt-image-2",
      imageSize: "landscape",
      imageQuality: "high",
      outputFormat: "webp",
      background: "opaque",
    },
  ],
});

const generated = result.content?.parts.find((part) => part.type === "inlineData");

The hosted public tool additionally exposes action, outputFormat, outputCompression, moderation, and inputImageMask. It is only available with OpenAI API text models. For a ChatGPT-authenticated outer model, use imageGenerationTool or createImageGenerationTool() so rendering happens through the direct subscription endpoint. GPT Image 2 always processes input images at high fidelity, and this library does not expose paid partial-image streaming until it can surface those intermediate images as events.

Streaming (response + thoughts + usage)

import { streamText } from "@ljoukov/llm";

const call = streamText({
  model: "gpt-6-luna",
  input: "Explain what a hash function is in one paragraph.",
});

for await (const event of call.events) {
  if (event.type === "delta" && event.channel === "thought") {
    process.stderr.write(event.text);
  }
  if (event.type === "delta" && event.channel === "response") {
    process.stdout.write(event.text);
  }
  if (event.type === "usage") {
    console.log("\n\nusage:", event.usage, "costUsd:", event.costUsd);
  }
}

const result = await call.result;
console.log("\nmodelVersion:", result.modelVersion);

Full conversation (multi-turn)

Pass a full message array via input.

import { generateText, type LlmInputMessage } from "@ljoukov/llm";

const input: LlmInputMessage[] = [
  {
    role: "system",
    content: "You are a concise assistant.",
  },
  {
    role: "user",
    content: "Summarize: Rust is a systems programming language.",
  },
  {
    role: "assistant",
    content: "Rust is a fast, memory-safe systems language.",
  },
  {
    role: "user",
    content: "Now rewrite it in 1 sentence.",
  },
];

const result = await generateText({ model: "gpt-6-luna", input });
console.log(result.text);

Files API

The library now exposes an OpenAI-like canonical files API:

import fs from "node:fs";
import { files, generateText, type LlmInputMessage } from "@ljoukov/llm";

const stored = await files.create({
  data: fs.readFileSync("report.pdf"),
  filename: "report.pdf",
  mimeType: "application/pdf",
});

const input: LlmInputMessage[] = [
  {
    role: "user",
    content: [
      { type: "text", text: "Summarize the PDF in 5 bullets." },
      { type: "input_file", file_id: stored.id, filename: stored.filename },
    ],
  },
];

const result = await generateText({ model: "gpt-6-luna", input });
console.log(result.text);

Canonical storage now uses GCS-backed objects with a 48h TTL.

• OpenAI and ChatGPT models resolve that file_id to a signed HTTPS URL.
• Gemini Developer API mirrors the file lazily into Gemini Files when needed.
• Vertex-backed Gemini mirrors the file lazily into VERTEX_GCS_BUCKET and uses gs://... URIs.

Available methods:

• files.create({ path | data, filename?, mimeType? })
• files.retrieve(fileId)
• files.delete(fileId)
• files.content(fileId)

Attachments (files / images)

Use inlineData parts to attach base64-encoded bytes (intermixed with text). inlineData.data is base64 (not a data URL).

Optional: set filename on inlineData to preserve the original file name when the provider supports it.

Note: inlineData is mapped based on mimeType.

• image/* -> image input (input_image)
• otherwise -> file input (input_file, e.g. application/pdf)

You can also pass OpenAI-style file/image parts directly:

• input_file with file_id
• input_image with file_id

When the combined inline attachment payload in a single request would exceed about 20 MiB of base64/data-URL text, the library automatically uploads those attachments to the canonical files store first and swaps the prompt to file references:

• OpenAI / ChatGPT: use signed HTTPS URLs for canonical files
• Gemini Developer API: mirrors to Gemini Files and sends fileData.fileUri
• Vertex AI: mirrors to VERTEX_GCS_BUCKET and sends gs://... URIs

import fs from "node:fs";
import { generateText, type LlmInputMessage } from "@ljoukov/llm";

const imageB64 = fs.readFileSync("image.png").toString("base64");

const input: LlmInputMessage[] = [
  {
    role: "user",
    content: [
      { type: "text", text: "Describe this image in 1 paragraph." },
      { type: "inlineData", mimeType: "image/png", data: imageB64 },
    ],
  },
];

const result = await generateText({ model: "gpt-6-luna", input });
console.log(result.text);

You can mix direct file_id parts with inlineData. Small attachments stay inline; oversized turns are upgraded to canonical files automatically. Tool loops do the same for large tool outputs, and they also re-check the combined size after parallel tool calls so a batch of individually-small images/files still gets upgraded to canonical-file references before the next model request if the aggregate payload is too large.

You can also control image analysis fidelity with request-level mediaResolution:

• low, medium, high, original, auto
• OpenAI / ChatGPT map this onto image detail
• Gemini maps this onto media resolution/tokenization settings

const result = await generateText({
  model: "gpt-6-sol",
  mediaResolution: "original",
  input,
});

OpenAI-style direct file-id example:

import { files, generateText, type LlmInputMessage } from "@ljoukov/llm";

const stored = await files.create({
  path: "doc.pdf",
});

const input: LlmInputMessage[] = [
  {
    role: "user",
    content: [
      { type: "text", text: "Summarize the attachment." },
      { type: "input_file", file_id: stored.id, filename: stored.filename },
    ],
  },
];

const result = await generateText({ model: "gemini-2.5-pro", input });
console.log(result.text);

PDF attachment example:

import fs from "node:fs";
import { generateText, type LlmInputMessage } from "@ljoukov/llm";

const pdfB64 = fs.readFileSync("doc.pdf").toString("base64");

const input: LlmInputMessage[] = [
  {
    role: "user",
    content: [
      { type: "text", text: "Summarize this PDF in 5 bullet points." },
      { type: "inlineData", mimeType: "application/pdf", data: pdfB64 },
    ],
  },
];

const result = await generateText({ model: "gpt-6-luna", input });
console.log(result.text);

Intermixed text + multiple images (e.g. compare two images):

import fs from "node:fs";
import { generateText, type LlmInputMessage } from "@ljoukov/llm";

const a = fs.readFileSync("a.png").toString("base64");
const b = fs.readFileSync("b.png").toString("base64");

const input: LlmInputMessage[] = [
  {
    role: "user",
    content: [
      { type: "text", text: "Compare the two images. List the important differences." },
      { type: "text", text: "Image A:" },
      { type: "inlineData", mimeType: "image/png", data: a },
      { type: "text", text: "Image B:" },
      { type: "inlineData", mimeType: "image/png", data: b },
    ],
  },
];

const result = await generateText({ model: "gpt-6-luna", input });
console.log(result.text);

Gemini

import { generateText } from "@ljoukov/llm";

const result = await generateText({
  model: "gemini-4-argon",
  thinkingLevel: "low",
  input: "Return exactly: OK",
});

console.log(result.text);

The text allowlist includes gemini-4-argon and gemini-3.8-flash. Both support text streaming, structured JSON, function tools, and low, medium, or high thinking (xhigh, max, and ultra map to high).

Anthropic Claude API

import { generateText } from "@ljoukov/llm";

const result = await generateText({
  model: "sonnet-5-5",
  thinkingLevel: "low",
  input: "Return exactly: OK",
});

console.log(result.text, result.usage, result.costUsd);

Supported API IDs are haiku-5-5, sonnet-5-5, opus-5-5, and fable-5-1. They resolve to Anthropic's native claude-* names on the wire. The API route uses the native Messages API with response/thought streaming, validated JSON outputs, and JSON function tools. Tool loops accept maxOutputTokens (default 32_000, maximum 128_000), including reasoning. Unsupported hosted tools and freeform tools return explicit errors.

Claude Code subscription models

import { generateText } from "@ljoukov/llm";

const result = await generateText({
  model: "claude-sonnet-5-5",
  thinkingLevel: "low",
  input: "Return exactly: OK",
});

console.log(result.provider); // "claude"
console.log(result.text, result.usage, result.costUsd);

Supported subscription IDs are claude-haiku-5-5, claude-sonnet-5-5, claude-opus-5-5, and claude-fable-5-1. Each call launches the Claude Code runtime through the Agent SDK and uses its existing local subscription login. The subscription and API routes have separate provider names and concurrency settings.

The subscription route supports text/thought streaming, schema-validated JSON, application function tools, terminal tools, cancellation, steering, and inherited subagent models. Only the caller's tools are enabled; Claude Code's built-in tools and local project customizations are disabled. Previous messages are supplied as a role-labeled transcript because the Agent SDK accepts a user prompt rather than a Messages API history; user images and PDFs remain native media blocks. Tool schemas must describe a Zod object. The Claude concurrency cap limits top-level sessions; nested calls with the inherited model share their ancestor's slot to avoid waiting on themselves.

Version 11 changes the earlier API naming: migrate an API call from claude-sonnet-5-5 to sonnet-5-5 (likewise Haiku, Opus, and Fable). Keeping the claude- prefix selects the subscription route. The old API route has no alias.

Subscription costUsd values are API-equivalent token estimates for comparison; they are not subscription charges or remaining plan allowance. To run integration tests against the local subscription, opt in explicitly:

LLM_INTEGRATION_CLAUDE_SUBSCRIPTION=1 LLM_INTEGRATION_MODELS=claude-sonnet-5-5 npm run test:integration:claude-subscription

Gemini and Claude token costs

All 48 listed text IDs and 4 image IDs have cost data. Tests check input, output, and cached-input rates independently, plus cache writes where supported and image fallback estimates when usage is absent. Private experimental-chatgpt-* IDs have unknown prices unless a known pricingModelId is supplied explicitly. ChatGPT and Claude subscription estimates use API-equivalent rates.

Standard global API rates checked October 22, 2026 against the official Gemini pricing and Claude pricing pages. Prices below are USD per million tokens. Output rates include billed reasoning tokens.

| Model | Input | Cached input | Output | Cache writes (5m / 1h) | | --- | ---: | ---: | ---: | ---: | | gemini-4-argon, through Nov 30, 2026 | $2 | $0.10 | $10 | — | | gemini-4-argon, from Dec 1, 2026 | $4 | $0.20 | $20 | — | | gemini-3.8-flash, through Dec 31, 2026 | $0.75 | $0.075 | $3.75 | — | | gemini-3.8-flash, from Jan 1, 2027 | $1.50 | $0.15 | $7.50 | — | | haiku-5-5, prompt ≤100,000 tokens | $0.10 | $0.01 | $0.50 | $0.125 / $0.20 | | haiku-5-5, prompt >100,000 tokens | $0.50 | $0.05 | $2.50 | $0.625 / $1 | | sonnet-5-5 | $2 | $0.10 | $10 | $2.50 / $4 | | opus-5-5 | $4 | $0.20 | $20 | $5 / $8 | | fable-5-1 | $10 | $0.25 | $50 | $12.50 / $20 |

The estimator automatically switches Gemini rates at the published UTC dates. Gemini Argon supports implicit caching only. Haiku's threshold includes uncached input, cache reads, and cache writes; the higher rates apply to the whole request. Claude usage.promptTokens includes all three categories, while usage.responseTokens already includes reasoning, so it is charged once. Cache-write usage is priced when reported; the library does not enable caching implicitly. Estimates cover token inference; search/tool charges, cache storage, taxes, and account-specific discounts are excluded.

Gemini latest aliases use the concrete response model version's rate when known, with the last documented alias target as a fallback. Image estimates use reported image tokens when available and published output-token counts by image size otherwise.

Fireworks

Use the short library model IDs below; they resolve to the exact direct Fireworks model paths. These direct serverless models and pricing were checked on October 22, 2026 against the Fireworks catalogue and authenticated inference /models endpoint. Routers and embedding/reranking models are excluded.

| Library model ID | Fireworks model suffix | Input / cached / output USD per million tokens | | --- | --- | --- | | deepseek-v4.1-flash-1017 | deepseek-v4p1-flash-1017 | $0.3 / $0.006 / $1.2 | | glm-5.3 | glm-5p3 | $1.4 / $0.26 / $4.4 | | kimi-k3 | kimi-k3 | $3 / $0.3 / $15 |

Version 10 removes retired kimi-k2.5, glm-5, and minimax-m2.1; use kimi-k3, glm-5.3, or deepseek-v4.1-flash-1017 for current research runs. The Fireworks allowlist is intentionally limited to these three models; obsolete default exports are removed.

Fireworks thinkingLevel is forwarded as reasoning_effort (ultra maps to max). When omitted, each model retains its native default. Reasoning in reasoning_content is included in thoughts and thought events, and is replayed unchanged across tool calls. The provider's reported reasoning-token counts are kept in usage.thinkingTokens. See Fireworks reasoning.

Fireworks text generation and tool loops request streaming completions, including when you await generateText() or runToolLoop(). Thought and response deltas arrive while inference is running. Fragmented tool calls are assembled before execution, and usage is collected from the final stream chunk.

Since 10.1, Fireworks runToolLoop() and streamToolLoop() accept optional maxOutputTokens, forwarded as max_tokens for each completion and including reasoning tokens. Omission preserves the provider default. It must be a positive safe integer. Anthropic also accepts this option starting in 10.2; other providers explicitly reject it. This option is not available on text-generation requests. Provider output limits may differ from the model vendor's own service and are not inferred from context size.

Fireworks tool-loop steps and model events expose the raw optional finishReason. In particular, "length" indicates an output or context budget limit, including completions containing only reasoning and no final text. It must not be interpreted as a completed mathematical answer. "stop" and "tool_calls" retain their provider meanings; unknown reasons are preserved. The library does not automatically retry or add a continuation when a limit is reached. See Fireworks completion limits.

import { generateText } from "@ljoukov/llm";

const result = await generateText({
  model: "kimi-k3",
  input: "Return exactly: OK",
});

console.log(result.text);

ChatGPT subscription models

Use an explicitly supported chatgpt- model id. These calls reuse ChatGPT/Codex subscription authentication, without OpenRouter or an OpenAI API key:

import { generateText } from "@ljoukov/llm";

const result = await generateText({
  model: "chatgpt-gpt-6-astra",
  thinkingLevel: "high",
  input: "Return exactly: OK",
});

console.log(result.text);

chatgpt-gpt-6-astra supports low, medium (default), high, xhigh, and max; ultra maps to max. none and minimal are not accepted by the public thinkingLevel type. Astra uses the existing ChatGPT streaming, JSON, and tool-loop transport.

For ChatGPT text calls, costUsd is an API-equivalent estimate, not a billed subscription charge. Astra API-equivalent rates use $10 input, $1 cached input, $12.50 cache writes, and $50 output per million tokens; requests above 272,000 input tokens apply 2× input/cache rates and 1.5× output rates. usage.cacheWriteTokens is populated only when the provider reports it; unreported cache writes are not inferred.

Every supported text variant has a -fast convenience alias that sends the same provider model with priority processing enabled (service_tier="priority"). Priority rates are twice standard rates. Official pricing includes cache writes and the long-input rate above 272,000 tokens. Private experimental model pricing is unknown (zero estimate). To calculate an explicit API-equivalent estimate, pass a supported pricingModelId to estimateCallCostUsd().

Supported OpenAI and ChatGPT model ids are fixed literal unions in code:

• OpenAI API text: gpt-6.1-sol, gpt-6-astra, gpt-6-sol, gpt-6-luna, gpt-5.6 (Sol alias), gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, and a -fast form of each.
• ChatGPT subscription text: the same named variants with a chatgpt- prefix, including their -fast forms; the generic chatgpt-gpt-5.6 alias is not supported.
• OpenAI API image: gpt-image-2; ChatGPT subscription image: chatgpt-gpt-image-2.

Version 9 removes GPT-5.5 and earlier text IDs. Migrate smaller-model calls to gpt-6-luna or chatgpt-gpt-6-luna, and general calls to gpt-6.1-sol or chatgpt-gpt-6.1-sol. Image IDs are unchanged. Standard text API estimates per million tokens are $2 / $0.10 / $10 (input / cached input / output) for GPT-6.1 Sol and $0.10 / $0.01 / $0.50 for GPT-6 Luna. Subscription estimates describe equivalent API usage, not a charge against the subscription.

JSON outputs

generateJson() validates the output with Zod and returns the parsed value.

• OpenAI API models use structured outputs (json_schema) when possible.
• Gemini uses responseJsonSchema.
• The Anthropic API uses native JSON schema outputs; Claude Code subscription calls use the SDK's structured output. Both validate the result with Zod.
• chatgpt-* models try to use structured outputs too; if the endpoint/account/model rejects json_schema, the call retries with best-effort JSON parsing.

import { generateJson } from "@ljoukov/llm";
import { z } from "zod";

const schema = z.object({
  ok: z.boolean(),
  message: z.string(),
});

const { value } = await generateJson({
  model: "gpt-6-luna",
  input: "Return a JSON object with ok=true and message='hello'.",
  schema,
});

console.log(value.ok, value.message);

For calls that should use a smaller reasoning budget, pass the cross-model thinkingLevel option:

const { value } = await generateJson({
  model: "gemini-flash-latest",
  input: "Return a JSON object with ok=true and message='hello'.",
  schema,
  thinkingLevel: "low",
});

thinkingLevel accepts low, medium, high, xhigh, max, and ultra. In ordinary text and tool-loop calls, ultra sends the provider's highest supported reasoning level (max for GPT-6 and GPT-5.6). In runAgentLoop() and streamAgentLoop(), it additionally enables the built-in subagent tools and Codex-style proactive delegation prompt. Pass subagentTool: false to opt out of delegation while retaining maximum provider reasoning.

Streaming JSON outputs

Use streamJson() to stream thought deltas and get best-effort partial JSON snapshots while the model is still generating.

import { streamJson } from "@ljoukov/llm";
import { z } from "zod";

const schema = z.object({
  ok: z.boolean(),
  message: z.string(),
});

const call = streamJson({
  model: "gpt-6-luna",
  input: "Return a JSON object with ok=true and message='hello'.",
  schema,
});

for await (const event of call.events) {
  if (event.type === "delta" && event.channel === "thought") {
    process.stdout.write(event.text);
  }
  if (event.type === "json" && event.stage === "partial") {
    console.log("partial:", event.value);
  }
}

const { value } = await call.result;
console.log("final:", value);

If you only want thought deltas (no partial JSON), set streamMode: "final".

const call = streamJson({
  model: "gpt-6-luna",
  input: "Return a JSON object with ok=true and message='hello'.",
  schema,
  streamMode: "final",
});

If you want to keep generateJson() but still stream thoughts, pass an onEvent callback.

const { value } = await generateJson({
  model: "gpt-6-luna",
  input: "Return a JSON object with ok=true and message='hello'.",
  schema,
  onEvent: (event) => {
    if (event.type === "delta" && event.channel === "thought") {
      process.stdout.write(event.text);
    }
  },
});

Telemetry

Telemetry is one shared API across the library:

• direct calls: generateText(), streamText(), generateJson(), streamJson(), generateImages()
• agent loops: runAgentLoop(), streamAgentLoop()

Configure it once for the process with configureTelemetry() and override or disable it per call with telemetry.

import { configureTelemetry, generateJson, runAgentLoop } from "@ljoukov/llm";
import { z } from "zod";

configureTelemetry({
  includeStreamEvents: false,
  sink: {
    emit: (event) => {
      // event.type:
      //   "llm.call.started" | "llm.call.stream" | "llm.call.completed" |
      //   "agent.run.started" | "agent.run.stream" | "agent.run.completed"
    },
    flush: async () => {},
  },
});

const { value } = await generateJson({
  model: "gpt-6-luna",
  input: "Return { ok: true }.",
  schema: z.object({ ok: z.boolean() }),
});

await runAgentLoop({
  model: "gpt-6-luna",
  input: "Inspect the repo and update the file.",
  filesystemTool: true,
});

Per-call opt-out:

await generateJson({
  model: "gpt-6-luna",
  input: "Return { ok: true }.",
  schema: z.object({ ok: z.boolean() }),
  telemetry: false,
});

See docs/telemetry.md for the event schema and adapter guidance.

Tools

There are three tool-enabled call patterns:

1. generateText() for provider-native/server-side tools (for example web search).
2. runToolLoop() for your runtime JS/TS tools (function tools executed in your process).
3. runAgentLoop() for full agentic loops (a convenience wrapper around runToolLoop() with built-in subagent orchestration and optional filesystem tools).

Architecture note:

• Built-in filesystem tools are not a separate execution system.
• runAgentLoop() can construct a filesystem toolset, merges your optional custom tools, and calls the same runToolLoop() engine.
• This behavior is model-agnostic at API level; profile selection only adapts tool shape for model compatibility.

Provider-Native Tools (generateText())

Use this when the model provider executes the tool remotely (for example search/code-exec/shell style tools).

import { generateText } from "@ljoukov/llm";

const result = await generateText({
  model: "gpt-5.6-sol",
  input: "Run python --version in a hosted shell and summarize the result.",
  tools: [{ type: "shell" }],
});

console.log(result.text);

{ type: "shell" } uses OpenAI hosted shell containers by default. It is only supported for OpenAI API models; ChatGPT-authenticated, Gemini, and Fireworks providers reject it.

When the shell writes artifacts under /mnt/data, generateText() returns the OpenAI container reference and the library can retrieve the resulting files:

import { downloadOpenAiContainerFile, generateText, listOpenAiContainerFiles } from "@ljoukov/llm";

const result = await generateText({
  model: "gpt-5.6-sol",
  input:
    "Use the shell to write /mnt/data/report.txt containing 'hello'. Reply only when done.",
  tools: [{ type: "shell" }],
});

const containerId = result.openAi?.containers.find((c) => c.toolType === "shell")?.containerId;
if (!containerId) throw new Error("The response did not include a hosted shell container.");

const files = await listOpenAiContainerFiles(containerId);
const report = files.find((file) => file.path === "/mnt/data/report.txt");
if (!report) throw new Error("The shell did not create report.txt.");

const bytes = await downloadOpenAiContainerFile({
  containerId,
  fileId: report.id,
});

For persistent multi-step shell work, create a container first, upload any input assets, then pass it back as a container-reference:

import {
  createOpenAiContainer,
  generateText,
  uploadOpenAiContainerFile,
} from "@ljoukov/llm";

const container = await createOpenAiContainer({ name: "latex-build", memoryLimit: "1g" });
const cover = await uploadOpenAiContainerFile({
  containerId: container.id,
  filename: "cover.png",
  data: coverPngBytes,
  mimeType: "image/png",
});

await generateText({
  model: "gpt-5.6-sol",
  input: `Use ${cover.path}, write LaTeX, run xelatex, and save /mnt/data/article.pdf.`,
  tools: [{ type: "shell", environment: { type: "container-reference", containerId: container.id } }],
});

Runtime Tools (runToolLoop())

Use this when the model should call your local runtime functions.

import { runToolLoop, tool } from "@ljoukov/llm";
import { z } from "zod";

const result = await runToolLoop({
  model: "gpt-6-luna",
  input: "What is 12 * 9? Use the tool.",
  tools: {
    multiply: tool({
      description: "Multiply two integers.",
      inputSchema: z.object({ a: z.number(), b: z.number() }),
      execute: ({ a, b }) => ({ value: a * b }),
    }),
  },
});

console.log(result.text);

Use customTool() only when you need freeform/non-JSON tool input grammar.

Mid-Run Steering (Queued Input)

You can queue user steering while a tool loop is already running. Steering is applied on the next model step (it does not interrupt the current generation/tool execution).

import { streamToolLoop, tool } from "@ljoukov/llm";
import { z } from "zod";

const call = streamToolLoop({
  model: "chatgpt-gpt-6-luna",
  input: "Start implementing the feature.",
  tools: {
    echo: tool({
      inputSchema: z.object({ text: z.string() }),
      execute: ({ text }) => ({ text }),
    }),
  },
});

// Append steering while run is active.
call.append("Focus on tests first, then refactor.");

const result = await call.result;
console.log(result.text);

If you already manage your own run lifecycle, you can create and pass a steering channel directly:

import { createToolLoopSteeringChannel, runAgentLoop } from "@ljoukov/llm";

const steering = createToolLoopSteeringChannel();
const run = runAgentLoop({
  model: "chatgpt-gpt-6-luna",
  input: "Implement the task.",
  filesystemTool: true,
  steering,
});

steering.append("Do not interrupt; apply this guidance on the next turn.");
const result = await run;

Agentic Loop (runAgentLoop())

runAgentLoop() is the high-level agentic API. It supports:

• optional filesystem workspace tools,
• built-in subagent orchestration (delegate work across spawned agents),
• your own custom runtime tools.

Subagents always inherit the parent run model. The subagent control tools do not expose a model override.

For interactive runs where you want to stream events and inject steering mid-run, use streamAgentLoop():

import { streamAgentLoop } from "@ljoukov/llm";

const call = streamAgentLoop({
  model: "chatgpt-gpt-6-luna",
  input: "Start implementation.",
  filesystemTool: true,
});

call.append("Prioritize a minimal diff and update tests.");
const result = await call.result;
console.log(result.text);

1) Filesystem agent loop

For read/search/write tasks in a workspace, enable filesystemTool. The library auto-selects a tool profile by model when profile: "auto":

• Codex-like models (chatgpt-gpt-6-astra, the GPT-5.6 family, gpt-6.1-sol, gpt-6.1-sol-fast, chatgpt-gpt-6.1-sol, chatgpt-gpt-6.1-sol-fast, gpt-6-sol, chatgpt-gpt-6-sol, chatgpt-gpt-6-sol-fast, and chatgpt-gpt-6-luna): Codex-compatible filesystem tool shape.
• Gemini models: Gemini-compatible filesystem tool shape.
• Other models: model-agnostic profile (currently Gemini-style).

Confinement/policy is set through filesystemTool.options:

• cwd: workspace root for path resolution.
• fs: backend (createNodeAgentFilesystem() or createInMemoryAgentFilesystem()).
• checkAccess: hook for allow/deny policy + audit.
• allowOutsideCwd: opt-out confinement (default is false).
• mediaResolution: default image fidelity for built-in view_image outputs.

Detailed reference: docs/agent-filesystem-tools.md.

Filesystem-only example:

import { createInMemoryAgentFilesystem, runAgentLoop } from "@ljoukov/llm";

const fs = createInMemoryAgentFilesystem({
  "/repo/src/a.ts": "export const value = 1;\n",
});

const result = await runAgentLoop({
  model: "chatgpt-gpt-6-luna",
  input: "Change value from 1 to 2 using filesystem tools.",
  filesystemTool: {
    profile: "auto",
    options: {
      cwd: "/repo",
      fs,
    },
  },
});

console.log(result.text);

2) Add subagent orchestration

Enable subagentTool to allow delegation via Codex-style control tools:

• spawn_agent, send_input, resume_agent, wait, close_agent
• optional limits: maxAgents, maxDepth, wait timeouts
• root and subagent tool loops default to 64 model steps when maxSteps is omitted
• spawn_agent.agent_type supports built-ins aligned with codex-rs-style roles: default, researcher, worker, reviewer
• delegation guidance keeps critical-path work local, favors independent sidecar tasks with disjoint write scopes, and waits only when an agent result blocks the next step

import { runAgentLoop } from "@ljoukov/llm";

const result = await runAgentLoop({
  model: "chatgpt-gpt-6-luna",
  input: "Plan the work, delegate in parallel where useful, and return a final merged result.",
  subagentTool: {
    enabled: true,
    maxAgents: 4,
    maxDepth: 2,
  },
});

console.log(result.text);

3) Combine filesystem + subagents

import { createInMemoryAgentFilesystem, runAgentLoop } from "@ljoukov/llm";

const fs = createInMemoryAgentFilesystem({
  "/repo/src/a.ts": "export const value = 1;\n",
});

const result = await runAgentLoop({
  model: "chatgpt-gpt-6-luna",
  input: "Change value from 1 to 2 using filesystem tools.",
  filesystemTool: {
    profile: "auto",
    options: {
      cwd: "/repo",
      fs,
    },
  },
  subagentTool: {
    enabled: true,
    maxAgents: 4,
    maxDepth: 2,
  },
});

console.log(result.text);

Agent Logging (Console + Files + Redirects)

runAgentLoop() enables logging by default. It writes:

• console lines,
• <workspace>/agent.log,
• per-call artifacts under <workspace>/llm_calls/<timestamp>-<n>/<model-id>/ by default.

Each LLM call writes:

• request.txt, request.metadata.json, tool_call_response.txt plus tool_call_response.json (when the request includes tool outputs), and input-<n>.<ext> attachments immediately,
• streamed thoughts.txt and response.txt deltas during generation, batched to disk every five seconds while data is pending,
• tool_call.txt plus tool_call.json when the model asks to call tools, output-<n>.<ext> for inline output media, and response.metadata.json at completion,
• error.txt plus response.metadata.json on failure.

Completion, failure, and an explicit logging-session flush() immediately drain pending file deltas. Thought lines in agent.log are also batched every five seconds and flushed before other events and at the end of a run. Stream events themselves remain immediate. Only reasoning that the provider sends can be logged; an already-running non-streaming request must finish or be restarted with the updated library to gain live traces.

image_url data URLs are redacted in text/metadata logs (data:...,...) so base64 payloads are not printed inline. Every canonical upload or provider mirror is also appended to agent.log as a [upload] ... line with source, backend, bytes, and latency. Direct generateText() / streamText() calls inherit the same upload logging when you run them inside an agent logging session, and their response.metadata.json includes an uploads summary.

import path from "node:path";
import { runAgentLoop } from "@ljoukov/llm";

await runAgentLoop({
  model: "chatgpt-gpt-6-luna",
  input: "Do the task",
  filesystemTool: true,
  logging: {
    workspaceDir: path.join(process.cwd(), "logs", "agent"), // optional; defaults to filesystem cwd or process.cwd()
    callLogsDir: "llm_calls", // optional; relative paths resolve from workspaceDir
    mirrorToConsole: false, // useful for CLI UIs that already render stream events
    sink: {
      append: (line) => {
        // Optional extra destination (file, socket, queue, etc.)
      },
      flush: async () => {},
    },
  },
});

Set logging: false to disable logger output for a run.

If you need exact control over tool definitions, build the filesystem toolset yourself and call runToolLoop() directly.

import {
  createFilesystemToolSetForModel,
  createInMemoryAgentFilesystem,
  runToolLoop,
} from "@ljoukov/llm";

const fs = createInMemoryAgentFilesystem({ "/repo/a.ts": "export const n = 1;\n" });
const tools = createFilesystemToolSetForModel("chatgpt-gpt-6-luna", {
  cwd: "/repo",
  fs,
});

const result = await runToolLoop({
  model: "chatgpt-gpt-6-luna",
  input: "Update n to 2.",
  tools,
});

Agent benchmark (filesystem extraction)

For filesystem extraction/summarization evaluation across Codex, Fireworks, and Gemini models:

npm run bench:agent

Standard full refresh (all tasks, auto-write LATEST_RESULTS.md, refresh traces/latest, prune old traces):

npm run bench:agent:latest

Estimate-only:

npm run bench:agent:estimate

See benchmarks/agent/README.md for options and output format.

Examples

Interactive CLI chat with mid-run steering, thought streaming, filesystem tools rooted at the current directory, subagents enabled, and Esc interrupt support:

npm run example:cli-chat

Testing

Unit tests:

npm run test:unit

Standard integration suite:

npm run test:integration

Run only the six Gemini/Claude models added in 10.2 (streams, JSON schemas, and function-tool round trips):

npm run test:integration:new-models

LLM_INTEGRATION_MODELS can narrow this matrix to a comma-separated subset. To test the Gemini Developer API when a Vertex service account is configured in .env.local, use GOOGLE_SERVICE_ACCOUNT_JSON= npm run test:integration:new-models.

Large-file live integration tests are opt-in because they upload multi-megabyte fixtures to real canonical/provider file stores:

LLM_INTEGRATION_LARGE_FILES=1 npm run test:integration

Those tests generate valid PDFs programmatically so the canonical upload path, signed-URL reuse, and automatic large attachment offload all exercise real provider APIs. The unit suite also covers direct-call upload logging plus runAgentLoop() upload telemetry/logging for combined-image overflow, and the integration suite includes provider format coverage for common document and image attachments.

License

MIT