@odla-ai/ai

⚠️ Early access — pre-1.0. Agents work from bounded runbooks; humans approve credentials, production changes, releases, and merges. APIs and exact package availability can change. Review the documented guarantees and limitations; this software is MIT-licensed and provided without warranty.

One general-purpose interface for AI inference across Anthropic (Claude), OpenAI (GPT), and Google (Gemini) — text, image, and audio — plus a tool-use agent engine and an eval harness. It's the place every odla app reaches for when it needs AI.

• Library surface. Import it in-process and bring your own keys; this package exports no generic hosted gateway. Separately, odla's System AI broker uses the same facade server-side for bounded o11y/security purposes and returns purpose grants rather than provider credentials.
• One canonical shape. Anthropic-flavored content blocks, forced tool calls with object inputs, a normalized streaming event vocabulary, cache-aware usage, and a closed error taxonomy — the same regardless of which provider answers.
• Capability-checked. Every request is validated against the model before dispatch, so "audio to Claude" fails fast with a clear error instead of an opaque provider 400.
• Server runtimes. Node 20+ and Cloudflare Workers. Provider adapters are runtime-lazy, but all three provider SDKs are package dependencies. Do not put long-lived provider keys in browser code; use your own authenticated proxy.

npm install @odla-ai/ai

Ask the runbooks first. odla's operational procedures live in a database, not in this file: npx @odla-ai/cli runbook ask "<question>" returns the current steps, and unlike anything written here it cannot be out of date. Use it before searching the web or working from memory. This README and the JSDoc in the shipped .d.ts are the version-matched API reference; a runbook is the procedure. Most tasks need an answer from both.

Quick start

import { init } from "@odla-ai/ai";

const ai = init({
  keys: {
    anthropic: process.env.ANTHROPIC_API_KEY,
    openai: process.env.OPENAI_API_KEY,
    google: process.env.GOOGLE_API_KEY,
  },
  defaultModel: "claude-opus-4-8",
});

// One call, any provider — pick the model, the provider is resolved for you.
const res = await ai.chat({
  model: "gpt-5",
  messages: [{ role: "user", content: "Say hello in one word." }],
  maxTokens: 64,
});
console.log(res.content); // [{ type: "text", text: "Hello" }]

Automatic observability

Every provider request made by chat, stream, extract, or search creates one OpenTelemetry CLIENT span when the host has configured an OTel tracer provider. The span includes provider/model/operation, all four token classes, catalog-priced cost (or an explicit unpriced marker), provider latency, and streaming time to first token. With no tracer provider, the OTel API is a no-op.

Use low-cardinality metadata to attribute a call to an app or feature:

await ai.chat({
  model: "gpt-5",
  messages,
  maxTokens: 512,
  telemetry: {
    operation: "support.answer",
    attributes: { "odla.app": "support", "deployment.environment": "prod" },
  },
});

Static defaults belong on init({ telemetry: ... }). Set telemetry: false on init or on one request to suppress automatic spans. Telemetry metadata is consumed by the facade and is never sent in a provider-native request.

Capacity control

init() can share a FIFO capacity scheduler across callers. Pools are keyed by the SHA-256 fingerprint of the active provider key plus the provider-native model id, so aliases share capacity while credentials remain isolated. The default 20% headroom makes configured TPM/RPM/TPD limits 80% effective; estimated tokens advance the next-start clock, and only positive actual-token excess advances it again.

const ai = init({
  keys: { openai: process.env.OPENAI_API_KEY },
  capacity: {
    limits: ({ model }) => model === "gpt-5-nano"
      ? { tokensPerMinute: 180_000_000, requestsPerMinute: 30_000,
          tokensPerDay: 15_000_000_000, maxParallelRequests: 64 }
      : undefined,
  },
});

Requests beyond the pacing or parallel ceiling wait FIFO. deadline includes queue time and signal cancels a queued request.

Streaming

for await (const ev of ai.stream({ model: "gemini-2.5-flash", messages: [...], maxTokens: 512 })) {
  if (ev.type === "content_block_delta" && ev.delta.type === "text_delta") {
    process.stdout.write(ev.delta.text);
  }
}

The provider event union (message_start → content_block_start → content_block_delta → content_block_stop → message_delta → message_stop, plus error) is identical across providers. Platform-hosted SSE adds a final receipt event containing usage, cost, queue wait, TTFT, and total latency.

Structured extraction (forced tool call)

const { value } = await ai.extract<{ sentiment: "positive" | "negative" | "neutral" }>({
  model: "claude-haiku-4-5",
  user: "Classify: 'I love this!'",
  tool: {
    name: "classify",
    parameters: {
      type: "object",
      properties: { sentiment: { type: "string", enum: ["positive", "negative", "neutral"] } },
      required: ["sentiment"],
    },
  },
});
// value.sentiment === "positive" (parsed and validated before it is returned)

extract and local agent tools validate provider arguments before returning or executing them. The dependency-free validator supports the common tool-schema subset: local $ref/$defs, type, enum, const, composition, object properties/required/additional properties, arrays, and string/number constraints. Unknown or unsupported schema keywords outside that documented subset fail closed. extract() throws ToolInputError; local agent tools return an error tool_result and their handlers do not execute.

Web search

const { text, citations } = await ai.search({ model: "claude-sonnet-5", user: "Who won the 2026 World Cup?" });

citations is the normalized source list, parsed from Anthropic's web_search_tool_result blocks and inline text citations, OpenAI's url_citation annotations, and Gemini's groundingMetadata. A citation carrying citedText or a startIndex/endIndex span was attributed to a specific claim; one carrying only a url was merely consulted, which is much weaker evidence that the answer came from there. It is [] when the provider reported nothing — a grounded-looking answer is not the same as a sourced one.

Two caveats worth knowing before you compare providers: Gemini reports vertexaisearch.cloud.google.com/grounding-api-redirect/… links rather than publisher URLs, so hosts only line up after you resolve those redirects; and streamed responses carry no citations, because the event set has no citation event yet.

Multimodal: text, image, audio

Content blocks carry the modality. Image is broadly supported; audio is accepted by OpenAI and Google, but not Anthropic — sending an audio block to a Claude model throws a CapabilityError before any network call.

// Image (works on Claude, GPT, Gemini)
await ai.chat({
  model: "gpt-4o",
  messages: [{ role: "user", content: [
    { type: "image", source: { type: "base64", mediaType: "image/png", data: pngB64 } },
    { type: "text", text: "What's in this image?" },
  ] }],
  maxTokens: 128,
});

// Audio (OpenAI: gpt-audio; Google: gemini-2.5-*)
await ai.chat({
  model: "gpt-audio",
  messages: [{ role: "user", content: [
    { type: "audio", source: { type: "base64", mediaType: "audio/wav", data: wavB64 } },
    { type: "text", text: "Transcribe this." },
  ] }],
  maxTokens: 256,
});

Agents: build, run, and test

A persona is a role as config-as-data: a model, a system prompt, its tools, and reasoning knobs. runAgent drives the tool-use loop (model turn → run tool handlers → feed results back → repeat) until the model stops or a step limit hits.

import { init, runAgent, type Persona, type ToolDef } from "@odla-ai/ai";

const ai = init({ keys: { anthropic: process.env.ANTHROPIC_API_KEY } });

const add: ToolDef = {
  name: "add",
  description: "Add two integers.",
  inputSchema: { type: "object", properties: { a: { type: "number" }, b: { type: "number" } }, required: ["a", "b"] },
  handler: (input) => ({ content: String((input.a as number) + (input.b as number)) }),
};

const calculator: Persona = {
  name: "calculator",
  model: "claude-opus-4-8",
  system: "Use the add tool for arithmetic, then state the result.",
  tools: [add],
  maxSteps: 4,
};

const run = await runAgent(ai, calculator, { input: "What is 21 + 21?" });
console.log(run.finalText);   // "…42"
console.log(run.toolCalls);   // [{ toolUse, output }]
console.log(run.usage);       // accumulated across turns

Provider calls and tool handlers receive a combined cancellation signal. Runs can also stop before additional effects when a run-wide budget is exhausted:

const run = await runAgent(ai, calculator, {
  input: "Reconcile these records",
  signal: request.signal,
  deadline: Date.now() + 30_000,
  budget: { maxInputTokens: 20_000, maxOutputTokens: 4_000, maxToolCalls: 8 },
});

Token limits are post-turn ceilings because provider input usage is known only after a response. maxInputTokens and maxTotalTokens therefore cannot tokenizer-bound the first prompt; they stop later turns and tool effects once reported usage reaches the ceiling. maxOutputTokens and the remaining total budget cap each request's maxTokens before dispatch. Limits must be integers (maxToolCalls may be zero; token limits must be positive). When a limit blocks tools, the loop records a matching error tool_result for every requested tool before persisting memory, so retries never inherit an unmatched effect request. stoppedReason is "budget_exhausted" when a limit prevents further work.

Tools can declare taint constraints (outputTaint / acceptsTaint) for a lightweight CaMeL-style prompt-injection gate, and personas can carry a pluggable MemoryScope.

Skills

A skill is a named bundle of tools + instructions. Attach skills to a persona and their tools are merged into the model's tool list and their instructions into the system prompt — the primitive for "add a skill and call it as tools."

import { entityCrudSkill, runAgent } from "@odla-ai/ai";
import { init as odlaInit } from "@odla-ai/db";

const db = odlaInit({ appId, adminToken: appKey, endpoint: ODLA_URL });

// Turn an odla-db entity into list/create/update/delete tools that change data.
const todos = entityCrudSkill({
  db,
  entity: "todos",
  fields: { text: { type: "string" }, done: { type: "boolean" } },
  required: ["text"],
});

const assistant = { name: "todo-bot", model: "claude-opus-4-8", skills: [todos] };
const run = await runAgent(ai, assistant, { input: "Add 'buy milk' and show my open todos." });
// the agent calls create_todo then list_todos, writing/reading via odla-db

entityCrudSkill generates list_<plural>, create_<singular>, update_<singular> (partial — pass only the fields that change, e.g. toggle done), and delete_<singular> tools whose handlers call db.transact/db.query. Or hand-roll a Skill = { name, instructions?, tools: ToolDef[] } for anything else.

Evals

import { evaluate, exactMatch, llmJudge } from "@odla-ai/ai";

const report = await evaluate({
  inference: ai,
  model: "claude-opus-4-8",
  system: "Answer with only the number.",
  grader: exactMatch(),
  cases: [
    { input: "What is 2+2?", expected: "4" },
    { input: "What is 10-3?", expected: "7" },
  ],
});
console.log(`${report.passed}/${report.total} passed`);

Graders: exactMatch, includes, structuredMatch, and llmJudge (which dogfoods the inference core to grade open-ended output against a rubric). Pass a persona instead of a model to evaluate a full agent.

Harness: configuration as data, and proof it was sent

A Persona carries live tool handlers, so it can be written in TypeScript but not in YAML, a database row, or a run record. A HarnessSpec is the same configuration as plain JSON; a ToolsetRegistry holds the code, and toPersona joins them.

import { toPersona, resolveHarness, verifyEcho } from "@odla-ai/ai";

const harness = { id: "fast", model: "gpt-5.5", effort: "low", toolset: "search" };

// Can this model run this configuration, exactly as written?
const res = resolveHarness(harness);
if (!res.ok) return skip(res.code); // "effort_unsupported", "temperature_ignored", …

const run = await runAgent(ai, toPersona(harness, { search: [searchTool] }), { input: q });

// Did the adapter actually send what was asked for?
const drift = verifyEcho(harness, res.spec, run.response.echo!);
if (drift.length) throw new Error(drift[0].detail);

resolveHarness answers rather than throws, so a caller enumerating many model × setting combinations can mark the illegal ones skipped instead of wrapping every one in try/catch. It has no "resolved with adjustments" result: a harness runs as specified or it does not run.

response.echo reports what the adapter actually built — native model id, the reasoning knob really sent, whether thinking was on, whether temperature survived. Comparing it to the request (verifyEcho) is what catches a silent substitution introduced below your code, which upfront validation cannot see. One it still surfaces today: reasoning-tier models on Chat Completions drop temperature. That is correct behavior, and it still makes a run non-comparable to one that ran on the model's default.

Per-step trace

AgentRun.usage says what a run cost in total; AgentRun.trace says where it went — one StepRecord per model turn, carrying that turn's usage, costUsd (absent when the model's pricing is unknown, never zero), durationMs, stopReason, the tool calls it triggered, and its echo.

const run = await runAgent(ai, persona, { input: q });

for (const step of run.trace) {
  console.log(step.index, step.durationMs, step.costUsd, step.toolCalls.length);
}

// A mid-run substitution a final-turn echo cannot see:
const lostReasoning = run.trace.filter((s) => s.echo?.thinkingActive === false);

Pass now on the run input to inject a clock; it defaults to Date.now.

Which OpenAI endpoint a model uses

OpenAI models are called on Chat Completions by default and on /v1/responses when the model needs it. That routing exists for one reason: Chat Completions refuses function tools whenever reasoning is in play on the gpt-5.6 generation, so the adapter used to disable reasoning to keep tools working. /v1/responses carries both, so those models go there instead and keep their reasoning. It also accepts temperature alongside an effort level, and its reasoning vocabulary covers the full low→max ladder rather than collapsing the top rungs.

Set api: "chat" | "responses" on a ModelSpec to pin the endpoint; omit it to let the adapter choose. Every built-in catalog model stays on Chat Completions — models move by explicit declaration, not by a wholesale switch.

Keys & roles

Keys are BYO — a static per-provider map and/or a dynamic resolveKey (multi-tenant / per-call). A provider with no resolvable key is simply unavailable: requesting one of its models throws a ConfigError.

const ai = init({
  resolveKey: (provider) => lookupTenantKey(currentTenant, provider),
});

"Roles" are personas (above) — config-as-data, not RBAC.

Storage, memory & BYO keys via odla-db

odla-ai follows odla-db's model: it's a self-contained npm package whose README and exported TypeScript declarations/JSDoc describe the installed API, and it integrates with @odla-ai/db for durable storage — without taking a runtime dependency on it. You install @odla-ai/db, build a client, and inject it; odla-ai talks to it through a structural interface. The rendered public reference is at https://odla.ai/docs/packages/ai.

The agent handshake (an agent never takes a platform secret):

import { requestToken, init as odlaInit } from "@odla-ai/db";
import { provisionAgentApp, odlaDbKeyResolver, OdlaDbMemory, persistRun, runAgent, init } from "@odla-ai/ai";

const ODLA_PLATFORM = "https://odla.ai";
const ODLA_DB_URL = "https://db.odla.ai";

// 1. Device authorization — the matching account signs in, reviews, and
//    approves the exact code; the agent receives a revocable platform token.
const { token } = await requestToken({
  endpoint: ODLA_PLATFORM,
  email: process.env.ODLA_USER_EMAIL!, // identity hint, not a password or token
  onCode: ({ userCode, verificationUriComplete }) => console.log(verificationUriComplete ?? userCode),
});

// 2. Provision: create app, mint an app key, push the agent schema, and store the
//    user's BYO provider keys as odla-db secrets (encrypted at rest).
const { appId, appKey } = await provisionAgentApp({
  endpoint: ODLA_PLATFORM,
  token,                                   // operator/dev token — needed to WRITE secrets
  providerKeys: { openai: OPENAI_KEY, anthropic: ANTHROPIC_KEY },
});

// 3. Runtime: read keys from secrets with the app key, persist memory + run traces.
const db = odlaInit({ appId, adminToken: appKey, endpoint: ODLA_DB_URL });
const ai = init({ resolveKey: odlaDbKeyResolver(db) });   // keys come from odla-db secrets

const persona = { name: "assistant", model: "gpt-5", memory: new OdlaDbMemory({ db, sessionId: "user-42" }) };
const run = await runAgent(ai, persona, { input: "Pick up where we left off." });
await persistRun(run, { db, sessionId: "user-42" });

BYO keys as secrets — the write/read split mirrors odla-db: storing a secret is a privileged action (operator or odla_dev_ token, via provisionAgentApp / putSecret); reading it at request time uses the ordinary read-only app key (odlaDbKeyResolver → db.secrets.get). Keys are AES-GCM-encrypted at rest and never returned to browser clients. OdlaDbMemory persists conversation turns and persistRun writes run traces, both as natural-key upserts (idempotent on retry).

Platform apps (odla.ai)

For apps registered on odla, hosted AI is the normal path. Admins choose a priced model allowlist and guardrails; each app chooses one of those models. The app uses its existing backend key, while provider credentials stay in the platform vault. Legacy and explicitly selected BYOK configs remain supported.

import { initFromPlatform } from "@odla-ai/ai";
import { init as odlaInit } from "@odla-ai/db";

const db = odlaInit({ appId: TENANT, adminToken: env.ODLA_API_KEY, endpoint: "https://db.odla.ai" });
const { ai, provider, model, revision } = await initFromPlatform({
  platform: "https://odla.ai",
  appId: APP_ID,          // the registry app id, not the tenant
  env: "prod",
  appKey: env.ODLA_API_KEY,
  db,                     // needed only if this app is configured for BYOK
});

The anonymous public-config projection carries the mode, attested config and policy revisions, default model, and only the approved model catalog. It is cached for about 60 seconds, so model changes need no redeploy. Hosted calls go to the Registry broker, which reserves quota before provider I/O, applies server-side output/deadline bounds, prices normalized token usage, and records a 90-day metadata ledger by app/environment/model/status. Prompts are not stored in that ledger. Non-streaming idempotent retries use a separate response receipt bounded to 512 KiB; streaming responses are not replayed.

Hosted inference sends the request's prompts, messages, tool definitions/results, and attachments to the selected third-party model provider. Do not send data that the app is not authorized to disclose to that provider. providerExtras is rejected in hosted mode so app code cannot route around platform policy.

BYOK still reads the configured provider key from db.secrets and filters the facade to that provider. Never put provider keys in a Worker var.

0.3 migration notes

• Tool/extraction JSON Schemas now fail closed. extract() throws ToolInputError; runAgent() records an error tool_result and does not run a local handler when its arguments do not validate.
• Platform calls cannot cross the provider selected in Studio.
• CancelledError (cancelled) and DeadlineExceededError (deadline_exceeded) distinguish caller limits from provider failures.
• Invalid run budgets now throw ConfigError; budget stops persist paired tool results and may therefore add one final user/tool-result turn to memory.

Models & capabilities

The catalog maps a canonical model id → provider, native id, and capabilities. The built-in DEFAULT_CATALOG covers current Anthropic, OpenAI, and Google models; extend or override it via buildCatalog / init({ catalog }).

| | image in | audio in | tools | thinking | effort | web search | structured | |---|---|---|---|---|---|---|---| | Claude (opus/sonnet/fable) | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | | Claude Haiku | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ✅ | | GPT-5 line | ✅ | ❌ | ✅ | — | ✅ | — | ✅ | | GPT-4o | ✅ | ❌ | ✅ | — | ❌ | — | ✅ | | gpt-audio | ❌ | ✅ | ✅ | — | ❌ | — | ✅ | | Gemini 2.5 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

Model ids drift. The catalog is data — verify ids against each provider's current docs (the claude-api skill for Anthropic; /v1/models for OpenAI; Google GenAI docs for Gemini) rather than trusting them from memory.

Pricing

Every catalog entry carries USD-per-MTok pricing, so a call's cost is a lookup plus arithmetic:

import { priceCall, modelPricing } from "@odla-ai/ai/pricing";

const usd = priceCall("claude-opus-4-8", res.usage); // number | undefined

priceCall returns undefined — never 0 — for a model with no known pricing, so missing catalog data can't masquerade as a free call. Cache-read and cache-write tokens are billed at their own rates when the model declares them, falling back to the input rate otherwise.

Import from @odla-ai/ai/pricing rather than the package root when you only need to price calls: the subpath pulls in the catalog and nothing else — no client, no adapters, and none of the three provider SDKs. That keeps cost accounting (e.g. @odla-ai/o11y) small enough to sit in a Worker bundle, and means there is one price list rather than a second one that drifts.

Errors

Every error is an OdlaAIError subclass carrying a stable code — branch on err.code, not message text: ConfigError, AuthError, RateLimitError, CapabilityError, ContextWindowError, InvalidRequestError, ToolInputError, CancelledError, DeadlineExceededError, ProviderError.

OdlaAIError.usage is present only when a provider response was already received and a later forced-tool or schema check failed. It is a frozen, sanitized token-usage record so accounting can charge a failed structured response accurately; it may be absent for transport, authentication, timeout, or preflight failures. toShape() and streaming OracleErrorShape deliberately omit it.

Develop

npm run typecheck   # tsc --noEmit (strict)
npm test            # vitest — offline, mock-provider based
npm run build       # tsup → dual ESM + CJS + d.ts
npm run smoke       # live check against real providers (needs .dev.vars)

For the live smoke, copy .dev.vars.example → .dev.vars and fill in whatever keys you have; providers without a key are skipped.