@socketsecurity/odai

odai - pronounced like the trickster; it lives in your machine and does your chores.

odai is local-only - the primary backend is Chrome's built-in AI (the Prompt API, stable since Chrome 148) via installed Google Chrome on every platform; llama-server (loopback) is the local fallback; Apple FM and Phi Silica (Copilot+) are opportunistic per-OS extras. No cloud, no remote endpoints, no keys.

The Prompt API is model-agnostic by design, so odai names its backend for the interface rather than the model. The on-device model is Gemini Nano today; Gemma 4 is the base for the next Gemini Nano and is already testable in Chrome Canary behind the "Gemma 4 for Built-in AI" flag.

@socketsecurity/odai is a local, on-device AI library for browser and Node. It wraps the browser's built-in AI Prompt API behind a type-safe, backend-agnostic seam, hardens small-model JSON output, and ships bench - an evaluation harness that scores any backend on real Socket workloads. It exists so Socket code can feature-detect, prompt, and parse on-device model responses without scattering DOM-specific checks across consumers.

Install

pnpm install @socketsecurity/odai
ODAI_CHROME_MODEL=gemma4 pnpm exec odai setup

odai setup discovers Google Chrome, prepares a dedicated persistent profile, downloads the selected on-device model when needed, verifies the responding model, closes Chrome, and prints a JSON receipt. Run it once after installation and again when the receipt check reports that the model is unavailable.

Applications that manage their own setup can use the Node API. The cleanup callback receives the measured storage deficit and may remove only data the application owns. Odai measures free space again before it starts Chrome.

import { setupChromeBuiltin } from '@socketsecurity/odai/node'

await setupChromeBuiltin({
  model: 'gemma4',
  async reclaimStorage(pressure) {
    await removeApplicationCaches(pressure.minimumBytes)
  },
})

Usage

import { createOdaiModel } from '@socketsecurity/odai'

const model = await createOdaiModel()
const { raw } = await model.promptStreaming(
  'Summarize this page in one sentence.',
)
console.log(raw)

createOdaiModel picks a backend by precedence: the explicit backend option, then the ODAI_BACKEND env var, then the availability probe order - chrome-builtin, llama-server, apple-fm, windows-phi-silica. Select simulator explicitly for testing.

Command suggestions

Node consumers can classify a request against their own action catalog:

import { classifyIntent, withOdaiModel } from '@socketsecurity/odai/node'

const result = await withOdaiModel(
  model =>
    classifyIntent(model, {
      query: 'Inspect this project for dependency risks.',
      candidates: [
        { id: 'inspect-project', description: 'Inspect project dependencies.' },
        {
          id: 'repair-project',
          description: 'Apply dependency security repairs.',
        },
      ],
    }),
  { timeoutMs: 5000 },
)

A successful result contains an allowed actionId or null for abstention. The caller owns argument validation and execution. Invalid output returns a failed task result. Cancellation and backend errors reject the operation.

withOdaiModel requires an already running, cancellable loopback llama-server backend. It performs no model installation or backend preparation. Its shared deadline covers discovery, prompts, and session cleanup. The callback's second argument supplies abortSignal for additional cancellable work. probeBackendAvailability checks the same eligible backend without creating a model session.

CLI

The package ships a odai bin for single-shot, keyless AI steps in scripts and CI. Input arrives on stdin or --input; the parsed result prints as one JSON line on stdout, diagnostics go to stderr.

git diff | odai commit-msg
printf 'Critical: 2\nHigh: 5\n' | odai triage
odai summarize --input README.md
odai patch --input src/greet.js --instruction "use a template literal"
odai classify-deps --input narrowed-dep-diff.json
odai lockstep --input prepared-lockstep.json
odai backends
printf '%s\n' '{"id":"a","task":"summarize","input":"release notes..."}' '{"id":"b","task":"commit-msg","input":"diff --git..."}' | odai batch

Every prompt runs under a hard budget - --timeout <ms> or the ODAI_TIMEOUT_MS env var, default 120000 - so a wedged engine can never hang a job. Exit codes are the CI contract:

| code | meaning | | ---- | --------------------------------------------------------------- | | 0 | success - parsed JSON on stdout | | 1 | model or task failure - invalid reply, timeout, or engine error | | 2 | usage error | | 69 | no backend available - treat as a clean skip in CI |

When nothing is provisioned the CLI prints the exact provisioning steps for each backend and exits 69; odai backends prints per-backend availability JSON with the reason for every unavailable engine.

odai batch reads a JSONL manifest (stdin or --input), runs every task over a single backend launch, and prints one JSON line per entry in manifest order - {"id","ok":true,"value":…} or {"id","ok":false,"error":…}. It exits 0 when the batch ran even if every task failed (failures are in-band lines), 2 on a malformed manifest (checked in full before any task runs), and 69 when no backend is available. --timeout is the per-task budget; --raw is not accepted.

Lockstep

odai lockstep analyzes one lockstep row with full or sparse materialization. It validates cited evidence and proposed patches before returning a result. The fleet runner verifies changes in a temporary copy with trusted commands. See lockstep assistance for preparation, verification, and model evaluation.

Use the conversation API when requests need shared context. It supports transcript restore, streaming, cancellation, and bounded history across native and replay providers.

Serve

odai serve turns any backend into a loopback HTTP server speaking both wire formats llama-server does, on one port: the Anthropic Messages API (POST /v1/messages, /v1/messages/count_tokens) and the OpenAI chat-completions API (POST /v1/chat/completions, /v1/chat/completions/input_tokens, GET /v1/models), plus /health. odai already talks to llama-server as a client, so serving the same routes lets anything pointed at a local OpenAI base URL treat odai as the server it would otherwise run. Tool calling works on both sides and is emulated for plain-text engines: the shim teaches the model a one-line JSON tool-call protocol in the system prompt and reads the reply back into tool_use blocks or tool_calls, with the same JSON repair hardening the tasks use.

odai serve                  # 127.0.0.1:8402, backend from the registry probe
odai serve --port 0         # let the OS pick a free port
odai serve --backend llama-server

Point any Anthropic-speaking client at the printed URL with ANTHROPIC_BASE_URL and any non-empty ANTHROPIC_API_KEY (loopback only, no auth). For example, communique runs its release-notes agent loop against local inference with no code changes:

communique --provider anthropic --model local \
  --base-url http://127.0.0.1:8402

An OpenAI-speaking client points at the same port's /v1 prefix, the way it would at llama-server:

OPENAI_BASE_URL=http://127.0.0.1:8402/v1 OPENAI_API_KEY=anything

Known limitation: the shim ignores max_tokens in the request - odai's session interface has no output-token budget - and never emits a max_tokens stop reason, so a reply cut short by the engine itself (for example llama-server's own n_predict default) arrives as end_turn.

Bench

Run the bench evaluation harness against the built-in simulator, or score a real backend with --backend:

pnpm run bench
pnpm run bench --backend=chrome-builtin
pnpm run bench --scenario=lockstep --backend=chrome-builtin --json

The default simulator checks the evaluation harness. Use an explicit backend to measure a real model. The lockstep cases check response contracts. Upstream conformance requires the separate fleet verifier.

See the performance practices for package measurements and retained-context experiments. The performance journal records measured results, rejected changes, and model limitations. The automation practices define deterministic setup, validation, and model fallback boundaries.

Development

pnpm install
pnpm run check
pnpm run test

License

MIT