@octomil/browser

Run ML models in the browser. WebGPU-accelerated, WASM fallback, zero server required.

What is this?

A TypeScript SDK for running ONNX models directly in the browser with WebGPU or WASM. The model downloads once, caches locally, and runs on the user's device. It is a good fit for classification, embeddings, image recognition, and other browser-safe ONNX workloads where low latency and offline support matter.

Install

pnpm add @octomil/browser
# or
npm install @octomil/browser

Authentication

OctomilClient requires an auth field. Two auth modes are supported:

Organization API key

Use this when your app has a backend that can provision org-scoped credentials.

auth: {
  type: 'org_api_key',
  apiKey: 'your-api-key',
  orgId: 'your-org-id',
  serverUrl: 'https://api.octomil.com', // optional, defaults to production
}

Device token

Use this for device-style bootstrap and registration flows.

auth: {
  type: 'device_token',
  deviceId: 'device-uuid',
  bootstrapToken: 'bootstrap-token',
  serverUrl: 'https://api.octomil.com', // optional
}

Quick Start (Hosted/Cloud)

The unified facade sends requests to the Octomil cloud API. Use a publishable key (client-safe, oct_pub_live_ prefix) -- never embed server API keys in browser code.

import { Octomil } from "@octomil/browser";

// Use a publishable key -- safe for client-side code
const client = new Octomil({ publishableKey: "oct_pub_live_..." });
await client.initialize();
const response = await client.responses.create({
  model: "phi-4-mini",
  input: "Hello",
});
console.log(response.outputText);

Embeddings (Hosted/Cloud)

const result = await client.embeddings.create({
  model: "nomic-embed-text-v1.5",
  input: "On-device AI inference at scale",
});
console.log(result.embeddings[0].slice(0, 5));

Array input is also supported:

const result = await client.embeddings.create({
  model: "nomic-embed-text-v1.5",
  input: ["first document", "second document"],
});

Migrating from OctomilClient

OctomilClient and the low-level ResponsesClient APIs still work exactly as before. The Octomil facade is a convenience wrapper for the cloud-backed Responses path — it delegates to ResponsesClient internally. For local ONNX inference, continue using OctomilClient directly.

Advanced Usage (OctomilClient)

import { OctomilClient } from '@octomil/browser';

const ml = new OctomilClient({
  model: 'https://models.octomil.com/sentiment-v1.onnx',
  auth: {
    type: 'org_api_key',
    apiKey: 'your-api-key',
    orgId: 'your-org-id',
    serverUrl: 'https://api.octomil.com',
  },
});

await ml.load();
const result = await ml.predict({ text: 'This product is incredible' });
console.log(result.label, result.score); // "1" 0.97
ml.close();

The SDK downloads the model, caches it via the Cache API, and picks the fastest backend available (WebGPU first, WASM fallback).

Features

Multiple input types

// Text
await ml.predict({ text: 'classify this' });

// Image (canvas, img element, or raw ImageData)
await ml.predict({ image: document.querySelector('canvas') });

// Raw tensors
await ml.predict({ raw: new Float32Array(784), dims: [1, 1, 28, 28] });

Automatic model caching

Models cache locally after the first download. Later loads can skip the network entirely.

const ml = new OctomilClient({
  model: 'https://models.octomil.com/sentiment-v1.onnx',
  auth: {
    type: 'org_api_key',
    apiKey: 'your-api-key',
    orgId: 'your-org-id',
  },
  cacheStrategy: 'cache-api', // default; also 'indexeddb' or 'none'
});
await ml.load();       // downloads once
await ml.isCached();   // true on next visit

Routing policies

The SDK exports the same routing policy names as the Python and Node SDKs for API parity. Use validateRoutingPolicy() to check policy names and assertBrowserCompatiblePolicy() to reject policies that require local execution.

import {
  validateRoutingPolicy,
  assertBrowserCompatiblePolicy,
  VALID_ROUTING_POLICIES,
} from '@octomil/browser';

// Validate a policy name (throws on unknown names like "quality_first")
const policy = validateRoutingPolicy("cloud_first"); // ok

// Assert browser compatibility (throws on "private" and "local_only")
assertBrowserCompatiblePolicy("cloud_only");    // ok
assertBrowserCompatiblePolicy("private");       // throws POLICY_DENIED

Supported policies: cloud_only, cloud_first, local_first, performance_first. The private and local_only policies require local on-device execution and are rejected with a clear error in the browser SDK.

Browser SDK limitations

The browser SDK is hosted/cloud only for the runtime planner. It differs from the Python and native SDKs in these ways:

• No local model artifact download or caching via the planner (ONNX-web models use a separate OctomilClient path)
• No local inference engine detection or benchmarking
• No offline plan resolution
• private and local_only routing policies are rejected -- these require capabilities the browser cannot provide
• The planner types (RouteMetadata, RuntimeSelection, etc.) are exported for type parity but the browser SDK does not implement a planner client that calls POST /api/v2/runtime/plan

Smart routing (device vs. cloud) (Hosted/Cloud)

The SDK can choose between local and cloud inference based on model size and device capabilities, then fall back cleanly when conditions change.

const ml = new OctomilClient({
  model: 'phi-4-mini',
  auth: {
    type: 'org_api_key',
    apiKey: 'oct_pub_live_...', // publishable key, safe for browser
    orgId: 'your-org-id',
    serverUrl: 'https://api.octomil.com',
  },
  routing: { prefer: 'fastest' }, // 'device' | 'cloud' | 'cheapest' | 'fastest'
});

Streaming and embeddings (Hosted/Cloud)

// Stream tokens via SSE
for await (const token of ml.predictStream('phi-4-mini', 'Explain quantum computing')) {
  process.stdout.write(token.token);
}

// Generate embeddings
const { embeddings } = await ml.embed('nomic-embed-text', ['query', 'document']);

Batch inference

const results = await ml.predictBatch([
  { text: 'great product' },
  { text: 'terrible experience' },
  { text: 'it was okay' },
]);

Federated learning with differential privacy

On-device training with built-in gradient clipping, noise injection, and secure aggregation. Raw user data never leaves the browser.

import { clipGradients, addGaussianNoise } from '@octomil/browser';
const noised = addGaussianNoise(clipGradients(delta, 1.0), 0.01);

Browser Support

| Browser | Backend | Notes | |---------|---------|-------| | Chrome 113+ | WebGPU | Full GPU acceleration | | Edge 113+ | WebGPU | Full GPU acceleration | | Firefox | WASM | WebGPU behind flag | | Safari 18+ | WASM | WebGPU partial support | | All modern browsers | WASM | Universal fallback via WASM SIMD |

The SDK auto-detects the best backend. Pass backend: 'webgpu' or backend: 'wasm' to override.

Limitations: Works well for models up to ~500MB. Large LLMs will hit browser memory limits. WebGPU performance varies by GPU. WASM is slower but universal.

Control plane (optional)

The Browser SDK has optional control plane integration for device registration, desired-state sync, and heartbeats. This is separate from the main inference path.

import { configure } from '@octomil/browser';

// Silent device registration (background, exponential backoff)
configure({
  auth: { type: 'publishable_key', key: 'oct_pub_live_...' },
  monitoring: { enabled: true },
});

Or use the control namespace directly:

await ml.control.register();
ml.control.startHeartbeat(300_000); // 5 min interval
const state = await ml.control.fetchDesiredState();
await ml.control.sync({ modelInventory: [...] });

SyncManager is a standalone opt-in export for periodic desired-state reconciliation (not automatic):

import { SyncManager } from '@octomil/browser';
const sync = new SyncManager({ intervalMs: 300_000, onEvent: (e) => console.log(e) });

Key difference from iOS/Android: The Browser SDK has no AppManifest. It is model-URL-driven — you pass a direct .onnx URL to the constructor. predict() is local ONNX inference only; predictStream() and embed() hit cloud endpoints.

Architecture

OctomilClient → ModelManager (download/cache) → InferenceEngine (ONNX Runtime Web)
             → RoutingClient (device vs. cloud) → StreamingEngine (SSE tokens)
             → ControlClient (register, heartbeat, sync) — optional
             → TelemetryReporter (opt-in metrics)
configure()  → SilentAuth (background registration) + HeartbeatManager
SyncManager  → Periodic desired-state reconciliation (standalone, opt-in)

Script Tag (no bundler)

<script src="https://unpkg.com/@octomil/browser/dist/octomil.min.js"></script>
<script>
  const ml = new OctomilClient({
    model: 'model.onnx',
    auth: {
      type: 'org_api_key',
      apiKey: 'your-api-key',
      orgId: 'your-org-id',
    },
  });
  ml.load().then(() => ml.predict({ text: 'hello' })).then(console.log);
</script>

API Reference

docs.octomil.com

Contributing

git clone https://github.com/octomil/octomil-browser.git && cd octomil-browser
pnpm install && pnpm test && pnpm run build

Releasing

Releases publish to npm from GitHub Releases via trusted publishing with npm provenance. Configure npm trusted publishing for @octomil/browser with repository octomil/octomil-browser and workflow .github/workflows/publish.yml.

pnpm install --frozen-lockfile
pnpm run typecheck
pnpm test
pnpm run build
pnpm run exports:check
pnpm run pack:check

Then create a GitHub Release for the package version in package.json. The publish workflow runs the same gates and publishes @octomil/browser with public scoped-package access and provenance.

License

MIT