inferis-ml

Run AI models in the browser. No server, no per-request cost, no data leaving the device.

Live Demo — try it in your browser.

import { createPool } from 'inferis-ml';
import { transformersAdapter } from 'inferis-ml/adapters/transformers';

const pool = await createPool({ adapter: transformersAdapter() });
const model = await pool.load<number[][]>('feature-extraction', {
  model: 'mixedbread-ai/mxbai-embed-xsmall-v1',
});

const embeddings = await model.run(['Hello world', 'Another sentence']);

Why

Existing browser runtimes (transformers.js, web-llm, onnxruntime-web) give you inference but leave everything else to you — worker management, postMessage boilerplate, model lifecycle, memory budgets, cross-tab dedup, WebGPU fallback, streaming.

inferis-ml handles all of it. You get a clean async API and focus on the product.

| Problem | Without inferis-ml | With inferis-ml | |---------|-------------------|-----------------| | UI freezes during inference | Main thread blocked | Runs in Web Workers | | 5 tabs = 5 model copies | 10 GB RAM, browser crashes | crossTab: true — one shared copy | | WebGPU not everywhere | Manual detection + swap | defaultDevice: 'auto' |

Install

npm install inferis-ml

# Pick your adapter (peer deps):
npm install @huggingface/transformers   # transformersAdapter
npm install @mlc-ai/web-llm             # webLlmAdapter
npm install onnxruntime-web             # onnxAdapter

Quick Start

LLM Streaming

import { createPool } from 'inferis-ml';
import { webLlmAdapter } from 'inferis-ml/adapters/web-llm';

const pool = await createPool({
  adapter: webLlmAdapter(),
  defaultDevice: 'webgpu',
  maxWorkers: 1,
});

const llm = await pool.load<string>('text-generation', {
  model: 'Llama-3.2-3B-Instruct-q4f32_1-MLC',
  onProgress: ({ phase }) => console.log(phase),
});

const stream = llm.stream({
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: 'Explain WebGPU in 3 sentences.' },
  ],
});

for await (const token of stream) {
  output.textContent += token;
}

Speech Transcription

const transcriber = await pool.load<{ text: string }>('automatic-speech-recognition', {
  model: 'openai/whisper-base',
  estimatedMemoryMB: 80,
});

const result = await transcriber.run(audioData);
console.log(result.text);

Abort Inference

const ctrl = new AbortController();
stopButton.onclick = () => ctrl.abort();

try {
  for await (const token of llm.stream(input, { signal: ctrl.signal })) {
    output.textContent += token;
  }
} catch (e) {
  if (e.name === 'AbortError') output.textContent += ' [stopped]';
}

Cross-Tab Deduplication

const pool = await createPool({
  adapter: transformersAdapter(),
  crossTab: true, // SharedWorker > leader election > per-tab fallback
});

Model State Changes

model.onStateChange((state) => {
  if (state === 'loading')  showSpinner();
  if (state === 'ready')    hideSpinner();
  if (state === 'error')    showError('Failed to load model');
  if (state === 'disposed') disableUI();
});

Features

• Runtime-agnostic — adapters for @huggingface/transformers, @mlc-ai/web-llm, onnxruntime-web, or your own
• Zero framework deps — works with React, Vue, Svelte, or vanilla JS
• WebGPU -> WASM fallback — auto-detected or configured explicitly
• Streaming — ReadableStream + for await for token-by-token output
• Memory budget — LRU eviction when models exceed the configured cap
• Cross-tab dedup — SharedWorker (tier 1), leader election (tier 2), per-tab (tier 3)
• AbortController — cancel any in-flight inference
• TypeScript — full type safety, generic output types

API Reference

createPool(config)

const pool = await createPool({
  adapter: transformersAdapter(),   // required
  workerUrl: new URL('inferis-ml/worker', import.meta.url),
  maxWorkers: navigator.hardwareConcurrency - 1,
  maxMemoryMB: 2048,
  defaultDevice: 'auto',           // 'webgpu' | 'wasm' | 'auto'
  crossTab: false,
  taskTimeout: 120_000,
});

pool.load<TOutput>(task, config)

Loads a model and returns a ModelHandle. If already loaded, returns the existing handle.

const model = await pool.load<number[][]>('feature-extraction', {
  model: 'mixedbread-ai/mxbai-embed-xsmall-v1',
  estimatedMemoryMB: 30,
  onProgress: (p) => { ... },
});

ModelHandle<TOutput>

| Method | Description | |--------|-------------| | run(input, options?) | Non-streaming inference. Returns Promise<TOutput>. | | stream(input, options?) | Streaming inference. Returns ReadableStream<TOutput>. | | dispose() | Unload model and free memory. | | onStateChange(cb) | Subscribe to state changes. Returns unsubscribe function. | | id | Unique model ID (task:model). | | state | Current state: idle \| loading \| ready \| inferring \| unloading \| error \| disposed. | | memoryMB | Approximate memory usage. | | device | Resolved device: webgpu or wasm. |

InferenceOptions

interface InferenceOptions {
  signal?: AbortSignal;
  priority?: 'high' | 'normal' | 'low';
}

detectCapabilities()

import { detectCapabilities } from 'inferis-ml';

const caps = await detectCapabilities();
if (caps.webgpu.supported) {
  console.log('GPU vendor:', caps.webgpu.adapter?.vendor);
} else {
  console.log('WASM SIMD:', caps.wasm.simd);
}

Custom Adapter

import type { ModelAdapter, ModelAdapterFactory } from 'inferis-ml';

export function myCustomAdapter(): ModelAdapterFactory {
  return {
    name: 'my-adapter',

    async create(): Promise<ModelAdapter> {
      const { MyRuntime } = await import('my-runtime');

      return {
        name: 'my-adapter',

        estimateMemoryMB(_task, config) {
          return (config.estimatedMemoryMB as number) ?? 50;
        },

        async load(task, config, device, onProgress) {
          onProgress({ phase: 'loading', loaded: 0, total: 1 });
          const instance = await MyRuntime.load(config.model as string, { device });
          onProgress({ phase: 'done', loaded: 1, total: 1 });
          return { instance, memoryMB: 50 };
        },

        async run(model, input) {
          return (model.instance as MyRuntime).infer(input);
        },

        async stream(model, input, onChunk) {
          for await (const chunk of (model.instance as MyRuntime).stream(input)) {
            onChunk(chunk);
          }
        },

        async unload(model) {
          await (model.instance as MyRuntime).dispose();
        },
      };
    },
  };
}

Framework Integrations

Official bindings with idiomatic APIs for popular frameworks:

| Package | Install | Docs | |---------|---------|------| | inferis-react | npm i inferis-react | README | | inferis-vue | npm i inferis-vue | README | | inferis-svelte | npm i inferis-svelte | README |

Each package provides context/provider setup, model lifecycle management, streaming, capability detection, and memory monitoring -- all wired into the framework's reactivity system.

// React
const { text, start } = useStream(model);

// Vue
const { text, start } = useStream(model);

// Svelte
const { text, start } = useStream(model);  // $text in template

Bundler & Framework Setup

inferis-ml is browser-only. In SSR frameworks, ensure initialization runs only on the client.

Vite

// vite.config.ts
export default {
  worker: { format: 'es' },
};

webpack 5

// webpack.config.js
module.exports = {
  experiments: { asyncWebAssembly: true },
};

Next.js

'use client';

import { useEffect, useState } from 'react';
import type { WorkerPoolInterface } from 'inferis-ml';

export default function AI() {
  const [pool, setPool] = useState<WorkerPoolInterface | null>(null);

  useEffect(() => {
    import('inferis-ml').then(({ createPool }) =>
      createPool({ adapter: { type: 'transformers' } })
    ).then(setPool);
  }, []);

  if (!pool) return <p>Loading...</p>;
  // use pool
}

Nuxt

<template>
  <ClientOnly>
    <InferenceComponent />
  </ClientOnly>
</template>

// composables/useInferis.ts
export async function useInferis() {
  const { createPool } = await import('inferis-ml');
  return createPool({ adapter: { type: 'transformers' } });
}

SvelteKit

import { browser } from '$app/environment';

let pool;
if (browser) {
  const { createPool } = await import('inferis-ml');
  pool = await createPool({ adapter: { type: 'transformers' } });
}

Popular Models

Models download from Hugging Face Hub on first use and are cached in the browser's Cache API. Subsequent loads are instant and work offline.

Embeddings / Semantic Search

| Model | Size | Notes | |-------|------|-------| | mixedbread-ai/mxbai-embed-xsmall-v1 | 23 MB | Best quality/size for English | | Xenova/all-MiniLM-L6-v2 | 23 MB | Popular multilingual | | Xenova/multilingual-e5-small | 118 MB | 100+ languages |

Text Generation (LLM)

Requires @mlc-ai/web-llm + defaultDevice: 'webgpu'.

| Model | Size | Notes | |-------|------|-------| | Llama-3.2-1B-Instruct-q4f32_1-MLC | 0.8 GB | Fastest | | Llama-3.2-3B-Instruct-q4f32_1-MLC | 2 GB | Good balance | | Phi-3.5-mini-instruct-q4f16_1-MLC | 2.2 GB | Strong reasoning | | gemma-2-2b-it-q4f16_1-MLC | 1.5 GB | Fast on mobile GPU |

Speech Recognition

| Model | Size | Notes | |-------|------|-------| | openai/whisper-tiny | 39 MB | Fastest | | openai/whisper-base | 74 MB | Good balance | | openai/whisper-small | 244 MB | Better accuracy |

Text Classification

| Model | Size | Notes | |-------|------|-------| | Xenova/distilbert-base-uncased-finetuned-sst-2-english | 67 MB | Sentiment | | Xenova/toxic-bert | 438 MB | Toxicity detection |

Translation

| Model | Size | Notes | |-------|------|-------| | Xenova/opus-mt-en-ru | 74 MB | EN -> RU | | Xenova/opus-mt-ru-en | 74 MB | RU -> EN | | Xenova/nllb-200-distilled-600M | 600 MB | 200 languages |

Image Classification

| Model | Size | Notes | |-------|------|-------| | Xenova/efficientnet-lite4 | 13 MB | Fastest, 1000 classes | | Xenova/mobilevit-small | 22 MB | Mobile-friendly |

Model Sources

Models are not locked to Hugging Face. Each adapter has its own sources:

• transformers.js — HF Hub ID or any direct URL
• web-llm — MLC registry, or register custom models
• onnxruntime-web — direct URL to .onnx file
• Custom adapter — load from anywhere (fetch, IndexedDB, bundled)

Caching

First visit:  download -> Cache API -> run  (5-60s)
Next visits:  Cache API -> run              (1-3s, no network)
Offline:      Cache API -> run              (works without internet)

Browser Support

| Feature | Chrome | Firefox | Safari | Edge | |---------|--------|---------|--------|------| | Core (Worker + WASM) | 57+ | 52+ | 11+ | 16+ | | WebGPU | 113+ | 141+ | 26+ | 113+ | | WASM SIMD | 91+ | 89+ | 16.4+ | 91+ | | SharedWorker | 4+ | 29+ | 16+ | 79+ | | Leader Election | 69+ | 96+ | 15.4+ | 79+ |

Minimum: Web Workers + WebAssembly (97%+ of browsers). All advanced features are progressive enhancements.

Performance Tips

• maxWorkers: 1 for GPU-bound workloads (LLMs)
• defaultDevice: 'webgpu' when targeting modern hardware
• estimatedMemoryMB for accurate LRU eviction
• crossTab: true for multi-tab apps (chat, editors)
• Reuse ModelHandle — re-loading a ready model is a no-op

When To Use

| Use case | Fit? | |----------|------| | Semantic search, chatbot, speech, classification, translation | Yes | | Private data (never leaves device) | Yes | | Offline after first load | Yes | | Server-side batch processing | No | | Models > 4 GB | No |

License

MIT