@localmode/webllm

WebLLM provider for local-first LLM inference. Uses 4-bit quantized models for efficient browser-based text generation.

Installation

pnpm install @localmode/webllm @localmode/core

Quick Start

import { generateText, streamText } from '@localmode/core';
import { webllm } from '@localmode/webllm';

// Generate text
const { text, usage } = await generateText({
  model: webllm.languageModel('Llama-3.2-1B-Instruct-q4f16_1-MLC'),
  prompt: 'Explain quantum computing in simple terms.',
});

console.log(text);
console.log(`Generated in ${usage.durationMs}ms`);

Streaming

import { streamText } from '@localmode/core';
import { webllm } from '@localmode/webllm';

const result = await streamText({
  model: webllm.languageModel('Llama-3.2-1B-Instruct-q4f16_1-MLC'),
  prompt: 'Write a haiku about programming.',
});

for await (const chunk of result.stream) {
  process.stdout.write(chunk.text);
}

Model Preloading

import { preloadModel, isModelCached, deleteModelCache } from '@localmode/webllm';

// Check if model is already cached
if (!(await isModelCached('Llama-3.2-1B-Instruct-q4f16_1-MLC'))) {
  // Preload with progress
  await preloadModel('Llama-3.2-1B-Instruct-q4f16_1-MLC', {
    onProgress: (p) => console.log(`Loading: ${p.progress?.toFixed(1)}%`),
  });
}

// Delete cached model
await deleteModelCache('Llama-3.2-1B-Instruct-q4f16_1-MLC');

Available Models

Tiny (< 500MB)

| Model | Size | Context | Description | | ----- | ---- | ------- | ----------- | | SmolLM2-135M-Instruct-q0f16-MLC | 78MB | 2K | Tiniest model, instant loading | | SmolLM2-360M-Instruct-q4f16_1-MLC | 210MB | 2K | Very small, surprisingly capable | | Qwen2.5-0.5B-Instruct-q4f16_1-MLC | 278MB | 4K | Tiny Qwen, great quality for size | | Qwen3-0.6B-q4f16_1-MLC | 350MB | 4K | Latest tiny model | | TinyLlama-1.1B-Chat-v1.0-q4f16_1-MLC | 400MB | 2K | Fast and capable chat |

Small (500MB – 1GB)

| Model | Size | Context | Description | | ----- | ---- | ------- | ----------- | | Llama-3.2-1B-Instruct-q4f16_1-MLC | 712MB | 4K | Great for simple tasks | | Qwen2.5-1.5B-Instruct-q4f16_1-MLC | 868MB | 4K | Multilingual | | Qwen2.5-Coder-1.5B-Instruct-q4f16_1-MLC | 868MB | 4K | Code-specialized |

Medium (1 – 2GB)

| Model | Size | Context | Description | | ----- | ---- | ------- | ----------- | | Qwen3-1.7B-q4f16_1-MLC | 1.1GB | 4K | Latest multilingual | | SmolLM2-1.7B-Instruct-q4f16_1-MLC | 1GB | 2K | Best small model (requires shader-f16) | | gemma-2-2b-it-q4f16_1-MLC | 1.44GB | 2K | Google Gemma 2 (requires shader-f16) | | Qwen2.5-3B-Instruct-q4f16_1-MLC | 1.7GB | 4K | High quality | | Qwen2.5-Coder-3B-Instruct-q4f16_1-MLC | 1.7GB | 4K | Mid-range code model | | Llama-3.2-3B-Instruct-q4f16_1-MLC | 1.76GB | 4K | Excellent quality | | Hermes-3-Llama-3.2-3B-q4f16_1-MLC | 1.76GB | 4K | Enhanced chat | | Ministral-3-3B-Instruct-2512-BF16-q4f16_1-MLC | 1.8GB | 4K | Latest Mistral 3B architecture | | Ministral-3-3B-Reasoning-2512-q4f16_1-MLC | 1.8GB | 4K | Reasoning-tuned 3B |

Large (> 2GB)

| Model | Size | Context | Description | | ----- | ---- | ------- | ----------- | | Phi-3.5-mini-instruct-q4f16_1-MLC | 2.1GB | 4K | Excellent reasoning | | Phi-3-mini-4k-instruct-q4f16_1-MLC | 2.2GB | 4K | Reasoning and coding | | Phi-3.5-vision-instruct-q4f16_1-MLC | 2.4GB | 1K | Vision — multimodal (text + images) | | Qwen3-4B-q4f16_1-MLC | 2.2GB | 4K | Best quality in medium param range | | Mistral-7B-Instruct-v0.3-q4f16_1-MLC | 4GB | 4K | Strong general-purpose | | Qwen2.5-7B-Instruct-q4f16_1-MLC | 4GB | 4K | Excellent multilingual | | Qwen2.5-Coder-7B-Instruct-q4f16_1-MLC | 4GB | 4K | Best-in-class code model | | DeepSeek-R1-Distill-Qwen-7B-q4f16_1-MLC | 4.18GB | 4K | Advanced reasoning | | DeepSeek-R1-Distill-Llama-8B-q4f16_1-MLC | 4.41GB | 4K | Best reasoning | | Llama-3.1-8B-Instruct-q4f16_1-MLC | 4.5GB | 4K | Strong general-purpose | | Qwen3-8B-q4f16_1-MLC | 4.5GB | 4K | Highest quality multilingual | | Hermes-3-Llama-3.1-8B-q4f16_1-MLC | 4.9GB | 4K | Hermes 3 8B, DPO-optimized chat | | gemma-2-9b-it-q4f16_1-MLC | 5GB | 1K | Google Gemma 2 9B, highest quality | | Qwen3.5-4B-q4f16_1-MLC | 2.39GB | 32K | Qwen 3.5 4B, latest high-quality multilingual generation | | Qwen3.5-9B-q4f16_1-MLC | 5.06GB | 32K | Qwen 3.5 9B, highest-quality preset. Requires capable GPU and 8GB+ RAM |

Vision (Image Input)

Phi 3.5 Vision supports multimodal input — send images alongside text:

import { streamText } from '@localmode/core';
import { webllm } from '@localmode/webllm';

const model = webllm.languageModel('Phi-3.5-vision-instruct-q4f16_1-MLC');
console.log(model.supportsVision); // true

const result = await streamText({
  model,
  prompt: '',
  messages: [{
    role: 'user',
    content: [
      { type: 'text', text: 'What is in this image?' },
      { type: 'image', data: base64Data, mimeType: 'image/jpeg' },
    ],
  }],
});

Structured Output (JSON mode)

Forward MLC's OpenAI-compatible response_format via providerOptions.webllm to force schema-conforming JSON through XGrammar-constrained decoding — far more reliable than prompting a small model for JSON:

import { generateObject, jsonSchema } from '@localmode/core';
import { webllm } from '@localmode/webllm';
import { z } from 'zod';

const schema = jsonSchema(z.object({ name: z.string(), age: z.number() }));

const { object } = await generateObject({
  model: webllm.languageModel('Qwen3-1.7B-q4f16_1-MLC'),
  schema,
  prompt: 'Generate a profile for a software engineer named Alex',
  providerOptions: {
    webllm: { response_format: { type: 'json_object', schema: JSON.stringify(schema.jsonSchema) } },
  },
});

Custom Configuration

import { createWebLLM } from '@localmode/webllm';

const myWebLLM = createWebLLM({
  onProgress: (p) => updateLoadingBar(p.progress),
});

const model = myWebLLM.languageModel('Llama-3.2-1B-Instruct-q4f16_1-MLC', {
  systemPrompt: 'You are a helpful coding assistant.',
  temperature: 0.5,
  maxTokens: 1024,
});

Default Settings

| Setting | Default | Description | | ------- | ------- | ----------- | | temperature | 0.7 | Sampling temperature | | topP | 0.95 | Nucleus sampling threshold | | maxTokens | 512 | Maximum tokens to generate | | contextLength | 4096 | Context window size | | useIndexedDBCache | false | Use IndexedDB instead of Cache API for model storage. Useful for Chrome extensions with MV3 restrictions where Cache.add() can fail during multi-gigabyte downloads. | | cacheBackend | 'cache' | Explicit cache backend selection ('cache', 'indexeddb', or 'cross-origin'). Overrides useIndexedDBCache when set. | | appConfig | undefined | Custom WebLLM AppConfig passed to CreateMLCEngine() for advanced model configuration. |

Utilities

import {
  preloadModel,
  isModelCached,
  deleteModelCache,
  getModelSize,
  isWebGPUAvailable,
} from '@localmode/webllm';

// Check WebGPU support
const gpuAvailable = await isWebGPUAvailable();

// Get estimated model size in bytes
const size = getModelSize('Llama-3.2-1B-Instruct-q4f16_1-MLC');

// Delete cached model data
await deleteModelCache('Llama-3.2-1B-Instruct-q4f16_1-MLC');

Requirements

• WebGPU support (Chrome 113+, Edge 113+)
• Sufficient GPU memory for the model
• Some models (SmolLM2-1.7B, Gemma 2 2B) require the shader-f16 WebGPU extension, which is not available on all devices (e.g., Qualcomm/Android). Use q4f32_1 variants as fallbacks for broader compatibility.

Acknowledgments

This package is built on WebLLM by MLC AI — high-performance LLM inference in the browser with WebGPU.

License

MIT