smoltalk-webllm

WebGPU-accelerated browser provider for smoltalk. Runs LLMs locally in the user's browser via @mlc-ai/web-llm.

Install

pnpm add smoltalk smoltalk-webllm

Usage

Register the provider before your first call, then use smoltalk normally:

import { registerProvider, text, userMessage } from "smoltalk";
import { WebLLMClient, loadModel } from "smoltalk-webllm";

registerProvider("webllm", WebLLMClient);

await loadModel("Llama-3.2-3B-Instruct-q4f32_1-MLC", {
  onProgress: (p) => console.log(p.text, p.loaded, p.total),
});

const result = await text({
  model: "Llama-3.2-3B-Instruct-q4f32_1-MLC",
  provider: "webllm",
  messages: [userMessage("Hello")],
});

Run the example

A single-file copy/paste-and-go example lives at examples/index.html. It loads smoltalk and smoltalk-webllm from a CDN, downloads Llama-3.2-1B-Instruct-q4f32_1-MLC (~700MB, cached after first run), and streams a chat completion to the page.

To try it locally:

# from the smoltalk repo root
npx serve packages/smoltalk-webllm/examples
# then open http://localhost:3000 in a WebGPU-capable browser

Available models

The model id must match a record in WebLLM's prebuiltAppConfig.model_list. To see all available models:

import { listModels } from "smoltalk-webllm";
console.log(await listModels()); // ["Llama-3.2-1B-Instruct-q4f32_1-MLC", ...]

Custom models

Pass a CustomModel object to loadModel to load a model not in the prebuilt list:

import { loadModel } from "smoltalk-webllm";

await loadModel({
  id: "MyLlama-3b",
  modelUrl: "https://huggingface.co/.../resolve/main/",
  modelLibUrl: "https://.../mymodel.wasm",
  contextWindow: 4096,
});

Limitations

• WebGPU required: throws SmolError at loadModel() time if navigator.gpu is undefined.
• AbortSignal: passing a signal to loadModel will reject the returned promise on abort, but the underlying download/compile may continue in the background. The engine will be unloaded if it eventually arrives so GPU memory is released.