@feltdb/webllm

Run a small language model entirely in the browser and use it as a FeltDB AI provider. Prompts and inference stay on the user's device; no frontier-model account or hosted inference API is required.

Install

npm install @feltdb/webllm

The package requires a browser with WebGPU. On first use, WebLLM downloads the selected model's artifacts and caches them in the browser.

Generate locally

import { WebLLMProvider } from '@feltdb/webllm';

const llm = new WebLLMProvider({
  onProgress: ({ progress, text }) => {
    console.log(`${Math.round(progress * 100)}%`, text);
  },
});

const answer = await llm.generate([
  { role: 'system', content: 'Return concise JSON.' },
  { role: 'user', content: 'Design a FeltDB task tracker.' },
], {
  responseFormat: { type: 'json_object' },
});

Initialization is lazy: the model loads on the first call to generate() or stream(). Call initialize() earlier to show loading progress sooner.

Choose a model

The default is SmolLM2-360M-Instruct-q4f32_1-MLC, a deliberately small model that needs roughly 580 MB of VRAM. Model selection is configurable:

import { RECOMMENDED_MODELS, WebLLMProvider } from '@feltdb/webllm';

const llm = new WebLLMProvider({
  model: RECOMMENDED_MODELS.balanced,
});

• smallest: the default 360M model
• balanced: SmolLM2 1.7B for stronger generation on capable devices
• lowMemoryWithShaderF16: a smaller 360M quantization for devices supporting the WebGPU shader-f16 feature

Pass any model ID from WebLLM's prebuilt catalog, or call getAvailableModels() to inspect the installed catalog.

FeltDB application generation

WebLLMProvider implements FeltDB's message-generation provider contract:

import { ApplicationFactory } from '@feltdb/ai';
import { WebLLMProvider } from '@feltdb/webllm';

const factory = new ApplicationFactory({
  provider: new WebLLMProvider(),
});

const application = await factory.generateApplication(
  'A collaborative, offline-first field inspection app',
);

@feltdb/ai is currently an experimental repository workspace. This package is independently usable through its generate() and stream() APIs.

Workers and lifecycle

Inference runs in the bundled Web Worker by default so execution does not block the UI. Set useWorker: false for main-thread execution, or pass an application-owned worker for custom hosting or CSP behavior.

Call interrupt() to stop generation and shutdown() to unload the model and terminate a package-owned worker.

Privacy and deployment

The model executes locally. Hosting and analytics code can still transmit data, and model files must be fetched before they are cached, so review your app's network and content-security policies. This package does not send prompts to FeltDB or an inference provider.

MIT licensed.