LangChain WebLLM

Run WebLLM models locally with WebGPU through LangChain, LangGraph, and Deep Agents.

npm install langchain-web-llm

• Yarn: also install @mlc-ai/web-llm and @langchain/core. Check compatible versions with yarn explain peer-requirements.
• CommonJS: use require('langchain-web-llm') on Node.js 22.12 or later. With Yarn PnP on Node 22.12–22.14 or 23.0–23.4, use import().
• TypeScript: moduleResolution: "bundler" retains more WebLLM option types than NodeNext. WebLLM's declarations may require skipLibCheck and leave some types unchecked.

Use

Create a worker that runs WebLLM:

// worker.ts
import { WebWorkerMLCEngineHandler } from '@mlc-ai/web-llm'

const handler = new WebWorkerMLCEngineHandler()
self.onmessage = (message: MessageEvent) => handler.onmessage(message)

Load Qwen3.5 with WebLLM 0.2.86, then pass its engine to the provider:

import { CreateWebWorkerMLCEngine } from '@mlc-ai/web-llm'
import { ChatWebLLM } from 'langchain-web-llm'

const modelId = 'Qwen3.5-4B-q4f16_1-MLC'
const worker = new Worker(new URL('./worker.ts', import.meta.url), { type: 'module' })
try {
  const engine = await CreateWebWorkerMLCEngine(worker, modelId, undefined, {
    conv_config: {
      role_empty_sep: '\n<think>\n',
      stop_token_ids: [248044, 248046],
    },
  })
  try {
    const model = new ChatWebLLM({
      engine,
      modelId,
      thinkingPrefilled: true,
      request: {
        temperature: 1,
        top_p: 0.95,
        presence_penalty: 1.5,
        repetition_penalty: 1,
        extra_body: { enable_thinking: true },
      },
    })
    console.log((await model.invoke('Hello')).text)
  } finally {
    await engine.unload()
  }
} finally {
  worker.terminate()
}

The first load fetches WebLLM's runtime and model files; the browser caches them for later use. Inference runs locally.

Use LangChain's usual bindTools, stream, batch, and withStructuredOutput APIs. For strict: true, use method: 'jsonSchema'. Call streamEvents(input) without a version for typed text, reasoning, tool-call, and usage streams. stream() yields partial tool-call chunks; IDs appear after validation. streamEvents(input).toolCalls returns validated calls. Thinking is returned as standard reasoning content blocks; .text contains the answer. Enable it with request.extra_body.enable_thinking on a model that supports it.

Thinking can reach the token or context limit before an answer; check response_metadata.finish_reason for length.

Long tool arguments can stream slowly with @langchain/core before 1.2.18; 1.2.18 fixes upstream quadratic chunk aggregation.

Deep Agents

Keep the agent and its tools in Node.js, and WebLLM in a private browser page:

npm install deepagents

// Browser: after loading the engine
import { createBrowserBridge } from 'langchain-web-llm'
window.webllmLangChainBridge = createBrowserBridge(engine)

// Node.js: page is a caller-owned Playwright Page
import { createDeepAgent } from 'deepagents'
import { BrowserChatWebLLM } from 'langchain-web-llm'

const model = new BrowserChatWebLLM(page, {
  modelId: 'Qwen3.5-4B-q4f16_1-MLC',
  profile: { maxInputTokens: 32768 }, // Match the loaded engine's context window.
  thinkingPrefilled: true,
  request: {
    temperature: 1,
    top_p: 0.95,
    presence_penalty: 1.5,
    repetition_penalty: 1,
    extra_body: { enable_thinking: true },
  },
})
const agent = createDeepAgent({ model })
const result = await agent.invoke({
  messages: [{ role: 'user', content: 'Write a short greeting to /greeting.txt, then read it.' }],
})

Deep Agents writes the text to its virtual filesystem in result.files, not to your computer.

Load the browser engine with context_window_size: 32768 and the Qwen3.5 conv_config above. The page must already contain the loaded model and bridge. Runnable examples show the full setup.

Support

| Feature | Support | | ------------------------------------ | -------------------------------------------------------------------------------------------------------------- | | Chat, streaming, tools, reasoning | Model-dependent | | Required tool choice | Qwen: one or more tools; Gemma/native tool models: one | | Cancellation and shared-engine calls | Calls are serialized | | Token counts | WebLLM usage; LangChain estimates with getNumTokens | | LangChain response cache (cache) | Known upstream cache defects; leave unset (#10092) | | Structured output | withStructuredOutput() through tools or native JSON | | Images | Model-dependent; URLs/base64/bytes; no detail | | WAV and Float32 PCM audio | Model-dependent; WAV base64/bytes/data URLs or native pcm_f32 | | Video and generic files | No native WebLLM input | | Embeddings | WebLLMEmbeddings and BrowserWebLLMEmbeddings for LangGraph memory | | Strict tool schemas | No |

When WebLLM reuses conversation state, input usage may count only new prompt tokens. Chat retries are off by default; maxRetries retries stream setup only. For log probabilities, browser transports must preserve non-finite numbers; Playwright does. A media message accepts one text segment; text on both sides of media blocks is rejected. Browser transport carries PCM as float WAV without changing samples or sample rate.

Supports LangChain Core ^1.2.8 and WebLLM ~0.2.86. Consumer checks cover Core 1.2.8, 1.3.6 and WebLLM 0.2.86, 0.2.88.

For Qwen agents with tools and a response schema, use responseFormat: toolStrategy(schema) (toolStrategy from langchain); native JSON cannot be combined with Qwen tools. For optional Qwen tool-call constraints, set request.constrainedToolCalls: true with WebLLM 0.2.86.

Use Gemma agent response schemas without other tools.

Use canonical JSON Pointer escapes ~0 and ~1 for ~ and / in $ref paths.

Models

See the verified models for exact IDs and features.

Install WebLLM 0.2.86 or later in the 0.2.x series. For WebLLM 0.2.86, use the bundled Gemma 4 E2B runtime. Audio works in ordinary chat; the default profile keeps it disabled for Deep Agents. Tools and thinking accept text and tool results only. Model-selected tools and arguments still need validation.

Tool errors are sent with an [ERROR] prefix because WebLLM has no tool-result status field.

Qwen3.5 preserves object/array arguments, type-less non-string enum/const values, and anyOf/oneOf unions without string branches as JSON. History rejects <|im_start|>, <|im_end|>, or <|endoftext|> in tool arguments, > in top-level argument names, and <parameter= / </parameter> in raw strings.

Embeddings

Load an embedding model, then use LangChain's standard interface:

import { WebLLMEmbeddings } from 'langchain-web-llm'

const embeddings = new WebLLMEmbeddings({
  engine,
  modelId: 'snowflake-arctic-embed-s-q0f32-MLC-b4',
})
const vector = await embeddings.embedQuery('a flower')

Arctic Embed M returns 768 dimensions; S returns 384. Both have a 512-token input limit. Snowflake's query prefix is applied automatically; documents stay unprefixed. Set queryPrefix: '' to send queries unchanged. Use BrowserWebLLMEmbeddings(page, options) for Node.js and LangGraph semantic memory. WebLLM splits each request to the loaded artifact's GPU batch limit. Lower batchSize to reduce temporary host memory and retry scope; choose a lower-batch artifact to reduce GPU memory. maxRetries retries only the failed request batch.

Native options

Set generation options with request at construction or per call. Per-call extra_body replaces the configured WebLLM fields; preserve_thinking is retained unless overridden. WebLLM's created timestamp is available as response_metadata.created in Unix seconds. For probabilities, set logprobs: true and top_logprobs: 2; read response_metadata.logprobs. onChunk exposes unchanged native chunks and timing; async handlers are awaited in order. Do not await work on the same engine from a handler. Browser models may read ahead while a handler runs.

For Gemma 4, request.extra_body.preserve_thinking: true also keeps earlier tool-call reasoning; the default keeps only the current user turn.

Set thinkingPrefilled: true at construction if your loaded Qwen template already opens <think>. This only controls output parsing; it does not change the engine or prompt. See Qwen3.5 engine setup for WebLLM 0.2.86.

Context-limit errors use LangChain's ContextOverflowError, with the native cause preserved.

BrowserChatWebLLM awaits onFailedAttempt in Node.js before retrying; throwing stops the call. Errors preserve JSON fields and cause values, plus LangChain retry markers.

For custom vision models in Deep Agents, set profile.imageInputs and profile.imageToolMessage to true. Tool images are sent in the next user turn.

Use withEngine(model.engine, callback) for native APIs, or window.webllmLangChainBridge.withEngine(callback) inside the browser page. Finish all work and consume streams inside the callback. Calls on one engine run in order. You own model loading, the engine, workers, and page; cancellation does not unload them.

With WebLLM 0.2.86, set request.extra_body.resumable.enabled to true to checkpoint chat, then call resumeChatCompletion() through withEngine(). Set resumable.strictPersistence to true to surface storage errors. WebLLM only resumes text-only requests without structured output or a custom logit processor; this does not restore LangChain or agent state. Gemma tool and thinking requests use raw completions and cannot resume. For Gemma, ignore_eos is available without tools; generation may run to max_tokens, which LangChain reports as length.