strands-web-llm

A community WebLLM provider for Strands SDK and Harness. Run inference in a WebGPU browser while tools run in your application.

Install

npm install strands-web-llm

• Yarn: also install compatible versions of @mlc-ai/web-llm, @strands-agents/sdk, and the SDK peers listed by yarn explain peer-requirements.
• CommonJS: use require('strands-web-llm') on Node.js 22.12 or later. With Yarn PnP on Node 22.12–22.14 or 23.0–23.4, use import().
• TypeScript: moduleResolution: "bundler" retains more WebLLM option types than NodeNext. WebLLM's declarations may require skipLibCheck and leave some types unchecked.

Browser

Create a worker that runs WebLLM:

// worker.ts
import { WebWorkerMLCEngineHandler } from '@mlc-ai/web-llm'

const handler = new WebWorkerMLCEngineHandler()
self.onmessage = (message: MessageEvent) => handler.onmessage(message)

Load Qwen3.5 with WebLLM 0.2.86, then pass its engine to Strands:

import { CreateWebWorkerMLCEngine } from '@mlc-ai/web-llm'
import { Agent } from '@strands-agents/sdk'
import { WebLLMModel } from 'strands-web-llm'

const modelId = 'Qwen3.5-4B-q4f16_1-MLC'
const contextWindowSize = 4096
const worker = new Worker(new URL('./worker.ts', import.meta.url), { type: 'module' })
try {
  const engine = await CreateWebWorkerMLCEngine(worker, modelId, undefined, {
    context_window_size: contextWindowSize,
    conv_config: {
      role_empty_sep: '\n<think>\n',
      stop_token_ids: [248044, 248046],
    },
  })
  try {
    const model = new WebLLMModel({
      engine,
      modelId,
      contextWindowLimit: contextWindowSize,
      thinkingPrefilled: true,
      temperature: 1,
      topP: 0.95,
      request: {
        presence_penalty: 1.5,
        repetition_penalty: 1,
        extra_body: { enable_thinking: true },
      },
    })
    const agent = new Agent({ model })
    console.log((await agent.invoke('Say hello.')).lastMessage.content)
  } finally {
    await engine.unload()
  }
} finally {
  worker.terminate()
}

You own model loading and unloading. The examples use Worker and Main thread engines; the adapter accepts any WebLLM MLCEngineInterface. If thinking reaches the token or context limit, Agent.invoke throws MaxTokensError; its partialMessage contains the received content. The first load fetches WebLLM's runtime and model files; the browser caches them for later use. Inference runs locally. See the runnable examples.

Node.js Harness

Install Harness for this example:

npm install @strands-agents/harness@~0.3.4

On a browser page controlled by your application, load the model with context_window_size: 16384 and the Qwen3.5 conv_config above, then expose its engine:

import { createBrowserBridge } from 'strands-web-llm'
window.webllmBridge = createBrowserBridge(engine)

Pass its Playwright page to Harness in Node.js:

import { createHarness } from '@strands-agents/harness'
import { BrowserModel } from 'strands-web-llm'

const model = new BrowserModel(page, {
  modelId: 'Qwen3.5-4B-q4f16_1-MLC',
  contextWindowLimit: 16384, // Match the loaded engine.
  maxTokens: 4096,
  thinkingPrefilled: true,
  temperature: 1,
  topP: 0.95,
  request: { presence_penalty: 1.5, repetition_penalty: 1, extra_body: { enable_thinking: true } },
})
// Qwen3.5 is text-only, so keep media out of Harness's read tool.
const agent = await createHarness({ model, builtinTools: { read: { media: false } } })
try {
  console.log((await agent.invoke('Say hello.')).lastMessage.content)
} finally {
  await agent.shutdown()
  await page.evaluate(() => window.webllmBridge.close())
}

Tools use your host permissions unless you configure a Harness sandbox. The Harness example includes browser startup and shutdown.

Models and memory

See the verified models for exact IDs and features. Qwen3.5 preserves object/array arguments, type-less non-string enum/const values, and anyOf/oneOf unions without string branches as JSON. History rejects <|im_start|>, <|im_end|>, or <|endoftext|> in tool arguments, > in top-level argument names, and <parameter= / </parameter> in raw strings. Install WebLLM 0.2.86 or later in the 0.2.x series. For WebLLM 0.2.86, use the bundled Gemma 4 E2B runtime. Audio works in ordinary chat, without tools or thinking.

WebLLMEmbeddingSearchStrategy connects WebLLM to Strands memory; see the memory example. This strategy embeds UTF-8 text only. Entries with invalid UTF-8 stay in storage but are excluded from semantic search. Snowflake Arctic Embed S and M have a 512-token input limit. Prefix queries with Represent this sentence for searching relevant passages: ; leave stored text unprefixed. Search returns positive cosine matches by default; set minScore to include lower scores. includeData: false reduces memory use; FileMemoryStore rereads the returned matches. Searches scan all entries, and vectors rebuild after restart. FileMemoryStore appends facts that share a heading, so keep each stored record within the model's input limit. Before changing the embedding model, wait for pending searches and writes, then call strategy.clearCache().

Support

Supports Strands SDK ^1.17.0 and WebLLM ~0.2.86. Consumer checks cover SDK 1.17.0, 1.24.0 and WebLLM 0.2.86, 0.2.88. Model availability depends on the installed WebLLM version. Harness 0.3.4 is used by the integration example.

Features depend on the exact model.

For Gemma 4, request.extra_body.preserve_thinking: true also keeps earlier tool-call reasoning; the default keeps only the current user turn.

Set thinkingPrefilled: true at construction if your loaded Qwen template already opens <think>. This only controls output parsing; it does not change the engine or prompt. See Qwen3.5 engine setup for WebLLM 0.2.86.

| Feature | Use | Limit | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- | | Chat, system prompts, history | Strands messages and systemPrompt | Unsupported blocks raise errors | | Provider citation blocks | Unsupported | This adapter does not produce them; cited history raises an error | | Streaming and cancellation | agent.stream(), agent.cancel() | One assistant response | | Tools and results | Strands tools and toolChoice | Qwen supports any; Gemma/native tool models require one tool | | Disable tools | request.tool_choice: 'none' | Keeps the Agent tool registry | | Thinking | request.extra_body.enable_thinking | Gemma required calls have no thinking before the call | | Sampling and token limits | request; maxTokens, temperature, topP | SDK aliases take precedence | | Output constraints | request.response_format; optional Qwen tool constraints | Qwen constraints need WebLLM 0.2.86; Qwen tools reject custom formats; Gemma tools/thinking use text | | Images and WAV audio | ImageBlock, AudioBlock, pcmAudio() | Model-dependent; PCM is converted to WAV | | Usage and timing | Strands metadata | WebLLM token counts and latency | | Response metadata | model.stream() stop-event additionalModelResponseFields (id, model, created) | created is Unix seconds | | Log probabilities | request.logprobs: true; stop-event additionalModelResponseFields.logprobs; onChunk | Native per-token values retained | | Embeddings | WebLLMEmbeddingSearchStrategy or withEngine() | Separate embedding model required | | Context recovery | countTokens(); default SlidingWindowConversationManager | Trims and retries on ContextWindowOverflowError; match contextWindowLimit; counts are estimates | | Other native APIs | withEngine() | Caller owns setup and lifecycle |

Tool errors are sent with an [ERROR] prefix because WebLLM has no tool-result status field. When WebLLM reuses conversation state, input usage may count only new prompt tokens. This can make context estimates too low. Strands adds injected context after estimation, so it is not counted. For log probabilities, browser transports must preserve non-finite numbers; Playwright does.

Signed/redacted reasoning, S3 media, video, documents, guardrails, and non-image tool-result media are unsupported. A media message accepts one text segment; text on both sides of media blocks is rejected. Tool-result images follow the adjacent tool replies; no listed model is verified for both tools and vision. Invalid or incomplete tool output reaches the caller; the adapter does not change tool names or arguments or supply answers. Validate model-selected arguments: Strands runtime validation requires a Zod inputSchema, and Gemma's template may omit some JSON Schema constraints.

Share the engine safely from tools, hooks, or application code:

import { withEngine } from 'strands-web-llm'

const result = await withEngine(engine, (engine) =>
  engine.embeddings.create({ model: embeddingModelId, input: 'hello' }),
)

For Node.js, run the callback on the browser page:

const stats = await page.evaluate(() =>
  window.webllmBridge.withEngine((engine) => engine.runtimeStatsText()),
)

Await all work and consume streams inside the callback. Return serializable results to Node.js. Calls sharing an engine run in order; do not invoke an Agent or another scope on that engine inside the callback. Direct engine calls bypass the queue. Stop callers before replacing models.

| Native feature | API or limit | | --------------------------------------------------- | ------------------------------------------------------------- | | Raw prompts, echo, multiple choices | Native completions; multiple streamed choices are unsupported | | Generation save and resume | WebLLM chat API; separate from Strands sessions | | Reload, unload, reset, progress, stats, GPU queries | Original engine APIs | | Logit processors | WebLLM engine setup | | Cache backends, integrity, custom models | WebLLM setup and cache APIs | | Main thread, Worker, Service Worker, extension | Any MLCEngineInterface; use WebLLM's factories and handlers | | Message name | Native chat; SDK messages have no name field |

With WebLLM 0.2.86, set request.extra_body.resumable.enabled to true to checkpoint chat, then call resumeChatCompletion() through withEngine(). Set resumable.strictPersistence to true to surface storage errors. WebLLM only resumes text-only requests without structured output or a custom logit processor. Gemma tool and thinking requests use raw completions and cannot resume. Strands rejects Gemma ignore_eos because the SDK treats a max-token finish as an unrecoverable error.

updateConfig() merges top-level settings; modelId selects an already loaded model. Use model-card sampling settings. WebLLM does not expose top_k or min_p. Async onChunk callbacks are awaited in order; don't await work on the same engine from one. Browser models may read ahead while a callback runs. Rejections fail the call. Cache hints do not configure WebLLM's KV cache. Recovering external tool side effects is the application's responsibility.

See the native engine interface and Strands provider contract.