@localmode/transformers

HuggingFace Transformers.js provider for LocalMode — run ML models locally in the browser.

Features

• Browser-Native - Run ML models directly in the browser with WebGPU/WASM
• Privacy-First - All processing happens locally, no data leaves the device
• Model Caching - Model files are cached in the browser Cache API (transformers-cache) for instant subsequent loads, with a resilient write path that never fails a model load (opt out: createTransformers({ resilientCache: false }))
• Optimized - Uses quantized models for smaller size and faster inference

Installation

pnpm install @localmode/transformers @localmode/core

Dependencies

| Package | Purpose | | ------- | ------- | | @huggingface/transformers (^4.2.0) | ML inference via ONNX Runtime (WebGPU/WASM) | | phonemizer | eSpeak-NG WASM for Kokoro TTS text-to-phoneme conversion |

Overview

@localmode/transformers provides model implementations for the interfaces defined in @localmode/core. It wraps HuggingFace Transformers.js to enable local ML inference in the browser.

Provider API

All models are created via the transformers provider object. Each factory method returns a model implementing a @localmode/core interface.

Embeddings — Docs

import { embed, embedMany } from '@localmode/core';
import { transformers } from '@localmode/transformers';

const embeddingModel = transformers.embedding('Xenova/bge-small-en-v1.5');

const { embedding } = await embed({ model: embeddingModel, value: 'Hello world' });
const { embeddings } = await embedMany({ model: embeddingModel, values: ['Hello', 'World'] });

| Method | Interface | Description | | ------ | --------- | ----------- | | transformers.embedding(modelId) | EmbeddingModel | Text embeddings |

Recommended Models:

• Xenova/all-MiniLM-L6-v2 - Fast, general-purpose (~22MB)
• Xenova/paraphrase-multilingual-MiniLM-L12-v2 - 50+ languages

Multimodal Embeddings (CLIP/SigLIP) — Docs

Embed both text and images into the same vector space for cross-modal search.

import { embed, embedImage, cosineSimilarity } from '@localmode/core';
import { transformers } from '@localmode/transformers';

const model = transformers.multimodalEmbedding('Xenova/clip-vit-base-patch32');

// Text embedding
const { embedding: textVec } = await embed({ model, value: 'a photo of a cat' });

// Image embedding (same vector space)
const { embedding: imgVec } = await embedImage({ model, image: catImageBlob });

// Cross-modal similarity
const similarity = cosineSimilarity(textVec, imgVec);

| Method | Interface | Description | | -------------------------------------------- | ---------------------------- | ------------------------ | | transformers.multimodalEmbedding(modelId) | MultimodalEmbeddingModel | Text + image embeddings |

Recommended Models:

• Xenova/clip-vit-base-patch32 - Fast, 512 dimensions
• Xenova/clip-vit-base-patch16 - Better accuracy, 512 dimensions

Reranking — Docs

import { rerank } from '@localmode/core';
import { transformers } from '@localmode/transformers';

const rerankerModel = transformers.reranker('Xenova/ms-marco-MiniLM-L-6-v2');

const { results } = await rerank({
  model: rerankerModel,
  query: 'What is machine learning?',
  documents: ['ML is a subset of AI...', 'Python is a language...'],
  topK: 5,
});

| Method | Interface | Description | | ------ | --------- | ----------- | | transformers.reranker(modelId) | RerankerModel | Document reranking |

Classification & NLP — Docs

import { classify, extractEntities } from '@localmode/core';
import { transformers } from '@localmode/transformers';

const sentiment = await classify({
  model: transformers.classifier('Xenova/distilbert-base-uncased-finetuned-sst-2-english'),
  text: 'I love this product!',
});

const entities = await extractEntities({
  model: transformers.ner('Xenova/bert-base-NER'),
  text: 'John works at Microsoft in Seattle',
});

| Method | Interface | Description | | ------ | --------- | ----------- | | transformers.classifier(modelId) | ClassificationModel | Text classification | | transformers.zeroShot(modelId) | ZeroShotClassificationModel | Zero-shot text classification | | transformers.ner(modelId) | NERModel | Named Entity Recognition |

Translation & Summarization

| Method | Interface | Description | Docs | | ------ | --------- | ----------- | ---- | | transformers.translator(modelId) | TranslationModel | Text translation | Docs | | transformers.summarizer(modelId) | SummarizationModel | Text summarization | Docs | | transformers.fillMask(modelId) | FillMaskModel | Masked token prediction | Docs | | transformers.questionAnswering(modelId) | QuestionAnsweringModel | Extractive QA | Docs |

Audio

import { transcribe, synthesizeSpeech } from '@localmode/core';
import { transformers } from '@localmode/transformers';

const transcription = await transcribe({
  model: transformers.speechToText('onnx-community/moonshine-tiny-ONNX'),
  audio: audioBlob,
  returnTimestamps: true,
});

const { audio, sampleRate } = await synthesizeSpeech({
  model: transformers.textToSpeech('onnx-community/Kokoro-82M-v1.0-ONNX'),
  text: 'Hello, how are you?',
  voice: 'af_heart', // 29 English voices (see KOKORO_VOICES)
  speed: 1.0,        // 0.5 – 2.0
});

| Method | Interface | Description | Docs | | ------ | --------- | ----------- | ---- | | transformers.speechToText(modelId) | SpeechToTextModel | Speech-to-text transcription | Docs | | transformers.textToSpeech(modelId) | TextToSpeechModel | Text-to-speech synthesis | Docs | | transformers.audioClassifier(modelId) | AudioClassificationModel | Audio classification | | | transformers.zeroShotAudioClassifier(modelId) | ZeroShotAudioClassificationModel | Zero-shot audio classification | | | transformers.vad(modelId) | VADProvider | Voice Activity Detection (Silero) | |

Voice Activity Detection (VAD)

Detect speech segments in real-time audio streams. Used with createLiveTranscriber() for open-mic and push-to-talk transcription.

import { createLiveTranscriber } from '@localmode/core';
import { transformers } from '@localmode/transformers';

const vad = transformers.vad('onnx-community/silero-vad');
const transcriber = await createLiveTranscriber({
  model: transformers.speechToText('onnx-community/moonshine-tiny-ONNX'),
  mode: 'open-mic',
  vad,
});

| Method | Interface | Description | | ------ | --------- | ----------- | | transformers.vad(modelId) | VADProvider | Voice Activity Detection (Silero VAD) |

Recommended Models:

| Model | Description | | ----- | ----------- | | onnx-community/silero-vad | Silero VAD v5 — recommended browser VAD (~1.8MB, 512-sample frames at 16 kHz) |

Options: threshold (speech probability, default 0.5), silenceTimeoutMs (end-of-utterance timeout, default 700).

Vision

import { classifyImage, captionImage } from '@localmode/core';
import { transformers } from '@localmode/transformers';

const classification = await classifyImage({
  model: transformers.imageClassifier('Xenova/vit-base-patch16-224'),
  image: imageBlob,
});

const caption = await captionImage({
  model: transformers.captioner('onnx-community/Florence-2-base-ft'),
  image: imageBlob,
});

| Method | Interface | Description | Docs | | ------ | --------- | ----------- | ---- | | transformers.imageClassifier(modelId) | ImageClassificationModel | Image classification | Docs | | transformers.zeroShotImageClassifier(modelId) | ZeroShotImageClassificationModel | Zero-shot image classification | Docs | | transformers.captioner(modelId) | ImageCaptionModel | Image captioning | Docs | | transformers.segmenter(modelId) | SegmentationModel | Image segmentation | Docs | | transformers.objectDetector(modelId) | ObjectDetectionModel | Object detection | Docs | | transformers.imageFeatures(modelId) | ImageFeatureModel | Image feature extraction | Docs | | transformers.imageToImage(modelId) | ImageToImageModel | Image super resolution | Docs | | transformers.depthEstimator(modelId) | DepthEstimationModel | Monocular depth estimation | |

OCR & Document QA

| Method | Interface | Description | Docs | | ------ | --------- | ----------- | ---- | | transformers.ocr(modelId) | OCRModel | OCR (TrOCR, GLM-OCR, LightOnOCR-2) | Docs | | transformers.documentQA(modelId) | DocumentQAModel | Document/Table question answering | Docs |

Text Generation — Docs

Run ONNX-format language models in the browser with WebGPU acceleration:

import { generateText, streamText } from '@localmode/core';
import { transformers } from '@localmode/transformers';

const model = transformers.languageModel('onnx-community/Qwen3.5-0.8B-ONNX');

// Single-shot generation
const { text } = await generateText({ model, prompt: 'What is 2+2?' });

// Streaming generation
const result = await streamText({ model, prompt: 'Write a haiku' });
for await (const chunk of result.stream) {
  process.stdout.write(chunk.text);
}

| Method | Interface | Description | | ------ | --------- | ----------- | | transformers.languageModel(modelId) | LanguageModel | Text generation (ONNX, WebGPU/WASM) |

Recommended ONNX LLMs (16 curated models):

| Model | Size | Context | Vision | | ----- | ---- | ------- | ------ | | onnx-community/granite-4.0-350m-ONNX-web | ~120MB | 4K | No | | onnx-community/Qwen3-0.6B-ONNX | ~570MB | 4K | No | | onnx-community/Qwen3.5-0.8B-ONNX | ~500MB | 32K | Yes | | onnx-community/granite-4.0-1b-ONNX-web | ~350MB | 4K | No | | onnx-community/Llama-3.2-1B-Instruct-ONNX | ~380MB | 8K | No | | onnx-community/TinyLlama-1.1B-Chat-v1.0-ONNX | ~350MB | 2K | No | | onnx-community/Qwen2.5-Coder-1.5B-Instruct | ~450MB | 4K | No | | onnx-community/DeepSeek-R1-Distill-Qwen-1.5B-ONNX | ~500MB | 4K | No | | onnx-community/Llama-3.2-3B-Instruct-ONNX | ~900MB | 8K | No | | onnx-community/Qwen3-4B-ONNX | ~1.2GB | 4K | No | | microsoft/Phi-3-mini-4k-instruct-onnx-web | ~1.2GB | 4K | No | | onnx-community/Qwen3.5-2B-ONNX | ~1.5GB | 32K | Yes | | onnx-community/gemma-4-E2B-it-ONNX | ~1.5GB | 128K | Yes | | onnx-community/Phi-4-mini-instruct-web-q4f16 | ~2.3GB | 4K | No | | onnx-community/Qwen3.5-4B-ONNX | ~2.5GB | 32K | Yes | | onnx-community/gemma-4-E4B-it-ONNX | ~3GB | 128K | Yes |

Vision support: Qwen3.5, Qwen2.5-VL, Qwen3-VL, and Gemma 4 models support image input via their built-in vision encoder. Check model.supportsVision for feature detection. See Vision docs for usage.

Model Utilities

import { preloadModel, isModelCached, getModelStorageUsage } from '@localmode/transformers';

const cached = await isModelCached('Xenova/bge-small-en-v1.5');

await preloadModel('Xenova/bge-small-en-v1.5', {
  onProgress: (p) => console.log(`${p.progress}% loaded`),
});

const usage = await getModelStorageUsage();

Resilient Model-File Cache

Model files are cached in the browser Cache API (cache name transformers-cache, honoring a user-set env.cacheKey) — not IndexedDB. The provider installs a resilient wrapper by default so a failing cache write (e.g. the intermittent NetworkError some browsers throw mid-write, or QuotaExceededError) never fails a model load: the model still loads from the network response, and one warning per URL per session is logged. Environments without caches (Node/SSR) skip installation and keep Transformers.js's own file cache.

import { createTransformers } from '@localmode/transformers';

// Opt out of the resilient cache (global — the transformers.js env is a singleton)
const provider = createTransformers({ resilientCache: false });

// Manual control (advanced)
import {
  createResilientModelCache,   // build a Cache-API-backed cache implementing match/put/delete
  installResilientModelCache,  // idempotently configure env.useCustomCache + env.customCache
  setResilientModelCacheEnabled, // enable/disable installation globally
} from '@localmode/transformers';

Recommended Models

Embeddings

| Model | Description | | ----- | ----------- | | Xenova/bge-small-en-v1.5 | Fast, general-purpose (~22MB, 384d) | | Xenova/paraphrase-multilingual-MiniLM-L12-v2 | 50+ languages (~120MB, 384d) | | Xenova/all-mpnet-base-v2 | Higher quality (~420MB, 768d) | | Snowflake/snowflake-arctic-embed-xs | Tiny retrieval embeddings (~23MB, 384d) |

Reranking

| Model | Description | | ----- | ----------- | | Xenova/ms-marco-MiniLM-L-6-v2 | Fast, small (~23MB, recommended) |

Text Classification

| Model | Description | | ----- | ----------- | | Xenova/distilbert-base-uncased-finetuned-sst-2-english | Sentiment analysis | | Xenova/twitter-roberta-base-sentiment-latest | Twitter sentiment |

Zero-Shot Classification

| Model | Description | | ----- | ----------- | | Xenova/mobilebert-uncased-mnli | Fast, mobile-friendly (~21MB) | | Xenova/nli-deberta-v3-xsmall | Mid-tier accuracy (~90MB) |

Named Entity Recognition

| Model | Description | | ----- | ----------- | | Xenova/bert-base-NER | Standard NER (PER, ORG, LOC, MISC) |

Translation

| Model | Description | | ----- | ----------- | | Xenova/opus-mt-en-de | English to German | | Xenova/opus-mt-en-fr | English to French | | Xenova/opus-mt-en-es | English to Spanish |

Summarization

| Model | Description | | ----- | ----------- | | Xenova/distilbart-cnn-6-6 | Best quality browser summarizer (~284MB) |

Fill-Mask

| Model | Description | | ----- | ----------- | | onnx-community/ModernBERT-base-ONNX | General purpose (mask: [MASK]) |

Question Answering

| Model | Description | | ----- | ----------- | | Xenova/distilbert-base-cased-distilled-squad | SQuAD trained (~65MB) |

Speech-to-Text

| Model | Description | | ----- | ----------- | | onnx-community/moonshine-tiny-ONNX | Fast, edge-optimized (~50MB) | | onnx-community/moonshine-base-ONNX | Best quality/size ratio (~237MB) |

Text-to-Speech

| Model | Description | | ----- | ----------- | | onnx-community/Kokoro-82M-v1.0-ONNX | Natural speech, 29 English voices (~86MB) |

Image Classification

| Model | Description | | ----- | ----------- | | Xenova/vit-base-patch16-224 | General image classification | | Xenova/siglip-base-patch16-224 | Zero-shot image classification (~400MB) |

Image Captioning

| Model | Description | | ----- | ----------- | | onnx-community/Florence-2-base-ft | High-quality captions (~223MB) |

Image Segmentation

| Model | Description | | ----- | ----------- | | Xenova/segformer-b0-finetuned-ade-512-512 | Semantic segmentation (ADE20K) |

Object Detection

| Model | Description | | ----- | ----------- | | onnx-community/dfine_n_coco-ONNX | State-of-the-art, tiny (~4.5MB) | | Xenova/detr-resnet-50 | Classic transformer-based detection |

Image Features

| Model | Description | | ----- | ----------- | | Xenova/siglip-base-patch16-224 | Image embeddings (768d) | | onnx-community/dinov2-base-ONNX | Self-supervised features |

Image Super Resolution

| Model | Description | | ----- | ----------- | | Xenova/swin2SR-lightweight-x2-64 | 2x upscale, fast | | Xenova/swin2SR-classical-sr-x4-64 | 4x upscale |

OCR

| Model | Description | | ----- | ----------- | | Xenova/trocr-small-printed | Printed text, line-level (~120MB) | | Xenova/trocr-small-handwritten | Handwritten text, line-level (~120MB) | | onnx-community/GLM-OCR-ONNX | Document-level OCR with table/formula recognition (~652MB) | | onnx-community/LightOnOCR-2-1B-ONNX | Fast document OCR, 11 languages (~700MB) |

Document QA

| Model | Description | | ----- | ----------- | | onnx-community/Florence-2-base-ft | Document QA (~223MB) | | Xenova/donut-base-finetuned-docvqa | Donut (~218MB) |

Model Constants

All recommended models are exported as constants for easy reference:

import {
  MODELS,                      // All models organized by task
  EMBEDDING_MODELS,
  CLASSIFICATION_MODELS,
  ZERO_SHOT_MODELS,
  NER_MODELS,
  RERANKER_MODELS,
  SPEECH_TO_TEXT_MODELS,
  TEXT_TO_SPEECH_MODELS,
  IMAGE_CLASSIFICATION_MODELS,
  ZERO_SHOT_IMAGE_MODELS,
  IMAGE_CAPTION_MODELS,
  TRANSLATION_MODELS,
  SUMMARIZATION_MODELS,
  FILL_MASK_MODELS,
  QUESTION_ANSWERING_MODELS,
  OBJECT_DETECTION_MODELS,
  SEGMENTATION_MODELS,
  OCR_MODELS,
  DOCUMENT_QA_MODELS,
  IMAGE_TO_IMAGE_MODELS,
  IMAGE_FEATURE_MODELS,
  VAD_MODELS,
  TRANSFORMERS_LLM_MODELS,
  MULTIMODAL_EMBEDDING_MODELS,
  KOKORO_LANG_MAP,
} from '@localmode/transformers';

// Use with provider
const model = transformers.embedding(EMBEDDING_MODELS.BGE_SMALL_EN);

Kokoro Voice Catalog

The KOKORO_VOICES export provides a catalog of 29 English voices with metadata for UI display:

import { KOKORO_VOICES, KOKORO_DEFAULT_VOICE } from '@localmode/transformers';
import type { KokoroVoice } from '@localmode/transformers';

// Each voice has: id, name, language, languageLabel, gender
const english = KOKORO_VOICES.filter((v) => v.language === 'en-US');
const females = KOKORO_VOICES.filter((v) => v.gender === 'female');

console.log(KOKORO_DEFAULT_VOICE); // 'af_heart'

Languages: American English, British English.

Advanced Usage

Custom Model Options

const model = transformers.embedding('Xenova/bge-small-en-v1.5', {
  quantized: true, // Use quantized model (smaller, faster)
  device: 'webgpu', // Use WebGPU for acceleration (falls back to WASM)
});

Language Model Options

Language models accept additional settings via LanguageModelSettings:

const model = transformers.languageModel('onnx-community/Qwen3.5-0.8B-ONNX', {
  contextLength: 32768,
  maxTokens: 1024,
  temperature: 0.7,
  device: 'webgpu',
  // dtype accepts a string or a per-component config object
  dtype: 'q4f16',
  // For multimodal models, use per-component dtype:
  // dtype: { embed_tokens: 'q4', vision_encoder: 'q4', decoder_model_merged: 'q4' },
});

Provider Options

Pass provider-specific options to core functions:

const { embedding } = await embed({
  model: transformers.embedding('Xenova/bge-small-en-v1.5'),
  value: 'Hello world',
  providerOptions: {
    transformers: {
      // Any Transformers.js specific options
    },
  },
});

Preloading Models

For better UX, preload models before use:

import { preloadModel, isModelCached } from '@localmode/transformers';
import { embed } from '@localmode/core';

if (!(await isModelCached('Xenova/bge-small-en-v1.5'))) {
  await preloadModel('Xenova/bge-small-en-v1.5', {
    onProgress: (p) => console.log(`Loading: ${p.progress}%`),
  });
}

// Subsequent calls are instant (loaded from cache)
const embeddingModel = transformers.embedding('Xenova/bge-small-en-v1.5');
const { embedding } = await embed({ model: embeddingModel, value: 'Hello' });

Exported Implementation Classes

For advanced use cases, implementation classes are available:

import {
  TransformersEmbeddingModel,
  TransformersClassificationModel,
  TransformersZeroShotModel,
  TransformersNERModel,
  TransformersRerankerModel,
  TransformersSpeechToTextModel,
  TransformersImageClassificationModel,
  TransformersZeroShotImageModel,
  TransformersCaptionModel,
  TransformersCLIPEmbeddingModel,
  TransformersLanguageModel,
  TransformersGenerativeOCRModel,
  isGenerativeOCRModel,
  TransformersSileroVAD,
  createSileroVAD,
} from '@localmode/transformers';

Browser Compatibility

| Browser | WebGPU | WASM | Notes | | ----------- | ------ | ---- | ---------------------------- | | Chrome 113+ | ✅ | ✅ | Best performance with WebGPU | | Edge 113+ | ✅ | ✅ | Same as Chrome | | Firefox | ❌ | ✅ | WASM only | | Safari 26+ | ✅ | ✅ | WebGPU available | | iOS Safari | ✅ | ✅ | WebGPU available (iOS 26+) |

Performance Tips

1. Use quantized models - Smaller and faster with minimal quality loss
2. Preload models - Load during app init for instant inference
3. Use WebGPU when available - 3-5x faster than WASM
4. Batch operations - Process multiple inputs together

Acknowledgments

This package is built on Transformers.js by HuggingFace — state-of-the-art ML models running in the browser via ONNX Runtime.

License

MIT