expo-sherpa-onnx

Warning This is an pre-release version. APIs may change without notice.

Expo module wrapping sherpa-onnx for on-device speech processing in React Native. Runs fully offline on Android and iOS — no network calls, no cloud APIs.

Features

• Offline Speech-to-Text — Whisper, SenseVoice, Paraformer, Transducer, NeMo CTC, Moonshine
• Streaming Speech-to-Text — real-time recognition from microphone input
• Text-to-Speech — VITS, Matcha, Kokoro, and more with streaming audio support
• Voice Activity Detection — Silero VAD and Ten VAD with file and live-mic modes
• Keyword Spotting — wake-word / hotword detection from a stream
• Speaker Embedding — extract voice embeddings for speaker verification and identification
• Speaker Diarization — segment audio by speaker with optional per-segment transcription
• File-based Processing — large audio files (1hr+) are read natively, bypassing the JS bridge
• Model download helper (optional) — useSherpaModel and ModelDownloadManager download and extract archives to the app sandbox with progress, retries, and optional SHA-256 verification

Platforms

| Platform | Status | | -------- | --------- | | Android | Supported | | iOS | Supported |

Native Dependencies

This module bundles prebuilt static libraries from sherpa-onnx:

• Version: 1.12.29
• Git SHA1: 75022de
• Source: https://github.com/k2-fsa/sherpa-onnx

The prebuilt binaries are included in the npm package. If you are installing via npx expo install, no extra setup is required.

Building from source (contributors)

The binaries are not checked into git. If you're contributing to this module and need to build or run the example app natively:

git clone https://github.com/k2-fsa/sherpa-onnx.git
cd sherpa-onnx
git checkout 75022de

# Android — build for each ABI, then copy .so files into android/src/main/jniLibs/
./build-android-arm64-v8a.sh
./build-android-armv7-eabi.sh
./build-android-x86-64.sh

# iOS — build xcframeworks, then copy into ios/ (use cp -RLa to resolve symlinks)
./build-ios.sh

See CONTRIBUTING.md for full step-by-step instructions with exact copy commands.

Installation

npx expo install expo-sherpa-onnx

The module requires native code, so it works with development builds (not Expo Go).

iOS troubleshooting

ld: library 'onnxruntime' not found — The CocoaPods prepare_command in ExpoSherpaOnnx.podspec creates libonnxruntime.a symlinks next to onnxruntime.a in each xcframework slice (the linker looks for the lib* name). After upgrading the package, run npx pod-install or cd ios && pod install, then Product → Clean Build Folder in Xcode. Ensure node_modules/expo-sherpa-onnx/ios/onnxruntime.xcframework exists and is not empty; reinstall the package if it is missing.

The Hermes script-phase message about “Based on dependency analysis” is a known Xcode warning and is unrelated to this module.

Quick Start

Using paths you control

Point engines at any directory on disk (bundled assets, OTA-updated folders, manual downloads):

import { createSTT, detectSttModel } from "expo-sherpa-onnx";

// Auto-detect model type from a directory
const detected = await detectSttModel("/path/to/model-dir");

// Build config (see example app for config builders)
const engine = await createSTT({
  modelConfig: {
    tokens: "/path/to/tokens.txt",
    whisper: {
      encoder: "/path/to/encoder.onnx",
      decoder: "/path/to/decoder.onnx",
    },
  },
});

// Transcribe from file (native read, no JS bridge overhead)
const result = await engine.transcribeFile("/path/to/audio.wav");
console.log(result.text);

await engine.destroy();

Using the model download helper

useSherpaModel registers a model by ID, then downloads and extracts .tar.bz2, .zip, or single-file URLs into {documentDirectory}models/downloads/ (see expo-file-system). The extract is stored under expectedDir; sherpa archives often add a single inner folder (for example sherpa-onnx-nemo-…). For STT, detectSttModel merges that inner folder name into its classification hint, so models are recognized correctly even when expectedDir does not match the release name (for example after changing the download URL but leaving an old expectedDir). When status is READY, localPath is the extracted folder (or file) you pass into detectSttModel / engine factories:

import { createSTT, detectSttModel, useSherpaModel } from "expo-sherpa-onnx";
import type { SherpaModelConfig } from "expo-sherpa-onnx";

const MODEL: SherpaModelConfig = {
  useCase: "asr-offline",
  url: "https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/sherpa-onnx-whisper-tiny.tar.bz2",
  archiveFormat: "tar.bz2",
  version: "1.0.0",
  sizeBytes: 77_000_000,
  expectedDir: "sherpa-onnx-whisper-tiny",
  // sha256: "...", // optional integrity check
};

function MyScreen() {
  const model = useSherpaModel("my-asr", MODEL);

  const run = async () => {
    if (model.status !== "READY" || !model.localPath) return;
    const dir = model.localPath.replace(/^file:\/\//, "");
    const detected = await detectSttModel(dir);
    // build config from detected + dir, then createSTT(...)
  };

  // model.triggerDownload() — start download
  // model.deleteModel() — remove from disk and reset state
  // model.updateUrl(url) — point at a different URL before download
}

For imperative use (no hook), call ModelDownloadManager.registerModel(id, config) then await ModelDownloadManager.ensureModelReady(id). You can preload a catalog with ModelDownloadManager.initialize(manifest) where manifest is a Record<string, SherpaModelConfig>.

Example app: each feature tab embeds a ModelDownloadCard that reuses useSherpaModel — download button, progress bar, optional URL edit, file tree after extract, and delete. That UI component lives under example/components/ only; your app can copy the pattern or build your own on top of the hook.

API Reference

Factory Functions

All factory functions return engine objects that must be destroy()'d when no longer needed.

Speech-to-Text

import { createSTT, createStreamingSTT } from 'expo-sherpa-onnx';

// Offline (file/batch processing)
const engine = await createSTT(config: OfflineRecognizerConfig);

// Online (streaming/real-time)
const engine = await createStreamingSTT(config: OnlineRecognizerConfig);

Text-to-Speech

import { createTTS } from 'expo-sherpa-onnx';

const engine = await createTTS(config: OfflineTtsConfig);

Voice Activity Detection

import { createVAD } from 'expo-sherpa-onnx';

const vad = await createVAD(config: VadModelConfig, bufferSizeInSeconds?: number);

Keyword Spotting

import { createKeywordSpotter } from 'expo-sherpa-onnx';

const spotter = await createKeywordSpotter(config: KeywordSpotterConfig);

Speaker Embedding

import {
  createSpeakerEmbeddingExtractor,
  createSpeakerEmbeddingManager,
} from 'expo-sherpa-onnx';

const extractor = await createSpeakerEmbeddingExtractor(config: SpeakerEmbeddingExtractorConfig);
const manager = await createSpeakerEmbeddingManager(dim: number);

Speaker Diarization

import { createOfflineSpeakerDiarization } from 'expo-sherpa-onnx';

const engine = await createOfflineSpeakerDiarization(config: OfflineSpeakerDiarizationConfig);

Engine Methods

OfflineSTTEngine

| Method | Signature | Description | | ------------------- | ------------------------------------------------------------------------------ | ----------------------------------- | | transcribeSamples | (samples: number[], sampleRate?: number) => Promise<OfflineRecognizerResult> | Transcribe raw PCM samples | | transcribeFile | (filePath: string) => Promise<OfflineRecognizerResult> | Transcribe a WAV file (native read) | | destroy | () => Promise<void> | Release native resources |

const result = await engine.transcribeFile("/path/to/audio.wav");
// result.text, result.tokens, result.timestamps, result.lang

OnlineSTTEngine

| Method | Signature | Description | | -------------- | ------------------------------------------------- | --------------------------- | | createStream | (hotwords?: string) => Promise<OnlineSTTStream> | Create a recognition stream | | destroy | () => Promise<void> | Release native resources |

OnlineSTTStream methods:

| Method | Signature | Description | | ---------------- | ----------------------------------------------------------- | --------------------------------- | | acceptWaveform | (samples: number[], sampleRate?: number) => Promise<void> | Feed audio samples | | inputFinished | () => Promise<void> | Signal end of audio | | decode | () => Promise<void> | Run the decoder | | isReady | () => Promise<boolean> | Check if enough data for decode | | isEndpoint | () => Promise<boolean> | Check if an endpoint was detected | | getResult | () => Promise<OnlineRecognizerResult> | Get current transcription | | reset | () => Promise<void> | Reset stream state | | destroy | () => Promise<void> | Release stream resources |

const stream = await engine.createStream();
await stream.acceptWaveform(audioChunk);
await stream.decode();
const result = await stream.getResult();

OfflineTTSEngine

| Property / Method | Signature | Description | | ------------------- | ------------------------------------------------------------------------------------------------- | ---------------------------------- | | sampleRate | readonly number | Output sample rate | | numSpeakers | readonly number | Number of available speaker voices | | generate | (text: string, sid?: number, speed?: number) => Promise<GeneratedAudio> | Synthesize speech | | generateStreaming | (text: string, callbacks: StreamingTTSCallbacks, sid?: number, speed?: number) => Promise<void> | Synthesize with streaming chunks | | destroy | () => Promise<void> | Release native resources |

const audio = await engine.generate("Hello world", 0, 1.0);
// audio.samples (number[]), audio.sampleRate

VADEngine

| Method | Signature | Description | | ------------------ | ------------------------------------------------ | ---------------------------------- | | acceptWaveform | (samples: number[]) => Promise<void> | Feed audio samples (for streaming) | | empty | () => Promise<boolean> | Check if segment queue is empty | | isSpeechDetected | () => Promise<boolean> | Current speech detection state | | pop | () => Promise<void> | Remove front segment from queue | | front | () => Promise<SpeechSegment> | Get front segment | | clear | () => Promise<void> | Clear segment queue | | reset | () => Promise<void> | Reset VAD state | | flush | () => Promise<void> | Flush remaining audio | | processFile | (filePath: string) => Promise<SpeechSegment[]> | Process entire WAV file natively | | destroy | () => Promise<void> | Release native resources |

// File-based (recommended for batch processing)
const segments = await vad.processFile("/path/to/audio.wav");
for (const seg of segments) {
  console.log(
    `Speech at sample ${seg.start}, duration: ${seg.samples.length} samples`
  );
}

KeywordSpotterEngine

| Method | Signature | Description | | -------------- | ----------------------------------------------- | --------------------------------- | | createStream | (keywords?: string) => Promise<KeywordStream> | Create a keyword detection stream | | destroy | () => Promise<void> | Release native resources |

KeywordStream methods:

| Method | Signature | Description | | ---------------- | ----------------------------------------------------------- | ------------------------ | | acceptWaveform | (samples: number[], sampleRate?: number) => Promise<void> | Feed audio samples | | isReady | () => Promise<boolean> | Check if ready to decode | | decode | () => Promise<void> | Run the decoder | | getResult | () => Promise<KeywordSpotterResult> | Get spotted keyword | | reset | () => Promise<void> | Reset stream state | | destroy | () => Promise<void> | Release stream resources |

SpeakerEmbeddingExtractorEngine

| Method | Signature | Description | | -------------------------- | ----------------------------------------- | --------------------------------------------- | | dim | () => Promise<number> | Get embedding dimension | | createStream | () => Promise<SpeakerEmbeddingStream> | Create extraction stream | | computeEmbeddingFromFile | (filePath: string) => Promise<number[]> | Extract embedding from WAV file (native read) | | destroy | () => Promise<void> | Release native resources |

SpeakerEmbeddingStream methods:

| Method | Signature | Description | | ---------------- | ----------------------------------------------------------- | ---------------------------- | | acceptWaveform | (samples: number[], sampleRate?: number) => Promise<void> | Feed audio samples | | isReady | () => Promise<boolean> | Check if ready to compute | | compute | () => Promise<number[]> | Compute the embedding vector | | destroy | () => Promise<void> | Release stream resources |

// File-based (single native round-trip)
const embedding = await extractor.computeEmbeddingFromFile(
  "/path/to/speaker.wav"
);
await manager.add("Alice", embedding);

SpeakerEmbeddingManagerEngine

| Method | Signature | Description | | ----------------- | ---------------------------------------------------------------------------- | --------------------------------- | | add | (name: string, embedding: number[]) => Promise<boolean> | Enroll a speaker | | addList | (name: string, embeddings: number[][]) => Promise<boolean> | Enroll with multiple embeddings | | remove | (name: string) => Promise<boolean> | Remove a speaker | | search | (embedding: number[], threshold: number) => Promise<string> | Find closest matching speaker | | verify | (name: string, embedding: number[], threshold: number) => Promise<boolean> | Verify against a specific speaker | | contains | (name: string) => Promise<boolean> | Check if speaker is enrolled | | numSpeakers | () => Promise<number> | Count enrolled speakers | | allSpeakerNames | () => Promise<string[]> | List all enrolled speaker names | | destroy | () => Promise<void> | Release native resources |

OfflineSpeakerDiarizationEngine

| Method | Signature | Description | | -------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------ | | getSampleRate | () => Promise<number> | Get expected sample rate | | process | (samples: number[]) => Promise<DiarizationSegment[]> | Diarize from samples | | processFile | (filePath: string) => Promise<DiarizationSegment[]> | Diarize a WAV file (native read) | | transcribeAndDiarizeFile | (asrHandle: number, filePath: string) => Promise<TranscribedDiarizationSegment[]> | Diarize + transcribe each segment natively | | setConfig | (config: OfflineSpeakerDiarizationConfig) => Promise<void> | Update clustering config | | destroy | () => Promise<void> | Release native resources |

// Diarize only
const segments = await engine.processFile("/path/to/meeting.wav");
// segments: [{ start: 0.0, end: 2.5, speaker: 0 }, ...]

// Diarize + transcribe (everything runs natively)
const asrEngine = await createSTT(asrConfig);
const transcript = await engine.transcribeAndDiarizeFile(
  asrEngine.handle,
  "/path/to/meeting.wav"
);
// transcript: [{ speaker: 0, start: 0.0, end: 2.5, text: "Hello everyone" }, ...]

Utility Functions

import {
  readWaveFile,
  getAvailableProviders,
  listModelsAtPath,
  detectSttModel,
  detectTtsModel,
} from "expo-sherpa-onnx";

// Read a WAV file into PCM samples
const wave = await readWaveFile("/path/to/audio.wav");
// wave.samples, wave.sampleRate

// List available hardware providers (e.g. 'cpu', 'coreml', 'nnapi')
const providers = getAvailableProviders();

// List model files/directories in a path
const items = await listModelsAtPath("/path/to/models", true);

// Auto-detect model type from directory contents
const sttModel = await detectSttModel("/path/to/whisper-model");
// sttModel.type === 'whisper', sttModel.files === { encoder: '...', decoder: '...' }

const ttsModel = await detectTtsModel("/path/to/vits-model");
// ttsModel.type === 'vits', ttsModel.files === { model: '...' }

File-Based Processing

For batch processing of audio files, always prefer the file-path methods. These read WAV data on the native side, avoiding the JS bridge memory bottleneck that would occur with large audio arrays.

| Engine | File-path method | What it replaces | | --------------------------------- | ------------------------------------------- | --------------------------------------------- | | OfflineSTTEngine | transcribeFile(path) | transcribeSamples(samples) | | VADEngine | processFile(path) | manual acceptWaveform + flush loop | | SpeakerEmbeddingExtractorEngine | computeEmbeddingFromFile(path) | createStream + acceptWaveform + compute | | OfflineSpeakerDiarizationEngine | processFile(path) | process(samples) | | OfflineSpeakerDiarizationEngine | transcribeAndDiarizeFile(asrHandle, path) | diarize + loop + transcribe per segment |

The streaming methods (acceptWaveform on online STT, KWS, and live VAD) are intended for real-time mic input in small chunks and remain unchanged.

Example App

The example/ directory contains a full demo app. Every feature screen includes a model download card wired through useSherpaModel: a default URL from sherpa-onnx releases (often one tap to download), optional URL override, progress, extracted file tree, and delete. You only sideload files manually if you want custom paths.

| Tab | Screen | Description | | ----------- | ------------------------- | --------------------------------------------------------------- | | Build | BuildVerificationScreen | Verify native build and sherpa-onnx version | | Offline ASR | OfflineASRScreen | Download Whisper tiny demo model; file or record-and-transcribe | | Stream ASR | StreamingASRScreen | Streaming Zipformer + live mic | | TTS | OfflineTTSScreen | VITS Piper demo model | | Stream TTS | StreamingTTSScreen | Streaming TTS with chunk callbacks | | VAD | VADScreen | Silero VAD; file and live-mic modes | | KWS | KeywordSpottingScreen | Keyword / wake-word spotting | | Speaker | SpeakerScreen | Speaker embedding enrollment and search | | Diarize | DiarizationScreen | Multiple downloads (segmentation, embedding, optional ASR) | | Lang ID | LanguageIdScreen | Spoken language identification | | Audio Tag | AudioTaggingScreen | Audio tagging | | Punct | PunctuationScreen | Offline punctuation | | Denoise | DenoisingScreen | Speech enhancement | | Pipeline | StreamingPipelineScreen | ASR + denoiser + online punctuation demo | | Accel | AccelerationScreen | Hardware acceleration providers |

Running the example

cd expo-sherpa-onnx/example

# iOS
npx expo run:ios

# Android
npx expo run:android

On first use, open a tab and tap Download on the card; models land under the app document directory in models/downloads/ (managed by ModelDownloadManager). You can still copy files yourself into any path the engines accept if you prefer not to use the helper.

Models

This module works with any sherpa-onnx compatible ONNX model. Ship models any way you like — app bundle, your own OTA, or HTTPS via useSherpaModel / ModelDownloadManager as in the example. Pre-trained release archives are listed here:

• ASR: sherpa-onnx ASR models
• TTS: sherpa-onnx TTS models
• VAD: silero-vad or sherpa-onnx VAD releases
• Speaker: speaker-recognition-models
• Diarization: sherpa-onnx-pyannote-segmentation-3-0

Testing

cd expo-sherpa-onnx
npx jest

The test suite uses mocks for native functions, enabling full coverage of the TypeScript API layer without native builds.

License

MIT