@qvac/tts-onnx

v0.9.0

Published

14 days ago

Text to Speech (TTS) addon for qvac

0High
0Medium
0Low

mafintosh

prdn

subash.77

tts-onnx

This library simplifies running Text-to-Speech (TTS) models within QVAC runtime applications. It provides an easy interface to load, execute, and manage TTS instances, supporting multiple data sources (called data loaders) and leveraging ONNX Runtime for efficient inference.

The package supports two TTS engines:

Chatterbox - Neural TTS with voice cloning from reference audio (24 kHz)
Supertonic - Diffusion-based TTS with pre-trained voice styles (44.1 kHz)

The engine is auto-detected based on the arguments you provide.

tts-onnx

Supported Platforms

| Platform | Architecture | Min Version | Status | GPU Support | |----------|-------------|-------------|--------|-------------| | macOS | arm64, x64 | 14.0+ | ✅ Tier 1 | CoreML | | iOS | arm64 | 17.0+ | ✅ Tier 1 | CoreML | | Linux | arm64, x64 | Ubuntu-22+ | ✅ Tier 1 | CPU only | | Android | arm64 | 12+ | ✅ Tier 1 | NNAPI | | Windows | x64 | 10+ | ✅ Tier 1 | DirectML |

Dependencies:

inference-addon-cpp: C++ addon framework
ONNX Runtime: Inference engine
Chatterbox TTS: Neural text-to-speech engine with voice cloning
Supertonic TTS: Diffusion-based text-to-speech engine with pre-trained voices
Bare Runtime (>=1.17.3): JavaScript runtime
Linux requires Clang/LLVM 22 with libc++

TTS Engines

This package supports two TTS engines. The engine is auto-detected based on the arguments you provide:

If you pass modelDir + voiceName, or textEncoderPath, the Supertonic engine is used.
Otherwise, the Chatterbox engine is used.

| Feature | Chatterbox | Supertonic | |---------|-----------|------------| | Architecture | Transformer-based language model | Diffusion-based latent denoising | | Sample Rate | 24,000 Hz | 44,100 Hz | | Voice Method | Voice cloning from reference audio | Pre-trained voice styles (.bin files) | | ONNX Models | 5 (tokenizer, speech_encoder, embed_tokens, conditional_decoder, language_model) | 3 (text_encoder, latent_denoiser, voice_decoder) | | Languages | English (en), Spanish (es), French (fr), German (de), Italian (it), Portuguese (pt), Russian (ru), Arabic (ar), Danish (da), Greek (el), Finnish (fi), Hebrew (he), Hindi (hi), Korean (ko), Malay (ms), Dutch (nl), Norwegian (no), Polish (pl), Swedish (sv), Swahili (sw), Turkish (tr) | English (en), Korean (ko), Spanish (es), Portuguese (pt), French (fr) | | Speed Control | N/A | Configurable via speed parameter | | Inference Steps | Single-pass | Configurable via numInferenceSteps (default: 5) | | Use Case | Voice cloning from a reference audio sample | General-purpose TTS with selectable voice presets | | Real Time Factor | Usually >1.0 | <1.0 |

Note: though Chatterbox models support up to 23 languages, this implementation still only support 7 due to onnx models limitations. We are actively working to provide new onnx models that support all 23 langauges from the original implementation.

Installation

Prerequisites

Install Bare Runtime:

npm install -g bare

Note : Make sure the Bare version is >= 1.17.3. Check this using:

bare -v

Installing the Package

Install the latest TTS package:

npm install @qvac/tts-onnx@latest

Building from Source

If you want to build the addon from source (for development or customization), follow these steps:

Prerequisites

Before building, ensure you have the following installed:

vcpkg - Cross-platform C++ package manager

git clone https://github.com/microsoft/vcpkg.git
cd vcpkg && ./bootstrap-vcpkg.sh -disableMetrics
export VCPKG_ROOT=/path/to/vcpkg
export PATH=$VCPKG_ROOT:$PATH

Build tools for your platform:
- Linux:
```
sudo apt install clang libc++-dev libc++abi-dev build-essential autoconf automake libtool pkg-config
```
- macOS: Xcode command line tools
- Windows: Visual Studio with C++ build tools
Node.js and npm (version 18+ recommended)
Bare runtime and build tools:
```
npm install -g bare-runtime bare-make
```

Building the Addon

Clone the repository:

git clone https://github.com/tetherto/qvac
cd qvac/packages/tts-onnx

Install dependencies:
```
npm install
```
Build the addon:
```
npm run build
```

This command will:

Generate CMake build files (bare-make generate)
Build the native addon (bare-make build)
Install the addon to the prebuilds directory (bare-make install)

Verifying the Build

After building, you can run the tests to verify everything works:

npm run test:unit
npm run test:integration  # Requires model files

Note: Integration tests require model files to be present in the models/ directory. See the Downloading Models section below.

Downloading Models

Model files must be present locally before running examples or integration tests. Download scripts fetch models from Hugging Face.

# Chatterbox only (English q4 by default)
npm run models:ensure:chatterbox

# Chatterbox with a specific variant
CHATTERBOX_VARIANT=fp32 npm run models:ensure:chatterbox

# Supertonic only (English by default)
npm run models:ensure:supertonic

# Multilingual models
TTS_LANGUAGE=multilingual npm run models:ensure:chatterbox
TTS_LANGUAGE=multilingual npm run models:ensure:supertonic

# Both English and multilingual for a single engine
TTS_LANGUAGE=all npm run models:ensure:chatterbox

# Both Chatterbox + Supertonic (English by default)
npm run models:ensure

# Both Chatterbox + Supertonic multilingual
TTS_LANGUAGE=multilingual npm run models:ensure

# Everything (both engines, both languages)
TTS_LANGUAGE=all npm run models:ensure

Environment Variables

| Variable | Default | Values | Description | |----------|---------|--------|-------------| | CHATTERBOX_VARIANT | q4 | fp32, fp16, q4, q4f16 | Chatterbox model quantization variant | | TTS_LANGUAGE | en | en, multilingual, all | Language set for model downloads (all downloads both) |

Models are saved to models/chatterbox/ (English), models/chatterbox-multilingual/, models/supertonic/, and models/supertonic-multilingual/.

Usage: Chatterbox

1. Import the Model Class

const { ONNXTTS } = require('@qvac/tts-onnx')
// or if importing directly:
// const ONNXTTS = require('./')

2. Create a Data Loader

Data Loaders abstract the way model files are accessed. You can use a FileSystemDataLoader to stream the model file(s) from your local file system.

const FilesystemDL = require('@qvac/dl-filesystem')
const fsDL = new FilesystemDL({
  dirPath: './path/to/model/files'
})

3. Create the `args` obj

const args = {
  loader: fsDL,
  opts: { stats: true },
  logger: console,
  cache: './models/',
  tokenizerPath: 'chatterbox/tokenizer.json',
  speechEncoderPath: 'chatterbox/speech_encoder.onnx',
  embedTokensPath: 'chatterbox/embed_tokens.onnx',
  conditionalDecoderPath: 'chatterbox/conditional_decoder.onnx',
  languageModelPath: 'chatterbox/language_model.onnx',
  referenceAudio: referenceAudioFloat32Array
}

The args obj contains the following properties:

loader: The Data Loader instance from which the model files will be streamed.
logger: This property is used to create logging functionality.
opts.stats: This flag determines whether to calculate inference stats.
cache: The local directory where the model files will be downloaded to.
tokenizerPath: Path to the Chatterbox tokenizer JSON file.
speechEncoderPath: Path to the speech encoder ONNX model.
embedTokensPath: Path to the embed tokens ONNX model.
conditionalDecoderPath: Path to the conditional decoder ONNX model.
languageModelPath: Path to the language model ONNX model.
referenceAudio: Float32Array of reference audio samples for voice cloning. See Reference Audio Guidelines below.
lazySessionLoading: (optional) Boolean to defer ONNX session creation until first use. Defaults to true on iOS and Android, false on all other platforms.

Reference Audio Guidelines

The quality of the reference audio directly affects voice cloning results. Poor recordings can cause the model to repeat the reference content instead of synthesizing the target text.

Recommended specs:

| Property | Recommendation | |----------|---------------| | Format | WAV (16-bit PCM, mono) | | Sample rate | 24,000 Hz (other rates are resampled automatically) | | Duration | 3--6 seconds of clear speech | | Quality | Uncompressed or high-bitrate source; avoid low-bitrate AAC (e.g. Voice Memos at 64 kbps) | | Content | A single continuous sentence; minimize silence and background noise |

Common pitfalls:

Lossy compression artifacts (AAC, MP3) surviving WAV conversion degrade the speech encoder output
Long recordings (>10s) dilute the voice embedding and slow inference
Background noise or multiple speakers confuse the encoder
Reference content that overlaps with the target text can cause the model to reproduce the reference verbatim

4. Create the `config` obj

The config obj consists of a set of parameters which can be used to tweak the behaviour of the TTS model.

const config = {
  language: 'en',
  useGPU: true,
}

| Parameter | Type | Default | Description | |------------------|---------|---------|------------------------------------------------| | language | string | 'en' | Language code (ISO 639-1 format) | | useGPU | boolean | false | Enable GPU acceleration based on EP provider |

5. Create Model Instance

const model = new ONNXTTS(args, config)

6. Load Model

await model.load()

Optionally you can pass the following parameters to tweak the loading behaviour.

closeLoader?: This boolean value determines whether to close the Data Loader after loading. Defaults to true
reportProgressCallback?: A callback function which gets called periodically with progress updates. It can be used to display overall progress percentage.

For example:

await model.load(false, progress => process.stdout.write(`\rOverall Progress: ${progress.overallProgress}%`))

Progress Callback Data

The progress callback receives an object with the following properties:

| Property | Type | Description | |-----------------------|--------|-----------------------------------------| | action | string | Current operation being performed | | totalSize | number | Total bytes to be loaded | | totalFiles | number | Total number of files to process | | filesProcessed | number | Number of files completed so far | | currentFile | string | Name of file currently being processed | | currentFileProgress | string | Percentage progress on current file | | overallProgress | string | Overall loading progress percentage |

7. Run TTS Synthesis

Pass the text to synthesize to the run method. Process the generated audio output asynchronously:

try {
  const textToSynthesize = 'Hello world! This is a test of the TTS system.'
  let audioSamples = []

  const response = await model.run({
    input: textToSynthesize,
    type: 'text'
  })

  // Process output using callback to collect audio samples
  await response
    .onUpdate(data => {
      if (data.outputArray) {
        // Collect raw PCM audio samples
        const samples = Array.from(data.outputArray)
        audioSamples = audioSamples.concat(samples)
        console.log(`Received ${samples.length} audio samples`)
      }
      if (data.event === 'JobEnded') {
        console.log('TTS synthesis completed:', data.stats)
      }
    })
    .await() // Wait for the entire process to complete

  console.log(`Total audio samples generated: ${audioSamples.length}`)
    
  // audioSamples now contains the complete audio as PCM data (16-bit, 16kHz, mono)
  // You can create WAV files, stream to audio APIs, etc.

  // Access performance stats if enabled
  if (response.stats) {
    console.log(`Inference stats: ${JSON.stringify(response.stats)}`)
  }

} catch (error) {
  console.error('TTS synthesis failed:', error)
}

8. Release Resources

Unload the model when finished:

try {
  await model.unload()
} catch (error) {
  console.error('Failed to unload model:', error)
}

Usage: Supertonic

Supertonic is a diffusion-based TTS engine that uses pre-trained voice styles instead of voice cloning. It produces high-quality speech at 44.1 kHz.

Model Directory Setup

Supertonic expects the following directory layout:

models/supertonic/
├── tokenizer.json
├── onnx/
│   ├── text_encoder.onnx
│   ├── text_encoder.onnx_data
│   ├── latent_denoiser.onnx
│   ├── latent_denoiser.onnx_data
│   ├── voice_decoder.onnx
│   └── voice_decoder.onnx_data
└── voices/
    ├── F1.bin
    ├── F2.bin
    ├── ...
    └── M5.bin

Models can be downloaded from the Hugging Face repository.

Basic Usage (modelDir)

The simplest way to use Supertonic is by passing a modelDir and voiceName. All model file paths are derived automatically from the directory structure.

const path = require('bare-path')
const { ONNXTTS } = require('@qvac/tts-onnx')

const SUPERTONIC_SAMPLE_RATE = 44100

const args = {
  modelDir: path.join(__dirname, 'models', 'supertonic'),
  voiceName: 'F1',
  speed: 1,
  numInferenceSteps: 5,
  opts: { stats: true },
  logger: console
}

const config = {
  language: 'en'
}

const model = new ONNXTTS(args, config)

await model.load()

const response = await model.run({
  input: 'Hello world! This is Supertonic TTS.',
  type: 'text'
})

let audioSamples = []
await response
  .onUpdate(data => {
    if (data && data.outputArray) {
      audioSamples = audioSamples.concat(Array.from(data.outputArray))
    }
  })
  .await()

// audioSamples contains PCM data (16-bit, 44100 Hz, mono)

await model.unload()

Explicit Paths Usage

Alternatively, you can provide explicit paths to each model file instead of using modelDir:

const args = {
  tokenizerPath: '/path/to/tokenizer.json',
  textEncoderPath: '/path/to/onnx/text_encoder.onnx',
  latentDenoiserPath: '/path/to/onnx/latent_denoiser.onnx',
  voiceDecoderPath: '/path/to/onnx/voice_decoder.onnx',
  voicesDir: '/path/to/voices',
  voiceName: 'M1',
  speed: 1.2,
  numInferenceSteps: 10,
  opts: { stats: true },
  logger: console
}

const model = new ONNXTTS(args, { language: 'es' })

Supertonic Args Reference

| Parameter | Type | Default | Description | |-----------|------|---------|-------------| | modelDir | string | - | Base directory containing tokenizer, onnx/, and voices/ subdirectories | | tokenizerPath | string | - | Path to tokenizer.json (auto-derived from modelDir) | | textEncoderPath | string | - | Path to text_encoder.onnx (auto-derived from modelDir) | | latentDenoiserPath | string | - | Path to latent_denoiser.onnx (auto-derived from modelDir) | | voiceDecoderPath | string | - | Path to voice_decoder.onnx (auto-derived from modelDir) | | voicesDir | string | - | Path to directory containing voice .bin files (auto-derived from modelDir) | | voiceName | string | 'F1' | Voice preset name (e.g., 'F1', 'M1') | | speed | number | 1 | Speech speed multiplier (1.0 = normal speed) | | numInferenceSteps | number | 5 | Number of diffusion denoising steps (higher = better quality, slower) | | loader | Loader | - | Optional data loader for streaming model files | | cache | string | '.' | Local directory for caching model files | | opts.stats | boolean | false | Enable inference performance statistics | | logger | object | - | Logger instance for debug output |

Available Voices

Supertonic includes 10 pre-trained voice styles:

| Voice | Gender | Description | |-------|--------|-------------| | F1 | Female | Female voice style 1 (default) | | F2 | Female | Female voice style 2 | | F3 | Female | Female voice style 3 | | F4 | Female | Female voice style 4 | | F5 | Female | Female voice style 5 | | M1 | Male | Male voice style 1 | | M2 | Male | Male voice style 2 | | M3 | Male | Male voice style 3 | | M4 | Male | Male voice style 4 | | M5 | Male | Male voice style 5 |

Output Format

The output is received via the onUpdate callback of the response object. The TTS system provides raw audio data in the form of PCM samples.

Output Events

The system generates different types of events during TTS synthesis:

1. Audio Output Events

When audio data is available, the callback receives raw PCM samples:

// Audio output event - contains only the raw PCM data
{
  outputArray: Int16Array([1234, -567, 890, -123, ...]) // 16-bit PCM samples
}

2. Job Completion Events

When synthesis completes, performance statistics are provided:

// Job completion event - contains performance statistics
{
  totalTime: 0.624621926,              // Total processing time in seconds
  tokensPerSecond: 219.33267837286903, // Processing speed
  realTimeFactor: 0.05818013468703428, // Real-time performance factor. Less than 1 means that streaming is possible
  audioDurationMs: 10736,              // Generated audio duration in milliseconds
  totalSamples: 171776                 // Total number of audio samples generated
}

Audio Format Specifications:

Sample Rate: 24,000 Hz (Chatterbox) or 44,100 Hz (Supertonic)
Format: 16-bit signed PCM, mono channel
Data Type: Int16Array containing raw audio samples

Working with Audio Data

Here's how to collect and process the audio output:

let audioSamples = []

const response = await model.run({
  input: 'Your text to synthesize',
  type: 'text'
})

await response
  .onUpdate(data => {
    if (data.outputArray) {
      // Check if this is an audio output event
      const samples = Array.from(data.outputArray)
      audioSamples = audioSamples.concat(samples)
      console.log(`Received ${samples.length} audio samples`)
    } else {
      // This is a completion event with statistics
      console.log('TTS completed with stats:', data)
    }
  })
  .await()

// audioSamples now contains all PCM samples as 16-bit integers
// Sample rate: 24000 Hz (Chatterbox) or 44100 Hz (Supertonic), mono PCM
console.log(`Total audio samples generated: ${audioSamples.length}`)

Other Examples

Chatterbox TTS - Voice cloning from reference audio (English or multilingual).
Chatterbox Langdetect TTS - Detects the input language with @qvac/langdetect-text, selects the English or multilingual Chatterbox bundle, and synthesizes terminal text.
Supertonic TTS - Pre-trained voice styles (English or multilingual).

# Chatterbox English (uses bundled jfk.wav as reference audio)
bare examples/chatterbox-tts.js english

# Chatterbox English with custom reference audio
bare examples/chatterbox-tts.js english path/to/reference.wav

# Chatterbox Multilingual
bare examples/chatterbox-tts.js multilingual

# Chatterbox with automatic language detection
bare examples/chatterbox-langdetect-tts.js "Hola mundo. Esta demo detecta el idioma automaticamente."

# Supertonic English
bare examples/supertonic-tts.js english

# Supertonic Multilingual
bare examples/supertonic-tts.js multilingual

Tests

# js integration tests
npm run test:integration

# C++ unit tests
npm run test:cpp

# C++ unit tests to collect code coverage
npm run coverage:cpp

Note: Integration tests require model files to be present in the models/ directory.

Glossary

Bare - Small and modular JavaScript runtime for desktop and mobile. Learn more.
QVAC - QVAC is our open-source AI-SDK for building decentralized AI applications.
ONNX - Open Neural Network Exchange is an open format built to represent machine learning models. Learn more.
Chatterbox - A neural text-to-speech system with voice cloning capabilities. Learn more.
Supertonic - A diffusion-based text-to-speech system with pre-trained voice styles. Learn more.
Corestore - Corestore is a Hypercore factory that makes it easier to manage large collections of named Hypercores. Learn more.

Resources

QVAC Examples Repo: https://github.com/tetherto/qvac-examples
ONNX Runtime: https://onnxruntime.ai/
Base ONNX Addon: https://github.com/tetherto/qvac/tree/main/packages/onnx
Chatterbox TTS: https://github.com/ResembleAI/chatterbox
Supertonic TTS: https://huggingface.co/onnx-community/Supertonic-TTS-ONNX

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

License

This project is licensed under the Apache-2.0 License - see the LICENSE file for details.

For questions or issues, please open an issue on the GitHub repository.

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

tts-onnx

Table of Contents

Supported Platforms

TTS Engines

Installation

Prerequisites

Installing the Package

Building from Source

Prerequisites

Building the Addon

Verifying the Build

Downloading Models

Environment Variables

Usage: Chatterbox

1. Import the Model Class

2. Create a Data Loader

3. Create the args obj

Reference Audio Guidelines

4. Create the config obj

5. Create Model Instance

6. Load Model

7. Run TTS Synthesis

8. Release Resources

Usage: Supertonic

Model Directory Setup

Basic Usage (modelDir)

Explicit Paths Usage

Supertonic Args Reference

Available Voices

Output Format

Output Events

1. Audio Output Events

2. Job Completion Events

Working with Audio Data

Other Examples

Tests

Glossary

Resources

Contributing

License

3. Create the `args` obj

4. Create the `config` obj