composite-voice

v0.2.1

Published

2 months ago

Extensible composite AI voice-agent browser SDK

0High
0Medium
0Low

justsomedev

voice ai agent stt tts llm speech transcription synthesis websocket browser sdk

CompositeVoice

An SDK for building AI voice agents — wire together any combination of input, STT, LLM, TTS, and output providers behind one unified 5-role pipeline.

import { CompositeVoice, AnthropicLLM } from 'composite-voice';

// Text agent — NullInput + NullOutput auto-filled
const agent = new CompositeVoice({
  providers: [new AnthropicLLM({ apiKey: 'sk-ant-...', model: 'claude-haiku-4-5-20251001' })],
});

await agent.initialize();
// Add NativeSTT + NativeTTS for voice, or cloud providers for production audio.

Why CompositeVoice?

Building a voice agent from scratch means solving many hard problems simultaneously: microphone capture, WebSocket reconnection, turn-taking logic, interleaving STT/LLM/TTS lifecycles, and stitching together provider SDKs that share nothing in common. It is easy to spend weeks before your first working demo.

CompositeVoice handles the plumbing. You declare the pipeline; the SDK runs it.

| Feature | What it means for you | | ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | 5-role pipeline | Audio flows through 5 roles: input → stt → llm → tts → output. Each role is a pluggable provider. Multi-role providers (e.g., NativeSTT = input+stt) reduce boilerplate. | | Provider-agnostic | Deepgram, AssemblyAI, Soniox, Gladia, Speechmatics, Rev AI, Anthropic, OpenAI, Groq, Gemini, Mistral, ElevenLabs, Cartesia, Speechify, Murf, LMNT, Smallest.ai, Rime, MiniMax, Fish Audio, Google Cloud, Azure Speech, AWS, or browser built-ins — mix and match freely. Swapping a provider is one constructor change. | | Type-safe throughout | Every event payload, config option, and provider interface is fully typed. TypeScript autocomplete works end-to-end. | | Zero-config text agent | Pass an empty providers array (or just an LLM) and the SDK defaults to a text-only agent — AnthropicLLM + NullInput + NullOutput. Add voice providers to progressively enhance. | | Smart text routing | LLM output is split into visual and spoken streams. Code fences are buffered and never sent to TTS. Markdown is stripped for natural speech while the UI gets full formatting. | | Event-driven | Subscribe to any stage of the pipeline: individual transcription words, LLM tokens, TTS audio chunks, queue stats, and state transitions. | | Race-condition-free | Audio frames are buffered in a queue during STT connection. No frames are ever lost, even when the WebSocket handshake takes time. | | Conversation memory | Multi-turn history that grows and trims automatically, included in every LLM call. | | Eager LLM generation | Start generating a response before the user finishes speaking — cuts perceived latency noticeably. | | Server-side proxy | Keep API keys completely off the client. Proxy middleware included for Express, Next.js, and plain Node.js — supports all providers. | | Server-side pipelines | Run the full pipeline in Node.js, Bun, or Deno with BufferInput and NullOutput — no browser APIs required. | | Platform inputs & outputs | Put the agent on a phone call (Twilio, Vonage), in a Discord voice channel or Teams meeting, or listening to Zoom/Google Meet — plus generic WebRTC in/out for LiveKit, Daily, and custom SFUs. | | Agent providers | Collapse STT + LLM + TTS into a single connection. DeepgramAgent, OpenAIRealtimeAgent, GeminiLiveAgent, and ElevenLabsAgent each use one WebSocket — the SDK auto-fills mic input and speaker output. | | Extensible | Abstract base classes for all 5 roles plus BaseAgentProvider for multi-role agents. The OpenAICompatibleLLM base class means any OpenAI-compatible API works out of the box. |

Installation

pnpm add composite-voice

Node.js 18 or later is required.

Most providers use native fetch or native WebSocket — no SDKs to install. Optional peer dependencies:

pnpm add @mlc-ai/web-llm      # WebLLMLLM — in-browser inference (>=0.2.74)
pnpm add ws                   # server-side proxy WebSocket support, Node.js only (>=8.0.0)
pnpm add @msgpack/msgpack     # FishAudioTTS — binary request encoding (>=3.0.0)

Anthropic, OpenAI, Groq, Gemini, Mistral, Deepgram, AssemblyAI, Soniox, Gladia, Speechmatics, Rev AI, ElevenLabs, Cartesia, Speechify, Murf, LMNT, Smallest.ai, Rime, MiniMax, Google Cloud, Azure Speech, and AWS (Amazon Polly, Amazon Transcribe) providers all work with zero peer dependencies.

Quick start

Simplest setup — text agent, zero providers

Pass just an LLM (or even an empty array) and the SDK defaults to a text-only agent. NullInput and NullOutput are auto-filled — no microphone, no speaker. You interact via agent.pushText() and subscribe to LLM events.

import { CompositeVoice, AnthropicLLM } from 'composite-voice';

// Just an LLM — NullInput + NullOutput auto-filled for text-only mode
const agent = new CompositeVoice({
  providers: [new AnthropicLLM({ apiKey: 'sk-ant-...', model: 'claude-haiku-4-5-20251001' })],
});

// Or even zero providers — AnthropicLLM (claude-haiku-4-5) auto-filled too
// const agent = new CompositeVoice({ providers: [] });

agent.on('llm.chunk', (e) => process.stdout.write(e.chunk));
agent.on('agent.stateChange', (e) => console.log('State:', e.state));

await agent.initialize();

Add voice — browser built-ins

Add NativeSTT and NativeTTS for a voice agent using the browser's Web Speech API and SpeechSynthesis. Works in Chrome and Edge.

import { CompositeVoice, NativeSTT, AnthropicLLM, NativeTTS } from 'composite-voice';

const agent = new CompositeVoice({
  providers: [
    new NativeSTT({ language: 'en-US' }),
    new AnthropicLLM({
      apiKey: 'sk-ant-...',
      model: 'claude-haiku-4-5-20251001',
      systemPrompt: 'You are a helpful voice assistant. Keep responses brief.',
      maxTokens: 200,
    }),
    new NativeTTS(),
  ],
});

agent.on('transcription.final', (e) => console.log('You said:', e.text));
agent.on('llm.chunk', (e) => process.stdout.write(e.chunk));
agent.on('agent.stateChange', (e) => console.log('State:', e.state));

await agent.initialize();
await agent.startListening();

See Example 00 for a full runnable demo with UI.

Production setup — cloud providers, auto-filled audio I/O

Supply cloud STT and TTS providers and the SDK auto-fills MicrophoneInput and BrowserAudioOutput for you. Audio is buffered between stages, eliminating the race condition where first frames could be lost during STT WebSocket handshake.

import { CompositeVoice, DeepgramSTT, AnthropicLLM, DeepgramTTS } from 'composite-voice';

// 3-provider config — MicrophoneInput + BrowserAudioOutput auto-filled
const agent = new CompositeVoice({
  providers: [
    new DeepgramSTT({
      apiKey: 'your-deepgram-key',
      options: {
        model: 'nova-3',
        smartFormat: true,
        interimResults: true,
        endpointing: 300,
      },
    }),
    new AnthropicLLM({
      apiKey: 'your-anthropic-key',
      model: 'claude-haiku-4-5-20251001',
      systemPrompt: 'You are a helpful voice assistant. Keep responses brief.',
      maxTokens: 200,
    }),
    new DeepgramTTS({
      apiKey: 'your-deepgram-key',
      options: { model: 'aura-2-thalia-en', encoding: 'linear16', sampleRate: 24000 },
    }),
  ],
});

await agent.initialize();
await agent.startListening();

See Example 20 for the full runnable demo.

5-role pipeline

Every voice agent follows the same 5-stage pipeline:

[InputProvider]  →  InputQueue  →  [STT]  →  [LLM]  →  [TTS]  →  OutputQueue  →  [OutputProvider]
     input             ↑           stt        llm        tts           ↑              output
                  buffers during                                 buffers during
                  STT connection                                 output setup

| Stage | Role | What it does | Example providers | | ----- | -------- | ---------------------------- | ----------------------------------------------- | | 1 | input | Captures audio from a source | MicrophoneInput, BufferInput, NativeSTT | | 2 | stt | Converts audio to text | DeepgramSTT, AssemblyAISTT, NativeSTT | | 3 | llm | Generates a text response | AnthropicLLM, OpenAILLM, WebLLMLLM | | 4 | tts | Converts text to audio | DeepgramTTS, ElevenLabsTTS, NativeTTS | | 5 | output | Plays audio to a destination | BrowserAudioOutput, NullOutput, NativeTTS |

Audio frames are buffered in queues between stages. The input queue holds frames while the STT WebSocket connects, then flushes them in order — no audio is ever lost. The output queue does the same for TTS-to-speaker handoff.

The agent state machine moves through well-defined states — idle -> ready -> listening -> thinking -> speaking — emitting events at every transition. Your UI subscribes to these events; the SDK manages the lifecycle.

Provider roles

Every provider declares a roles property listing which pipeline slots it fills. The SDK resolves the full 5-role pipeline from a flat providers array.

Multi-role providers

Some providers handle multiple roles. NativeSTT manages its own microphone internally (Web Speech API), so it covers both input and stt. NativeTTS manages its own speaker output, so it covers tts and output. These are explicit providers you add when you want voice — they are not auto-filled defaults.

// Voice agent with browser built-ins — NativeSTT covers input+stt, NativeTTS covers tts+output
const agent = new CompositeVoice({
  providers: [
    new NativeSTT(),        // roles: ['input', 'stt']
    new AnthropicLLM({...}), // roles: ['llm']
    new NativeTTS(),        // roles: ['tts', 'output']
  ],
});

Explicit 5-provider config

When using providers that handle a single role each, supply all 5:

// 5-provider config — each role filled by a separate provider
const agent = new CompositeVoice({
  providers: [
    new MicrophoneInput(),        // roles: ['input']
    new DeepgramSTT({...}),       // roles: ['stt']
    new AnthropicLLM({...}),      // roles: ['llm']
    new DeepgramTTS({...}),       // roles: ['tts']
    new BrowserAudioOutput(),     // roles: ['output']
  ],
});

Defaults

When you omit certain roles, the SDK auto-fills them:

input + stt both uncovered → auto-fills with NullInput (text-only, no microphone)
tts + output both uncovered → auto-fills with NullOutput (text-only, no speaker)
llm uncovered → auto-fills with AnthropicLLM (claude-haiku-4-5)
stt provided without input → auto-fills with MicrophoneInput
tts provided without output → auto-fills with BrowserAudioOutput

This means the minimal config is an empty providers array (text-only agent with AnthropicLLM):

// Zero-config — AnthropicLLM + NullInput + NullOutput auto-filled
const agent = new CompositeVoice({ providers: [] });

// Or just an LLM — NullInput + NullOutput auto-filled for text-only mode
const agent = new CompositeVoice({
  providers: [new AnthropicLLM({ apiKey: '...', model: 'claude-haiku-4-5-20251001' })],
});

Providers

Audio Input

| Provider | Environment | Roles | Peer dependency | | ----------------- | ------------- | --------------- | --------------- | | MicrophoneInput | Browser | input | None | | BufferInput | Node/Bun/Deno | input | None | | WebRTCInput | Browser | input | None | | NativeSTT | Browser | input + stt | None |

MicrophoneInput wraps the browser's getUserMedia + AudioContext into a provider. BufferInput accepts pushed ArrayBuffer data for server-side pipelines. WebRTCInput extracts PCM from a remote WebRTC MediaStreamTrack — the building block for joining LiveKit, Daily, or any custom SFU. NativeSTT manages its own microphone internally.

Speech-to-Text (STT)

| Provider | Transport | Browser support | Peer dependency | | ------------------- | --------------------- | ------------------- | --------------- | | NativeSTT | Web Speech API | Chrome, Edge | None | | DeepgramSTT | WebSocket | All modern browsers | None | | DeepgramFlux | WebSocket | All modern browsers | None | | AssemblyAISTT | WebSocket | All modern browsers | None | | ElevenLabsSTT | WebSocket | All modern browsers | None | | SonioxSTT | WebSocket | All modern browsers | None | | GladiaSTT | HTTP init + WebSocket | All modern browsers | None | | SpeechmaticsSTT | WebSocket | All modern browsers | None | | RevAISTT | WebSocket | All modern browsers | None | | OpenAIRealtimeSTT | WebSocket | All modern browsers | None | | GoogleSTT | HTTP (REST, batch) | All modern browsers | None | | AzureSTT | WebSocket | All modern browsers | None | | TranscribeSTT | WebSocket | All modern browsers | None |

All STT providers emit an utteranceComplete: true flag on transcription results to signal when an utterance is ready for LLM processing. This flag is the canonical trigger for LLM generation. The speechFinal event is retained for display purposes but is deprecated as the LLM trigger — use utteranceComplete instead.

NativeSTT options:

new NativeSTT({
  language: 'en-US', // BCP-47 language tag
  continuous: true, // keep listening between pauses
  interimResults: true, // emit partial results while speaking
  startTimeout: 5000, // ms before erroring if no audio detected
});

DeepgramSTT options (V1/Nova):

new DeepgramSTT({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  language: 'en-US',
  options: {
    model: 'nova-3', // nova-3 = best accuracy
    smartFormat: true,
    punctuation: true,
    interimResults: true,
    endpointing: 300, // ms of silence before speech_final fires
    vadEvents: true,
  },
});

DeepgramFlux options (V2/Flux — supports eager LLM):

new DeepgramFlux({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  options: {
    model: 'flux-general-en',
    eagerEotThreshold: 0.5, // enables eager end-of-turn signals
    eotThreshold: 0.7,
  },
});

AssemblyAISTT options:

new AssemblyAISTT({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  language: 'en', // language code
  sampleRate: 16000, // audio sample rate in Hz
  wordBoost: ['CompositeVoice'], // boost recognition of specific words
  interimResults: true, // partial transcripts while speaking
});

ElevenLabsSTT options:

new ElevenLabsSTT({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  model: 'scribe_v2_realtime', // default model
  audioFormat: 'pcm_16000', // audio format
  language: 'en', // BCP 47, ISO 639-1, or ISO 639-3
  commitStrategy: 'vad', // 'vad' (default) or 'manual'
  includeTimestamps: true, // word-level timestamps
});

SonioxSTT options:

new SonioxSTT({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  model: 'stt-rt-v5', // default real-time model
  audioFormat: 'pcm_s16le', // raw format, or 'auto' for container formats
  sampleRate: 16000, // audio sample rate in Hz
  languageHints: ['en', 'es'], // bias recognition (auto-detects 60+ languages)
  enableEndpointDetection: true, // default — finalizes utterances on silence
});

GladiaSTT options:

new GladiaSTT({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  model: 'solaria-1', // default live transcription model
  encoding: 'wav/pcm', // 'wav/pcm' (default), 'wav/alaw', or 'wav/ulaw'
  sampleRate: 16000, // audio sample rate in Hz
  languages: ['en', 'es'], // pin or restrict language detection
  endpointing: 0.3, // seconds of silence before finalizing an utterance
});

SpeechmaticsSTT options:

new SpeechmaticsSTT({
  apiKey: 'your-key', // temporary key (JWT) or async factory; omit and use proxyUrl for server-side key injection
  region: 'eu', // real-time SaaS region ('eu' or 'us')
  language: 'en', // ISO language code (50+ languages)
  audioFormat: 'pcm_s16le', // raw encoding, or 'file' for container formats
  sampleRate: 16000, // audio sample rate in Hz
  operatingPoint: 'enhanced', // accuracy/latency trade-off
  endOfUtteranceSilenceTrigger: 0.75, // default — finalizes utterances on silence
});

RevAISTT options:

new RevAISTT({
  apiKey: 'your-key', // omit and use proxyUrl for server-side token injection
  sampleRate: 16000, // raw audio sample rate in Hz (8000-48000)
  language: 'en', // en, fr, de, it, ja, ko, cmn, pt, es
  filterProfanity: false, // replace profanities with asterisks (English only)
  removeDisfluencies: false, // strip "ums" and "uhs" (English only)
  maxSegmentDurationSeconds: 10, // force final hypotheses every 5-30 s
});

OpenAIRealtimeSTT options:

new OpenAIRealtimeSTT({
  apiKey: 'your-key', // omit and use proxyUrl, or pass an async ephemeral-secret factory
  model: 'gpt-4o-mini-transcribe', // or gpt-4o-transcribe, whisper-1, gpt-realtime-whisper
  language: 'en', // optional ISO 639-1 hint
  turnDetection: { type: 'server_vad' }, // default — commits audio at turn boundaries
  noiseReduction: 'near_field', // optional — 'near_field' or 'far_field'
});

Audio input must be 24 kHz mono PCM — pair with MicrophoneInput({ sampleRate: 24000 }).

GoogleSTT options (batch — transcribes complete recordings, ≤60s per request):

new GoogleSTT({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  language: 'en-US', // BCP-47 language code
  encoding: 'WEBM_OPUS', // 'LINEAR16' | 'FLAC' | 'OGG_OPUS' | 'WEBM_OPUS' | 'MP3' | ...
  sampleRate: 48000, // audio sample rate in Hz
  model: 'latest_short', // 'latest_short' | 'latest_long' | 'telephony' | ...
  enableWordTimeOffsets: true, // word timings in result metadata
});

GoogleSTT is a batch provider: each transcribe(blob) call uploads a complete recording to Google's synchronous speech:recognize endpoint (max 60 seconds / 10 MB) and emits one final, utteranceComplete: true result. There is no live variant because Google's streaming API (StreamingRecognize) is gRPC-only — no public WebSocket endpoint exists. For real-time streaming, use DeepgramSTT, AssemblyAISTT, SonioxSTT, or ElevenLabsSTT.

AzureSTT options:

new AzureSTT({
  apiKey: 'your-key', // string key, or async factory returning a bearer token; omit and use proxyUrl
  region: 'eastus', // required in direct mode — your Speech resource region
  language: 'en-US', // BCP 47 recognition locale
  recognitionMode: 'conversation', // 'conversation' | 'interactive' | 'dictation'
  outputFormat: 'simple', // 'detailed' adds NBest alternatives + confidence
});

TranscribeSTT options (Amazon Transcribe streaming):

new TranscribeSTT({
  credentials: async () => fetchTempCredentials(), // AWS credentials or async factory; omit with proxyUrl
  region: 'us-east-1', // AWS region (required in direct mode)
  languageCode: 'en-US', // language of the input audio
  mediaEncoding: 'pcm', // 'pcm' | 'ogg-opus' | 'flac'
  sampleRate: 16000, // audio sample rate in Hz
  enablePartialResultsStabilization: true, // lower-latency interim results
  partialResultsStability: 'high', // 'high' | 'medium' | 'low'
});

Language Models (LLM)

| Provider | Transport | Peer dependency | Notes | | --------------------- | ----------------- | ----------------- | -------------------------------------------------------- | | AnthropicLLM | HTTP streaming | None | Claude models. Streams by default. Pipeline default. | | OpenAILLM | HTTP | None | GPT models. | | GroqLLM | HTTP | None | Groq-hosted models. Ultra-fast inference. | | GeminiLLM | HTTP | None | Google Gemini models via OpenAI-compatible API. | | MistralLLM | HTTP | None | Mistral open and commercial models. | | WebLLMLLM | In-browser WebGPU | @mlc-ai/web-llm | Fully offline. No API keys. Runs entirely client-side. | | OpenAICompatibleLLM | HTTP | None | Base class — extend for any OpenAI-compatible endpoint. |

AnthropicLLM options:

new AnthropicLLM({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  model: 'claude-haiku-4-5-20251001',
  systemPrompt: 'You are a helpful voice assistant.',
  maxTokens: 200,
  temperature: 0.7,
  stream: true, // default: true
});

OpenAILLM options:

new OpenAILLM({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  model: 'gpt-4o-mini',
  systemPrompt: 'You are a helpful voice assistant.',
  maxTokens: 200,
  temperature: 0.7,
});

GroqLLM options:

new GroqLLM({
  apiKey: 'your-key', // or groqApiKey; omit and use proxyUrl for proxy mode
  model: 'llama-3.3-70b-versatile', // default model
  systemPrompt: 'You are a helpful voice assistant.',
  maxTokens: 200,
});

GeminiLLM options:

new GeminiLLM({
  apiKey: 'your-key', // or geminiApiKey; omit and use proxyUrl for proxy mode
  model: 'gemini-2.0-flash', // default model
  systemPrompt: 'You are a helpful voice assistant.',
  maxTokens: 200,
});

MistralLLM options:

new MistralLLM({
  apiKey: 'your-key', // or mistralApiKey; omit and use proxyUrl for proxy mode
  model: 'mistral-small-latest', // default model
  systemPrompt: 'You are a helpful voice assistant.',
  maxTokens: 200,
});

WebLLMLLM options:

new WebLLMLLM({
  model: 'Llama-3.2-1B-Instruct-q4f16_1-MLC', // runs entirely in-browser via WebGPU
  onLoadProgress: (progress) => console.log(progress), // model download progress
  chatOpts: { context_window_size: 2048 }, // optional model config overrides
});

Text-to-Speech (TTS)

| Provider | Transport | Browser support | Peer dependency | | --------------- | -------------------- | ------------------- | -------------------------------------- | | NativeTTS | SpeechSynthesis API | All modern browsers | None | | DeepgramTTS | WebSocket | All modern browsers | None | | OpenAITTS | HTTP (REST) | All modern browsers | None | | ElevenLabsTTS | WebSocket | All modern browsers | None | | CartesiaTTS | WebSocket | All modern browsers | None | | SpeechifyTTS | HTTP (REST) | All modern browsers | None | | MurfTTS | HTTP (REST) | All modern browsers | None | | LMNTTTS | HTTP (REST) | All modern browsers | None | | SmallestTTS | HTTP (REST) | All modern browsers | None | | RimeTTS | HTTP (REST) | All modern browsers | None | | MiniMaxTTS | HTTP (REST) | All modern browsers | None | | FishAudioTTS | HTTP (REST, msgpack) | All modern browsers | @msgpack/msgpack (optional, >=3.0.0) | | GoogleTTS | HTTP (REST) | All modern browsers | None | | AzureTTS | HTTP (REST) | All modern browsers | None | | PollyTTS | HTTP (REST) | All modern browsers | None |

NativeTTS options:

new NativeTTS({
  rate: 1.0, // speech rate (0.1 – 10)
  pitch: 1.0, // voice pitch (0 – 2)
  volume: 1.0, // volume (0 – 1)
  preferLocal: true, // prefer on-device voices over cloud voices
});

DeepgramTTS options:

new DeepgramTTS({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  options: {
    model: 'aura-2-thalia-en',
    encoding: 'linear16',
    sampleRate: 24000,
  },
});

OpenAITTS options:

new OpenAITTS({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  model: 'tts-1', // 'tts-1' (fast) or 'tts-1-hd' (quality)
  voice: 'alloy', // 'alloy' | 'echo' | 'fable' | 'onyx' | 'nova' | 'shimmer'
  responseFormat: 'mp3', // 'mp3' | 'opus' | 'aac' | 'flac' | 'wav'
  speed: 1.0, // 0.25 – 4.0
});

ElevenLabsTTS options:

new ElevenLabsTTS({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  voiceId: 'your-voice-id', // required — ElevenLabs voice ID
  modelId: 'eleven_turbo_v2_5', // fast low-latency model
  stability: 0.5, // voice stability (0 – 1)
  similarityBoost: 0.75, // similarity boost (0 – 1)
  outputFormat: 'pcm_16000', // 'pcm_16000' | 'pcm_22050' | 'pcm_24000' | 'mp3_44100_128' | ...
});

CartesiaTTS options:

new CartesiaTTS({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  voiceId: 'your-voice-id', // required — Cartesia voice ID
  modelId: 'sonic-2', // 'sonic-2' | 'sonic' | 'sonic-multilingual'
  language: 'en', // language code
  outputSampleRate: 16000, // sample rate in Hz
  speed: 1.0, // speech speed multiplier
  emotion: ['positivity:high'], // emotion tags for voice expression
});

SpeechifyTTS options:

new SpeechifyTTS({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  voiceId: 'geffen_32', // required — Speechify voice ID (catalog or cloned)
  model: 'simba-english', // 'simba-english' | 'simba-multilingual' | 'simba-3.0' | 'simba-3.2'
  audioFormat: 'mp3', // 'mp3' | 'wav' | 'ogg' | 'aac'
  language: 'en-US', // optional — auto-detected when omitted
});

MurfTTS options:

new MurfTTS({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  voiceId: 'en-US-natalie', // required — Murf voice ID from GET /v1/speech/voices
  modelVersion: 'GEN2', // Murf model generation (default)
  format: 'mp3', // 'mp3' | 'wav' | 'flac' | 'alaw' | 'ulaw'
  style: 'Conversational', // optional — per-voice speaking style
  rate: 0, // -50 to 50 speech rate
  pitch: 0, // -50 to 50 voice pitch
  variation: 1, // 0 to 5 — pause/pitch/speed variation
});

LMNTTTS options:

new LMNTTTS({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  voice: 'leah', // required — LMNT voice ID (catalog or cloned)
  model: 'blizzard', // LMNT's current speech model
  format: 'mp3', // 'mp3' | 'wav' | 'aac' | 'ulaw' | 'webm' | 'pcm_s16le' | 'pcm_f32le'
  sampleRate: 24000, // 8000 | 16000 | 24000
  language: 'en', // optional — auto-detected when omitted
  temperature: 0.7, // expressiveness (lower = more neutral)
  topP: 0.9, // stability (lower = more consistent)
});

SmallestTTS options:

new SmallestTTS({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  voiceId: 'meher', // required — Waves voice ID (catalog or cloned)
  model: 'lightning_v3.1', // 'lightning_v3.1' | 'lightning_v3.1_pro'
  outputFormat: 'wav', // 'wav' | 'mp3' | 'pcm' | 'ulaw' | 'alaw'
  sampleRate: 44100, // 8000 | 16000 | 24000 | 44100
  speed: 1.0, // 0.5 – 2.0
  language: 'en', // ISO 639-1 code matching the voice
});

RimeTTS options:

new RimeTTS({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  speaker: 'astra', // required — Rime voice from the per-model catalogs
  model: 'arcana', // 'coda' | 'arcana' | 'arcanav3' | 'arcanav2' | 'mistv3' | 'mistv2'
  audioFormat: 'mp3', // 'mp3' | 'wav' | 'ogg' | 'webm' | 'pcm' | 'mulaw'
  language: 'en', // ISO 639-1 or 639-2/3 code
  samplingRate: 24000, // output sample rate in Hz
});

MiniMaxTTS options:

new MiniMaxTTS({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  voiceId: 'English_expressive_narrator', // required — MiniMax system or cloned voice ID
  model: 'speech-02-hd', // speech-2.8/2.6/02/01, each in -hd and -turbo variants
  audioFormat: 'mp3', // 'mp3' | 'wav' | 'flac' | 'pcm'
  emotion: 'calm', // optional emotion control
  groupId: 'your-group-id', // optional — only for older group-scoped API keys
});

FishAudioTTS options:

Requires the optional peer dependency @msgpack/msgpack (pnpm add @msgpack/msgpack) — Fish Audio's API takes msgpack-encoded request bodies, which also carry binary reference audio for instant voice cloning.

new FishAudioTTS({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  referenceId: 'your-voice-id', // voice model from the Fish Audio catalog (optional)
  model: 's2.1-pro', // 's1' | 's2-pro' | 's2.1-pro' | 's2.1-pro-free' — sent as the `model` HTTP header
  format: 'mp3', // 'mp3' | 'wav' | 'pcm' | 'opus'
  latency: 'balanced', // 'normal' (most stable) | 'balanced' (~300ms time-to-first-audio)
  speed: 1.0, // prosody speed multiplier (0.5 – 2.0)
});

GoogleTTS options:

new GoogleTTS({
  apiKey: 'your-key', // omit and use proxyUrl for server-side key injection
  languageCode: 'en-US', // BCP-47 language/region code
  voiceName: 'en-US-Chirp3-HD-Kore', // Chirp 3: HD, Neural2, Studio, WaveNet, ...
  audioEncoding: 'MP3', // 'MP3' | 'OGG_OPUS' | 'LINEAR16' | 'MULAW' | 'ALAW'
  speakingRate: 1.0, // 0.25 – 4.0
  pitch: 0, // semitones (-20 to +20)
});

AzureTTS options:

new AzureTTS({
  apiKey: 'your-key', // string key, or async factory returning a bearer token; omit and use proxyUrl
  region: 'eastus', // required in direct mode — your Speech resource region
  voiceName: 'en-US-AriaNeural', // required — Azure neural voice
  outputFormat: 'audio-24khz-48kbitrate-mono-mp3', // mp3, riff (wav), ogg/webm opus, raw pcm
  style: 'cheerful', // optional speaking style (voice-dependent)
  rate: 1.1, // optional prosody rate multiplier
});

PollyTTS options (Amazon Polly):

new PollyTTS({
  credentials: async () => fetchTempCredentials(), // AWS credentials or async factory; omit with proxyUrl
  region: 'us-east-1', // AWS region (required in direct mode)
  voiceId: 'Joanna', // required — Polly voice ID
  engine: 'neural', // 'neural' | 'generative' | 'long-form' | 'standard'
  outputFormat: 'mp3', // 'mp3' | 'ogg_vorbis' | 'ogg_opus' | 'pcm'
  textType: 'text', // 'text' | 'ssml'
});

Agent Providers

Agent providers collapse the three middle pipeline roles (stt + llm + tts) into a single connection. Instead of wiring up separate STT, LLM, and TTS providers, one agent provider handles the entire server-side loop over a single WebSocket. The SDK auto-fills MicrophoneInput and BrowserAudioOutput for audio I/O, so a working voice agent requires just one provider in your config.

| Provider | Transport | Roles | Peer Dependency | Description | | --------------------- | --------- | --------------------- | --------------- | ----------------------------------------------------------------------------------------------------- | | DeepgramAgent | WebSocket | stt + llm + tts | None | Deepgram Voice Agent API — configurable listen/think/speak stages, server-side functions | | OpenAIRealtimeAgent | WebSocket | stt + llm + tts | None | OpenAI Realtime API — native speech-to-speech (gpt-realtime), server/semantic VAD, function calling | | GeminiLiveAgent | WebSocket | stt + llm + tts | None | Gemini Live API — native audio dialog over BidiGenerateContent, tool calling | | ElevenLabsAgent | WebSocket | stt + llm + tts | None | ElevenLabs Conversational AI — dashboard-configured agents, overrides, client tools |

// One provider = a complete voice agent (mic + speakers auto-filled)
const voice = new CompositeVoice({
  providers: [
    new OpenAIRealtimeAgent({
      proxyUrl: '/api/proxy/openai-realtime',
      voice: 'marin',
      instructions: 'You are a helpful voice assistant. Keep answers brief.',
    }),
  ],
});

The auto-filled microphone captures at whatever rate the agent expects (24 kHz for OpenAIRealtimeAgent, 16 kHz for the rest). If you supply your own input provider instead, the agent adapts to the rate it reports — resampling or re-labelling the stream where the protocol allows it, and warning where it does not.

All four speak the same SDK surface — transcription.*, llm.*, and tts.* events fire as usual, and each also exposes provider-specific events via onDeepgramAgentEvent / onAgentEvent (live captions, interruptions, latency metrics, tool calls). GeminiLiveAgent authenticates via ?key= in direct mode or the {pathPrefix}/gemini-live proxy route; ElevenLabsAgent accepts a public agentId, a signed URL (or async factory), an xi-api-key, or the {pathPrefix}/elevenlabs proxy route.

Audio Output

| Provider | Environment | Roles | Peer dependency | | -------------------- | ------------- | ---------------- | --------------- | | BrowserAudioOutput | Browser | output | None | | NullOutput | Node/Bun/Deno | output | None | | WebRTCOutput | Browser | output | None | | NativeTTS | Browser | tts + output | None |

BrowserAudioOutput wraps the browser's AudioContext for speaker playback. NullOutput silently discards audio for server-side pipelines. WebRTCOutput renders TTS audio into a publishable WebRTC MediaStreamTrack so the agent's voice can join any WebRTC room. NativeTTS manages its own speaker output internally via the SpeechSynthesis API.

Platform Inputs & Outputs

Platform providers connect the pipeline to call and chat platforms. Duplex providers cover input + output with a single instance; receive-only platforms cover input and pair with NullOutput. Providers that bind to a per-call handle (a Twilio/Vonage socket, Discord voice connection, Zoom RTMS session, WebRTC track) expose attach() — and startListening() accepts that handle directly as a typed optional parameter, so agent.startListening(socket) attaches and starts capture in one call.

| Provider | Environment | Roles | Peer dependency | | ------------------- | ------------- | ---------------------- | ------------------------------------------------------------- | | TwilioMediaStream | Node/Bun/Deno | input + output | None | | VonageAudioSocket | Node/Bun/Deno | input + output | None | | DiscordVoice | Node | input + output | @discordjs/voice, prism-media | | ZoomRtmsInput | Node/Bun/Deno | input (receive-only) | None | | GoogleMeetInput | Browser | input (receive-only) | None | | TeamsCall | Browser | input + output | @azure/communication-calling, @azure/communication-common |

TwilioMediaStream puts the pipeline on a phone call via Twilio Media Streams: your server accepts Twilio's <Connect><Stream> WebSocket and passes each socket to attach(socket) — caller audio arrives as mu-law 8 kHz, and TTS audio is sent back on the same socket (mu-law passthrough, or linear16 auto-converted). VonageAudioSocket does the same for the Vonage Voice API WebSocket endpoint — linear16 at the NCCO-negotiated rate (8/16/24 kHz), with outbound frames paced at 20 ms. DiscordVoice puts the agent in a Discord voice channel: your bot joins with joinVoiceChannel and hands the connection to the provider, which decodes speakers' Opus to PCM for STT and plays TTS back through an AudioPlayer. ZoomRtmsInput streams live Zoom meeting audio (16 kHz PCM, mixed or per-participant) via Realtime Media Streams — RTMS is receive-only by design, so pair it with NullOutput. GoogleMeetInput joins a Meet conference through the Meet Media API (WebRTC, Developer Preview, receive-only) and mixes the conference audio into the pipeline. TeamsCall joins a Microsoft Teams meeting as an external participant via Azure Communication Services interop — meeting audio flows to STT and the agent speaks back through a custom outgoing audio stream.

Provider fallback chains

Voice agents die publicly when one vendor has an outage. FallbackSTT wraps an ordered list of live STT providers and automatically fails over to the next one when the active provider can't connect, times out, or errors mid-session — the pipeline keeps running on the backup:

import { CompositeVoice, FallbackSTT, DeepgramSTT, AssemblyAISTT } from 'composite-voice';

const agent = new CompositeVoice({
  providers: [
    new FallbackSTT([
      new DeepgramSTT({ proxyUrl: '/api/proxy/deepgram' }), // primary
      new AssemblyAISTT({ proxyUrl: '/api/proxy/assemblyai' }), // backup
    ]),
    new AnthropicLLM({ proxyUrl: '/api/proxy/anthropic', model: 'claude-haiku-4-5' }),
    new DeepgramTTS({ proxyUrl: '/api/proxy/deepgram' }),
  ],
});

// Know when a swap happened
agent.on('provider.fallback', ({ from, to, reason }) => {
  console.warn(`STT failed over from ${from} to ${to} (${reason})`);
});

Failover triggers:

| Reason | When | | ------------------- | --------------------------------------------------------------------------------------------------------------------- | | 'init-error' | A provider failed to initialize during agent.initialize() | | 'connect-error' | The active provider's connect() rejected | | 'connect-timeout' | connect() exceeded connectTimeout (default 10 s) | | 'stream-error' | The active provider's stream died mid-session — an unexpected socket close, an error result, or a failing sendAudio |

Behavior details:

A dropped connection is detected immediately. Every live STT provider reports an unexpectedly closed socket as a ws_closed error result, including a clean server-initiated close (quota exhausted, session limit) that raises no error event of its own. Providers do not silently auto-reconnect in the background, so the chain never spends a vendor outage dropping audio into a dead socket.
Audio is buffered during a mid-session swap and replayed to the backup once it connects, so speech during the failover window is not lost. If the buffer overflows, the oldest chunks are dropped so the replay stays contiguous with live audio.
Container headers are re-injected. When the input is a container format (WebM/OGG/WAV), the cached header is replayed to the backup before any buffered audio, so it can demux the stream it joins mid-flight.
Failover is sticky — a failed provider stays out of rotation for the rest of the session (no flapping during a vendor outage). Call fallback.resetToPrimary() between sessions to probe whether the primary has recovered.
The chain only errors when exhausted. If every provider in the chain fails, the normal error events (transcription.error, agent.error) fire.
Input audio format metadata is applied to every provider in the chain (Deepgram, AssemblyAI, Azure, ElevenLabs, Gladia, Speechmatics, Soniox, OpenAI Realtime, Transcribe, and Rev AI), so a backup connects already knowing the encoding/sample rate.
Turn-taking sees through the chain. A chain of full-duplex providers stays full-duplex (barge-in keeps working); it only falls back to pausing capture if a member needs it.

Options: new FallbackSTT(providers, { connectTimeout?: number, debug?: boolean }).

Reusing a chain across agents: onFallback() and setReconnectHeaderSource() each return an unsubscribe function. CompositeVoice calls both when initialize fails and when dispose() runs. The chain's own dispose() drops leftover listener references. Call resetToPrimary() between sessions to put the primary back in rotation.

Constraints: all chained providers must be live (WebSocket) STT providers covering only the stt role — multi-role providers like NativeSTT manage their own microphone and can't be swapped mid-session.

Configuration

Full CompositeVoice configuration reference:

const agent = new CompositeVoice({
  // Required: array of provider instances covering the 5 pipeline roles
  providers: [
    new MicrophoneInput({ sampleRate: 16000 }),
    new DeepgramSTT({ apiKey: '...' }),
    new AnthropicLLM({ apiKey: '...' }),
    new DeepgramTTS({ apiKey: '...' }),
    new BrowserAudioOutput(),
  ],

  // Audio buffer queue tuning (for separate input/STT and TTS/output providers)
  queue: {
    input: { maxSize: 2000, overflowStrategy: 'drop-oldest' },
    output: { maxSize: 500 },
  },

  // Conversation memory (disabled by default)
  conversationHistory: {
    enabled: true,
    maxTurns: 10, // 0 = unlimited; each turn = one user + assistant exchange
    maxTokens: 4000, // approximate token budget (chars/4 heuristic)
    preserveSystemMessages: true, // keep system messages during trimming (default: true)
  },

  // Eager/speculative LLM generation (requires DeepgramFlux)
  eagerLLM: {
    enabled: true,
    cancelOnTextChange: true, // restart generation if the preflight transcript was wrong
    similarityThreshold: 0.8, // word-overlap threshold for accepting speculative response
  },

  // Turn-taking: whether to pause the mic during TTS playback
  turnTaking: {
    pauseCaptureOnPlayback: 'auto', // 'auto' | true | false (default: 'auto')
    autoStrategy: 'conservative', // 'conservative' | 'aggressive' | 'detect' (default: 'conservative')
  },

  // Logging
  logging: {
    enabled: true,
    level: 'info', // 'debug' | 'info' | 'warn' | 'error'
  },

  // LLM→TTS backpressure
  pipeline: {
    maxPendingChunks: 10, // pause LLM if TTS has 10+ unprocessed chunks
  },

  // Guardrails: async filters between LLM output and TTS
  guardrails: {
    filters: [createPIIRedactionGuardrail(), createModerationGuardrail({ moderate })],
    mode: 'streaming', // 'streaming' (low latency) | 'buffered' (absolute blocking)
    timeoutMs: 1000, // per-guardrail time limit
    onError: 'passthrough', // 'passthrough' (fail open) | 'block' (fail closed)
  },

  // WebSocket reconnection (applies to Deepgram providers)
  reconnection: {
    enabled: true, // required — enables reconnection
    maxAttempts: 5,
    initialDelay: 1000, // ms before first retry
    maxDelay: 30000, // cap on retry interval
    backoffMultiplier: 2, // exponential backoff factor
  },

  // Automatic error recovery
  autoRecover: true,

  // Recovery strategy (when autoRecover is true)
  recovery: {
    maxAttempts: 3,
    initialDelay: 1000,
    backoffMultiplier: 2,
    maxDelay: 10000,
  },
});

Events

Subscribe to any part of the voice pipeline with a type-safe event system:

agent.on('event.name', handler); // subscribe
agent.off('event.name', handler); // unsubscribe
agent.once('event.name', handler); // fire once, then auto-unsubscribe

Agent events

| Event | Payload | Description | | ------------------- | ---------------------------------- | -------------------------------------------- | | agent.ready | — | SDK initialized and ready to start listening | | agent.stateChange | { state, previousState } | Agent moved to a new state | | agent.error | { error, recoverable, context? } | System-level error |

Transcription events

| Event | Payload | Description | | --------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------- | | transcription.start | — | Transcription session opened | | transcription.interim | { text, confidence? } | Partial transcript — updates word by word while the user is speaking | | transcription.final | { text, confidence? } | Confirmed transcript segment | | transcription.speechFinal | { text, confidence? } | Full utterance ended. The utteranceComplete flag on the transcription result is what triggers LLM processing. | | transcription.preflight | { text, confidence? } | Early end-of-turn signal (DeepgramFlux only) | | transcription.error | { error } | Transcription error |

LLM events

| Event | Payload | Description | | -------------- | ------------------------ | ---------------------------------- | | llm.start | { prompt } | LLM generation started | | llm.chunk | { chunk, accumulated } | Text token received from the model | | llm.complete | { text, tokensUsed? } | Full response assembled | | llm.error | { error } | LLM error |

TTS events

| Event | Payload | Description | | -------------- | -------------- | ------------------------------------------------------ | | tts.start | { text } | Synthesis started | | tts.audio | { chunk } | Audio chunk ready for playback | | tts.metadata | { metadata } | Audio format metadata | | tts.complete | — | Synthesis complete (playback may still be in progress) | | tts.error | { error } | TTS error |

Guardrail events

| Event | Payload | Description | | ------------------- | ---------------------------------------------------------- | ---------------------------------------------------------------- | | guardrail.applied | { guardrail, stage, original, text, reason?, metadata? } | A guardrail rewrote text on its way to TTS | | guardrail.blocked | { guardrail, stage, original, reason?, metadata? } | A guardrail suppressed text instead of speaking it | | guardrail.error | { guardrail, stage, error, policy } | A guardrail threw or timed out; policy says how it was handled |

See Guardrails for the filter interface and configuration.

Audio events

| Event | Payload | Description | | ---------------------- | ----------- | ---------------------- | | audio.capture.start | — | Microphone opened | | audio.capture.stop | — | Microphone closed | | audio.capture.error | { error } | Capture failure | | audio.playback.start | — | Audio playback started | | audio.playback.end | — | Audio playback ended | | audio.playback.error | { error } | Playback failure |

Queue events

| Event | Payload | Description | | ---------------- | ------------------------------------------------------------------- | ----------------------------------------------- | | queue.overflow | { queueName, droppedChunks, currentSize } | Queue exceeded maxSize, chunks were dropped | | queue.stats | { queueName, size, totalEnqueued, totalDequeued, oldestChunkAge } | Pipeline health snapshot from getQueueStats() |

agent.on('queue.overflow', (e) => {
  console.warn(`Queue "${e.queueName}" dropped ${e.droppedChunks} chunks (size: ${e.currentSize})`);
});

agent.on('queue.stats', (e) => {
  console.log(`Queue "${e.queueName}": ${e.size} buffered, ${e.totalEnqueued} total`);
});

Provider events

| Event | Payload | Description | | ------------------- | ----------------------------------- | -------------------------------------------------------------------------- | | provider.fallback | { role, from, to, reason, error } | A fallback chain swapped to a backup provider |

Turn events

| Event | Payload | Description | | -------------- | --------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- | | turn.metrics | { turnId, transcript, modality, eagerUsed, interrupted, timestamps, durations } | Per-turn latency breakdown — see Latency observability |

VAD events

Emitted when local VAD is enabled.

| Event | Payload | Description | | ----------------- | ----------------------------- | ----------------------------------------------- | | vad.speechStart | { probability } | Local model confirmed the user started speaking | | vad.speechEnd | { durationMs } | Local model detected the user stopped speaking | | vad.bargeIn | { probability, agentState } | Local speech interrupted the agent mid-response |

Latency observability

Every conversation turn emits a turn.metrics event with the full timing breakdown of the pipeline — so you can measure what your users actually experience, tune the eager pipeline, and compare provider choices on real traffic instead of vendor benchmarks:

preflight ──▶ sttFinal ──▶ llmFirstToken ──▶ ttsFirstAudio ──▶ playbackStart ──▶ playbackEnd
 (eager)      (turn t0)                                        ▲
                └──────────────── voiceToVoice ────────────────┘

agent.on('turn.metrics', ({ turnId, durations, eagerUsed, interrupted }) => {
  if (interrupted) return; // barge-in / cancelled turns carry partial data
  console.log(`turn ${turnId}: ${durations.voiceToVoice}ms voice-to-voice`);
  console.log(`  STT final → first LLM token: ${durations.sttFinalToFirstToken}ms`);
  console.log(`  first token → first audio:   ${durations.firstTokenToFirstAudio}ms`);
  console.log(`  first audio → playback:      ${durations.firstAudioToPlayback}ms`);
  if (eagerUsed) console.log('  (eager pipeline was used this turn)');
});

What you get per turn:

timestamps — absolute epoch-ms marks for each captured stage: preflight (early end-of-turn signal, the closest observable proxy for "user stopped speaking"), sttFinal (the turn anchor), llmStart, llmFirstToken, llmComplete, ttsFirstAudio, playbackStart, playbackEnd, turnEnd.
durations — derived phase durations: sttFinalToFirstToken, firstTokenToFirstAudio, firstAudioToPlayback, voiceToVoice (the headline number: STT final → audible playback), llmTotal, turnTotal.
eagerUsed — whether a speculative (preflight-triggered) generation was adopted. When it was, sttFinalToFirstToken can be negative — that negative number is exactly the latency the eager pipeline saved. Compare the distribution with eagerLLM on and off to decide whether it's worth it for your STT provider.
interrupted — turns cut short by barge-in, stopSpeaking(), or a pipeline error still report whatever marks were captured.
Typed turns via sendMessage() are reported too, with modality: 'text'.

Fields are present only when the topology captures them — e.g. multi-role outputs like NativeTTS synthesize and play in one step, with no audio crossing the pipeline, so ttsFirstAudio (and the firstTokenToFirstAudio / firstAudioToPlayback durations) are unavailable there. playbackStart still is: it comes from the output provider's own playback callback.

Note: to observe playback, the SDK registers the output provider's onPlaybackStart / onPlaybackEnd / onPlaybackError callbacks — which hold one callback each. Subscribe to the audio.playback.* events on the agent rather than registering on the provider directly; registering on the provider replaces the SDK's handler and stops both the events and the playback metrics.

Exporting to tracing systems: timestamps are absolute epoch milliseconds (the same clock as every SDK event), so each phase converts directly to a span — no clock translation needed. An OpenTelemetry example:

agent.on('turn.metrics', ({ turnId, timestamps: t, durations }) => {
  const tracer = trace.getTracer('composite-voice');
  const turn = tracer.startSpan('voice.turn', { startTime: t.sttFinal });
  if (t.llmStart && t.llmComplete) {
    tracer
      .startSpan('voice.llm', { startTime: t.llmStart }, trace.setSpan(context.active(), turn))
      .end(t.llmComplete);
  }
  if (t.ttsFirstAudio && t.playbackEnd) {
    tracer
      .startSpan('voice.tts', { startTime: t.ttsFirstAudio }, trace.setSpan(context.active(), turn))
      .end(t.playbackEnd);
  }
  turn.setAttributes({
    'voice.turn_id': turnId,
    'voice.voice_to_voice_ms': durations.voiceToVoice ?? -1,
  });
  turn.end(t.turnEnd);
});

Related observability hooks: queue.stats / queue.overflow for pipeline buffering health, and audio.playback.start / audio.playback.end for raw playback boundaries.

Agent states

The agent moves through a well-defined state machine. Every transition emits an agent.stateChange event so your UI can always reflect what the agent is doing.

idle -> ready -> listening -> thinking -> speaking
                    ^                       |
                    |_______________________|
                              |
                           (error)

| State | Description | | ----------- | ---------------------------------------------------- | | idle | Not yet initialized | | ready | Initialized, waiting to start | | listening | Capturing audio and transcribing | | thinking | LLM is generating a response | | speaking | TTS audio is playing | | error | Recoverable error — call startListening() to retry |

agent.on('agent.stateChange', ({ state, previousState }) => {
  console.log(`${previousState} -> ${state}`);
});

The error state is recoverable. The agent does not shut down on errors — it waits for you to call startListening() again, which lets you add your own retry UI or backoff logic.

Conversation history

Enable multi-turn memory so the LLM remembers previous exchanges within a session. Without this, each user utterance is sent to the LLM in isolation.

const agent = new CompositeVoice({
  providers: [stt, llm, tts],
  conversationHistory: {
    enabled: true,
    maxTurns: 10, // keep last 10 user + assistant pairs; 0 = unlimited
  },
});

Each completed turn is automatically appended and included in the next LLM call:

You:  "My name is Sam."
AI:   "Nice to meet you, Sam!"
You:  "What's my name?"
AI:   "Your name is Sam."   // LLM remembers the earlier exchange

Access and manage history programmatically:

// Retrieve the full conversation as an array of LLMMessage objects
const history = agent.getHistory();

// Wipe the history without reinitializing the agent
agent.clearHistory();

See Example 01 for a demo with a full chat-thread UI.

Eager LLM pipeline

With the DeepgramFlux provider, the SDK can begin LLM generation before the user finishes speaking. DeepgramFlux emits a preflight event — an early end-of-turn prediction — which the SDK uses to speculatively start the LLM. If the final transcript is sufficiently similar to the preflight (based on similarityThreshold), the response continues uninterrupted. If it differs significantly, generation restarts with the correct text.

Enabling eager LLM:

const agent = new CompositeVoice({
  providers: [
    new DeepgramFlux({
      apiKey: 'your-key',
      options: { model: 'flux-general-en', eagerEotThreshold: 0.5 },
    }),
    llm,
    tts,
  ],
  eagerLLM: {
    enabled: true,
    cancelOnTextChange: true, // restart if the preflight text diverged
    similarityThreshold: 0.8, // 80% word overlap required to keep response
  },
});

How it works:

User is still speaking
        |
        v
preflight fires  -->  LLM starts generating (speculative)
        |
        v
speech_final arrives
        |
        +---> text unchanged?  -->  LLM continues streaming uninterrupted
        |
        +---> text changed?    -->  LLM cancelled, restarts with correct text

The result is noticeably lower perceived latency on natural speech patterns where the end of an utterance is predictable. See Example 21 for a demo with real-time pipeline timing.

Smart text routing

LLM responses often contain markdown, code blocks, and formatting that sounds terrible when read aloud by TTS. CompositeVoice automatically splits LLM output into separate visual and spoken streams:

Code fences are buffered entirely and never sent to TTS — no more hearing "opening backtick backtick backtick javascript function hello..."
Markdown is stripped from the spoken stream (headings, bold, links, lists) while the visual stream preserves full formatting for the UI
Partial fences are held in a buffer until the closing fence arrives, so incomplete code blocks never leak to either stream

// Subscribe to the visual stream (full markdown + code) for your UI
voice.o