@livekit/agents-plugin-assemblyai
v1.10.0
Published
AssemblyAI plugin for LiveKit Agents for Node.js
Maintainers
Keywords
Readme
AssemblyAI plugin for LiveKit Agents
Support for Streaming Speech-to-Text and Streaming Text-to-Speech from AssemblyAI.
More information is available in the docs for the STT integration and for AssemblyAI's Streaming TTS.
Installation
pnpm add @livekit/agents-plugin-assemblyaiPre-requisites
You'll need to specify an AssemblyAI API Key. It can be set as environment variable:
ASSEMBLYAI_API_KEY.
Usage
import { voice } from '@livekit/agents';
import * as assemblyai from '@livekit/agents-plugin-assemblyai';
const session = new voice.AgentSession({
stt: new assemblyai.STT({ speechModel: 'universal-3-6-pro', mode: 'balanced' }),
tts: new assemblyai.TTS({
model: 'assemblyai-tts-1',
voice: '49c443a4-8a1d-5f7e-b70b-98bdbd9823c3', // Audrey
}),
// Let AssemblyAI decide when the user has finished speaking.
turnHandling: { turnDetection: 'stt' },
// Use the TTS word timings for the transcript and for what an interrupted reply said.
useTtsAlignedTranscript: true,
// ... llm, vad, etc.
});Speech-to-text
assemblyai.STT streams Universal-3.6 Pro by default (speechModel). Useful options:
mode:"min_latency","balanced"or"max_accuracy".languageCodes: steer transcription toward one or more expected languages ("es"or["en", "es"]).prompt: context about the audio, such as its domain or scenario, to help recognize likely vocabulary (up to 1,750 characters).keytermsPrompt: names and terms the model should get right.
Text-to-speech
assemblyai.TTS streams speech over a pooled WebSocket and cancels in-flight audio when the user
interrupts. Useful options:
voice: a voice's id, from the voice library or a custom (cloned) voice; defaults to Audrey. Slugs such as"aai-linda-3"also work, but can change, while ids don't.language: a code such as"es", set independently of the voice.model: defaults toassemblyai-tts-1.baseUrl: the US data zone by default;wss://tts.eu.assemblyai.comkeeps data in the EU.wordTimestamps: per-word timings (defaulttrue). WithAgentSession({ useTtsAlignedTranscript: true }), LiveKit uses them to sync the transcript and to record an interrupted reply up to the word the user heard.
Call tts.prewarm() to open the session before the first reply.
