@memberjunction/ai-assemblyai

MemberJunction provider package for AssemblyAI, currently exposing the Voice Agent API (launched April 2026) as a realtime, full-duplex, tool-calling model.

What this package provides

• AssemblyAIRealtime — a BaseRealtimeModel driver (registered via @RegisterClass(BaseRealtimeModel, 'AssemblyAIRealtime')) for AssemblyAI's single-websocket speech-to-speech stack: Universal-3 Pro streaming ASR, server-side turn detection and barge-in, LLM reasoning, JSON-Schema tool calling, and conversational TTS.
• AssemblyAIRealtimeSession — the IRealtimeSession implementation backing the server-bridged topology.

The matching browser-direct client driver (AssemblyAIRealtimeClient, ClassFactory key 'assemblyai') ships in @memberjunction/ai-realtime-client.

Provider characteristics

| Concern | How this provider does it | | --- | --- | | Endpoint | One websocket: wss://agents.assemblyai.com/v1/ws?token=… | | Session config | Native per-session session.update (prompt, tools, voice, turn detection) — no server-side agent object to manage | | Audio | PCM16 mono 24 kHz, base64, both directions (fixed — no negotiation) | | Tool calling | JSON-Schema function tools; tool.call arguments arrive parsed; tool.result takes a JSON string | | Tools mid-session | Mutable — RegisterTools re-declares natively via session.update | | Narration (RequestSpokenUpdate) | Native via reply.create instructions | | Context notes (SendContextNote) | Emulated via the mutable system_prompt ("Background updates" section) | | Typed text (client SendText) | Emulated via reply.create instructions (no typed-input wire event) | | Interruption | reply.done status: 'interrupted' (authoritative); the client driver also flushes on input.speech.started while output is active for snappier barge-in | | Usage events | None — flat hourly session billing ($4.50/hr); OnUsage never fires | | Client-direct | Supported — one-time temp token minted via GET /v1/token (Bearer API key) | | Session end | Close()/Disconnect() send session.end first — skipping it leaves a billable 30-second resume hold |

Configuration

• API key env alias: AI_VENDOR_API_KEY__AssemblyAIRealtime
• RealtimeSessionParams.Config passthrough keys: voice, greeting, turn_detection (object), keyterms (string array)
• Model / APIName plays no wire role (single endpoint, no model selection) — the metadata row uses voice-agent

Docs

• Product: https://www.assemblyai.com/products/voice-agent-api
• API reference: https://www.assemblyai.com/docs/voice-agents/voice-agent-api
• Token endpoint: GET https://agents.assemblyai.com/v1/token?expires_in_seconds=…
• Full MJ realtime architecture (topologies, co-agent model, four-provider capability matrix): guides/REALTIME_CO_AGENTS_GUIDE.md