@eleven-am/vox-rtc-server

Trusted TypeScript SDK for Vox-hosted WebRTC conversations. It creates sessions over HTTP and controls them over PondSocket, including through the browser gateway.

It is not the SDK for ordinary transcription or synthesis requests.

Install

npm install @eleven-am/vox-rtc-server

Pass apiKey explicitly or set VOX_API_KEY.

Browser application gateway

Mount one WebSocket gateway on the application's HTTP server:

import { createServer } from "node:http";
import express from "express";
import { createVoxRtcGateway } from "@eleven-am/vox-rtc-server";

const app = express();
const server = createServer(app);

const gateway = createVoxRtcGateway({
  voxHttpBase: "http://vox-service.vox.svc.cluster.local:11435",
  apiKey: process.env.VOX_API_KEY,
  path: "/api/vox/rtc",
  onSessionCreated: async ({ request, session }) => {
    await registerCall(request, session.sessionId);
    session.configure({
      sttModel: "parakeet-stt:tdt-0.6b-v3",
      ttsModel: "kokoro-tts:v1.0",
      voice: "af_heart",
      turnProfile: "browser_default",
      speechContext: true,
    });
    session.onTranscript((event) => {
      console.log(event.transcript, event.speechContext);
    });
  },
  onSessionClosed: async ({ request, session, reason }) => {
    await releaseCall(request, session.sessionId, reason);
  },
});

const detachGateway = gateway.attach(server);

The browser needs only:

new VoxRtcBrowserClient({ signalingEndpoint: "/api/vox/rtc" });

The gateway owns its Vox client, controlled sessions, PondSocket connection, and signaling lifecycle. onSessionCreated receives the original incoming request and the complete trusted control session. Applications may inspect the request or ignore it. If the hook throws, the gateway rolls the session back. Closure hooks and session cleanup run exactly once.

On application shutdown:

detachGateway();
await gateway.close();

The browser sees only public ICE/session metadata. The gateway retains the Vox hostname, API key, and internal socket endpoint. Audio remains direct browser-to-Vox WebRTC media.

The gateway preserves the browser client's RTC negotiation generation on the offer, every trickled candidate, and the null end-of-candidates marker. Generation-aware negotiations reject missing or malformed candidate generations before they reach Vox; generation-less legacy negotiation remains supported.

PondSocket control session

Applications that do not need the gateway can use the same server session directly:

import { VoxRtcServerClient } from "@eleven-am/vox-rtc-server";

const vox = new VoxRtcServerClient({
  httpBase: "http://vox-service.vox.svc.cluster.local:11435",
  apiKey: process.env.VOX_API_KEY,
});
const { bootstrap, session } = await vox.createControlledSession();

session.onTranscript((event) => console.log("user said", event.transcript));
session.onBrowserEvent((event) => console.log(event.event, event.payload));
session.sendTextResponse("Hello from the backend.", {
  allowInterruptions: true,
});
session.sendClientEvent({
  event: "render.url",
  payload: { url: "https://example.com" },
});

Applications doing their own signaling can pass the same generation through the direct control API:

session.sendOffer(offer, { generation: 1 });
session.sendIceCandidate(candidate, { generation: 1 });
session.sendIceCandidate(null, { generation: 1 });

sendClientEvent is server-to-browser. Browser-originated data-channel events arrive through onBrowserEvent.

Set speechContext: true in the session configuration to analyze each final user turn for prosody and audio events. The optional result is exposed as event.speechContext on onTranscript; partial transcript events do not carry speech context.

speechContext is the VoxRtcSpeechContext discriminated union. Schema v2 separates speaker analysis from environmental audio:

session.onTranscript(({ speechContext }) => {
  if (!speechContext) return;

  for (const span of speechContext.emotions ?? []) {
    console.log("emotion", span.label, span.startMs, span.endMs);
  }
  for (const span of speechContext.vocal ?? []) {
    console.log("vocal event", span.label, span.startMs, span.endMs);
  }
  for (const sound of speechContext.sounds ?? []) {
    console.log("environment", sound.label, sound.score);
  }

  if (speechContext.status !== "complete") {
    console.log("unavailable tracks", speechContext.unavailable);
  }
});

emotions and vocal contain timestamped speaker spans. sounds contains timestamped environmental events with a score from 0 to 1. A partial result names the unavailable "speaker" or "sounds" track; a failed result names both. Unsupported or malformed speech-context data is omitted without dropping the transcript event.

Generation correlation

startResponse, appendResponseText, commitResponse, cancelResponse, replaceResponseText, and sendTextResponse accept an optional generationId (sent on the wire as generation_id). Response lifecycle events (onResponseCreated, onResponseCommitted, onResponseDone, onResponseCancelled, onResponseAudioClear, onInterruptionDetected, onInterruptionFalsePositive) expose generationId when the server knows it.

Instead of fire-and-forget, gate delta pumping on the start acknowledgement:

const result = await session.startResponseAndWait({
  timeoutMs: 5_000,
  output: {
    model: "qwen3-tts:0.6b-clone",
    voice: "samantha",
    language: "fr",
    speed: 0.9,
    params: { temperature: 0.7 },
  },
});
if (result.accepted) {
  console.log("effective output", result.output);
  session.appendResponseText("Hello.", { generationId: result.generationId });
  session.commitResponse({ generationId: result.generationId });
} else {
  console.warn("start rejected", result.error.code, result.error.message);
}

startResponseAndWait sends response.start with a generationId (generated when not supplied) and resolves on the correlated response.created, on the correlated typed error, or with { accepted: false } and the start_ack_timeout code when no ack arrives within timeoutMs (default 10 000 ms). output is optional. Vox resolves omitted output fields from the session configuration and returns the immutable effective selection in result.output and onResponseCreated.

Error handling

onError events carry message, a stable code, recoverable, and an optional generationId scoping the failure to one response generation. The known codes are exported as VOX_ERROR_CODES (with the VoxErrorCode type and the isVoxErrorCode guard): response_rejected_turn_state, response_rejected_user_speech, response_stale_generation, response_already_active, response_failed, command_invalid, and session_failed.

Only recoverable === false (or the transport itself closing) should be treated as call-ending. Recoverable errors are per-command failures: handle them, abort the affected generation when generationId matches, and keep the session running. Older Vox servers omit code and recoverable; the SDK defaults recoverable to true in that case, so a missing field never ends the call.

session.onError((event) => {
  if (!event.recoverable) {
    endCall(event.message);
    return;
  }
  if (event.generationId) {
    abortGeneration(event.generationId);
  }
});

onSignalingError is a separate, terminal event. Vox emits rtc.signaling_error ({ message, generation }) only when a WebRTC signaling failure ends the session, and closes the session immediately after. The event carries no code and no recoverable field — treat every onSignalingError as call-ending.

Reconnection

The underlying PondSocket client reconnects the socket with exponential backoff and re-joins the control channel automatically after a transient drop: a joined channel moves to a stalled state while the socket is down and re-sends its join request once the socket reconnects. The SDK relies on that behavior rather than tearing the session down. Observe the transitions with onConnectionChange:

const off = vox.onConnectionChange((state) => {
  console.log("Vox control connection", state);
});