@demystify/voice-client

Browser/edge client for dmstfy-voice-core voice agents: typed events, WebSocket transport (binary PCM + JSON events), barge-in aware.

import { voiceClient } from "@demystify/voice-client";

const call = await voiceClient.start({ agent: "arkivo-support", baseUrl, token });
call.on("interruption", () => ui.showListening());
call.on("transcript.final", ({ text, role }) => ui.append(role, text));
call.sendAudio(micChunk);
call.end();

Speaker integration (barge-in critical)

Audio arrives faster than real time, so your app keeps a local speaker buffer. On barge-in the server sends playback.clear; the SDK surfaces it as a playback_clear event and your speaker must flush/abort its buffer — the audio it holds belongs to the interrupted utterance and must never play:

const speaker = createPcmSpeaker(); // your AudioWorklet/queue implementation
call.on("audio", ({ data }) => speaker.enqueue(data)); // s16le mono PCM
call.on("playback_clear", () => speaker.flush()); // agent goes silent NOW
call.on("agent_speaking", ({ speaking }) => ui.setTalking(speaking));

The server also paces audio in ~50 ms sub-chunks so the local buffer stays shallow — but without handling playback_clear, an interrupt would still leave up to a sub-chunk (plus your device latency) of stale audio playing.

Events: agent_speaking · user_speaking · interruption · playback_clear · transcript.partial / transcript.final · tool_call · session.ended (+ audio, error).

Auth: voiceClient.start performs the join handshake by sending {"type":"auth","token":...} as the first WebSocket message — the short-lived join token never appears in the URL (URLs leak into proxy/server logs).

Microphone capture, echo cancellation & why WebRTC is v2 (honest notes)

This SDK moves PCM over a WebSocket; capturing the mic is your app's job (getUserMedia + AudioWorklet, downsample to s16le mono 16 kHz). Be clear about what that does and does not give you:

• What you get. getUserMedia({ audio: { echoCancellation: true, noiseSuppression: true, autoGainControl: true } }) enables the browser's AEC/NS/AGC on the capture path. With a headset this is a complete solution and barge-in works exactly as advertised.
• What you do NOT reliably get. Browser AEC needs a reference signal of what the device is playing. That reference is wired up dependably for WebRTC playout; audio you play yourself through the Web Audio API (as a WS client must) is only sometimes included, varying by browser/OS. So in an open-speaker setup the mic can re-capture the agent's own voice, the server VAD hears "speech", and the agent can barge in on itself.
• v1 mitigations, in order of preference: use headsets for demos; keep echoCancellation: true anyway (it often helps, it never hurts); raise the agent's barge_in.max_backchannel_energy / VAD threshold; or gate your sendAudio while agent_speaking is true if your product tolerates half-duplex turn-taking.
• Why WebRTC is v2, not v1. WebRTC buys the standardized AEC-with-playout path (echo-safe speakerphone), jitter buffers + Opus/FEC over UDP instead of TCP head-of-line blocking, and NAT traversal. It also costs an SFU/ICE deployment story and a second transport implementation on both ends. v1 proves the hard part — the interruption pipeline and the client-visible stop — over the simplest transport that can carry it; the WebRTC seams are already reserved (WebRtcTransportOptions here, a Pipecat-based sketch behind the runtime's pipecat extra) so v2 swaps the transport, not the architecture.

WebSocketTransport — exactly what it is

WebSocketTransport (the default) speaks the dmstfy-voice-core control-plane session WebSocket. Facts, from the code on both ends:

• Endpoint. voiceClient.start first calls POST {baseUrl}/v1/sessions on the control plane; the response's join.ws_url is /v1/sessions/{session_id}/ws, and the transport connects to ws(s)://<baseUrl host>/v1/sessions/{session_id}/ws (FastAPI @app.websocket route in the voice-core runtime), optionally with a ?traceparent= query param to continue the session trace.
• Handshake. The first frame the client sends must be the text frame {"type":"auth","token":"<join token>"} — the token is never in the URL. The server closes with 4401 on bad auth, 4404 for an unknown/ended session, and 4409 on a second join (one session = one call; tokens can't be replayed into a live session).
• Framing. Binary frames carry s16le mono PCM audio in both directions (mic upstream via sendAudio, agent audio downstream, surfaced as the audio event). Text frames carry JSON events: upstream {"type":"auth"} and {"type":"end"}; downstream agent_speaking, user_speaking, interruption, playback.clear (emitted to your app as playback_clear), transcript.partial, transcript.final, tool_call, session.ended — typed in VoiceClientEvents.
• Browser-ready: yes. It uses the standard WebSocket API (globalThis.WebSocket, binaryType = "arraybuffer") — no ws npm dependency. It runs unchanged in browsers and in Node ≥ 22 (global WebSocket); an optional makeWebSocket constructor argument injects a custom factory for other runtimes.
• What it does NOT do. It is not a WebRTC media transport: no getUserMedia, no audio capture or playback, no acoustic echo cancellation, no jitter buffer, no automatic reconnection. Your app captures the mic and plays the PCM (see the speaker-integration section above); a dropped socket ends the call (session.ended, reason transport_closed).

MockTransport is the in-memory twin for tests. WebRTC is v2 — WebRtcTransportOptions is reserved, unimplemented. See the module README in modules/dmstfy-voice-core/ for the full picture.

Writing your own Transport

Transport is a public seam (pass it as voiceClient.start({ transport })). The contract, as WebSocketTransport/MockTransport implement it and as Call consumes it:

type TransportMessage =
  | { kind: "audio"; data: ArrayBuffer } // inbound agent audio (s16le PCM)
  | { kind: "event"; event: Record<string, unknown> }; // inbound parsed JSON event

interface Transport {
  connect(url: string): Promise<void>; // resolve when the link is up, reject on failure
  sendAudio(pcm: ArrayBuffer | Uint8Array): void; // outbound mic audio
  sendEvent(event: Record<string, unknown>): void; // outbound JSON control event
  onMessage(fn: (message: TransportMessage) => void): void; // register (additive)
  onClose(fn: () => void): void; // register (additive)
  close(): void; // tear the link down
}

Lifecycle and obligations for an implementation:

1. voiceClient.start calls connect(url) once, then immediately sendEvent({ type: "auth", token }), then constructs the Call (which registers its onMessage/onClose handlers). Your transport must therefore accept handler registration after connect and deliver only messages that arrive after registration (the auth reply and everything else comes later).
2. onMessage/onClose are additive: keep a list, call every registered handler per message, in registration order. They are never unregistered.
3. Inbound binary must be dispatched as { kind: "audio", data: ArrayBuffer }; inbound structured events as { kind: "event", event } with event.type set (that's what Call switches on). Deliver playback.clear immediately — it is latency-critical for barge-in.
4. Fire the onClose handlers exactly once when the link terminates for any reason (local close() included); Call turns that into session.ended with reason transport_closed if the server hadn't already ended the session.
5. sendAudio/sendEvent are only called after connect has resolved (the SDK awaits it). In WebSocketTransport, a send before connect() is a silent no-op (this.ws?.send) and a send after close is discarded by the underlying socket; matching that is sufficient. Reconnection is out of scope for a v1 transport: a session is one call.

MockTransport in src/transport.ts is a complete minimal reference (plus serverEvent/serverAudio test helpers).