@casola/avatar-client

Browser SDK for Casola live-avatar sessions, speaking avatar protocol v2: one WebSocket per session carrying JSON control messages and binary media frames (mic uplink, avatar audio, fMP4 video). Sessions render as MSE video when the box streams video, or as a poster image + PCM audio when it doesn't — the SDK handles both from the same API.

v1 boxes: this SDK speaks protocol v2 only. Against a fleet that still serves the legacy two-socket wire (/mse + /mic_stream), stay on @casola/[email protected].

Install

npm i @casola/avatar-client

Quickstart

import { AvatarSession, connectViaToken } from '@casola/avatar-client';

// Your backend mints the session (POST /api/v1/sessions with protocol_versions: [2])
// and hands the browser the box URL + short-lived session token.
const { connect_url, session_token } = await fetch('/my-backend/start-session').then(r => r.json());

const session = new AvatarSession({
  videoEl: document.querySelector('video#avatar'),
  connect: connectViaToken({ connectUrl: connect_url, sessionToken: session_token }),
  workletUrl: '/mic-worklet.js', // serve dist/worklet/mic-worklet.js from your origin
  callbacks: {
    onStateChange(next) { updateUI(next); },
    onPartial(text)     { console.log('partial:', text); },
    onTurn(t)           { console.log('turn:', t.text); },
    onFirstFrame()      { hideSpinner(); },
    onMicReady()        { showReadyToSpeak(); },
    onAudioBlocked()    { showTapForSound(); }, // iOS refused unmuted playback — media runs muted
    onClose(reason)     { console.log('ended:', reason); },
    onError(err)        { console.error(err); },
  },
});

await AvatarSession.ensureMicPermission();
await session.start();

// Typed turns ride the same session socket.
const turn = await session.sendText('Tell me about your priorities.');
console.log(turn.reply);

// End the session
session.leave();

The worklet file (dist/worklet/mic-worklet.js) must be served from the same origin as the page, or from a URL explicitly allowed by the browser's AudioWorklet loader.

API

connectViaToken(opts)

Returns a ConnectStrategy that points the session at the box's well-known /v2/session endpoint using the short-lived JWT minted by your server.

| Option | Type | Default | |--------|------|---------| | connectUrl | string | — | | sessionToken | string | — | | sessionCapSeconds | number | — (superseded by the box's cap_seconds once connected) |

AvatarSession

new AvatarSession(opts: AvatarSessionOpts)

| Member | Description | |--------|-------------| | .start() | Begin the session: open the session socket, handshake, start streaming. | | .sendText(text) | Send a typed turn over the session socket; resolves with {text, reply} plus optional speech metadata. | | .leave() | End the session and fire onClose('generic'). | | .destroy() | Tear down without callbacks (use in component cleanup). | | .setMuted(muted) | Mute or unmute the mic mid-session (muted frames are sent as silence, keeping timing continuous). | | .enableMic(stream?) | Back the mic channel with a microphone now — the stream given, or one asked of getUserMedia (call from a tap or click). The wire switches from zeroed frames to the microphone without a reconnect; rejects with the raw error when the capture cannot come up, and when the session has no mic channel. | | .micBacked | Whether the frames on the mic channel are a microphone right now. false while a promised permittedStream is pending, after a refusal, after the track ended, in a receive-only session — the channel then carries zeroed frames. | | .setLangs(langs) | Re-pin the ASR recognition language(s) mid-session; [] = auto-detect. | | .setResponseLanguage(lang) | Preferred reply language (BCP-47; '' returns the choice to the model). | | .setRuntimeInstruction(text) | Replace hidden system-level guidance for subsequent turns without generating a reply or transcript entry; an empty string clears it. | | .unmuteAudio() | Unmute avatar audio after onAudioBlocked; call from a tap/click handler. | | .bufferedVoiceMs() | Avatar voice still queued to play, in ms, or null when there is no playout clock to ask (a video session, or one that has not accepted). null is "unknown", not "none". Feeds attachCaptions's remainingVoiceMs. | | .state | Current WidgetState. | | .sessionCapSeconds | The session cap — the box's authoritative cap_seconds once connected. | | .personaKey | The avatar version the box bound, echoed in the handshake (the persona-pinning ack). | | .videoCodec | The downlink video codec the box negotiated — 'h264', 'hevc' or 'av1'; undefined before the handshake and in poster mode. A box that does not negotiate reports 'h264'. | | .negotiated | What the box negotiated for this session — { micCodec, videoCodec, hasVideo, posterMode, features }; null before the accept. | | .stats() | A snapshot of the diagnostic stream: the connect timeline, close code, last keepalive RTT, the negotiated shape and running counters. Safe to call after the session ends. | | AvatarSession.ensureMicPermission() | Request mic permission before start(). | | AvatarSession.primeVideoElement(video) | Call synchronously in the call-button tap handler, before any await: clears WebKit's per-element gesture restrictions so iOS Safari honors the SDK's unmute (otherwise the first call in a fresh browsing context plays muted, and pre-fix rendered as a slideshow). | | AvatarSession.decodableVideoCodecs() | The downlink codecs this browser can decode, which is what videoCodec: 'auto' offers the box. Diagnostics — report it next to .videoCodec to explain where a session landed. [] without MSE. | | AvatarSession.mediaSupported() | false on browsers without MSE. Poster-mode sessions (poster + audio) work regardless — the SDK simply doesn't offer video in the handshake. |

attachDisclosure(target, options)

Populates an application-owned element with the canonical persistent ● REC · AI · name label. The SDK owns the required wording, safe DOM construction, accessibility label, and update lifecycle; your application owns placement and styling through the casola-disclosure class. When recording is disabled, the REC segment is omitted. details accepts one string or an array of strings and always renders them as plain text after the avatar name.

Style hooks are casola-disclosure on the target and casola-disclosure__recording-group, __recording-dot, __recording, __ai, __name, __detail, and __separator on its generated children.

const disclosure = attachDisclosure(document.querySelector('#call-label'), {
  name: 'Mia',
  recording: true,
  details: ['Customer support', 'English'],
});

disclosure.update({ recording: false, visible: true });
disclosure.destroy();

attachSessionControls(target, options)

Populates an application-owned element with optional native mute and hang-up buttons. The SDK owns accessible state, button labels, pending hang-up protection, safe DOM construction, and the update lifecycle. The host owns the behavior through onMutedChange and onHangup, so product-specific cleanup, feedback, reporting, and navigation are not bypassed.

Style hooks are casola-session-controls on the target and casola-session-controls__mute, __mute-label, __hangup, and __hangup-label on generated children. State hooks are [data-muted] and [data-ending] on the target.

const controls = attachSessionControls(document.querySelector('#call-controls'), {
  mute: true,
  hangup: true,
  muted: false,
  labels: { hangup: 'End reading' },
  onMutedChange: (muted) => session.setMuted(muted),
  onHangup: async () => finishProductFlow(),
});

controls.update({ muted: true, muteDisabled: false });
controls.destroy();

Both helpers are protocol-agnostic DOM utilities — they behave identically to 0.1.4/0.1.5.

attachCaptions(target, options)

Populates an application-owned element with the streaming caption surface: the live ASR partial, each settled user turn, and the avatar's reply revealed in step with the utterance that speaks it.

Reply text arrives whole, with no per-word timing, so rendering turn.reply directly either races ahead of the voice or waits on an invented delay. This helper uses what the wire actually provides — speech_start / speech_end and the speechId they share with the turn:

• a reply waits for its speech_start, the real audio onset, when the box marks utterances (startTimeoutMs is the backstop if that marker never lands);
• it does not wait at all when the box has sent no marker this session;
• speech_end retimes the words still queued to finish inside the voice that is left, so the text cannot keep crawling after the speaker has stopped. That marker means the box stopped producing audio, not that the user stopped hearing it, so the budget is the player's own buffered voice via remainingVoiceMs — falling back to the fixed tailMs guess when no supplier is given or it answers null;
• a reply for an utterance that already ended opens straight into that tail pace.

Style hooks are casola-captions on the target and casola-captions__line on each line, with --partial, --user, --reply and --fading modifiers. The target becomes a polite live region (role="log"); a line still arriving is aria-hidden and announces once, whole, when it settles.

let session: AvatarSession | null = null;

const captions = attachCaptions(document.querySelector('#captions'), {
  maxLines: 6,
  holdMs: 2000, // 0 keeps lines until they are trimmed
  fadeMs: 600,
  // Read late: the ribbon is attached before the session it paces against exists.
  remainingVoiceMs: () => session?.bufferedVoiceMs() ?? null,
});

session = new AvatarSession({
  callbacks: {
    onPartial: (text) => captions.partial(text),
    onTurn: (turn) => captions.turn(turn),
    onSpeechStart: (id) => captions.speechStart(id),
    onSpeechEnd: (id) => captions.speechEnd(id),
  },
  // …
});

captions.update({ visible: false });
captions.destroy();

Wiring onSpeechStart / onSpeechEnd is what buys the alignment; without them the helper still renders, it just reveals every reply as soon as it arrives.

captions.line({ text, kind, speaker }) writes a line your application authored — a written fallback when a spoken reply fails, an interstitial — into the same ribbon, with no reveal schedule.

attachSessionUI(container, options)

Mounts the disclosure, controls and captions together, wires them to a session, and decides which states show what. The three helpers stay individually exported; this is composition.

import { attachSessionUI, AvatarSession } from '@casola/avatar-client';
import '@casola/avatar-client/styles.css';

const ui = attachSessionUI(document.querySelector('#call'), {
  name: 'Mia',
  controls: { hangup: false },   // this product has its own end-call button
  onHangup: () => endCallFlow(),
});

const session = new AvatarSession({ videoEl, connect });
ui.bind(session);
session.start();

It owns no layout — it creates one child per part and stops, because a laptop mock, a modal and a 16:9 shadow root place these very differently. It owns no product copy either.

"Live" is not always WidgetState.live. If your product waits for the first frame and the microphone before it considers the call started, say so with visibleWhen, or drive it yourself with ui.setLive(true) / ui.setLive(null).

session.on(event, handler)

Subscribe after construction; the returned function unsubscribes. The constructor callbacks still work and fire first. Events: state, partial, turn, firstFrame, micReady, speechStart, speechEnd, audioFrameSent, audioBlocked, muteChange, micBacking, micLevel, diagnostic, close, error. A throwing handler is caught, so one bad subscriber cannot break the session.

The microphone can arrive late

The mic channel is declared in the hello and stays up for the whole session; what changes is whether a microphone is behind it. Unbacked, the session sends the same zeroed 100 ms frames mute sends, so the box hears silence rather than a stalled clock, and micBacked is false.

• Pass permittedStream as a Promise when the permission prompt is still open at start() (a visitor who has not answered the browser's sheet yet). The session runs unbacked and attaches the stream when it resolves; resolve null to say "no stream, and do not prompt again".
• A refusal, a worklet failure, a track that ends (device unplugged, OS revoked, iOS backgrounded) or an Opus encoder that dies no longer end the session. The channel drops to zeros and micBacking says why (permission, unavailable, unsupported, failed, track_ended, encoder_failed).
• enableMic(stream?) backs the channel later — from a "Turn on microphone" button, say. It rebuilds a dead encoder on the way. micReady fires on every attach.

const session = new AvatarSession({ permittedStream: stillWaitingForTheSheet, videoEl, connect });
session.on('micBacking', ({ backed, reason }) => setMicIndicator(backed, reason));
retryButton.onclick = () => session.enableMic().catch(showMicError);

A quiet microphone costs almost nothing

The hello offers mic_dtx_v1. Where the box grants it (negotiated.micDtx), a silent 100 ms window — a muted or unbacked channel, the pauses between sentences — goes out as an empty frame: same cadence, same seq, 16 bytes instead of ~300 of encoded silence, about 25 kbit/s down to 1.3 while nobody is talking. The box expands it to silence before anything reads it, so its endpointing sees exactly what it would have; the gate's floor sits under every bar the box decides on. Native Opus DTX is deliberately not used — a suppressed packet is a missing window, and a missing window is what a stalled sender looks like. stats().counters.micFramesEmpty counts the empties; micDtx: false never offers the feature.

Microphone level

micLevel reports the input loudness about 20 times a second while a stream backs the channel: { rms, peak }, linear full-scale 0..1, measured on the raw capture samples (20 * log10(rms) is dBFS). It is what a "the mic hears you" meter wants, and — via peak — what noticing an attached microphone that hears nothing at all wants (a headset muted on its own switch, the wrong input device). Zeros while muted; no events while unbacked or while the capture context is not rendering, so treat "no events" as no news rather than as silence. The measurement is skipped entirely when nobody subscribes.

session.on('micLevel', ({ rms }) => meter.style.setProperty('--level', String(Math.min(1, rms * 4))));

AvatarSession.preflight(options?)

Microphone permission, MSE support and the browser gate in one call, before you spend a fleet seat:

const pre = await AvatarSession.preflight();
if (!pre.ok) return showError(myCopy[pre.error.kind] ?? myCopy.generic);
const session = new AvatarSession({ permittedStream: pre.stream ?? undefined, videoEl, connect });

Hold pre.stream and pass it as permittedStream so the session does not prompt twice. pre.video === false means poster mode is the only option here — a degradation, not a failure.

Errors

callbacks.onError and the error event receive an AvatarError: a real Error with a kind and a terminal flag. Branch on kind instead of sniffing DOMException names.

| kind | meaning | |---|---| | mic-permission, mic-unavailable, mic-failed | the microphone — the only kinds "check your mic" is correct for (isMicError). Since 0.9.0 the session does not end on these by itself: the mic channel runs unbacked and micBacking reports why (preflight() and enableMic() still reject with them) | | unsupported-browser | this browser cannot do what the session needs | | connect, handshake | the socket was refused or closed before accept, or the box never completed the handshake | | timeout | a connect watchdog fired: the socket never opened (30 s), or the box accepted and never sent a first video frame (20 s). stage says which ('open', 'first-media'; a slow prewarm is a non-terminal 'prewarm') | | unauthorized, protocol-mismatch, persona-unavailable, capacity, policy | the box refused: close 4001 / 4002 / 4003 / 4004 / 4008 | | server, media | in-band error or a playback hiccup — usually terminal: false |

Treat an unrecognized kind as unknown; the list grows. Errors a timer produced also carry stage (AvatarErrorStage): where in the connect sequence it fired, for your analytics and the support line you show the user. An error a socket close produced carries closeCode (the raw WebSocket code behind the kind), and a server error carries serverCode (the box's in-band error code) — the number and the wire code the flattened message used to lose.

Diagnostics

callbacks.onDiagnostic and the diagnostic event deliver an AvatarDiagnostic: a bounded, operational view of the session the SDK previously kept to itself. Branch on d.type and default-ignore an unrecognized one — the union grows. Every diagnostic carries at and, when you pass sessionId / traceId in the options, those too; none of it goes on the wire.

| type | what it reports | |---|---| | connect_phase | a connect milestone (socket_open, accept, first_frame, first_audio, mic_ready) and its ms from socket creation | | socket_closed | the close code, whether it was afterAccept, and the reason's length (never its text) | | protocol_violation | a malformed or out-of-sequence server message | | server_error | an in-band box error code, and whether it failed a pending sendText (inFlightRequest) | | go_away | the box asked the client to leave, with an optional deadlineS | | session_end | the raw end reason and the EndReason it mapped to | | negotiated | the codecs in force, hasVideo, posterMode, accepted features | | rtt | a keepalive round-trip in ms | | text_failed | a sendText that timed out or hit a dead transport | | playback_rejected | video.play() was rejected — the DOMException name, readyState, muted | | media_error | a SourceBuffer, video-element or codec-support error (source) | | buffer_evicted | MSE evicted buffered media under quota pressure | | stall | the pause watchdog resumed playback (resume / resume_muted) | | mic_context | the mic AudioContext was suspended at start, and whether resume() recovered it | | mic_track | the mic track ended, muted/unmuted, or the device set changed | | mic_backing | the mic channel gained (backed: true) or lost its capture stream, with a bounded reason — the fact that tells a live microphone from zeroed frames | | frame_dropped | the unit assembler discarded a fragment |

const session = new AvatarSession({
  videoEl,
  connect,
  sessionId,          // from your mint — joins a diagnostic to the session
  traceId,            // your support/trace id
  logger: (level, message, detail) => myLog(level, message, detail),
  callbacks: {
    onDiagnostic: (d) => {
      if (d.type === 'socket_closed') report('socket_closed', { code: d.code });
    },
  },
});

// After the call, one snapshot for your analytics row:
const stats = session.stats();   // { connect, closeCode, rttMs, negotiated, counters }

AvatarSessionOpts.logger routes the SDK's own internal logs (the driver, players, mic pipeline, state machine) through one (level, message, detail?) sink. Absent, the SDK logs to a dev-gated console exactly as before.

Muting: user choice vs application suppression

setMuted() is the user's choice. suppressMic(true) holds the microphone closed without discarding it, and suppressMic(false) restores whatever they had set — use it around app-driven turns rather than setMuted(true) … setMuted(previous), which loses the user's intent whenever the two interleave. Read userMuted / micSuppressed / micMuted, or subscribe to muteChange.

Default styles

import '@casola/avatar-client/styles.css';        // ordinary pages
import { adoptSessionUIStyles } from '@casola/avatar-client';
adoptSessionUIStyles(myShadowRoot);               // shadow roots, which a stylesheet never reaches

Theme with custom properties on the container rather than overriding rules — --casola-accent, --casola-caption-font, --casola-caption-bg, --casola-caption-blur, --casola-caption-max-height, --casola-speaker-*, --casola-control-*, --casola-disclosure-*, --casola-rec-color.

Key types

WidgetState

'idle' | 'selecting' | 'verifying' | 'waiting' | 'ready' | 'connecting' | 'live' | 'ended' | 'error'

EndReason

'cap' | 'edge_disconnect' | 'kicked' | 'expired' | 'dropped' | 'generic'

AvatarSessionOpts

{
  videoEl: HTMLVideoElement;
  connect: ConnectStrategy;
  langs?: string[];           // ASR language pin; [] / omitted = auto-detect
  responseLanguage?: string;  // preferred reply language (BCP-47)
  workletUrl?: string;        // default '/mic-worklet.js'
  mic?: boolean;              // default true; false = receive-only (no mic channel at all, text via sendText)
  permittedStream?: MediaStream | Promise<MediaStream | null>;
                              // from ensureMicPermission(), avoids a second prompt; a Promise = a
                              // prompt still open (unbacked until it resolves; null = do not prompt)
  micCodec?: 'auto' | 'pcm16'; // default 'auto': Opus when WebCodecs + the box allow it, else pcm16
  micDtx?: boolean;           // default true: offer mic_dtx_v1 — silent windows go out as empty
                              // frames where the box grants it; false never offers it
  videoCodec?: 'auto' | 'h264'; // default 'auto': offer every codec this browser decodes (AV1 /
                              // HEVC / H.264) and let the box pick; 'h264' offers none
  prewarm?: () => Promise<void> | void;
  dev?: boolean;              // log unexpected state transitions + protocol violations
  sessionId?: string;         // stamped on every diagnostic; never on the wire
  traceId?: string;           // your support/trace id, stamped on every diagnostic
  logger?: (level: 'debug' | 'warn', message: string, detail?: object) => void;
  callbacks?: { ... };        // see AvatarSessionOpts for the full set, incl.
                              // onSpeechStart/onSpeechEnd and onDiagnostic
}

Migrating from 0.1.x

0.2.0 replaces the two-socket v1 wire with the single v2 session socket. Breaking changes:

• Your backend must mint with protocol_versions: [2] and the fleet must contain protocol-v2 boxes; the SDK no longer speaks /mse + /mic_stream.
• connectViaToken lost edgePaths (the path is well-known) — pass sessionCapSeconds as before.
• AvatarSessionOpts lost textTransport (typed turns are always in-band) and lang (use langs / responseLanguage).
• The deprecated onQueueStatus callback was removed as promised in its deprecation note.

attachDisclosure and attachSessionControls are unchanged from 0.1.4/0.1.5 — they render DOM and never touch the wire.

License

MIT © 2026 Casola