@zeligate/zeli-avatar

The JavaScript and TypeScript SDK for a Zeli avatar box: real-time, lip-synced talking-head avatars over WebRTC.

It is the sibling of the Python package zeli-avatar. Client construction, connect(), talk(), the event names and the whole management surface line up: every method on client.management here is the same operation under the same name in snake_case there, and both packages' test suites read the other's source and fail if that stops being true.

Two things do not line up yet, and both are listed rather than smoothed over. The Python package raises a dedicated error when a photo avatar is refused for missing consent; this one reports that as an AuthenticationError, which reads as a bad key. And its gateway error is spelled GatewayUnavailable against GatewayUnavailableError here.

• Zero runtime dependencies. A browser already has everything it needs: WebRTC, WebSockets and fetch.
• Runs in Node too, given a WebRTC implementation you pass in (see below).
• Ships types. Written in TypeScript, published with declarations.

Install

npm install @zeligate/zeli-avatar

Quickstart (browser)

<video id="stage" playsinline></video>

import { ZeliClient } from "@zeligate/zeli-avatar";

const client = new ZeliClient({
  serverUrl: "https://your-box.example.com",
  sessionToken: "<minted by your backend, see below>",
});

const { session, muted } = await client.streamToVideoElement("stage");

await session.talk("Hello, welcome to Zeli.");

That is the whole thing: connect, attach, play. streamToVideoElement takes an element, an element id or a CSS selector, sets playsInline so iOS does not go fullscreen, and starts playback.

Check muted. Browsers refuse to start an unmuted video until the visitor has interacted with the page, so on a cold page load the avatar plays silently and muted comes back true. Show a control and restore sound from the click:

if (muted) {
  unmuteButton.hidden = false;
  unmuteButton.onclick = () => session.unmute();
}

This is a state the browser puts you in rather than an error, which is why it is reported instead of thrown. Muting quietly would leave you thinking the audio pipeline was broken.

Getting a session token

Never put a zsk_live_ key in page JavaScript: every visitor can read it, and a full key can create avatars, upload voices and mint more tokens. Mint a short-lived, stream-only token on your server instead. This SDK does it for you:

// your backend, not the browser
import { ZeliClient } from "@zeligate/zeli-avatar";

const admin = new ZeliClient({
  serverUrl: "https://your-box.example.com",
  apiKey: process.env.ZELI_API_KEY,
});

const { token, expiresAt, scope } = await admin.management.createSessionToken({
  expiresInSeconds: 600,
});

Hand token to the page. It expires in minutes and the box rejects every mutating route for it, so a leaked one cannot do damage. expiresInSeconds is clamped by the box, currently to at most 600; leave it out and the box applies its own default.

If you would rather not run the SDK on your backend, the call underneath is a POST to /v1/streaming.create_token with your key in an X-Api-Key header and an empty JSON body, answering { "data": { "token": "zsk_temp_...", "expires_at": ..., "scope": "tts" } }.

Keeping the session alive past the token

A session token lives for minutes. A conversation can easily outlive it, and when it runs out the avatar simply stops answering: the next request is refused and the page is told the API key is wrong, which is both unhelpful and untrue.

The SDK cannot mint a replacement, and that is by design rather than an omission. Minting needs the full API key, and the whole point of a session token is that the full key never reaches the browser. So the renewal has to come from your server, and tokenProvider is how it gets there:

const client = new ZeliClient({
  serverUrl: "https://your-box.example.com",
  sessionToken: firstToken,
  tokenProvider: async () => {
    const response = await fetch("/api/zeli-token");   // YOUR endpoint
    return (await response.json()).token;
  },
});

That endpoint is yours: it calls management.createSessionToken() with the full key, exactly as above, and returns the token to the page. The SDK then calls your provider when the token it holds is within fifteen seconds of expiring, and again if the box refuses it anyway. Both paths are needed: the second covers the case where the browser's clock and the box's clock disagree, which would otherwise strand a session this SDK believed still had time left.

Four things are worth knowing:

• It is optional. Leave it out and the client behaves exactly as it always has.
• It is never called twice at once. Two requests refused at the same moment share one refresh, and both then proceed. Your endpoint is not hammered, and you do not end up with a spare token nobody used.
• A failure is named as yours. If your endpoint is down the SDK throws TokenProviderError, not AuthenticationError, so nobody goes looking at an API key that was never part of the call. The original failure rides along as cause.
• It is refused beside apiKey. A full key does not expire, so there is nothing to refresh, and refreshing one would mean treating the full key and the browser token as the same kind of credential. That pairing throws a ConfigurationError.

Your provider may return the whole minted object rather than a bare string:

tokenProvider: async () => {
  const response = await fetch("/api/zeli-token");
  return await response.json();   // { token, expiresAt, scope }
},

Worth doing. A bare string cannot say when it expires, so the SDK can only react to a refusal; returning the expiry keeps the renewal ahead of the failure for every token after the first, which is the one a long session actually lives on.

client.credential exposes secondsRemaining, presented and refresh() if you want to drive it yourself, for instance to renew before something slow rather than during it.

Listening in

import { ZeliEvent } from "@zeligate/zeli-avatar";

client.on(ZeliEvent.MESSAGE_RECEIVED, (m) => console.log(`${m.role}: ${m.content}`));
// Worth registering: a box that could not prepare your avatar streams a
// stand-in face and says so ONLY here.
client.on(ZeliEvent.SERVER_WARNING, (message) => console.warn("[box]", message));

await session.sendUserMessage("Introduce yourself in one sentence.");
await session.stopStreaming();

Doing it by hand

streamToVideoElement is connect() plus the attach. If you want the session without the element, connect() is unchanged:

const session = await client.connect();
videoEl.srcObject = session.mediaStream as MediaStream;
await videoEl.play();   // you now own the autoplay problem described above

Quickstart (Node)

Node has fetch (18+) and WebSocket (22+), but no WebRTC. Supply one:

npm install @zeligate/zeli-avatar werift

import { RTCPeerConnection } from "werift";
import { ZeliClient, type RTCPeerConnectionLike } from "@zeligate/zeli-avatar";

const client = new ZeliClient({
  // serverUrl omitted: ZELI_SERVER_URL names the box. See below.
  apiKey: process.env.ZELI_API_KEY,
  avatar: { avatarId: "01-presenter-male__confident" },
  peerConnectionFactory: (config) =>
    new RTCPeerConnection(config) as unknown as RTCPeerConnectionLike,
});

Under Node there is no MediaStream class, so session.mediaStream is null and session.tracks is the whole story.

Full examples: examples/browser.ts, examples/node.ts.

Which box am I talking to

There is no default base URL, and that is deliberate. This product is deployed per environment, so a client that picks one for you picks an environment for you, and the only way to find out which is to look at the traffic. Name it one of two ways:

// In the code, when the app knows its own target:
const client = new ZeliClient({ serverUrl: "https://your-box.example.com", sessionToken });

// Or from the environment, when the deployment knows and the code should not:
//   export ZELI_SERVER_URL=https://your-box.example.com
const client = new ZeliClient({ sessionToken });

Say neither and the constructor throws a ConfigurationError naming both options. Say both and they must agree: two different URLs throw rather than one silently beating the other, because the developer who exported ZELI_SERVER_URL to aim a script somewhere else would otherwise be overruled with nothing printed. Whichever way it arrives, the URL must start with http:// or https://, and client.serverUrl reports the one in force.

Pass env to read the variable from somewhere other than process.env, such as a bundler's import.meta.env or a worker's bindings. The variable name is exported as SERVER_URL_ENV_VAR, the resolution as resolveServerUrl, and the Python SDK reads the same variable under the same rules.

What you set per session

Two things are chosen per session:

| You set | Effect | | --- | --- | | avatar.avatarId | Which face renders | | avatar.voiceId | Which voice speaks |

Everything else a persona has, including its system prompt, model, language and render settings, is configured once in the portal rather than per session. Naming one of those in avatar throws a ConfigurationError saying so, rather than being accepted and silently ignored. The list is exported as INERT_AVATAR_FIELDS.

API

new ZeliClient(options)

| option | meaning | | --- | --- | | serverUrl | Base URL of the box, http:// or https://. Required unless ZELI_SERVER_URL is set; see below. | | env | Where to read ZELI_SERVER_URL. Defaults to the process environment. | | apiKey | A full API key (zsk_live_...). Server side only. Omit only for a box running with auth open. | | sessionToken | A short-lived zsk_temp_... token, the browser-safe credential. A bare string, or the whole { token, expiresAt, scope } the mint route returns. Wins over apiKey when both are set, because the only reason to have both is a copy-paste mid-migration and the safer one should win. | | tokenProvider | () => Promise<string> that asks YOUR backend for a fresh session token. Called as expiry approaches and on a refusal, never twice at once. Optional; omitting it leaves today's behaviour unchanged. Refused beside apiKey. See above. | | avatar | { avatarId?, voiceId? }, see above. | | connectTimeoutMs | Connection handshake budget in milliseconds. Default 30 000. | | iceServers | ICE servers for NAT traversal. Defaults to a public STUN server. | | peerConnectionFactory | Supply WebRTC where the runtime has none. | | webSocketFactory | Supply a WebSocket where the runtime has none. | | fetch | Supply fetch, for a proxy or a test. |

Methods: on(event, listener) (returns an unsubscribe function), off, addListener, removeListener, connect(sessionOptions?), streamToVideoElement(target, sessionOptions?), getMessageHistory(), listAvatars(), listVoices(), waitUntilAvatarReady(avatarId, options?). Fields: serverUrl (the box actually in force), management (below) and credential (secondsRemaining, presented, refresh()).

client.management

Everything that is not the live conversation. It takes a long-lived API key, so it belongs on your server and must never reach a browser bundle.

| Operation | What it does | | --- | --- | | createSessionToken({ expiresInSeconds? }) | Mint the browser credential. Returns { token, expiresAt, scope }. | | getSettings() / updateSettings(patch) | Read and merge the box's configuration for this caller. A partial patch: omitted fields keep their value. | | getStatus() | Whether the language model and the voice engine are both ready. | | clearConversation() | Forget this caller's history. Also interrupts a reply in progress. | | listVoices() | Every voice the engine offers. An id from here goes into avatar.voiceId. | | previewVoice(voiceId?) | Audition a voice. Returns { bytes, contentType }. | | createVoice({ file, name }) | Clone a voice from a reference clip of at least 6 seconds. Minutes of work. | | prepareVoice(voiceId) | Ask the engine to make a voice ready to speak. | | listAvatars() | Every avatar, with its tone variants and what is still preparing. | | createAvatar({ file, filename?, tones?, consentToken?, framing?, gestureAmplitude? }) | Create from a portrait or a video. Answers immediately with preparing. | | createPhotoAvatar({ file, avatarId, ... }) | Build from a single portrait and wait for every clip. Gated by a feature flag and a consent check. | | deleteAvatar(avatarId) | Delete an uploaded avatar. Never a stock one. | | listAvatarClips(avatarId) | How each emotional variant is getting on. Sparse: a missing tone was never requested. | | getAvatarClip(avatarId, tone?) | One rendered clip, as MP4 bytes. | | retryAvatarClip(avatarId, tone) | Regenerate one tone. Answers 202; a 409 means one is already running. | | getAvatarPreview(avatarId) | A still frame, as image bytes. | | uploadAudio(file, filename?) | Upload an audio file for the box to play. |

ManagementApi and every option and result type it names are exported from the package entry point, so a consumer can write (api: ManagementApi) => ... rather than reach for any. The Python package offers the same eighteen operations under the same names in snake_case.

Session

| member | meaning | | --- | --- | | sessionId | Box-issued id for this session. | | requestedAvatar | The avatar id this session was opened with. | | substitutedAvatar | The face actually on screen when it is not the one you asked for; null otherwise. | | tones | Tone names this avatar accepts in talk({ tone }). | | tracks / mediaStream | Inbound media. | | isStreaming(), hasControlChannel | State. Note isStreaming is a method, so it takes parentheses. | | sendUserMessage(text) | Through the conversational model; the avatar speaks the reply. | | talk(text, { tone }) | Straight to TTS, bypassing the model and the transcript. | | createTalkMessageStream({ tone }) | Push text incrementally as it is produced. | | interruptPersona(correlationId?) | Barge-in. Pass a correlation id to interrupt one specific message rather than whatever is speaking. | | newConversation() | Close this conversation and start a fresh one, keeping the session live. Returns the new conversation id, or null from a box that does not report one. The persona and system prompt survive: only the turns rotate, so the avatar keeps its character and loses its memory. Clears the local transcript too, so getMessageHistory() agrees with the box. | | waitUntilClosed() | Resolves when the session ends, however it ends. | | stopStreaming() | Tear down. Also wired to Symbol.asyncDispose, so await using works on a runtime that supports it. |

Five of those were named differently in 0.1.0. The old names still work and still compile, so nothing you have already written breaks, but each one is deprecated and will go at 1.0:

| 0.1.0 name | use instead | | --- | --- | | isActive (a getter) | isStreaming() (a method) | | sendMessage | sendUserMessage | | createTalkStream | createTalkMessageStream | | interrupt | interruptPersona | | close | stopStreaming |

Events (ZeliEvent)

CONNECTION_ESTABLISHED, SESSION_READY, TRACK, MESSAGE_RECEIVED, MESSAGE_STREAM_EVENT_RECEIVED, MESSAGE_HISTORY_UPDATED, AVATAR_SPEECH_STARTED, AVATAR_SPEECH_ENDED, TALK_STREAM_INTERRUPTED, EMOTION_DETECTED, SERVER_WARNING, ERROR, CONNECTION_CLOSED.

Every one of those is emitted by a real server event. There is no USER_SPEECH_STARTED, because microphone input is not part of this SDK yet.

Errors

All derive from ZeliError: ConfigurationError, AuthenticationError, ConnectionError, TimeoutError (a ConnectionError, so a broad instanceof ConnectionError arm still catches it), SessionError, GatewayUnavailableError, TokenProviderError.

TokenProviderError is deliberately none of the others. It means YOUR token endpoint failed, so pointing it at the box or at the API key would send you to look at something that was never involved. The original failure is on cause.

Design notes

One session at a time. Each avatar instance serves a single live session. Starting a new one ends any session already running on it, so design for a single concurrent viewer per instance, and run more instances to serve more.

Ending is not always instant. When a session ends without a clean shutdown, the SDK emits CONNECTION_CLOSED and isStreaming() turns false, but detection can take up to about 30 seconds. Handle the gap rather than expecting an immediate callback.

Check what you were given. session.requestedAvatar is what you asked for and session.substitutedAvatar is set when a different face is on screen. session.tones lists the tones this avatar accepts. Both also raise a SERVER_WARNING, so listening for that event is the simplest way to catch either.

Wait for readiness. An avatar may still be preparing when you ask for it. Use client.waitUntilAvatarReady(avatarId), or avatarIsReady(catalogue, id) against the result of listAvatars().

Creating avatars is in this SDK, on the server side. See client.management above for the whole list. It takes a long-lived API key, so it belongs on your server and must never reach a browser bundle. The session client is the half that ships to the browser, on a short-lived session token your server mints.

Not in this SDK yet: microphone input. There is no capture path and no input-audio event, so a voice-in experience needs your own audio pipeline feeding session.talk or a talk stream.

Support

Questions and bug reports: [email protected]

Licence

Apache-2.0. Copyright Zeligate Pty Ltd. See LICENSE.