Kaltura Avatar SDK

A TypeScript SDK for integrating Kaltura's real-time avatar streaming into your web application.

Features

• 🎭 Simple API - Create sessions and control avatars with just a few lines of code
• 🎥 WebRTC Streaming - Real-time avatar video
• 🗣️ Text-to-Speech - Send text for the avatar to speak, with LLM streaming support
• 🎵 Audio Support - Send MP3 files for the avatar to speak
• 📚 Catalog Service - Browse available visuals and voices programmatically
• 🌍 Multi-Language TTS - 33 supported languages for text-to-speech

Installation

npm install @unisphere/models-sdk-js

Authentication

You'll need a Kaltura Session (KS) to authenticate with the Avatar API.

Generate a KS: Follow the Kaltura Session Creation Guide

⚠️ Security Warning For production applications, never expose your Kaltura Session in client-side code. Use the backend-created session pattern where the KS stays on your server.

Quick Start

The SDK supports two initialization patterns depending on where the session is created.

Option 1: Frontend-Only (demos, prototypes)

The SDK creates the session using your Kaltura Session directly from the browser.

import { KalturaAvatarSession } from '@unisphere/models-sdk-js';

const session = new KalturaAvatarSession({
  apiKey: 'your-kaltura-session',
  baseUrl: 'https://api.avatar.us.kaltura.ai/',
});

await session.createSession({
  visualId: 'visual-123',
  voiceId: 'voice-456',           // optional
  language: 'en',                 // optional TTS language
  videoContainerId: 'avatar-container', // auto-attaches video element
});

await session.sayText('Hello from Kaltura Avatar!');
await session.endSession();

Option 2: Backend-Created Session (recommended for production)

Your backend creates the session and passes the credentials to the frontend. The Kaltura Session never leaves your server.

Backend (Node.js):

const response = await fetch('https://api.avatar.us.kaltura.ai/v1/avatar-session/create', {
  method: 'POST',
  headers: {
    Authorization: `ks ${process.env.AVATAR_KS}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    clientId: 'kaltura-avatar-sdk',
    visualConfig: { id: 'visual-123' },
    voiceConfig: { id: 'voice-456', modelId: 'eleven_flash_v2_5', language: 'en' },
  }),
});

const { sessionId, token } = await response.json();
// Send sessionId and token to the frontend

Frontend:

import { KalturaAvatarSession } from '@unisphere/models-sdk-js';

const { sessionId, token } = await fetch('/api/create-avatar-session').then(r => r.json());

// No apiKey needed — backend already created the session
const session = new KalturaAvatarSession({
  baseUrl: 'https://api.avatar.us.kaltura.ai/',
});

await session.initSession(
  { sessionId, token },
  { videoContainerId: 'avatar-container' },
);

await session.sayText('Hello from Kaltura Avatar!');
await session.endSession();

HTML Setup (both options):

<div id="avatar-container" style="width: 512px; height: 512px;"></div>

API Reference

Constructor

new KalturaAvatarSession(config?: AvatarConfig)
// Legacy (deprecated): new KalturaAvatarSession(apiKey?: string, config?: AvatarConfig)

Recommended Usage (Config Object):

const session = new KalturaAvatarSession({
  apiKey: 'your-kaltura-session',  // optional, for FE-only flow
  baseUrl: 'https://api.avatar.us.kaltura.ai/',
  logLevel: 'debug',
});

Configuration Options (AvatarConfig):

• apiKey (optional) — Your Kaltura Session (KS). Required for FE-only flow; omit when backend creates sessions.
• baseUrl — Avatar API root URL (e.g., https://api.avatar.us.kaltura.ai/). Defaults to http://localhost:6100/.
• catalogUrl (optional) — Base URL for the Catalog Service. Defaults to baseUrl if not provided.
• logLevel — 'debug' | 'info' | 'warn' | 'error' (default: 'info')
• retryConfig — Retry/backoff settings for API calls
• unloadBeacon — Auto end session on page unload (default: true)

⚠️ Deprecation Notice: Passing apiKey as the first parameter is deprecated. Use the config object pattern instead: new KalturaAvatarSession({ apiKey: '...', ...config }). The legacy signature will continue to work but will log a deprecation warning.

createSession(options)

Creates a session using the SDK (requires apiKey). Establishes WebRTC connection and optionally auto-attaches video.

await session.createSession({
  visualId: 'visual-123',
  voiceId: 'voice-456',           // optional
  language: 'en',                 // optional TTS language
  videoContainerId: 'avatar-container', // optional, auto-attaches video
});

| Parameter | Type | Required | Description | | ------------------ | ------------ | -------- | ----------- | | visualId | string | Yes | ID of the visual to use | | voiceId | string | No | Voice ID for TTS | | language | TTSLanguage | No | TTS language code (e.g., 'en', 'es', 'ja') | | videoContainerId | string | No | DOM element ID for auto-attaching video |

⚠️ Deprecation Notice: avatarId is deprecated in favor of visualId. It still works but will log a warning. It will be removed in a future major version.

initSession(session, options?)

Initializes a pre-created session from the backend. No apiKey required.

await session.initSession(
  { sessionId: 'session-123', token: 'jwt-token' },
  { videoContainerId: 'avatar-container' }, // optional
);

attachAvatar(containerId)

Manually attaches the avatar video to a container div (creates a <video> element inside it).

session.attachAvatar('avatar-container');

sayText(text, turnId?, isFinal?)

Makes the avatar speak the provided text.

// Simple usage
await session.sayText('Hello, how can I help you?');

// LLM streaming — use the same turnId for all chunks, isFinal: true on the last one
const turnId = `turn-${Date.now()}`;
await session.sayText('Hello, ', turnId, false);
await session.sayText('how can I help you?', turnId, true);

| Parameter | Type | Default | Description | | --------- | ------- | ------- | ----------- | | text | string | — | Text to speak | | turnId | string | auto | Identifies a speech turn. Use the same ID for all chunks of one turn. | | isFinal | boolean | true | false signals more chunks are coming; true triggers speech. |

sayAudio(audioFile, turnId, duration)

Makes the avatar speak from an MP3 file or Blob. Audio must be MP3 at 44.1 kHz.

const arrayBuffer = await audioBlob.arrayBuffer();
const audioCtx = new AudioContext();
const decoded = await audioCtx.decodeAudioData(arrayBuffer);
await audioCtx.close();
const duration = decoded.duration;

const audioFile = new File([audioBlob], 'speech.mp3', { type: 'audio/mpeg' });
const turnId = `turn-${Date.now()}`;
await session.sayAudio(audioFile, turnId, duration);

interrupt()

Interrupts the avatar's current speech immediately.

await session.interrupt();

endSession()

Ends the session and cleans up all resources (stops keep-alive, disconnects WebRTC, ends backend session).

await session.endSession();

⚠️ Always call endSession() when you're done. Sessions left open consume backend resources and count against your usage. The SDK automatically sends a best-effort beacon on page unload, but calling endSession() explicitly is the only reliable guarantee.

State Methods

session.getSessionId();      // string | null
session.getSessionState();   // IDLE | CREATING | READY | ENDED | ERROR
session.getConnectionState(); // DISCONNECTED | CONNECTING | CONNECTED | FAILED | CLOSED

Events

session.on('stateChange', (state: SessionState) => { ... });
session.on('connectionChange', (state: ConnectionState) => { ... });
session.on('speakingStart', () => { /* avatar started speaking */ });
session.on('speakingEnd', () =>   { /* avatar stopped speaking */ });
session.on('error', (error: AvatarError) => {
  console.error(error.code, error.message);
});

Catalog Service

Use CatalogService to browse available visuals and voices before creating a session.

import { CatalogService } from '@unisphere/models-sdk-js';

const catalog = new CatalogService({
  catalogUrl: 'https://api.avatar.us.kaltura.ai/',
  apiKey: 'your-kaltura-session',
});

// Fetch all available visuals
const visuals = await catalog.listVisuals();

// Fetch all available voices
const voices = await catalog.listVoices();

// Use in session creation
await session.createSession({
  visualId: visuals[0].itemId,
  voiceId: voices[0].itemId,
  videoContainerId: 'avatar-container',
});

Configuration (CatalogServiceConfig):

| Parameter | Type | Required | Description | | ------------- | ----------- | -------- | ----------- | | catalogUrl | string | Yes | Catalog API root URL | | apiKey | string | Yes | Kaltura Session (KS) | | retryConfig | RetryConfig | No | Retry/backoff settings | | logLevel | string | No | Log level | | timeoutMs | number | No | Request timeout (default: 10000) |

Methods:

• listVisuals(pager?) — Returns Promise<CatalogItem[]> of visual items
• listVoices(pager?) — Returns Promise<CatalogItem[]> of voice items
• listItems(filter?, pager?) — Returns Promise<CatalogListResponse> with filtering support

TTS Languages

Use TTSLanguages for type-safe language codes:

import { TTSLanguages } from '@unisphere/models-sdk-js';

await session.createSession({
  visualId: 'visual-123',
  voiceId: 'voice-456',
  language: TTSLanguages.SPANISH, // 'es'
  videoContainerId: 'avatar-container',
});

Supported languages: English (en), Japanese (ja), Chinese (zh), German (de), Hindi (hi), French (fr), Korean (ko), Portuguese (pt), Italian (it), Spanish (es), Indonesian (id), Dutch (nl), Turkish (tr), Filipino (fil), Polish (pl), Swedish (sv), Bulgarian (bg), Romanian (ro), Arabic (ar), Czech (cs), Greek (el), Finnish (fi), Croatian (hr), Malay (ms), Slovak (sk), Danish (da), Tamil (ta), Ukrainian (uk), Russian (ru), Hungarian (hu), Norwegian (no), Vietnamese (vi).

Complete Example

import { KalturaAvatarSession, SessionState, ConnectionState, AvatarError } from '@unisphere/models-sdk-js';

const session = new KalturaAvatarSession({
  apiKey: 'your-kaltura-session',
  baseUrl: 'https://api.avatar.us.kaltura.ai/',
  logLevel: 'info',
});

session.on('stateChange', (state: SessionState) => {
  console.log('State:', state);
});

session.on('connectionChange', (state: ConnectionState) => {
  console.log('Connection:', state);
});

session.on('speakingStart', () => console.log('Avatar speaking...'));
session.on('speakingEnd', () =>   console.log('Avatar done speaking'));

session.on('error', (error: AvatarError) => {
  console.error('Error:', error.code, error.message);
});

try {
  await session.createSession({
    visualId: 'visual-123',
    voiceId: 'voice-456',
    language: 'en',
    videoContainerId: 'avatar-container',
  });

  console.log('Session ready:', session.getSessionId());

  await session.sayText('Welcome to Kaltura Avatar!');

  // Interrupt if needed
  await session.interrupt();

  // Send an audio file (MP3, 44.1 kHz)
  const turnId = `turn-${Date.now()}`;
  const audioFile = new File([audioBlob], 'speech.mp3', { type: 'audio/mpeg' });
  const duration = 3.0; // seconds — use AudioContext.decodeAudioData() in real usage
  await session.sayAudio(audioFile, turnId, duration);

  await session.endSession();
} catch (error) {
  if (error instanceof AvatarError) {
    console.error('Avatar error:', error.code, error.message);
  }
}

Browser Support

Chrome/Edge 80+, Firefox 75+, Safari 14+. Requires WebRTC.

License

AGPL-3.0