Kaltura Avatar SDK

A TypeScript SDK for integrating with Kaltura's speech-to-video (STV) avatar service.

Features

• 🎭 Simple API - Create sessions and control avatars with just a few lines of code
• 🎥 WebRTC Streaming - Real-time avatar video via WHEP protocol
• 🗣️ Text-to-Speech - Automatic text chunking for optimal processing
• 🎵 Audio Support - Send audio files for avatar to speak
• 🔄 Auto-Retry - Built-in retry logic with exponential backoff
• 💓 Keep-Alive - Automatic session maintenance every 10 seconds
• 🎬 Auto-Attach - Optional automatic video element creation and attachment
• 🛠️ TypeScript - Full type safety and IntelliSense support
• 📦 Modular - Clean SDK-of-SDKs architecture

Installation

npm install @unisphere/models-sdk-js

Quick Start

Option 1: Auto-Attach Video (Recommended)

import { KalturaAvatarSession } from '@unisphere/models-sdk-js';

// 1. Create SDK instance
const session = new KalturaAvatarSession('your-api-key', {
  baseUrl: 'https://api.avatar.example.com/v1/avatar-session'
});

// 2. Create session with auto-attach
// The SDK will automatically create a video element inside the container
await session.createSession({
  avatarId: 'avatar-123',
  voiceId: 'voice-456', // optional
  videoContainerId: 'avatar-container' // ID of div element
});

// 3. Make avatar speak
await session.sayText('Hello from Kaltura Avatar!');

// 4. End session
await session.endSession();

Option 2: Manual Attach

import { KalturaAvatarSession } from '@unisphere/models-sdk-js';

// 1. Create SDK instance
const session = new KalturaAvatarSession('your-api-key', {
  baseUrl: 'https://api.avatar.example.com/v1/avatar-session'
});

// 2. Create session
await session.createSession({
  avatarId: 'avatar-123',
  voiceId: 'voice-456' // optional
});

// 3. Attach video to container
// The SDK will create a video element inside the specified div
session.attachAvatar('avatar-container');

// 4. Make avatar speak
await session.sayText('Hello from Kaltura Avatar!');

// 5. End session
await session.endSession();

HTML Setup:

<div id="avatar-container" style="width: 512px; height: 512px;"></div>

API Reference

Constructor

new KalturaAvatarSession(apiKey: string, config?: AvatarConfig)

Parameters:

• apiKey - Your Kaltura Avatar API key
• config - Optional configuration:
  • baseUrl - Backend API URL (required for non-default endpoints)
  • iceServers - Custom ICE servers for WebRTC (optional, defaults to backend-provided TURN servers)
  • iceTransportPolicy - 'all' or 'relay' (default: 'all')
  • retryConfig - Retry configuration for API calls and connections
  • logLevel - 'debug' | 'info' | 'warn' | 'error' (default: 'info')

Methods

createSession(options: CreateSessionOptions): Promise<void>

Create a new avatar session and establish WebRTC connection. Optionally auto-attach video to a container.

Options:

• avatarId (required) - ID of the avatar to use
• voiceId (optional) - ID of the voice to use for TTS
• videoContainerId (optional) - ID of the div element to automatically attach video to

// With auto-attach
await session.createSession({
  avatarId: 'avatar-123',
  voiceId: 'voice-456', // optional
  videoContainerId: 'avatar-container' // optional - auto-attach video
});

// Without auto-attach
await session.createSession({
  avatarId: 'avatar-123',
  voiceId: 'voice-456' // optional
});

Note: When a session is created, the SDK automatically:

1. Initializes the backend client
2. Retrieves WHEP URL and TURN server configuration from the backend
3. Establishes WebRTC connection
4. Starts automatic keep-alive (every 10 seconds)
5. Attaches video if videoContainerId is provided

attachAvatar(containerId: string): void

Attach avatar video to a container div by creating a video element inside it. If a video element already exists in the container, it will be reused.

Parameters:

• containerId - ID of the div element to attach video to

// The SDK will create/find a video element inside this div
session.attachAvatar('avatar-container');

HTML:

<div id="avatar-container" style="width: 512px; height: 512px;">
  <!-- Video element will be created here automatically -->
</div>

Note: The created video element will have:

• autoplay and playsinline attributes
• Width and height set to 100%
• Appropriate styling for fullscreen display within the container

sayText(text: string): Promise<void>

Send text for the avatar to speak. Text is automatically chunked into 3-word segments.

await session.sayText('Hello, how are you doing today?');
// Internally split into: ['Hello, how are', 'you doing today']

say(mp3File: File | Blob): Promise<void>

Send audio file for the avatar to speak.

const audioFile = new File([audioBlob], 'speech.mp3', { type: 'audio/mpeg' });
await session.say(audioFile);

interrupt(): Promise<void>

Interrupt the avatar's current speech.

await session.interrupt();

endSession(): Promise<void>

End the session and cleanup resources. This will:

• Stop the automatic keep-alive interval
• Disconnect the WebRTC connection
• End the session on the backend
• Clean up all resources

await session.endSession();

State Management

getSessionId(): string | null

Get the current session ID.

const sessionId = session.getSessionId();
console.log('Session ID:', sessionId);

getSessionState(): SessionState

Get current session state.

const state = session.getSessionState();
// States: IDLE, CREATING, READY, ENDED, ERROR

getConnectionState(): ConnectionState

Get current WebRTC connection state.

const connState = session.getConnectionState();
// States: DISCONNECTED, CONNECTING, CONNECTED, FAILED, CLOSED

Events

on(event: SessionEvent, callback: Function): void

Register event handlers.

// State changes
session.on('stateChange', (state: SessionState) => {
  console.log('Session state:', state);
});

// Connection changes
session.on('connectionChange', (state: ConnectionState) => {
  console.log('Connection state:', state);
});

// Errors
session.on('error', (error: AvatarError) => {
  console.error('Error:', error.message, error.code);
});

Complete Example

import {
  KalturaAvatarSession,
  SessionState,
  ConnectionState,
  AvatarError,
} from '@unisphere/models-sdk-js';

async function runAvatarDemo() {
  // Create SDK instance with custom config
  const session = new KalturaAvatarSession('your-api-key', {
    baseUrl: 'https://api.avatar.example.com/v1/avatar-session',
    logLevel: 'info',
    retryConfig: {
      maxAttempts: 3,
      initialDelayMs: 500,
      maxDelayMs: 5000,
      backoffMultiplier: 2,
    },
  });

  // Setup event listeners
  session.on('stateChange', (state: SessionState) => {
    console.log('Session state changed:', state);
  });

  session.on('connectionChange', (state: ConnectionState) => {
    console.log('Connection state changed:', state);
  });

  session.on('error', (error: AvatarError) => {
    console.error('SDK Error:', error.message);
  });

  try {
    // Create session with auto-attach
    console.log('Creating session...');
    await session.createSession({
      avatarId: 'avatar-123',
      voiceId: 'voice-456',
      videoContainerId: 'avatar-container' // Auto-attach to div
    });

    // Session is now ready, keep-alive started automatically
    console.log('Session ID:', session.getSessionId());
    console.log('Session State:', session.getSessionState());
    console.log('Connection State:', session.getConnectionState());

    // Say some text (automatically chunked)
    await session.sayText('Welcome to Kaltura Avatar! How can I help you today?');

    // Wait for user interaction...
    await new Promise(resolve => setTimeout(resolve, 5000));

    // Interrupt if needed
    await session.interrupt();

    // Say something else
    await session.sayText('Thanks for trying the Kaltura Avatar SDK!');

    // Wait before ending
    await new Promise(resolve => setTimeout(resolve, 3000));

    // End session (automatically stops keep-alive)
    await session.endSession();
    console.log('Session ended successfully');

  } catch (error) {
    if (error instanceof AvatarError) {
      console.error('Avatar Error:', error.code, error.message);
    } else {
      console.error('Unexpected error:', error);
    }
  }
}

// HTML: <div id="avatar-container" style="width: 512px; height: 512px;"></div>
runAvatarDemo();

Architecture

The SDK follows an "SDK of SDKs" pattern with clean separation of concerns:

KalturaAvatarSession (Public API)
├── AvatarControlSDK (HTTP/JWT)
│   ├── Session lifecycle (create, init, end)
│   ├── Avatar control (sayText, say, interrupt)
│   └── Keep-alive management
└── AvatarRTCSDK (WebRTC)
    ├── SignalingManager (WHEP protocol)
    └── PeerConnectionManager (RTCPeerConnection)

Session Initialization Flow

1. Create Session - Establishes backend session and receives JWT token
2. Init Client - Retrieves WHEP URL and TURN server configuration from backend
3. Initialize RTC SDK - Creates RTC SDK with backend-provided configuration
4. Connect WebRTC - Establishes WHEP connection using backend-provided endpoint
5. Start Keep-Alive - Automatically sends keep-alive every 10 seconds
6. Auto-Attach (optional) - Creates video element if videoContainerId provided

Benefits

• Separation of Concerns - Each layer has a single responsibility
• Backend-Managed ICE Servers - TURN configuration provided by backend, not client
• Automatic Keep-Alive - Session maintenance handled automatically
• Testability - Mock and test each layer independently
• Extensibility - Easy to add new features or swap implementations
• Maintainability - Clear boundaries and good TypeScript typing

Advanced Usage

Custom ICE Servers (Override Backend)

By default, ICE servers (TURN configuration) are provided by the backend during initClient. However, you can override them:

const session = new KalturaAvatarSession('your-api-key', {
  baseUrl: 'https://api.avatar.example.com/v1/avatar-session',
  // Override backend-provided ICE servers
  iceServers: [
    { urls: 'stun:stun.example.com:19302' },
    {
      urls: ['turn:turn.example.com:80', 'turn:turn.example.com:443'],
      username: 'user',
      credential: 'pass',
    },
  ],
  iceTransportPolicy: 'relay', // Force TURN only
});

Note: The SDK will fall back to your custom ICE servers only if the backend doesn't provide them.

Using Individual SDKs

For advanced use cases, you can use the underlying SDKs directly:

import {
  AvatarControlSDK,
  AvatarRTCSDK,
} from '@unisphere/models-sdk-js';

// HTTP control
const controlSDK = new AvatarControlSDK({
  baseUrl: 'http://localhost:6100/v1/avatar-session',
  apiKey: 'your-api-key',
  logLevel: 'debug',
});

// Create session
const createRes = await controlSDK.createSession({
  avatarId: 'avatar-123',
  voiceId: 'voice-456',
});

// Initialize client to get WHEP URL and TURN servers
const initRes = await controlSDK.initClient(createRes.sessionId);

// WebRTC - initialized with backend-provided config
const rtcSDK = new AvatarRTCSDK({
  whepUrl: initRes.whepUrl,
  iceServers: [{
    urls: [
      `turn:${initRes.turn.url}:80?transport=udp`,
      `turn:${initRes.turn.url}:443?transport=udp`,
      `turn:${initRes.turn.url}:80?transport=tcp`,
    ],
    username: initRes.turn.username,
    credential: initRes.turn.credential,
  }],
});

await rtcSDK.connect();

Keep-Alive Management

The SDK automatically sends keep-alive signals to the backend every 10 seconds once a session is created. This ensures:

• Session remains active
• Backend doesn't terminate idle sessions
• Resources are properly maintained

// Keep-alive is started automatically when session is READY
await session.createSession({
  avatarId: 'avatar-123',
  videoContainerId: 'avatar-container'
});
// Keep-alive now running every 10 seconds

// Keep-alive is stopped automatically when session ends
await session.endSession();
// Keep-alive stopped

Configuration:

The keep-alive interval is set to 10 seconds and cannot be configured. If you need to manually manage keep-alive, use the AvatarControlSDK directly:

import { AvatarControlSDK } from '@unisphere/models-sdk-js';

const controlSDK = new AvatarControlSDK({
  baseUrl: 'https://api.avatar.example.com/v1/avatar-session',
  apiKey: 'your-api-key',
});

// Manual keep-alive
await controlSDK.keepAlive(sessionId);

Error Handling

All errors are instances of AvatarError with a specific error code:

try {
  await session.sayText('Hello');
} catch (error) {
  if (error instanceof AvatarError) {
    switch (error.code) {
      case AvatarErrorCode.INVALID_STATE:
        console.log('Wrong state - create session first');
        break;
      case AvatarErrorCode.API_REQUEST_FAILED:
        console.log('API request failed - check network');
        break;
      case AvatarErrorCode.RTC_CONNECTION_FAILED:
        console.log('WebRTC connection failed');
        break;
      default:
        console.log('Error:', error.message);
    }
  }
}

Browser Support

• Chrome/Edge 80+
• Firefox 75+
• Safari 14+
• Requires WebRTC support

License

AGPL-3.0

Support

For issues and questions, please visit our GitHub repository.