@spatialwalk/avatarkit-rtc

Unified RTC adapter for avatarkit - supports LiveKit, Agora and other RTC providers.

📚 Documentation: https://docs.spatialreal.ai/web-sdk/introduction

For server-side RTC SDK integration, please refer to the online documentation.

Installation

⚠️ IMPORTANT: This package does NOT work standalone. You MUST install the dependencies below.

Required Dependencies

| Package | Purpose | Required | |---------|---------|----------| | @spatialwalk/avatarkit | Web SDK for avatar rendering | ✅ Yes | | @spatialwalk/avatarkit-rtc | This package - RTC adapter | ✅ Yes | | livekit-client | LiveKit RTC SDK | ⚡ Choose one | | agora-rtc-sdk-ng | Agora RTC SDK | ⚡ Choose one |

One-line Installation (Recommended)

# With LiveKit
pnpm add @spatialwalk/avatarkit @spatialwalk/avatarkit-rtc livekit-client

# With Agora
pnpm add @spatialwalk/avatarkit @spatialwalk/avatarkit-rtc agora-rtc-sdk-ng

Step-by-step Installation

Step 1: Install the Web SDK (required dependency)

pnpm add @spatialwalk/avatarkit

Step 2: Install the RTC adapter

pnpm add @spatialwalk/avatarkit-rtc

Step 3: Install your chosen RTC provider SDK

# For LiveKit
pnpm add livekit-client

# For Agora
pnpm add agora-rtc-sdk-ng

Common Installation Errors

| Error | Cause | Solution | |-------|-------|----------| | No matching version found for @spatialwalk/avatarkit-rtc@^0.x.x | Wrong version number | Use pnpm add @spatialwalk/avatarkit-rtc (no version) | | No matching version found for @spatialwalk/avatarkit@^0.x.x | Wrong version number | Use pnpm add @spatialwalk/avatarkit (no version) | | Cannot find module '@spatialwalk/avatarkit' | Missing dependency | Install it: pnpm add @spatialwalk/avatarkit | | Avatar placeholder - SDK integration required | SDK not initialized | Call AvatarSDK.initialize() before creating AvatarView |

Get Credentials

Before using the SDK, you need to obtain authentication credentials:

App ID

• Visit the Developer Platform to create your App
• Get your App ID from the platform dashboard
• The App ID is required for SDK initialization

Session Token

• Session Token is required for avatar loading and SDK authentication
• Session Token should be obtained from your backend server
• Your backend server should request the Session Token from the AvatarKit Server
• Session Token has a maximum validity of 1 hour

Authentication Flow

Your Client → Your Backend Server → AvatarKit Server → returns Session Token (max 1 hour)

Note: App ID and Session Token are paired and must be used together. Each App ID corresponds to a specific list of avatars that can be driven.

Quick Start

When initializing @spatialwalk/avatarkit, use DrivingServiceMode.host since network communication is managed by this SDK.

import { AvatarPlayer, LiveKitProvider, AgoraProvider } from '@spatialwalk/avatarkit-rtc';
import { AvatarSDK, AvatarView, AvatarManager, DrivingServiceMode, Environment } from '@spatialwalk/avatarkit';

// 0. Initialize SDK (use host mode)
await AvatarSDK.initialize(appId, {
  environment: Environment.cn,  // or Environment.intl
  drivingServiceMode: DrivingServiceMode.host,
});

// 1. Set session token (required for avatar loading)
AvatarSDK.setSessionToken(sessionToken);

// 2. Create Avatar and AvatarView
const avatar = await AvatarManager.shared.load(characterId);
const avatarView = new AvatarView(avatar, container);

// 3. Create Provider (choose LiveKit or Agora)
const provider = new LiveKitProvider();  // or new AgoraProvider()

// 4. Create Player
const player = new AvatarPlayer(provider, avatarView, {
  logLevel: 'warning',  // optional
});

// 5. Connect to RTC server
await player.connect({
  url: 'wss://your-livekit-server.com',
  token: 'your-token',
  roomName: 'room-name',
});

// 6. Start microphone
await player.startPublishing();

// 7. Stop microphone
await player.stopPublishing();

// 8. Disconnect
await player.disconnect();

API Reference

AvatarPlayer

Main entry class that manages RTC connection and avatar rendering.

Constructor

new AvatarPlayer(provider: RTCProvider, avatarView: AvatarView, options?: AvatarPlayerOptions)

Parameters:

| Parameter | Type | Description | |-----------|------|-------------| | provider | LiveKitProvider \| AgoraProvider | RTC provider instance | | avatarView | AvatarView | AvatarView instance from avatarkit | | options | AvatarPlayerOptions | Optional configuration |

AvatarPlayerOptions

interface AvatarPlayerOptions {
  /** Start speaking transition frames, default 5 (~200ms at 25fps) */
  transitionStartFrameCount?: number;
  
  /** End speaking transition frames, default 40 (~1600ms at 25fps) */
  transitionEndFrameCount?: number;
  
  /** Log level: 'info' | 'warning' | 'error' | 'none', default 'warning' */
  logLevel?: LogLevel;
}

Properties

| Property | Type | Description | |----------|------|-------------| | isConnected | boolean | Whether connected to RTC server |

Methods

connect(config)

Connect to RTC server.

await player.connect(config: RTCConnectionConfig): Promise<void>

LiveKit Config:

interface LiveKitConnectionConfig {
  url: string;      // LiveKit server URL (wss://...)
  token: string;    // Auth token
  roomName: string; // Room name
}

Agora Config:

interface AgoraConnectionConfig {
  appId: string;    // Agora App ID
  channel: string;  // Channel name
  token?: string;   // Auth token (required in production)
  uid?: number;     // User ID (optional, 0 = auto assign)
}

disconnect()

Disconnect RTC connection and clean up all resources.

await player.disconnect(): Promise<void>

reconnect()

Reconnect to RTC server using the last connection configuration. Useful for recovering from connection issues or stream stalls.

await player.reconnect(): Promise<void>

Note: Throws an error if no previous connection exists. Call connect() first.

startPublishing()

Start publishing microphone audio. Automatically requests microphone permission.

await player.startPublishing(): Promise<void>

stopPublishing()

Stop publishing microphone audio and release the device.

await player.stopPublishing(): Promise<void>

publishAudio(track)

Publish custom audio track (advanced usage). For non-microphone audio sources like audio file playback.

await player.publishAudio(track: MediaStreamTrack): Promise<void>

Supported audio sources:

| Source | How to obtain | |--------|---------------| | 🎤 Microphone | Use startPublishing() for convenience | | 🎵 <audio> element | audioElement.captureStream() | | 🖥️ Screen share audio | getDisplayMedia({ audio: true }) | | 🎹 Web Audio API | audioContext.createMediaStreamDestination().stream |

Example:

// Play audio from browser <audio> element
const audioEl = document.querySelector('audio');
const stream = audioEl.captureStream();
await player.publishAudio(stream.getAudioTracks()[0]);

unpublishAudio()

Stop publishing custom audio track.

await player.unpublishAudio(): Promise<void>

Note: unpublishAudio() does not stop the track - caller is responsible for managing track lifecycle:

await player.unpublishAudio();
stream.getTracks().forEach(t => t.stop()); // Clean up externally

getConnectionState()

Get current connection state.

player.getConnectionState(): ConnectionState

ConnectionState enum:

| Value | Description | |-------|-------------| | 'disconnected' | Not connected | | 'connecting' | Connecting | | 'connected' | Connected | | 'reconnecting' | Reconnecting | | 'failed' | Connection failed |

getNativeClient()

Get underlying RTC client object for platform-specific features.

player.getNativeClient(): unknown

Return value:

| Provider | Return type | |----------|-------------| | LiveKit | Room instance (exported as LiveKitRoom) | | Agora | IAgoraRTCClient instance (exported as AgoraClient) |

Type safety notes:

• Direct Provider usage: Returns concrete type, no assertion needed
• Via AvatarPlayer: Returns unknown, requires manual type assertion

Example:

// Method 1: Use Provider directly (recommended, full type hints)
import { LiveKitProvider, LiveKitRoom } from '@spatialwalk/avatarkit-rtc';

const provider = new LiveKitProvider();
const room = provider.getNativeClient();  // Type: LiveKitRoom | null
console.log('Remote participants:', room?.remoteParticipants.size);

// Method 2: Via AvatarPlayer (requires assertion)
import { AvatarPlayer, LiveKitRoom } from '@spatialwalk/avatarkit-rtc';

const room = player.getNativeClient() as LiveKitRoom | null;
console.log('Remote participants:', room?.remoteParticipants.size);

// Agora example
import { AgoraProvider, AgoraClient } from '@spatialwalk/avatarkit-rtc';

const provider = new AgoraProvider();
const client = provider.getNativeClient();  // Type: AgoraClient | null
console.log('Connection state:', client?.connectionState);

on(event, handler)

Listen to events.

player.on(event: string, handler: Function): void

Supported events:

| Event | Callback params | Description | |-------|-----------------|-------------| | 'connected' | () | Connection successful | | 'disconnected' | () | Disconnected | | 'error' | (error: Error) | Error occurred | | 'connection-state-changed' | (state: ConnectionState) | Connection state changed | | 'stalled' | () | Data stream stalled (no frames for 3s). Avatar auto-transitions to idle. |

Example: Handling stream stalls

player.on('stalled', async () => {
  console.log('Stream stalled, attempting reconnection...');
  try {
    await player.reconnect();
  } catch (error) {
    console.error('Reconnection failed:', error);
  }
});

off(event, handler)

Remove event listener.

player.off(event: string, handler: Function): void

LiveKitProvider

LiveKit RTC provider implementation.

import { LiveKitProvider } from '@spatialwalk/avatarkit-rtc';

const provider = new LiveKitProvider();

AgoraProvider

Agora RTC provider implementation.

import { AgoraProvider } from '@spatialwalk/avatarkit-rtc';

const provider = new AgoraProvider({
  debugLogging: true,  // optional: enable debug logs
});

AgoraProviderOptions:

interface AgoraProviderOptions {
  /** Enable verbose debug logging, default false */
  debugLogging?: boolean;
}

Type Guards

Type guard functions for config type checking.

import { isLiveKitConfig, isAgoraConfig } from '@spatialwalk/avatarkit-rtc';

if (isLiveKitConfig(config)) {
  // config is LiveKitConnectionConfig
}

if (isAgoraConfig(config)) {
  // config is AgoraConnectionConfig
}

Native Client Types

SDK exports type aliases for underlying RTC clients, useful when using getNativeClient():

import type { LiveKitRoom, AgoraClient } from '@spatialwalk/avatarkit-rtc';

// LiveKitRoom = Room type from livekit-client
// AgoraClient = IAgoraRTCClient type from agora-rtc-sdk-ng

Log Levels

SDK log level is configured via AvatarPlayerOptions.logLevel:

| Level | Output | |-------|--------| | 'info' | All logs (connection state, frame processing, debug info) | | 'warning' | Warnings + errors (default) | | 'error' | Errors only | | 'none' | Disable all logs |

const player = new AvatarPlayer(provider, avatarView, {
  logLevel: 'info',  // Enable for debugging
});

Complete Example

LiveKit Integration

import { AvatarPlayer, LiveKitProvider } from '@spatialwalk/avatarkit-rtc';
import { AvatarSDK, AvatarView, AvatarManager, DrivingServiceMode, Environment } from '@spatialwalk/avatarkit';

async function initLiveKit() {
  // Initialize SDK (use host mode)
  await AvatarSDK.initialize('your-app-id', {
    environment: Environment.cn,
    drivingServiceMode: DrivingServiceMode.host,
  });

  // Set session token (required)
  AvatarSDK.setSessionToken('your-session-token');

  // Create Avatar
  const avatar = await AvatarManager.shared.load('character-id');
  const container = document.getElementById('avatar-container')!;
  const avatarView = new AvatarView(avatar, container);

  // Create Player
  const provider = new LiveKitProvider();
  const player = new AvatarPlayer(provider, avatarView, {
    logLevel: 'info',
    transitionStartFrameCount: 5,
    transitionEndFrameCount: 40,
  });

  // Listen to events
  player.on('connected', () => console.log('Connected!'));
  player.on('disconnected', () => console.log('Disconnected!'));
  player.on('error', (err) => console.error('Error:', err));

  // Connect
  await player.connect({
    url: 'wss://your-livekit-server.com',
    token: 'your-token',
    roomName: 'my-room',
  });

  // Start microphone
  await player.startPublishing();

  // Stop microphone
  // await player.stopPublishing();

  // Disconnect
  // await player.disconnect();
}

Agora Integration

import { AvatarPlayer, AgoraProvider } from '@spatialwalk/avatarkit-rtc';
import { AvatarSDK, AvatarView, AvatarManager, DrivingServiceMode, Environment } from '@spatialwalk/avatarkit';

async function initAgora() {
  // Initialize SDK (use host mode)
  await AvatarSDK.initialize('your-app-id', {
    environment: Environment.cn,
    drivingServiceMode: DrivingServiceMode.host,
  });

  // Set session token (required)
  AvatarSDK.setSessionToken('your-session-token');

  // Create Avatar
  const avatar = await AvatarManager.shared.load('character-id');
  const container = document.getElementById('avatar-container')!;
  const avatarView = new AvatarView(avatar, container);

  // Create Player
  const provider = new AgoraProvider({ debugLogging: true });
  const player = new AvatarPlayer(provider, avatarView);

  // Connect
  await player.connect({
    appId: 'your-agora-app-id',
    channel: 'my-channel',
    token: 'your-token',  // Required in production
    uid: 0,  // 0 = auto assign
  });

  // Start microphone
  await player.startPublishing();
}

Browser Compatibility

| Feature | Chrome | Firefox | Safari | Edge | |---------|--------|---------|--------|------| | LiveKit (VP8 + RTCRtpScriptTransform) | ✅ 94+ | ✅ 117+ | ✅ 15.4+ | ✅ 94+ | | Agora (H.264 + SEI) | ✅ 74+ | ✅ 78+ | ✅ 14.1+ | ✅ 79+ |

Note: LiveKit requires browser support for RTCRtpScriptTransform API (Safari 15.4+ supported).

License

MIT