RTVI Client

A comprehensive TypeScript/JavaScript client library for Real-Time Voice Interface (RTVI) applications. This package provides both React hooks and vanilla JavaScript clients for building voice-enabled applications with support for multiple LLM providers.

Features

• 🎤 Real-time voice interaction with audio streaming
• 🤖 Multiple LLM providers (OpenAI, Claude, Gemini, Llama, Gemini Live)
• ⚛️ React hooks for easy integration
• 🎯 TypeScript support with full type definitions
• 🔄 Event-driven architecture for real-time updates
• 📱 Cross-platform (Web, React Native compatible)
• 🎥 Video support (optional)
• 📝 Message handling with transcript support

Installation

npm install @convai/rtvi-client
# or
yarn add @convai/rtvi-client
# or
pnpm add @convai/rtvi-client

Quick Start

React Usage (Recommended)

The easiest way to get started is with the useRtviBot hook:

import React, { useState } from 'react';
import { useRtviBot } from '@convai/rtvi-client';

function VoiceChat() {
  const [apiKey, setApiKey] = useState('your-api-key');
  const [characterId, setCharacterId] = useState('your-character-id');

  const {
    isConnected,
    isConnecting,
    error,
    botStatus,
    messages,
    connect,
    disconnect,
    sendMessage,
  } = useRtviBot({
    apiKey,
    characterId,
    llmProvider: 'openai',
    connectionType: 'audio',
    enableMic: true,
    onMessage: (message) => {
      console.log(`${message.sender}: ${message.text}`);
    },
    onBotStatusChange: (status) => {
      console.log(`Bot status: ${status}`);
    },
  });

  const handleConnect = async () => {
    try {
      await connect({ characterId, llmProvider: 'openai' });
    } catch (err) {
      console.error('Failed to connect:', err);
    }
  };

  return (
    <div>
      <button onClick={isConnected ? disconnect : handleConnect}>
        {isConnecting ? 'Connecting...' : isConnected ? 'Disconnect' : 'Connect'}
      </button>
      
      <p>Status: {botStatus}</p>
      
      <div>
        {messages.map((msg) => (
          <div key={msg.id}>
            <strong>{msg.sender}:</strong> {msg.text}
          </div>
        ))}
      </div>
      
      <button 
        onClick={() => sendMessage('Hello!')} 
        disabled={!isConnected}
      >
        Send Message
      </button>
    </div>
  );
}

Vanilla JavaScript Usage

For non-React applications, use the client classes directly:

import { ChatbotClient, RTVIEvent } from '@convai/rtvi-client';

const client = new ChatbotClient({
  apiKey: 'your-api-key',
  characterId: 'your-character-id',
  llmProvider: 'openai',
  connectionType: 'audio',
  enableMic: true,
  callbacks: {
    onTrackStart: (track, participant) => {
      console.log('Audio track started:', track);
    },
    onError: (error) => {
      console.error('Client error:', error);
    },
  },
});

// Set up event listeners
client.on(RTVIEvent.Connected, () => {
  console.log('Connected to bot');
});

client.on(RTVIEvent.BotLlmText, (data) => {
  console.log('Bot said:', data.text);
});

client.on(RTVIEvent.UserTranscript, (data) => {
  if (data.final) {
    console.log('User said:', data.text);
  }
});

// Connect and start conversation
async function startConversation() {
  try {
    await client.connect();
    client.sendMessage('Hello, how are you?');
  } catch (error) {
    console.error('Failed to connect:', error);
  }
}

startConversation();

API Reference

React Hook

useRtviBot(options: UseRtviBotOptions): UseRtviBotReturn

The main React hook for RTVI functionality.

Options:

interface UseRtviBotOptions {
  apiKey: string;                    // Your API key
  characterId?: string;              // Character/bot ID
  llmProvider?: LLMProvider;         // LLM provider to use
  connectionType?: ConnectionType;   // 'audio' | 'text' | 'video'
  enableMic?: boolean;              // Enable microphone (default: true)
  enableCam?: boolean;              // Enable camera (default: false)
  videoContainer?: HTMLElement;      // Container for video elements
  onConnectionStatusChange?: (connected: boolean) => void;
  onMessage?: (message: Message) => void;
  onError?: (error: string) => void;
  onBotStatusChange?: (status: BotStatus) => void;
}

Returns:

interface UseRtviBotReturn {
  // Connection state
  client: ChatbotClient | null;
  isConnected: boolean;
  isConnecting: boolean;
  error: string | null;
  sessionId: string | null;
  
  // Bot state
  botStatus: BotStatus;              // 'idle' | 'listening' | 'speaking' | 'thinking'
  thinkingText: string;
  isMicActive: boolean;
  
  // Messages
  messages: Message[];
  
  // Actions
  connect: (options?: { characterId?: string; llmProvider?: LLMProvider }) => Promise<void>;
  disconnect: () => Promise<void>;
  sendMessage: (text: string) => void;
  clearMessages: () => void;
  
  // Audio/Video controls
  enableMic: (enable: boolean) => void;
  
  // Event handlers (for custom usage)
  on: (event: RTVIEvent, handler: (...args: any[]) => void) => void;
  off: (event: RTVIEvent, handler: (...args: any[]) => void) => void;
}

Client Classes

ChatbotClient

High-level client for chatbot interactions.

class ChatbotClient {
  constructor(options: ChatbotClientOptions);
  
  async connect(): Promise<ConnectResponse>;
  async disconnect(): Promise<void>;
  sendMessage(text: string): void;
  on(event: RTVIEvent, handler: Function): void;
  off(event: RTVIEvent, handler: Function): void;
  isConnected(): boolean;
  getSessionId(): string | null;
  enableMic(enable: boolean): void;
  setOptions(options: Partial<ChatbotClientOptions>): void;
}

RtviClient

Lower-level RTVI protocol client.

class RtviClient {
  constructor(options: RtviClientOptions);
  
  async connect(payload?: ConnectPayload): Promise<ConnectResponse>;
  async disconnect(): Promise<DisconnectResponse>;
  async healthCheck(): Promise<any>;
  sendMessage(message: string | RTVIMessage): void;
  on(event: RTVIEvent, handler: Function): void;
  off(event: RTVIEvent, handler: Function): void;
  isConnected(): boolean;
  getSessionId(): string | null;
  enableMic(enable: boolean): void;
}

Types

Core Types

// Message structure
interface Message {
  id: string;
  text: string;
  sender: "user" | "bot";
  timestamp: Date;
}

// Bot status
type BotStatus = "idle" | "listening" | "speaking" | "thinking";

// Supported LLM providers
type LLMProvider = "openai" | "claude" | "gemini" | "llama" | "gemini-live";

// Connection types
type ConnectionType = "audio" | "text" | "video";

Configuration Types

interface ChatbotClientOptions {
  apiKey: string;
  baseUrl?: string;                  // API base URL (default: convai.com)
  characterId: string;
  connectionType?: ConnectionType;
  llmProvider?: LLMProvider;
  videoContainer?: HTMLElement;
  enableMic?: boolean;
  enableCam?: boolean;
  callbacks?: {
    onTrackStart?: (track: MediaStreamTrack, participant?: any) => void;
    onTrackStop?: (track: MediaStreamTrack, participant?: any) => void;
    onError?: (error: any) => void;
  };
}

Events

The library uses an event-driven architecture. Here are the main events:

enum RTVIEvent {
  Connected = 'connected',
  Disconnected = 'disconnected',
  Error = 'error',
  
  // Bot events
  BotLlmText = 'bot-llm-text',           // Bot is speaking (text chunks)
  BotStartedSpeaking = 'bot-started-speaking',
  BotStoppedSpeaking = 'bot-stopped-speaking',
  
  // User events
  UserTranscript = 'user-transcript',     // User speech transcript
  UserStartedSpeaking = 'user-started-speaking',
  UserStoppedSpeaking = 'user-stopped-speaking',
  
  // Media events
  TrackStarted = 'track-started',         // Audio/video track started
  TrackStopped = 'track-stopped',         // Audio/video track stopped
}

Advanced Usage

Custom Event Handling

const { client, on, off } = useRtviBot(options);

// Custom event handler
const handleCustomEvent = (data) => {
  console.log('Custom event:', data);
};

// Add event listener
on(RTVIEvent.BotLlmText, handleCustomEvent);

// Remove event listener
off(RTVIEvent.BotLlmText, handleCustomEvent);

Video Support

function VideoChat() {
  const videoRef = useRef<HTMLDivElement>(null);
  
  const { connect } = useRtviBot({
    apiKey: 'your-api-key',
    characterId: 'your-character-id',
    connectionType: 'video',
    enableCam: true,
    videoContainer: videoRef.current,
  });

  return (
    <div>
      <div ref={videoRef} style={{ width: '100%', height: '400px' }} />
      <button onClick={() => connect()}>Start Video Chat</button>
    </div>
  );
}

Multiple LLM Providers

// Switch between providers dynamically
const { connect } = useRtviBot({
  apiKey: 'your-api-key',
  characterId: 'your-character-id',
});

// Connect with OpenAI
await connect({ llmProvider: 'openai' });

// Later, reconnect with Claude
await connect({ llmProvider: 'claude' });

Error Handling

const { error, connect } = useRtviBot({
  apiKey: 'your-api-key',
  characterId: 'your-character-id',
  onError: (errorMessage) => {
    // Handle errors globally
    console.error('RTVI Error:', errorMessage);
    // Show user-friendly error message
    alert(`Connection error: ${errorMessage}`);
  },
});

// Handle connection errors
const handleConnect = async () => {
  try {
    await connect();
  } catch (err) {
    console.error('Failed to connect:', err);
    // Handle specific connection errors
  }
};

Examples

Check out the /examples directory for complete working examples:

• React Example: Full-featured React app with UI
• Vanilla JavaScript Example: Basic implementation without React

Running Examples

# React example
cd examples/react
npm install
npm run dev

# Vanilla example
cd examples/vanilla
npm install
npm run dev

Configuration

Environment Variables

You can set default configuration using environment variables:

RTVI_API_KEY=your-api-key
RTVI_BASE_URL=https://your-api-endpoint.com
RTVI_CHARACTER_ID=your-default-character

API Endpoints

The library connects to the following endpoints by default:

• Connect: /connect - Establish connection
• Disconnect: /disconnect - Close connection
• Health Check: /healthz - Service health status

Troubleshooting

Common Issues

1. Microphone not working

   • Ensure enableMic: true in options
   • Check browser permissions for microphone access
   • Verify HTTPS connection (required for microphone)

2. Connection failures

   • Verify API key is correct
   • Check character ID exists
   • Ensure network connectivity
   • Check browser console for detailed errors

3. Audio not playing

   • Browser may require user interaction before audio playback
   • Check if audio elements are being created properly
   • Verify audio permissions

Debug Mode

Enable detailed logging:

const { client } = useRtviBot({
  // ... other options
  onError: (error) => console.error('RTVI Error:', error),
  onConnectionStatusChange: (connected) => console.log('Connected:', connected),
  onBotStatusChange: (status) => console.log('Bot Status:', status),
});

Browser Compatibility

• Chrome: Full support
• Firefox: Full support
• Safari: Full support (iOS 14.3+)
• Edge: Full support

License

MIT License - see LICENSE file for details.

Support

For issues and questions:

• GitHub Issues: Repository Issues
• Documentation: Full Documentation
• Discord: Community Discord