Agora Conversational AI TypeScript SDK

The Agora Agents SDK for TypeScript lets you build real-time voice agents on Agora Conversational AI with a high-level Agent / AgentSession API and a generated REST client.

Installation

npm install agora-agents

Quick Start

Start with the Agent builder: create a client with app credentials, configure ASR, LLM, and TTS with vendor classes, then start a session. Omit vendor API keys for supported Agora-managed models, or provide keys when you want BYOK. Set Agora interaction language with turnDetection.language; provider-specific STT language values remain under asr.params. Ares uses only the REST asr.language value sourced from turnDetection.language.

import { AgoraClient, Agent, Area, DeepgramSTT, ExpiresIn, MiniMaxTTS, OpenAI } from 'agora-agents';

const AGENT_PROMPT = `You are a concise, technically credible voice assistant. Keep replies short unless the user asks for detail.`;

const GREETING = 'Hi there! I am your Agora voice assistant. How can I help?';

export async function startConversation(): Promise<string> {
  const appId = process.env.AGORA_APP_ID!;
  const appCertificate = process.env.AGORA_APP_CERTIFICATE!;

  const client = new AgoraClient({
    area: Area.US,
    appId,
    appCertificate,
  });

  const agent = new Agent({
    client,
    turnDetection: {
      language: 'en-US',
      config: {
        speech_threshold: 0.5,
        start_of_speech: {
          mode: 'vad',
          vad_config: {
            interrupt_duration_ms: 160,
            prefix_padding_ms: 300,
          },
        },
        end_of_speech: {
          mode: 'vad',
          vad_config: {
            silence_duration_ms: 480,
          },
        },
      },
    },
    advancedFeatures: {
      enable_rtm: true,
      enable_tools: true,
    },
    parameters: {
      data_channel: 'rtm',
      enable_error_message: true,
    },
  })
    .withStt(
      new DeepgramSTT({
        model: 'nova-3',
        language: 'en',
      }),
    )
    .withLlm(
      new OpenAI({
        model: 'gpt-4o-mini',
        systemMessages: [{ role: 'system', content: AGENT_PROMPT }],
        greetingMessage: GREETING,
        failureMessage: 'Please wait a moment.',
        maxHistory: 50,
        params: {
          max_tokens: 1024,
          temperature: 0.7,
          top_p: 0.95,
        },
      }),
    )
    .withTts(
      new MiniMaxTTS({
        model: 'speech_2_6_turbo',
        voiceId: 'English_captivating_female1',
      }),
    );

  const session = agent.createSession({
    name: `conversation-${Date.now()}`,
    channel: `demo-channel-${Date.now()}`,  // Unique channel name
    agentUid: '123456',                     // Unique agent UID. Can be a random number or a specific user ID.
    remoteUids: ['*'],                     // '*' is a wildcard, or use a specific user ID.
    idleTimeout: 30,
    expiresIn: ExpiresIn.hours(1),
    debug: false,
  });

  return await session.start();
}

Why no token or vendor key in the example?

AgoraClient generates the required ConvoAI REST auth and RTC join tokens automatically when you provide appId and appCertificate. For supported Agora-managed global models, leave vendor API keys unset; provide keys when you want BYOK. CN MiniMax TTS is not Agora-managed and always requires key. CN custom LLM routing reuses CustomLLM, so apiKey is also required.

Regional agent builders

client.area controls API routing only. You may combine any supported vendor class with any area. See docs/guides/regional-routing.md for examples.

AI Studio pipeline IDs

Use pipelineId when you want a published AI Studio pipeline to provide the base agent configuration:

import { AgoraClient, Agent, Area } from 'agora-agents';

const client = new AgoraClient({
  area: Area.US,
  appId: process.env.AGORA_APP_ID!,
  appCertificate: process.env.AGORA_APP_CERTIFICATE!,
});

const agent = new Agent({
  client,
  pipelineId: 'studio-pipeline-id',
});

const session = agent.createSession({
  name: `conversation-${Date.now()}`,
  channel: `demo-channel-${Date.now()}`,
  agentUid: '1',
  remoteUids: ['100'],
});

You can override it per session:

const session = agent.createSession({
  name: `conversation-${Date.now()}`,
  channel: `demo-channel-${Date.now()}`,
  agentUid: '1',
  remoteUids: ['100'],
  pipelineId: 'session-pipeline-id',
});

AgentKit sends the resolved value as the top-level /join field pipeline_id, not inside properties. Explicit Agent config such as .withLlm(), .withTts(), .withStt(), .withMllm(), and advancedFeatures may send properties fields that override the saved pipeline settings.

BYOK version

Use the same Agent builder shape, but provide credentials explicitly when you want vendor-managed billing and routing instead of Agora-managed models.

import { AgoraClient, Agent, Area, DeepgramSTT, MiniMaxTTS, OpenAI } from 'agora-agents';

const client = new AgoraClient({
  area: Area.US,
  appId: process.env.AGORA_APP_ID!,
  appCertificate: process.env.AGORA_APP_CERTIFICATE!,
});

const SUPPORT_PROMPT = 'You are a concise support assistant.';
const GREETING = 'Hi there! I am your Agora voice assistant. How can I help?';

const agent = new Agent({ client, turnDetection: { language: 'en-US' } })
  .withStt(
    new DeepgramSTT({
      apiKey: process.env.DEEPGRAM_API_KEY!,
      model: 'nova-3',
      language: 'en',
    }),
  )
  .withLlm(
    new OpenAI({
      apiKey: process.env.OPENAI_API_KEY!,
      url: 'https://api.openai.com/v1/chat/completions',
      model: 'gpt-4o-mini',
      systemMessages: [{ role: 'system', content: SUPPORT_PROMPT }],
      greetingMessage: GREETING,
      maxTokens: 1024,
      temperature: 0.7,
      topP: 0.95,
    }),
  )
  .withTts(
    new MiniMaxTTS({
      model: 'speech_2_6_turbo',
      voiceId: 'English_captivating_female1',
    }),
  );

Migrating from agora-agent-server-sdk? Install and import agora-agents instead — see changelog migration notes or installation guide.

BYOK

If you want to bring your own vendor credentials instead of using Agora-managed models, use the BYOK guide:

• BYOK Guide

MLLM (Realtime / Multimodal)

Use withMllm() for OpenAI Realtime, Gemini Live, Vertex AI, or xAI Grok — no STT, LLM, or TTS vendor needed. MLLM mode is enabled automatically.

import { AgoraClient, Agent, Area, OpenAIRealtime } from 'agora-agents';

const client = new AgoraClient({
  area: Area.US,
  appId: process.env.AGORA_APP_ID!,
  appCertificate: process.env.AGORA_APP_CERTIFICATE!,
});

const agent = new Agent({ client }).withMllm(
  new OpenAIRealtime({
    apiKey: process.env.OPENAI_API_KEY!,
    model: 'gpt-4o-realtime-preview',
    greetingMessage: 'Hello! Ready to chat.',
  }),
);

See the MLLM Flow guide for full examples with Gemini Live, Vertex AI, and xAI Grok.

Avatars are not supported with MLLM. The avatar publisher requires the cascading ASR + LLM + TTS pipeline; combining withMllm() with withAvatar() throws at Agent.toProperties() and AgentSession.start().

Avatars

AgentKit supports LiveAvatar, Generic Avatar, Anam, Akool, SenseTime (CN), and deprecated HeyGen. Avatar agoraToken is optional: when omitted, session.start() generates a token using the same ConvoAI token format as the agent token, scoped to the avatar agoraUid. Avatars require the cascading ASR + LLM + TTS pipeline (not MLLM).

See the Avatar Integration guide for sample-rate requirements and Generic Avatar setup.

Documentation

• Overview
• Authentication
• Quick Start
• BYOK Guide
• MLLM Flow
• Low-Level API

Reference

• SDK Reference
• Agora Conversational AI Docs