Agent Server SDK for TypeScript

Conversational AI SDK for TypeScript, enabling you to build voice-powered AI agents with support for cascading flows (ASR -> LLM -> TTS) using domestic (China) vendors.

Installation

npm i -s agent-server-sdk

Quick Start

Use the builder pattern with Agent and AgentSession:

import {
  AgentClient,
  Area,
  Agent,
  ExpiresIn,
  AliyunLLM,
  MiniMaxTTS,
  FengmingSTT,
} from 'agent-server-sdk';

const client = new AgentClient({
  area: Area.CN,
  appId: 'your-app-id',
  appCertificate: 'your-app-certificate',
});

const agent = new Agent({
  name: 'support-assistant',
  instructions: '你是一个有用的语音助手。',
  greeting: '你好！有什么可以帮助你的？',
  maxHistory: 10,
})
  .withStt(new FengmingSTT({ language: 'zh-CN' }))
  .withLlm(new AliyunLLM({
    url: 'https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions',
    apiKey: 'your-aliyun-key',
    model: 'qwen-max',
  }))
  .withTts(new MiniMaxTTS({
    key: 'your-minimax-key',
    model: 'speech-01-turbo',
    voiceSetting: { voice_id: 'male-qn-qingse' },
  }));

const session = agent.createSession(client, {
  channel: 'support-room-123',
  agentUid: '1',
  remoteUids: ['100'],
  idleTimeout: 120,
  expiresIn: ExpiresIn.hours(12),
});

const agentSessionId = await session.start();
await session.stop();

Session lifecycle

start() joins the agent to the channel and returns a session ID. The session stays active until stop() is called.

Option 1 — Hold the session in memory:

const agentSessionId = await session.start();
await session.stop();

Option 2 — Store the session ID and stop by ID (stateless servers):

const agentSessionId = await session.start();
res.json({ agentSessionId });

// stop-session handler
const client = new AgentClient({
  area: Area.CN,
  appId: '...',
  appCertificate: '...',
});
await client.stopAgent(agentSessionId);

Supported Vendors

LLM

• AliyunLLM — Alibaba Cloud (Qwen)
• BytedanceLLM — Volcengine (Doubao)
• DeepSeekLLM — DeepSeek
• TencentLLM — Tencent (Hunyuan)
• CustomLLM — Custom LLM endpoint

TTS

• MiniMaxTTS — MiniMax
• TencentTTS — Tencent Cloud
• BytedanceTTS — Volcengine
• MicrosoftTTS — Microsoft Azure
• CosyVoiceTTS — Alibaba Cloud CosyVoice
• BytedanceDuplexTTS — Volcengine Duplex Streaming
• StepFunTTS — StepFun

STT

• FengmingSTT — Fengming ASR
• TencentSTT — Tencent Cloud ASR
• MicrosoftSTT — Microsoft Azure Speech
• XfyunSTT — iFlytek traditional ASR
• XfyunBigModelSTT — iFlytek Big Model ASR
• XfyunDialectSTT — iFlytek Dialect ASR

Avatar

• SensetimeAvatar — Sensetime digital human

Documentation

API reference documentation is available here.

Reference

A full reference for this library is available here.

Exception Handling

import { AgoraError } from "agent-server-sdk";

try {
    await client.agents.start(...);
} catch (err) {
    if (err instanceof AgoraError) {
        console.log(err.statusCode);
        console.log(err.message);
        console.log(err.body);
    }
}

Contributing

While we value open-source contributions to this SDK, this library is generated programmatically. Additions made directly to this library would have to be moved over to our generation code, otherwise they would be overwritten upon the next generated release. Feel free to open a PR as a proof of concept, but know that we will not be able to merge it as-is. We suggest opening an issue first to discuss with us!