@openbmb/realtime-sdk

High-level TypeScript SDK for OpenBMB real-time models.

This package provides a typed session API over the MiniCPM-o Realtime V1 WebSocket protocol. It manages queueing, session initialization, lifecycle transitions, interruption, and graceful shutdown so application code does not need to construct wire-level session.init, input.append, or session.close frames.

Features

• Typed audio and video session APIs
• Automatic handling of queue and initialization events
• Streaming assistant text and audio events
• Official server usage events and a session usage snapshot
• Explicit pause, resume, interrupt, and close operations
• Idempotent lifecycle methods
• Connection and close timeouts
• Custom WebSocket transport support for server-side runtimes and tests
• ESM distribution with generated TypeScript declarations

Installation

npm install @openbmb/realtime-sdk

pnpm add @openbmb/realtime-sdk

Quick Start

The SDK uses the browser's native WebSocket implementation by default. baseUrl is the HTTP API base URL; the SDK converts it to the corresponding WebSocket endpoint.

import { RealtimeClient } from '@openbmb/realtime-sdk';

const client = new RealtimeClient({
  apiKey: process.env.MODELBEST_API_KEY ?? '',
  baseUrl: 'https://api.modelbest.cn',
});

const session = await client.realtime.audio.connect({
  model: 'MiniCPM-o-4.5-Realtime',
  systemPrompt: 'You are a helpful voice assistant.',
});

session.on('assistant.text.delta', ({ text }) => {
  process.stdout.write(text);
});

session.on('assistant.audio.delta', ({ audio }) => {
  playPcm16k(audio);
});

session.on('error', ({ code, message }) => {
  console.error(`[${code}] ${message}`);
});

session.on('usage', ({ usage }) => {
  console.log('Usage:', usage);
});

session.sendAudio(audioBase64);
session.interrupt();
await session.close();

playPcm16k is application-provided audio playback code. The audio event contains a raw Base64-encoded 16 kHz, mono, Float32 PCM audio chunk.

Usage is read only from the server's official realtime usage payload. The SDK does not estimate tokens from audio duration, media size, or frame count. session.usage is undefined until usage is observed. A top-level usage payload is treated as a cumulative session snapshot; usage in response.done.response.usage is treated as one completed response turn and aggregated across turns. close() remains a lifecycle method and resolves to void; read session.usage after it resolves when the server has sent usage.

Audio Sessions

Use client.realtime.audio.connect() for microphone or audio-only conversations.

const session = await client.realtime.audio.connect({
  model: 'MiniCPM-o-4.5-Realtime',
  systemPrompt: 'Answer clearly and keep responses concise.',
  config: {
    ttsEnabled: true,
    temperature: 0.7,
  },
});

microphone.onChunk((audioBase64) => {
  session.sendAudio(audioBase64);
});

session.on('user.transcript', ({ text }) => {
  console.log('User:', text);
});

session.on('assistant.text.delta', ({ text, endOfTurn }) => {
  renderAssistantText(text, endOfTurn);
});

Audio input must be supplied as raw Base64-encoded 16 kHz, mono, Float32 PCM data. The SDK does not resample, encode, or otherwise transform media data.

Video Sessions

Use client.realtime.video.connect() when the model should receive image frames. Video sessions retain the audio methods and additionally expose sendAudioVideo().

const session = await client.realtime.video.connect({
  model: 'MiniCPM-o-4.5-Realtime',
  systemPrompt: 'Describe relevant changes in the camera view.',
  config: {
    visionHd: true,
  },
});

session.sendAudioVideo(audioBase64, jpegBase64);

sendAudioVideo() sends the Base64-encoded 16 kHz mono Float32 PCM audio and JPEG frame in one input.append. The full-duplex video protocol requires every video input to include audio. sendVideoFrame() therefore throws a RealtimeError instead of sending an invalid empty-audio frame. Audio sessions do not expose either video method, which keeps audio-only integrations type-safe.

Client Options

interface RealtimeClientOptions {
  apiKey: string;
  baseUrl: string;
  connectTimeout?: number;
  closeTimeout?: number;
  socketFactory?: RealtimeSocketFactory;
}

| Option | Required | Default | Description | | ---------------- | -------- | ----------------- | ------------------------------------------------------------------------------ | | apiKey | Yes | - | API key used to authenticate the WebSocket session. | | baseUrl | Yes | - | HTTP API base URL, for example https://api.modelbest.cn. | | connectTimeout | No | 30000 | Maximum time in milliseconds to wait for session.created. | | closeTimeout | No | 10000 | Maximum time in milliseconds to wait for close confirmation. | | socketFactory | No | Browser WebSocket | Custom WebSocket factory for runtimes without a compatible global WebSocket. |

The SDK accepts a base URL with or without a trailing /v1. It creates a WebSocket URL in the following form:

wss://api.modelbest.cn/v1/realtime?mode=audio&model=MiniCPM-o-4.5-Realtime

Session Configuration

interface RealtimeConnectOptions {
  model: string;
  systemPrompt?: string;
  config?: {
    lengthPenalty?: number;
    maxTokens?: number;
    temperature?: number;
    maxKvTokens?: number;
    ttsEnabled?: boolean;
    visionHd?: boolean;
  };
  referenceAudioBase64?: string;
  ttsReferenceAudioBase64?: string;
  maxSliceNums?: number;
}

Configuration fields are passed to the MiniCPM-o session initialization request. Media reference values must be raw Base64 strings without a data URI prefix.

Session Lifecycle

connect() resolves only after the server has acknowledged the session with session.created. Queueing and initialization are handled internally.

Typical lifecycle:

connecting -> queued -> initializing -> ready
                                      -> listening
                                      -> speaking
                                      -> closing -> closed

The SDK guarantees the following lifecycle behavior:

• sendAudio() and sendAudioVideo() are ignored until the session is ready.
• pause() stops local input delivery and is safe to call repeatedly.
• resume() re-enables local input delivery.
• interrupt() emits interrupted immediately and applies the protocol's force_listen flag to the next input frame.
• close() is safe to call repeatedly and waits for server confirmation or the configured close timeout.
• A failed connection rejects connect() with RealtimeError.
• Unknown server events are ignored so protocol extensions do not break the application event loop.

Events

Register listeners with session.on(). The method returns an unsubscribe function.

| Event | Payload | Description | | ----------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------ | | queued | position, estimatedWaitSeconds, queueLength | The session is waiting for an available worker. Optional fields may be absent. | | worker.assigned | None | A worker has been assigned and initialization is starting. | | ready | sessionId | The session is ready to accept input. | | assistant.audio.delta | audio | A Base64-encoded assistant audio chunk. | | assistant.text.delta | text, endOfTurn | A streamed assistant text fragment. | | user.transcript | text | A recognized user transcript fragment or turn result. | | turn.completed | None | The current response turn has completed. | | usage | usage | Official server usage; the payload is the current session snapshot. | | interrupted | None | An interruption was requested locally. | | paused | None | Local input delivery has been paused. | | resumed | None | Local input delivery has resumed. | | closed | reason | The session has completed shutdown. | | error | code, message | A transport, protocol, or session error occurred. |

Example:

const unsubscribe = session.on('assistant.text.delta', (event) => {
  updateTranscript(event.text);
});

unsubscribe();

The normalized usage shape is:

interface RealtimeUsage {
  prompt_tokens: number;
  completion_tokens: number;
  total_tokens: number;
  audio_input_tokens?: number;
  audio_output_tokens?: number;
  vision_input_tokens?: number;
}

Error Handling

Connection failures are reported by rejecting connect(). Errors after the session is ready are delivered through the error event.

import { RealtimeError } from '@openbmb/realtime-sdk';

try {
  const session = await client.realtime.audio.connect({
    model: 'MiniCPM-o-4.5-Realtime',
  });

  session.on('error', (event) => {
    reportRealtimeError(event.code, event.message);
  });
} catch (error) {
  if (error instanceof RealtimeError) {
    console.error({
      code: error.code,
      phase: error.phase,
      message: error.message,
    });
  }
  throw error;
}

RealtimeError.phase is either connecting or session. Typical connection error codes include invalid_api_key, invalid_model, socket_create_failed, connect_timeout, and session_closed_before_ready. The server may provide additional error codes.

Runtime Requirements

The package is ESM-only and publishes TypeScript declarations.

• Modern browsers with a native WebSocket implementation work without additional transport configuration.
• Server-side runtimes must provide a compatible global WebSocket or pass a socketFactory.
• The SDK does not include an audio encoder, decoder, resampler, media recorder, or playback engine. Choose media tooling appropriate for your runtime.

Security Guidance

An API key passed to the browser is visible to the browser environment and should not be treated as a secret. For production browser applications, establish the WebSocket through your own authenticated server or another short-lived credential flow. Never commit API keys to source control or place them in public URLs.

Protocol Abstraction

The high-level RealtimeClient and session interfaces are the recommended integration surface. The package also exports low-level protocol helpers such as buildSessionInit, buildInputAppend, buildSessionClose, and parseServerEvent for applications that need controlled protocol-level integration. These helpers are intentionally separate from the normal session lifecycle API.

License

Apache-2.0