@simfinity/constellation-client

Installation

npm install @simfinity/constellation-client
# or
yarn add @simfinity/constellation-client

Purpose & Usage

This package is a code wrapper to integrate the Simfinity constellation server. The constellation server is a proxy managing streaming sessions with third party LLMs. This package provides the programmatic functions covering the complete lifecycle of a streaming session:

• Open/start session
• Callbacks to continuously send and receive streamed data over a persistent connection
• Close/end session

Server implementation insight

The Constellation server is chat-room & session manager: upon receiving a session-start request, it creates a persistent chat-room, initiates the persistent connection with the LLM and configures it accordingly (e.g. system instruction, temperature, audio, transcript subscription...). Clients may lose connection with Constellation, but the chat-room will remain opened on server side, this allows the client to reconnect and resume the session. Clients MUST notify the server that a session has ended, so that Constellation can release allocated resources.

Example

Key steps in pseudo-code:

const client = new WebClient({
     sessionEndpoint: "https://simfinity.constellation.com", 
     streamingEndpoint: "wss://simfinity.constellation.com:30003",
     key: "my-key",
     llm: "openai",
});

try {
    /* ... */
    
    // Start a chat session
    await client.startSession(true);
    await client.connect(true, {
        onStreamClosed: (reason: string) => {
            console.log("Stream connection lost");
        },
        onAudioResponseStart: () => {
            console.log("The model is talking");
        },
        onAudioResponseChunk: (audioChunk: string) => {
            audioPlayer.enqueue(audioChunk);
        },
        onAudioResponseEnd: () => {
            console.log("The model is done talking");
        }
    });

    /* ... */
    
    client.sendAudioChunk("{PCM16 Base64-encoded data}");
    client.commitAudioChunksSent();
    
    /* ... */
}
catch {
    
}
finally {
    await endSession();
}

Types

Configuration

Configuration required to initiate a connection with the server: In the client, values would typically be stored in secret stores & environment variables

export interface WebClientConfig {
     sessionEndpoint:    string;
     streamingEndpoint:  string;
     key:    string;
     llm:    LlmType;
     model:  string;
}

Event hooks

Callback functions to catch all the propagated server events. Except for the onStreamClosed event, assigning hooks is optional: non-observed events will be silently ignored & lost.

export interface EventHandlers {
     onStreamClosed: (reason: string) => void;
     onAudioResponseStart?: () => void;
     onAudioResponseChunk?: (audioChunk: string) => void;
     onAudioResponseEnd?: () => void;
     onTranscriptInput?: (transcript: string) => void;
     onTranscriptResponse?: (transcript: string) => void;
     onTechnicalError?: (error: string) => void;
}

Audio

• The server expect exclusively base64 encoded PCM16 format & sends responses of the same format in return
• The server implements VAD - voice activation detection. Configured to detect 1s silences as a response trigger
• Therefore, input audio data chunks can be streamed immediately without buffering
• Client should however implement voice detection as well to reduce network consumption
  • 500ms ring buffer continuously filled with audio input
  • Noise detection with minimum threshold

Text & Transcript

• The transcript inputs/responses callbacks carry both the text exchanges and transcription of audio exchanges
• In an audio session, text and audio inputs will trigger:
  • a mirrored transcript text through onTranscriptInput
  • an audio response through onAudioResponseChunk
  • a text transcript of the audio response through onTranscriptResponse
• In a text-only session, a text input will trigger:
  • a mirrored transcript text through onTranscriptInput
  • a text response through the onTranscriptResponse callback