@stormee_ai/gemini-graphql-client

A powerful, reusable Gemini GraphQL client library for browser, VS Code extensions, and Node.js with streaming audio and transcription support.

Features

• Universal: Works in browser, VS Code extensions, and Node.js
• Dual Audio Modes:
• Browser: Automatic real-time audio streaming with Web Audio API
• VS Code: Manual audio queue for custom speaker integration
• Live Transcription: Streaming transcription updates as they arrive
• Auto-Reconnection: Automatic reconnection with exponential backoff
• Type-Safe: Full TypeScript support with comprehensive types
• Error Handling: Structured error codes for programmatic handling
• Event-Driven: Rich callback system for all events
• Production Ready: Built with Rollup, tested, and optimized

Installation

npm install @stormee_ai/gemini-graphql-client

Or with yarn:

yarn add @stormee_ai/gemini-graphql-client

Quick Start

Browser Usage

import { GeminiClient } from '@stormee_ai/gemini-graphql-client';

const client = new GeminiClient({
  websocketUrl: 'wss://your-graphql-endpoint',
  environment: 'browser',
  guestSessionId: 'user-session-123',
  assistantName: 'StormeeAI',
  organizationName: 'YourOrg',
  
  callbacks: {
    onTranscription: (data) => {
      console.log('Transcription:', data.text);
      if (data.isFinal) {
        console.log('Final transcription received');
      }
    },
    
    onAudioChunk: (chunk) => {
      // Audio plays automatically in browser mode
      console.log('Audio chunk received');
    },
    
    onComplete: (data) => {
      console.log('Response complete!', data);
    },
    
    onError: (error) => {
      console.error('Error:', error.code, error.message);
    },
  },
});

// Initialize and connect
await client.initialize();

// Send request
await client.sendRequest({
  message: 'Hello Gemini!',
  chatHistory: [
    { role: 'user', message: 'Previous message' },
    { role: 'assistant', message: 'Previous response' },
  ],
});

// Disconnect when done
await client.disconnect();

VS Code Extension Usage

import { GeminiClient } from '@stormee_ai/gemini-graphql-client';
import * as vscode from 'vscode';

const client = new GeminiClient({
  websocketUrl: 'wss://your-graphql-endpoint',
  environment: 'vscode',
  guestSessionId: 'vscode-user-123',
  assistantName: 'StormeeAI',
  organizationName: 'YourOrg',
  
  callbacks: {
    onTranscription: (data) => {
      // Update UI with transcription
      vscode.window.showInformationMessage(data.text);
    },
    
    onAudioChunk: (chunk) => {
      // Chunks are queued automatically
      console.log('Chunk added to queue');
    },
    
    onComplete: async (data) => {
      // Get all audio chunks
      const chunks = client.consumeAllAudioChunks();
      
      // Play using VS Code's speaker API
      for (const chunk of chunks) {
        await playSpeechChunk(chunk.chunk);
      }
    },
  },
});

await client.initialize();
await client.sendRequest({ message: 'Hello!' });

// Manual chunk consumption
const nextChunk = client.getNextAudioChunk();
if (nextChunk) {
  await playSpeechChunk(nextChunk.chunk);
}

API Reference

GeminiClient

Main client class for interacting with Gemini.

Constructor

new GeminiClient(config: GeminiClientConfig)

Methods

| Method | Description | Returns | |--------|-------------|---------| | initialize() | Connect to Gemini GraphQL | Promise<void> | | sendRequest(payload) | Send request to Gemini | Promise<void> | | disconnect() | Disconnect from Gemini | Promise<void> | | getState() | Get current client state | ClientState | | isReady() | Check if client is ready | boolean | | getSessionHandle() | Get current session handle | string \| null | | getNextAudioChunk() | Get next chunk (VS Code only) | AudioChunkData \| null | | getAllAudioChunks() | Get all chunks (VS Code only) | AudioChunkData[] | | consumeAllAudioChunks() | Get and clear queue (VS Code only) | AudioChunkData[] | | setVolume(volume) | Set volume 0-1 (Browser only) | void | | destroy() | Cleanup all resources | Promise<void> |

Configuration

GeminiClientConfig

interface GeminiClientConfig {
  // Required
  websocketUrl: string;
  environment: 'browser' | 'vscode' | 'node';
  guestSessionId: string;
  assistantName: string;
  organizationName: string;
  callbacks: GeminiEventCallbacks;
  
  // Optional
  sessionId?: string;
  aiModel?: string;
  user?: {
    name?: string;
    email?: string;
    id?: string;
  };
  audioConfig?: AudioConfiguration;
  autoReconnect?: boolean;
  maxReconnectAttempts?: number;
  reconnectDelay?: number;
  debug?: boolean;
}

Event Callbacks

interface GeminiEventCallbacks {
  onTranscription?: (data: TranscriptionData) => void;
  onAudioChunk?: (data: AudioChunkData) => void;
  onAudioComplete?: (audioBlob: Blob, transcription: string) => void;
  onComplete?: (data: CompletionData) => void;
  onError?: (error: GeminiError) => void;
  onConnectionStateChange?: (state: 'connecting' | 'connected' | 'disconnected' | 'reconnecting') => void;
  onTemplateSelected?: (data: TemplateData) => void;
  onThinkingMessage?: (message: string) => void;
}

Request Payload

interface GeminiRequestPayload {
  message: string;
  questions?: string;
  sessionHandle?: string;
  chatHistory?: ChatHistoryMessage[] | null;
  delayOnInitialMessage?: number;
  customMetadata?: Record<string, any>;
}

🛡️ Error Handling

The library uses structured error codes:

import { ErrorCodes } from '@stormee_ai/gemini-graphql-client';

client.callbacks.onError = (error) => {
  switch (error.code) {
    case ErrorCodes.CONNECTION_FAILED:
      console.log('Connection failed, will retry');
      break;
    
    case ErrorCodes.AUDIO_DECODE_ERROR:
      console.log('Audio decode error');
      break;
    
    case ErrorCodes.CLIENT_NOT_READY:
      console.log('Client not initialized');
      break;
    
    default:
      console.log('Unknown error:', error.message);
  }
};

Available Error Codes

• CONNECTION_FAILED - Connection failed
• CONNECTION_CLOSED - Connection closed unexpectedly
• RECONNECT_FAILED - Max reconnection attempts reached
• INVALID_REQUEST - Invalid request payload
• AUDIO_DECODE_ERROR - Audio decoding failed
• AUDIO_PLAYBACK_ERROR - Audio playback failed
• GRAPHQL_ERROR - GraphQL request error
• CLIENT_NOT_READY - Client not initialized
• INVALID_CONFIG - Invalid configuration
• And more...

Advanced Usage

Custom Audio Processing

import { AudioProcessor } from '@stormee_ai/gemini-graphql-client';

class CustomAudioProcessor extends AudioProcessor {
  protected async handleChunk(chunkData: AudioChunkData): Promise<void> {
    // Custom audio processing logic
    console.log('Processing chunk:', chunkData);
  }
  
  protected onStart(): void { }
  protected onStop(): void { }
  protected onReset(): void { }
  protected handleError(error: GeminiError): void { }
}

State Management

// Subscribe to state changes
const unsubscribe = client.onStateChange((state) => {
  console.log('State updated:', {
    isConnected: state.isConnected,
    isReceivingAudio: state.isReceivingAudio,
    audioChunksReceived: state.audioChunksReceived,
  });
});

// Unsubscribe later
unsubscribe();

Audio Utilities

import { 
  decodeBase64Audio, 
  convertPCMtoWAV,
  detectAudioFormat 
} from '@stormee_ai/gemini-graphql-client';

// Decode base64 audio
const audioBytes = decodeBase64Audio(base64String);

// Convert PCM to WAV
const wavBlob = convertPCMtoWAV(pcmBytes, 24000, 1, 16);

// Detect audio format
const format = detectAudioFormat(audioBytes); // 'wav', 'mp3', 'pcm', etc.

Testing

npm test

Building

npm run build

Output:

• dist/index.js - CommonJS bundle
• dist/index.esm.js - ES Module bundle
• dist/index.d.ts - TypeScript declarations

📝 Examples

See the examples/ directory for complete examples:

• browser-example.html - Browser integration
• vscode-example.ts - VS Code extension
• node-example.ts - Node.js server