eLLM Studio Chat Stream Package

Welcome to the eLLM Studio Chat Stream package! 🎉 This package enables streaming chat functionality with your AI assistant in eLLM Studio via WebSocket and GraphQL. It's designed for both frontend and backend implementations.

🚀 Features

• Real-time Streaming: Receive messages in a streaming fashion.
• AI Assistant Integration: Seamlessly connects with the AI models deployed within your organization.
• Customizable: Pass custom prompts or previous messages to enhance conversations.
• Error Handling: Catch and manage errors using callbacks.
• Image Upload: Upload images to the server.
• Image Deletion: Delete previously uploaded images from the server.

📦 Installation

npm i @e-llm-studio/streaming-response

🛠️ Streaming Usage

Here’s how you can use the startChatStream function to set up the AI chat stream:

import { startChatStream } from '@e-llm-studio/streaming-response';

const params = {
    WEBSOCKET_URL: 'wss://your-ellm-deployment/graphql', // Required: WebSocket URL for your deployment
    organizationName: 'TechOrg',                         // Required: Organization name where the assistant is created
    chatAssistantName: 'MyAssistant',                    // Required: Assistant name
    selectedAIModel: 'gpt-4',                            // Required: AI model selected for the assistant
    replyMessage: '',                                    // Optional: Pass the previous response from AI.
    userName: 'John Doe',                                // Required: Username of the person using the assistant
    userEmailId: '[email protected]',                     // Required: User's email
    userId: 'user-123',                                  // Required: Unique identifier for the user
    query: 'What is the weather like today?',            // Required: The user's question or prompt
    requestId: `requestId-${uuid()}`,                    // Required: Unique ID for the request
    customPrompt: 'Personalized prompt here',            // Optional: Add custom context to your prompt
    enableForBackend: false,                             // Optional: Set to true if you're using it on the backend (Node.js)
    images: { 'image1': 'ImageName1.jpg', 'image2': 'ImageName2.jpg' } // Optional: Pass image data if relevant.
    guestSessionId: `organizationName-chatAssistantName-${uuid()}` // Optional: Custom unique id for storing chat history
    chatPreviewId:  `${uuid()}`                          //Optional: Custom unique Id for tracking chat sessions
    aiModel: encodeParam(selectedAIModel),
    isDocumentRetrieval:true,                            // To bypass the document retrieval.
    onStreamEvent: (data) => console.log('Stream event:', data),  // Required: Callback for handling stream data
    onStreamEnd: (data) => console.log('Stream ended:', data),    // Optional: Callback for when the stream ends
    onError: (error) => console.error('Stream error:', error),    // Optional: Callback for handling errors
    customMetaData: ["test1", "test2"] || {""}           // Optional: Add custom metadata for tracking or context purposes
    multiAgentToggle: boolean                            // Optional: Toggle key for orchestrator by pass
    selectedAgent: String                                // Optional: selected agent for handling the query 
    runtimeAgents:[JSON]                                 //Optional: Agents array which can be utilized at runtime
    isAudioAgent:boolean                                 //Optional: For enabling the audio agent
    agentSettings: {                                     //Optional: Override orchestrator and agent configurations
        parallelToolsCalling: true,
        isConsultativeModeOn: false,
        consultativePrompt: "Custom consultative prompt...",
        orchestratorPrompt: "Custom orchestrator prompt...",
        modelName: "gpt-4o",
        modelType: "OpenAI",
        overrideAgentConfig: {
            "AgentName": {
                "properties": {
                    "when_to_use": {"value": "Use when user asks about..."},
                    "when_not_to_use": {"value": "Don't use when..."}
                }
                "tool_name":{
                   "DocRetrievalTool":{
                       "properties":{
                            "retrieval_limit":{"value":1}
                }
            }
                }
            }
        }
    },
  textToSpeechAgentId,                                     //Optional: Here we need to Pass the RTTA agent ID -> type: String                      
  useTextToSpeech,                                         //Optional: Flag for RTTA is ON/OFF -> type: Boolean     
  XCrossAccessToken?: string;                             //Optional: accessToken for authenticate the user -> type: string     
};
startChatStream(params);

import { startChatStreamSSE } from '@e-llm-studio/streaming-response';
const params = {
    API_URL: 'https://your-ellm-deployment',             // Required: API URL for your deployment without backslash
    organizationName: 'TechOrg',                         // Required: Organization name where the assistant is created
    chatAssistantName: 'MyAssistant',                    // Required: Assistant name
    selectedAIModel: 'gpt-4',                            // Required: AI model selected for the assistant
    replyMessage: '',                                    // Optional: Pass the previous response from AI.
    userName: 'John Doe',                                // Required: Username of the person using the assistant
    userEmailId: '[email protected]',                     // Required: User's email
    userId: 'user-123',                                  // Required: Unique identifier for the user
    query: 'What is the weather like today?',            // Required: The user's question or prompt
    requestId: `requestId-${uuid()}`,                    // Required: Unique ID for the request
    customPrompt: 'Personalized prompt here',            // Optional: Add custom context to your prompt
    enableForBackend: false,                             // Optional: Set to true if you're using it on the backend (Node.js)
    images: { 'image1': 'ImageName1.jpg', 'image2': 'ImageName2.jpg' } // Optional: Pass image data if relevant.
    guestSessionId: `organizationName-chatAssistantName-${uuid()}` // Optional: Custom unique id for storing chat history
    chatPreviewId:  `${uuid()}`                          //Optional: Custom unique Id for tracking chat sessions
    aiModel: encodeParam(selectedAIModel),
    isDocumentRetrieval:true,                            // To bypass the document retrieval.
    onStreamEvent: (data) => console.log('Stream event:', data),  // Required: Callback for handling stream data
    onStreamEnd: (data) => console.log('Stream ended:', data),    // Optional: Callback for when the stream ends
    onError: (error) => console.error('Stream error:', error),    // Optional: Callback for handling errors
    customMetaData: ["test1", "test2"] || {""}           // Optional: Add custom metadata for tracking or context purposes
    multiAgentToggle: boolean                            // Optional: Toggle key for orchestrator by pass
    selectedAgent: String                                // Optional: selected agent for handling the query 
    runtimeAgents:[JSON]                                 //Optional: Agents array which can be utilized at runtime
    isAudioAgent:boolean                                 //Optional: For enabling the audio agent
    agentSettings: {                                     //Optional: Override orchestrator and agent configurations
        parallelToolsCalling: true,
        isConsultativeModeOn: false,
        consultativePrompt: "Custom consultative prompt...",
        orchestratorPrompt: "Custom orchestrator prompt...",
        modelName: "gpt-4o",
        modelType: "OpenAI",
        overrideAgentConfig: {
            "AgentName": {
                "properties": {
                    "when_to_use": {"value": "Use when user asks about..."},
                    "when_not_to_use": {"value": "Don't use when..."}
                }
                "tool_name":{
                   "DocRetrievalTool":{
                       "properties":{
                            "retrieval_limit":{"value":1}
                }
            }
                }
            }
        }
    },
  textToSpeechAgentId,                                     //Optional: Here we need to Pass the RTTA agent ID -> type: String                      
  useTextToSpeech,                                         //Optional: Flag for RTTA is ON/OFF -> type: Boolean     
  XCrossAccessToken?: string;                             //Optional: accessToken for authenticate the user -> type: string    
  userPrefferedLanguage,                                  //Optional
  agent_parameter,                                        //Optional: useAgent to bypass orchestrator and use particular agent details
  orchestratorMetadata,                                   //Optional: provide orchestrator metadata
  additionalHeaders = {},                                 //Optional
};
startChatStreamSSE(params);

import { stopStreamingSSE } from '@e-llm-studio/streaming-response';
const params = {
 requestId: ""                                             //Required: requestId for question which you need to stop streaming
  API_URL: 'https://your-ellm-deployment',                 // Required: API URL for your deployment without backslash
}

🔑 Parameters

Required Parameters:

• API_URL/WEBSOCKET_URL: WebSocket URL of your eLLM deployment. Example: wss://dev-egpt.techo.camp/graphql.
• organizationName: Name of the organization where the assistant is created.
• chatAssistantName: Name of the assistant you're interacting with.
• selectedAIModel: The AI model used (e.g., GPT-4).
• userName: The name of the user interacting with the assistant.
• userEmailId: Email ID of the user.
• userId: Unique user ID.
• query: The question or prompt you want to send to AI.
• requestId: Unique request ID, e.g., requestId-${uuid()}.
• onStreamEvent: Callback function to capture incoming stream events.

Optional Parameters:

• replyMessage: If you want to include a previous response with the new query, pass it here. Leave empty for normal chat scenarios.
• customPrompt: Use this to add additional context to the prompt sent to the AI.
• images: Pass image data if relevant.
• isDocumentRetrieval: To bypass the document retrieval.
• Annotations & Citations: Dynamically display annotations with tooltip-based citations.
• enableForBackend: Set to true if you're using this package in backend e.g. NodeJs. Defaults to false, which is suitable for frontend use e.g. React/Next.js.
• onStreamEnd: Callback for when the stream ends. Useful for handling final events or cleanup.
• onError: Callback for capturing any errors during the stream.
• guestSessionId: Custom unique id for storing chat history
• chatPreviewId: Custom unique Id for tracking chat sessions
• runtimeAgents: Agents array which needs to be utilized at runtime
• isAudioAgent: For enabling the audio agent
• agentSettings: Object to override orchestrator and agent configurations (see detailed section below)
• orchestratorMetadata: orchestrator metadata

⚙️ Agent Settings Configuration

How to Use Specific Agents on Runtime

The agentSettings parameter allows you to override the default orchestrator and agent configurations stored in MongoDB. This provides dynamic control over AI behavior, model selection, and agent-specific settings at runtime.

🎯 Key Benefits

• Runtime Configuration: Override settings without modifying database configurations
• Dynamic Model Selection: Change orchestrator LLM modelType and modelNames
• Agent Customization: Modify agent behavior, prompts, and usage criteria
• Flexible Deployment: Different settings for different users or contexts
• A/B Testing: Test different configurations easily

📋 AgentSettings Object Properties

The agentSettings parameter accepts an object with the following optional properties: When calling the startChatStream function, one can include a runtimeAgents parameter in the params object:

agentSettings: {
  isParallelToolCall?: boolean,           // Enable/disable parallel agent execution
  isConsultativeModeOn?: boolean,         // Toggle consultative conversation mode
  consultativePrompt?: string,            // Custom prompt for consultative mode
  additionalPoints?: string,              // Additional context or instructions
  orchestratorPrompt?: string,            // Custom orchestrator system prompt
  modelName?: string,                     // Override orchestrator LLM model (e.g., "gpt-4o")
  modelType?: string,                     // Override orchestrator LLM provider (e.g., "OpenAI", "Anthropic")
  filterCitations?: boolean,              // Enable/disable citation filtering
  filterCitationsThreshold?: number,      // Threshold for citation confidence (0-100)
  ShareOrchestratorHistory?: boolean,     // Share conversation context between agents
  directAgentTrace?: string,              // Custom agent trace identifier
  overrideAgentConfig?: object,           // Agent-specific configuration override (agent_config)
  credentials?: object,                   // Custom API credentials for models
}

import { startChatStream } from '@e-llm-studio/streaming-response';


// Include runtime agents in the params
const params = {
  WEBSOCKET_URL: `${process.env.NEXT_PUBLIC_WEBSOCKET_URL}`,
  organizationName: 'yourOrganization',
  chatAssistantName: chatAssistantName,
  selectedAIModel: 'bestai',
  // ... other parameters ...
  runtimeAgents: runtimeAgents,
  agentSettings:agentSettings,
  onStreamEvent: (data) => {
    // Handle streaming data
  },
  onStreamEnd: () => setIsStreaming(false),
  onError: (error) => console.error('Stream error:', error),
};

// Start the streaming process
startChatStream(params);

Runtime Agents Structure

The runtimeAgents parameter should be an Agents array which can be utilized at runtime

• _id: A unique identifier for the agent
• title: The name/title of the agent

Example:

const runtimeAgents = [
  { _id: 'ObjectId', title: 'RAGAgent' }, 
  { _id: 'ObjectId2', title: 'SQLAgent' }
];

🛠️ Stop Streaming

The stopStreaming function allows you to gracefully stop an ongoing chat stream and clean up WebSocket connections.

Function Signature

export const stopStreaming = async (
  reqid: string,
  WEBSOCKET_URL: string,
  enableForBackend?: boolean
): Promise<{ success: boolean, message?: string, error?: any }>;

Parameters

• reqid (string): The request ID used in startChatStream
• WEBSOCKET_URL (string): The WebSocket URL of your eLLM deployment
• enableForBackend (boolean, optional): Set to true when using in Node.js environment (default: false)

Return Value

Returns a Promise that resolves to an object with:

• success (boolean): Indicates whether the operation was successful
• message (string, optional): Success or error message
• error (any, optional): Error details if operation fails

Example Usage

import { startChatStream, stopStreaming } from '@e-llm-studio/streaming-response';

const handleChat = async () => {
  const requestId = `request-${uuid()}`;
  
  // Start chat stream
  startChatStream({
    WEBSOCKET_URL: 'wss://your-ellm-deployment/graphql',
    requestId,
    // ... other required params
  });

  // Stop chat stream when needed
  const handleStop = async () => {
    try {
      const result = await stopStreaming(
        requestId,
        'wss://your-ellm-deployment/graphql',
        false 
      );
      
      if (result.success) {
        console.log("Stream stopped successfully:", result.message);
      } else {
        console.error("Failed to stop stream:", result.error);
      }
    } catch (error) {
      console.error("Error stopping stream:", error);
    }
  };
};

Important Notes

1. Connection Cleanup: The function automatically cleans up both the original streaming connection and the stop request connection.
2. Timeout: Includes a built-in timeout (1 second) to prevent hanging connections.
3. Backend Usage: Set enableForBackend: true when using in Node.js environment.
4. Error Handling: Properly handles and reports errors during the stopping process.
5. WebSocket Management: Ensures all WebSocket connections are properly disposed.

Best Practices

1. Always store the requestId when starting a stream to use it for stopping.
2. Use the same WEBSOCKET_URL for both starting and stopping.
3. Handle errors appropriately in your application.
4. Clean up any UI states after stopping the stream.
5. Implement proper error boundaries in your React components.

🛠️ Image Upload

You can upload an image to the server using the uploadImage function:

import { uploadImage } from "./index";

const imageFile = document.querySelector('input[type="file"]').files[0];

uploadImage(baseURL, imageFile)
  .then((message) => console.log(message))
  .catch((error) => console.error(error));

🛠️ Image Deletion

To delete an image from the server, use the deleteImage function:

import { baseURL, name, deleteImage } from "./index";

deleteImage("User Name", "image.jpg")
  .then((message) => console.log(message))
  .catch((error) => console.error(error));

🛠️ Consultative Mode

The Consultative Mode allows you to toggle a specialized conversation style with the AI assistant. This mode ensures the AI follows a specific set of guidelines to better understand user requirements, ask clarifying questions, and provide tailored responses.

Default Prompt: If no custom prompt is provided, the following default prompt is used:

"You are responsible for understanding the user’s needs by making sure that you gather enough relevant information before responding to achieve the goal described above. To achieve this, you will always ask clarifying follow-up questions, making sure to ask only one question at a time to maintain a focused and organized conversation. The follow-up questions should be short and crisp within 2 lines only. You should always follow a CARE framework which is about Curious:- being curious and interested to know about user needs, Acknowledge the responses given by the user and Respond with Empathy. Do not ask too many unnecessary question

🛠️ Delete Message

This API allows you to delete a specific message from a chat.

📌 Endpoint DELETE /api/chat/delete-message

🔹 Request Headers Header Type Description Authorization Bearer Token Required authentication token

🛠️ Usage:

Example:

import { ChatbotService } from "@e-llm-studio/streaming-response";

const chatbotService = new ChatbotService(
  "https://your-api-url.com", // API base URL
  "username", // Your username
  "password" // Your password
);

const toggleConsultativeMode = async () => {
  try {
    const chatBotId = "chatbot-id-123"; // Chatbot ID
    const assistantName = "MyAssistant"; // Assistant Name
    const organizationName = "TechOrg"; // Organization Name
    const currentMode = false; // Current consultative mode state
    const isFetch = false; // Whether to fetch or toggle
    const customPrompt = "Custom consultative mode prompt here"; // Optional

    const isModeEnabled = await chatbotService.toggleConsultativeMode(
      chatBotId,
      customPrompt,
      assistantName,
      organizationName,
      isFetch,
      currentMode
    );

    console.log("Consultative Mode:", isModeEnabled ? "Enabled" : "Disabled");
  } catch (error) {
    console.error("Error toggling consultative mode:", error);
  }
};

toggleConsultativeMode();

# Parallel Tool Streaming Response Handling

getAssistantObj(data, accordianMessageArray)
Purpose
The getAssistantObj function processes streaming responses from the orchestrator and agents. It dynamically manages messages in an accordion-style structure, supporting:

Orchestrator Messages:
Handles normal messages, thinking breaks, and final responses.
Agent Messages:
Supports parallel tool responses.
Manages explainability IDs for tracking responses.
Parameters
Parameter	Type	Description
data	any	Incoming streaming response data.
accordianMessageArray	any	Array to store processed messages for UI rendering.
Functionality Breakdown
Orchestrator Response Handling

Direct messages are added normally.
Messages with thinking breaks are stored in accordianMessageArray with finalResponseStatus: false.
Final orchestrator messages are processed separately.
Agent Response Handling

Messages are collapsed by default.
Supports explainability tracking using explainabilityMap to ensure responses are updated in the right place.
If a message corresponds to an existing explainability ID, it updates the previous entry.
Otherwise, it appends a new response.
Explainability Tracking

Uses a Map<string, { responded_by: string; index: number }> to track agent responses.
Ensures correct messages are updated instead of duplicated.
import { getAssistantObj } from "@e-llm-studio/llmstudioui";

const data = {
  chatStream: {
    continueChatResult: {
      message: { content: "Hello from AI" },
      is_orchestrator: true,
      isEnd: false,
      thinking_break: true,
      responded_by: "AI Assistant",
      header_message: "Processing...",
      explainability_id: "12345",
    },
  },
};

const accordianMessages = [];
const response = getAssistantObj(data, accordianMessages);

console.log(response);

 const  orchestratorAndAgentsMessages = [
    {
      "responsedata": "Hello from AI",
      "responseFrom": "AI Assistant",
      "header_message": "Processing...",
      "finalResponseStatus": false,
      "explainability_id": "12345",
      "isParallelToolResponse": false
    }
  ]

🛠️ Get Agent Settings

This function allows to retrieve agent settings for a given assistant on the basis of assistant name and organization name. Assuming you have intiaized ChatbotService, you can directly use the function from the same class.

🛠️ Usage:

Example:

import { ChatbotService } from "@e-llm-studio/streaming-response";

const chatbotService = new ChatbotService(
  "https://your-api-url.com", // API base URL
  "username", // Your username
  "password" // Your password
);

const getAgentSettings = async () => {
  try {
    const assistantName='assistantName';
    const organizationName='techolution'

    const agentSettings = await chatbotService.getAgentSettings(assistantName, organizationName);

    console.log("Agent settings:", agentSettings);
  } catch (error) {
    console.error("Error in retrieving agent settings:", error);
  }
};

🛠️ Get Models

This function allows to retrieve models supported by LLM Studio. The response would be a api response with message field and other fields which would contain models in an array

🛠️ Usage:

Example:

import { ChatbotService } from "@e-llm-studio/streaming-response";

const chatbotService = new ChatbotService(
  "https://your-api-url.com", // API base URL
  "username", // Your username
  "password" // Your password
);

const getModels = async () => {
  try {
    const models = await chatbotService.getModels();
    console.log("Models: ", models);
  } catch (error) {
    console.error("Error in retrieving models:", error);
  }
};

🧠 getChatbotData

This function fetches detailed chatbot configuration and metadata for a given assistant in a specified organization from the LLM Studio backend. It returns a full chatbot configuration object including settings, capabilities, and preferences associated with the assistant.

🛠️ Usage:

Example:

import { ChatbotService } from "@e-llm-studio/streaming-response";

const chatbotService = new ChatbotService(
  "https://your-api-url.com", // API base URL
  "username", // Your username
  "password" // Your password
);

import { ChatbotService } from "@e-llm-studio/streaming-response";

const chatbotService = new ChatbotService(
  "https://your-api-url.com", // API base URL
  "username", // Your username
  "password" // Your password
);

const getChatbotConfig = async () => {
  try {
    const chatbotData = await chatbotService.getChatbotData("assistantName", "organizationName");
    console.log("Chatbot Configuration: ", chatbotData);
  } catch (error) {
    console.error("Error in retrieving chatbot data:", error);
  }
};
 📌 Parameters
| Name               | Type     | Description                        |
| ------------------ | -------- | ---------------------------------- |
| `assistantName`    | `string` | The name of the assistant chatbot. |
| `organizationName` | `string` | The name of the organization.      |

📦 Response

Returns a JSON object containing the chatbot’s configuration and metadata including model, voice agent settings, transcription preferences, etc.

Note

If you want to have very basic details of chatbot then use getBasicChatbotDetails. Usage will be same just change getChatbotData with getBasicChatbotDetails.

🔐 getAccessToken

This function performs user authentication by sending login credentials to the LLM Studio backend. Upon successful authentication, it retrieves and stores a bearer access token for future API requests.

🛠️ Usage

import { ChatbotService } from "@e-llm-studio/streaming-response";

const chatbotService = new ChatbotService(
  "https://your-api-url.com", // API base URL
  "username", // Your username
  "password" // Your password
);

const login = async () => {
  try {
    const accessToken = await chatbotService.getAccessToken("your-username", "your-password");
    console.log("Access Token: ", accessToken);
  } catch (error) {
    console.error("Login failed:", error);
  }
};

📌 Parameters

| Name | Type | Description | | ---------- | -------- | -------------------------- | | username | string | The user's login username. | | password | string | The user's login password. |

📦 Response

Returns a string — the bearer access token used for authenticated API requests.

👥 Community & Support

For any questions or issues, feel free to reach out via our GitHub repository or join our community chat! We’re here to help. 😊