symbl-real-time-nebula
v1.0.0
Published
A real time copilot assistant for sales agent to help make quick suggestions whiles they are on a call with a customer.
Downloads
2
Readme
symbl-real-time-nebula
A real time copilot assistant for sales agent to help make quick suggestions whiles they are on a call with a customer.
Requirements for initialising the SDK
To initialise the sdk(CopilotAssistantAgent
) you need to pass an object with the following parameters.
symblAccessToken
- Symbl Token. You can get it from Platform (required)nebulaApiKey
- Nebula API Key. You can get it from Nebula (required)conversationId
- Conversation ID from symbl session. (required)connectionId
- Connection ID from symbl session. (required)trackers
- An array of trackers to be used for the conversation. (optional)enableAllTrackers
- A boolean to enable all trackers. (optional) Which istrue
by default.enableInterimResults
- A boolean to enable interim results. (optional) Which istrue
by default.
NOTE: You can't pass both trackers
and enableAllTrackers
to be true
at the same time. If you pass both, trackers
will be used.
Trackers Object
The trackers
array is a collection of tracker objects. Each tracker is used to identify specific phrases or words within a given context. Trackers are primarily characterized by a name
and a vocabulary
list. An optional id
can also be included for each tracker, typically sourced from a management API.
Structure
Each element in the trackers
array is an object with the following fields:
name
(string): The name of the tracker, representing the category or type of phrases it tracks.vocabulary
(array of strings): A list of phrases or words that this tracker is responsible for identifying.id
(string, optional): A unique identifier for the tracker, usually obtained from a management API.
When the sdk is initialised, it will automatically connect to the nebula websocket and start listening for suggestions based on the first stage and the meta data provided.
const sdk = CopilotAssistantAgent({
symblAccessToken, // required
nebulaApiKey, // required
conversationId, // required
connectionId, // required
trackers, // optional
enableAllTrackers, // optional
enableInterimResults,
});
Callback functions
onSuggestions Callback
The onSuggestions
method accepts a callback function that is invoked when suggestions are available.
Syntax
sdk.onSuggestions(callback);
The callback function takes an object as a parameter. This object contains a suggestion
property which is of type string
{
suggestion: "Hi ...";
}
suggestion
- Is of type string and contains the suggestion from nebula.
sdk.onSuggestions(({ suggestion }) => {
console.log(suggestion);
});
onCompletedTranscriptions
This method is called when the sdk receives a transcription from symbl with the isFinal
as true. It returns the following parameters.
Syntax:
sdk.onCompletedTranscriptions(callback);
The callback function takes an array of objects as a parameter. This array contains the transcriptions from symbl of format below and is called messages
{
messages: [
{
from: {
id: string,
name: string,
userId: string,
},
payload: {
content: string,
contentType: "text/plain",
},
id: string,
channel: {
id: "realtime-api",
},
metadata: {
disablePunctuation: true,
originalContent: string,
words: string,
originalMessageId: string,
},
dismissed: false,
duration: {
startTime: string,
endTime: string,
timeOffset: number,
duration: number,
},
entities: [],
},
];
}
messages
- Is of type array and contains the transcriptions from symbl of format; messagesfrom
- Is of type object and contains the user or speaker information.id
- Is of type string and contains the id of the user or speaker.name
- Is of type string and contains the name of the user or speaker.userId
- Is of type string and contains the userId of the user or speaker.
payload
- Is of type object and contains the payload of the transcription.content
- Is of type string and contains the content of the transcription.contentType
- Is of type string and contains the contentType of the transcription.
id
- Is of type string and contains the id of the transcription.channel
- Is of type object and contains the channel of the transcription.id
- Is of type string and contains the id of the channel.
metadata
- Is of type object and contains the metadata of the transcription.disablePunctuation
- Is of type boolean and contains the disablePunctuation of the transcription.originalContent
- Is of type string and contains the originalContent of the transcription.words
- Is of type string and contains the words of the transcription.originalMessageId
- Is of type string and contains the originalMessageId of the transcription.
dismissed
- Is of type boolean and contains the dismissed of the transcription.duration
- Is of type object and contains the duration of the transcription.startTime
- Is of type string and contains the startTime of the transcription.endTime
- Is of type string and contains the endTime of the transcription.timeOffset
- Is of type number and contains the timeOffset of the transcription.duration
- Is of type number and contains the duration of the transcription.
entities
- Is of type array and contains the entities of the transcription.
sdk.onCompletedTranscriptions(({ messages }) => { console.log(messages); });
onRealtimeTranscript
This method is called when the sdk receives a transcription from symbl in realtime as the users speak. It returns the following parameters.
Syntax:
sdk.onRealtimeTranscript(callback);
The callback function takes an object as a parameter. This object contains a message
property which is of type object
{
message: {
type: "recognition_result",
isFinal: false,
payload: {
raw: {
alternatives: [
{
words: [],
transcript: "Hello",
confidence: 0,
},
],
},
},
punctuated: {
transcript: "Hello",
},
user: {
name: "agent",
userId: "[email protected]",
"id": "1a726dac-1c95-484d-a99e-69e36950bf32"
}
}}
message
- Is of type object and contains the transcription from symbl.type
- Is of type string and contains the type of the transcription.isFinal
- Is of type boolean and contains the isFinal of the transcription.payload
- Is of type object and contains the payload of the transcription.raw
- Is of type object and contains the raw of the transcription.alternatives
- Is of type array and contains the alternatives of the transcription.words
- Is of type array and contains the words of the transcription.transcript
- Is of type string and contains the transcript of the transcription.confidence
- Is of type number and contains the confidence of the transcription.
punctuated
- Is of type object and contains the punctuated of the transcription.transcript
- Is of type string and contains the transcript of the transcription.
user
- Is of type object and contains the user or speaker information.name
- Is of type string and contains the name of the user or speaker.userId
- Is of type string and contains the userId of the user or speaker.
id
- Is of type string and contains the id of the transcription.
To access the actual transcript, you can use the following code snippet.
sdk.onRealtimeTranscript(({ message }) => {
const { transcript } = message.puntuated;
console.log(transcript);
});
You can also access the user or speaker name using the following code snippet.
sdk.onRealtimeTranscript(({ message }) => {
const { name } = message.user;
console.log(name);
});
onSymblResponse
This method is called when the sdk receives any response from symbl. It returns the following parameters.
Syntax:
sdk.onSymblResponse(callback);
// or
sdk.onSymblResponse(({ response }) => {
console.log(response);
});
The callback function takes an object as a parameter. This object contains a response
property.
To check the possible details of the response
visit this url Syml Streaming API Docs
Also note to look of for the following in the docs after visiting the url;
Response reference
onTrackerResponse
This method is called when the sdk receives a tracker response from symbl. It returns the following parameters.
Syntax:
sdk.onTrackerResponse(callback);
// or
sdk.onTrackerResponse(({ trackers }) => {
console.log(trackers);
});
The callback function takes an array of objects. This object contains a trackers
property which is of type array
{
trackers: [
{
name: "test",
matches: [
{
type: "vocabulary",
value: "test",
messageRefs: [
{
id: "7c4b04c6-cc86-4ef8-8a97-a26c1eb9a21f",
text: "It was a test.",
offset: 9,
},
],
insightRefs: [],
},
],
},
];
}
trackers
- Is of type array and contains the trackers from symbl.name
- Is of type string and contains the name of the tracker.matches
- Is of type array and contains the matches of the tracker.type
- Is of type string and contains the type of the match.value
- Is of type string and contains the value of the match.messageRefs
- Is of type array and contains the messageRefs of the match.id
- Is of type string and contains the id of the messageRefs.text
- Is of type string and contains the text of the messageRefs.offset
- Is of type number and contains the offset of the messageRefs.
insightRefs
- Is of type array and contains the insightRefs of the match.
Functions
getTranscriptions
This method returns the transcriptions received from symbl. It accepted an option parameter called fromMemory
which is false by default. When set to true, it returns the transcriptions from memory. When set to false, it returns the transcriptions making request to symbl api. It returns as an array of objects, with the format;
[
{
"from": {
"id": string,
"name": string,
"userId": string
},
"payload": {
"content": string,
"contentType": "text/plain"
},
"id": string,
"channel": {
"id": "realtime-api"
},
"metadata": {
"disablePunctuation": true,
"originalContent": string,
"words": string,
"originalMessageId": string
},
"dismissed": false,
"duration": {
"startTime": string,
"endTime": string,
"timeOffset": number,
"duration": number
},
"entities": []
}
]
Syntax:
const transcriptions = await sdk.getTranscriptions(fromMemory);
// or
const transcriptions = await sdk.getTranscriptions(true);
stop
This method stops the sdk from listening to the symbl websocket and closes the connection.
Syntax:
sdk.stop();
callNebulaForSuggestions
This function calls the nebula api for suggestions. The suggestions are received from the onSuggestions
callback function. It returns the following parameters.
Parameters:
options
(Object): The options for calling Nebula.systemPrompt
(string): The system prompt for Nebula which is optional.messages
(Array): The messages to send to Nebula.messages[].role
(string): The role of the message (either "human" or "assistant").messages[].text
(string): The text content of the message.
You can get more information about Nebula Chat model Returns:
- A promise that resolves when the response from Nebula is processed. The return type is
Promise<void>
.
Example:
await sdk.callNebulaAPI({
systemPrompt: "Help answer question asked politely",
messages: [
{
role: "human",
text: "Agent: Hello how are you \n User: This is Samson",
},
],
});
Latency
One important requirement for the solution is to keep latency to the minimum while keeping some reasonable response from Nebula LLM.
How we measure latency?
We broke it down in different blocks:
- Capture and time to send audio to Symbl.ai (depending on your location/network that can range between 100-500 ms)
- Receive tracker events and transcription (500 - 1000 ms)
- Prompt generation and Nebula response time (~80 ms, worst case). This is the time it takes until we get first character of the Nebula response.
- Display Latency (~0ms(For the JS, sdk) and ~900ms when using the python server sidem using socket.io)
- Also before the initialisation process, that is, the connection to the websocket symbl conversation, there is a delay of about 1-2 seconds.
Results
Average latency (excluding the initial audio communication): ~1865ms
Note: External factors such as network variability, server load, and the performance of the client's device significantly impact end-to-end latency.