Async JS Client for Voice Backend

Contents

• Intent
• Components
• Usage
• Running an Example

Intent

The rationale behind an async implementation of the JS voice client is to decouple the following sets of tasks...

• generating an audio stream
• processing this audio stream (resample, conversion, etc.)
• streaming to server over a websocket connection
• handling ASR returned from server

...such that each of the above run independently in their own thread insulated from each other and communicate via message passing.

Components

The async JS client is composed of the following components, each portraying well-defined and exclusive roles.

Streamer

Streamer is the entrypoint of the async JS client. It's role is two-fold:

• initialise the Worklet and the Executor components
• manage communication and control between all components

When a new dictation session is started by the end-user, following steps are executed:

1. Streamer is initialised along with the Worklet and the Executor
2. It initialises the audio context post which audio generation begins
3. It sends the audio packets to the Worklet for further processing
4. It receives the processed and buffered packets from the Worklet and sends it to the Executor for ASR reception, processing and presentation.
5. It repeats again from (3.)

When a dictation session is stopped by the end-user, following steps are executed:

1. Streamer stops the audio context
2. It sends DONE messages to Worklet and Executor asking them to close gracefully

NOTE: Audio source in the existing implementation is in the form of a recorded audio played in a loop. To support audio generation through a microphone, please uncomment pertinent code in Streamer.

Worklet

Worklet is employed mainly for audio processing, employing AudioWorklets. In this context, it can be used to resample the incoming audio stream before sending the same to the server. It also keeps the processed audio packets buffered for a short period of time before sending it back to the Streamer to decrease overhead of message passing between OS-level threads running itself and the Streamer.

Executor

This component deals with the following tasks:

• obtain processed audio packets from the Streamer and stream it to the server over a websocket connection
• obtain ASR from the server, process it well before pasting it to an editor screen

Executor manages a websocket connection with the server in a Web Worker. Audio packets received from the Streamer are buffered in a read-queue to be sent over to the server. It houses 3 daemons running periodically:

1. Consumer: It consumes audio packets from the read-queue and streams it to the server over a websocket connection
2. Healthcheck: It oversees the websocket connection and closes it forcefully when the server is unreachable.
3. IdleThread: It keeps track of the fact that data is sent regularly over the websocket connection. If the connection remains idle for a given period of time, this daemon closes the connection gracefully thus freeing up network and server resources.

Usage

Initialise Audio Stream

This can be employed when a new dictation session begins by the user with an intent of creating a clinical document.

const recorderInstance = new AugnitoRecorder(
  {
    serverURL: WS_URL,
    enableLogs: false,
    isDebug: false,
    bufferInterval: 1,
    EOS_Message: "EOS",
    socketTimeoutInterval: 10000,
    shouldSendAudioDataSequence: false,
  },
  heavyOp
);
recorderInstance.toggleStartStopAudioStream(); //if you want to start/stop audio on button click
or;
recorderInstance.togglePauseResumeAudioStream(); //if you want to pause/resume audio on button click

• WS_URL is the server websocket endpoint to which the client connects to, to stream audio.
• heavyOp is a CPU-intensive operation which is run on the received ASR before displaying it on the editor. A dummy example is as follows:
• enableLogs is set to true if you want to see logs in console else set to false
• isDebug is set to true if you want to save recorded audio to a file else set to false
• bufferInterval is set to any non negative integer, indicates the buffer size interval in seconds, default value is 1 sec
• EOS_Message is set to any string that indicates EOS or can be undefined, default value is EOS
• socketTimeoutInterval is set to any socket timeout interval as desired, default value is 10000
• shouldSendAudioDataSequence is set to true if you want the audio packet sequence sent with every packet as header else set to false

const heavyOp = (text) => {
  let num = -1;
  let iters = 0;
  let sum = 0;
  for (let i = 0; i < text.length; i++) {
    sum += text.charCodeAt(i);
  }
  console.debug(`Running Heavy Operation for "${text}" with sum: ${sum}`);
  while (num != sum) {
    num = Math.floor(Math.random() * 100000);
    iters++;
  }
  console.log(`Iterations completed for sum(${sum}): ${iters}`);
  return text;
};

Audio Stream Pause and Resume

This can be employed when the user pauses and resumes the dictation session emulating a toggle of the microphone button.

recorderInstance.pauseAudio();

/*
Do something interesting here...
*/

recorderInstance.resumeAudio();

Stop Audio Stream

This can be employed when the user has completed their dictation session after finishing off compiling the medical document.

recorderInstance.stopAudio();

Library Callbacks

These callbacks can be employed to read socket connection state change event, session event, speech text output event or any other error event from the library.

recorderInstance.onStateChanged = (connected) => {
  //connected is true when socket is opened and false when socket is closed
};
recorderInstance.onSessionEvent = (response) => {
  //handle all meta events
};
recorderInstance.onError = (error) => {
  //handle any error message
};
recorderIns.onPartialResult = (text) => {
  // hypothesis text output generated as the user speaks
};
recorderInstance.onFinalResult = (textResponse) => {
  //text reponse can be parsed to read the speech output json
};

Compile library

Below steps to compile library and the output is generated in dist folder

cd AugnitoRecorderJS
npm run build

Running an Example

cd AugnitoRecorderJS
python3 -m http.server

Once done, fire up the webpage at: http://localhost:8000. The webpage has following controls for speech:

• Start Stream: This begins playing the recorded audio which is streamed to the server and the ASR obtained pertaining to it is pasted on screen live. This control marks the start of a new session with the server and so a new WS connection is created for audio streaming and ASR reception. The recording and the ensuing ASR can be paused and resumed using the same control.
• Stop Stream: This stops the recorded audio from playing marking the end of the client session. The connection with the server is severed when this control is engaged.
• Go Crazy...: This control randomly starts, pauses, resumes and stops the audio streaming session in a loop. It employs the above two controls as it's building block and is a hands-free approach to communicate with the server in the form of a virtual user.