@ambivio/voice

Let your users dictate into your web application with Ambivio Voice.

Your page registers its fields as dictation targets. When the user presses Ambivio's dictation hotkey (or a PowerMic button) while one of those fields has the cursor, Ambivio records, transcribes and formats the dictation, and hands the result to your page, which writes it into the field. The user keeps working in your application; nothing is pasted and the field does not even need the cursor any more when the text arrives.

The SDK records no audio itself. It is a bridge to Ambivio Desktop running on the same machine, which keeps the hotkeys, the microphone, the user's settings and skills, and the dictation history.

Requirements

• Ambivio Desktop on the same machine as the browser (or in the same remote desktop session), with Diktat SDK in anderen Anwendungen switched on under Settings → Dictation. It is off by default.
• Chrome or Edge 147+, or Firefox 151+. Safari is not supported; the SDK reports unsupported there.
• The first time, the browser asks the user whether your site may connect to apps on this device. IT can pre-approve your site instead, see Browser permission.

Install

npm install @ambivio/voice

import { AmbivioVoice } from '@ambivio/voice'

Without a build step, load the script build, which defines the global AmbivioVoice:

<script src="https://cdn.jsdelivr.net/npm/@ambivio/[email protected]/dist/ambivio-voice.js"></script>

Quick start

const voice = AmbivioVoice.connect({ app: 'Example RIS', badge: { text: 'RIS' } })

const target = voice.registerDictationTarget({
	...AmbivioVoice.textInput(document.querySelector('#impression')),
	label: 'Impression',
})

// When the user leaves the examination:
target.dispose()

That is all a plain <input> or <textarea> needs. Put the cursor in the field, press Ambivio's smart dictation hotkey, speak, press it again.

Connecting

const voice = AmbivioVoice.connect({
	app: 'Example RIS', // shown in Ambivio's logs and its "Last dictations" menu
	badge: { text: 'RI' }, // or { icon: 'data:image/svg+xml,…' }
})

voice.status // 'connecting' | 'ready' | 'notRunning' | 'unsupported'
voice.onStatusChange((status) => updateMicIcon(status)) // returns an unsubscribe function
voice.launch() // from a click: starts Ambivio if it is installed but not running
voice.disconnect() // for good, e.g. when a single-page app tears down

| Status | Meaning | | --- | --- | | connecting | Looking for Ambivio, or waiting for the user to answer the browser's permission prompt. | | ready | Connected. Dictation into registered targets works. | | notRunning | Ambivio is not running, or Diktat SDK in anderen Anwendungen is switched off. The SDK keeps retrying in the background. | | unsupported | This browser cannot reach Ambivio: Safari, a denied permission, or an Ambivio too old or too new for this SDK. |

The connection survives Ambivio restarting: the SDK reconnects and registers everything again by itself. Targets can be registered at any time, also before Ambivio is running.

Badge. One or two characters, or a PNG or SVG data URL of at most 16 KB, drawn at about 16 px next to Ambivio's cursor indicator while a dictation goes to your page. It tells the user where the text is going while they look at another window. Anything else is ignored.

launch() opens ambivio-desktop://bridge, which starts Ambivio without bringing its window forward. Call it from a click, for example on a "Start Ambivio" button you show while the status is notRunning. The browser may ask once whether your page may open Ambivio. It cannot switch on the SDK setting; if Ambivio is running with the setting off, it does nothing.

Dictation targets

const target = voice.registerDictationTarget({
	label: 'Findings', // names the field to the model and in the history
	format: 'html', // 'plain' (default) | 'html'
	isFocused: () => editor.hasFocus(),
	insert: ({ text, position }) => editor.insertAt(position ?? editor.end(), text),

	// Optional, but only together:
	getValue: () => ({ before: editor.htmlBefore(), after: editor.htmlAfter(), position: editor.bookmark() }),
	replace: (text) => editor.setHtml(text),

	skills: [{ name: 'Findings style', text: 'One paragraph per organ system.' }],
	actions: [/* see below */],
	onStateChange: (state) => {}, // 'recording' | 'processing' | 'done' | 'failed' | 'cancelled'
})

await target.startDictation() // optional, for a microphone button in your page
await target.stopDictation()
target.dispose()

• isFocused is asked when the user presses the hotkey. Answer quickly and synchronously. The SDK also requires the page to have focus, so a background tab never counts.
• insert is the only required write. It is called when the result arrives, whether or not the field still has the cursor. Insert where you are told, not at the current cursor.
• getValue and replace give the field editing commands ("delete the last sentence", "make the second paragraph a list"). The model then sees the field's content with the cursor marked, and may answer with a complete replacement. Without them the field only takes inserts.
• position is whatever your editor uses to remember a place: an offset, a ProseMirror position, a CKEditor model position, a bookmark. It never leaves the page. The SDK takes it from getValue when the dictation starts and hands it back to insert, so the text lands where the cursor was when the user started speaking, even if they clicked elsewhere since. After each step the SDK calls getValue again, so a second insert of the same dictation goes after the first.
• Your users' typing is never overwritten. Before replace, the SDK calls getValue again. If before + after changed since the dictation started, the replacement is skipped, the state becomes failed, and the text waits in Ambivio's "Last dictations" menu.
• format: 'html' lets the model format: headings, paragraphs, bold, italics, underline and lists (h1–h3, p, br, strong, em, u, ul, ol, li). Sanitize before you insert it. The model is instructed to stay within those tags, but your page is where that is enforced.
• dispose ends a target for good. A result still on its way for it is never delivered, not even to a target registered later with the same label. Dispose your targets when the user leaves an examination, and a late result can never land in the next patient's report.

States

| State | Meaning | | --- | --- | | recording | Ambivio is recording for this target. | | processing | Recording stopped, the dictation is being transcribed. | | done | Everything was written into the field. | | failed | Something could not be delivered: the page was gone, the target disposed, a callback threw, or a replacement was skipped. The text is in Ambivio's "Last dictations" menu. | | cancelled | The user cancelled with Escape, or the recording was too short. Nothing was sent. |

Microphone button

startDictation() starts a smart dictation into the target without the hotkey. Ambivio only honours it while the browser is the app in front, so call it from a click. Stop with stopDictation(), the hotkey, or Escape to cancel.

micButton.addEventListener('click', async () => {
	if (recording) return void target.stopDictation()
	const outcome = await target.startDictation()
	if (outcome !== 'started') showHint(outcome)
})

| Outcome | Meaning | | --- | --- | | started | Recording. onStateChange('recording') fires as well. | | busy | Another dictation is being recorded or processed. | | notFrontmost | The browser was not the app in front. | | notConnected | Ambivio is not connected; see voice.status. | | unavailable | The target is disposed, or Ambivio has dictation switched off. | | failed | Ambivio could not start recording; its indicator tells the user why. |

Rich text editors

The SDK never touches the DOM except through textInput. A rich editor gets a small adapter against its own API. With TipTap, for example:

// htmlOf(editor, from, to) and sanitize(html) are yours: a serializer for a range of the
// document, and your HTML sanitizer.
const editorTarget = voice.registerDictationTarget({
	label: 'Findings',
	format: 'html',
	isFocused: () => editor.isFocused,
	getValue: () => {
		const at = editor.state.selection.to
		const doc = editor.state.doc
		return {
			before: htmlOf(editor, 0, at),
			after: htmlOf(editor, at, doc.content.size),
			position: at,
		}
	},
	insert: ({ text, position }) =>
		editor.chain().insertContentAt(position ?? editor.state.doc.content.size, sanitize(text)).run(),
	replace: (html) => editor.commands.setContent(sanitize(html)),
})

before and after only need to be stable for the same content: the SDK compares them to detect the user's edits, and the model reads them to understand the text around the cursor.

Skills and actions

Skills are standing instructions for the model, actions are things your page can do. Give them for the whole application with configure, or per target. The user's own Ambivio skills always apply on top, and win where they conflict.

voice.configure({
	skills: [{ name: 'Report style', text: 'Formal radiology German, no abbreviations.' }],
	actions: [
		{
			name: 'setPriority',
			description: 'Marks the examination as urgent or routine',
			params: {
				level: { type: 'option', options: [{ value: 'urgent', label: 'Urgent' }, { value: 'routine', label: 'Routine' }] },
			},
			run: ({ level }) => setPriority(level),
		},
		{
			name: 'fillReport',
			description: 'Fills the report when findings and impression are dictated together',
			params: {
				findings: { type: 'text', format: 'html' },
				impression: { type: 'text', format: 'html' },
				technique: { type: 'text', optional: true },
			},
			run: ({ findings, impression, technique }) => fillReport({ findings, impression, technique }),
		},
	],
})

• option parameters take one of the declared values. The SDK only ever passes declared values to run; anything else is dropped before it reaches the page.
• text parameters take dictated text, plain or HTML. They are how one utterance fills several fields: "Findings: … Impression: …" becomes one fillReport call. Sanitize HTML values like any other.
• Required parameters are always present when run is called. Optional ones the model left out are absent.
• A target's action wins over an application action of the same name.
• Text only ever goes into the target the dictation is bound to. Anything else, such as another field, goes through an action.
• Steps run in the order the user spoke them, and each waits for the previous one, including an async run.

Limits, per application and per target: 10 skills of up to 4,000 characters, 20 actions with up to 10 parameters, 100 options per parameter. A field value sent along for editing commands may have up to 100,000 characters. The SDK warns in the console about anything it has to drop.

Browser permission

Chrome, Edge and Firefox ask the user once per site before a page may connect to apps on the same device (Local Network Access). The SDK waits for the answer; status stays connecting meanwhile. If the user denies it, status becomes unsupported.

IT can pre-approve your site so nobody is asked:

• Chrome: policy LoopbackNetworkAllowedForUrls, e.g. Software\Policies\Google\Chrome\LoopbackNetworkAllowedForUrls\1 = https://ris.example.com.

• Edge: the same policy under Software\Policies\Microsoft\Edge\LoopbackNetworkAllowedForUrls.

• Firefox: policy LocalNetworkAccess with SkipDomains:

  { "policies": { "LocalNetworkAccess": { "SkipDomains": ["ris.example.com"] } } }

If the SDK runs inside a cross-origin iframe, the iframe needs allow="loopback-network".

How it works

The SDK connects over a WebSocket to 127.0.0.1 on port 43581, or 43582 when that one is taken. Nothing goes through the internet on the way between your page and Ambivio; the dictation itself is processed by Ambivio the way all its dictations are.

On terminal servers with many sessions, one Ambivio holds the port and hands each page on to the Ambivio of its own session. Your page does not notice; it just connects.

Any page the user allowed to connect can register targets, but a dictation only ever goes to a page that had focus when the user pressed the hotkey, and the microphone only starts from the browser that is in front. A recording always shows on Ambivio's cursor indicator.