@iloveagents/foundry-web-voice

Spoken conversation for Foundry UI, backed by Microsoft Foundry Voice Live.

Optional by construction: no other @iloveagents/foundry-* package imports this one, so an app that never installs it has an unchanged composer and an unchanged runtime.

npm install @iloveagents/foundry-web-voice

import { bootstrapShell } from "@iloveagents/foundry-web-shell";
import { createVoiceModule } from "@iloveagents/foundry-web-voice";

bootstrapShell({
  modules: [
    createVoiceModule({
      connection: { proxyUrl: import.meta.env.VITE_VOICE_PROXY_URL },
    }),
  ],
});

That is the whole integration. A voice button appears in the composer (holding the Send corner until there is something to send), and while a call is live the conversation is replaced by a theme-aware visualiser — or the avatar, if you configure one — and comes straight back when the call ends. Pass stage: false to keep the transcript visible instead.

The two modes

| | relay (default) | realtime | | ----------------------- | ----------------------------------------------------- | ------------------------------- | | Who answers | your AG-UI agent — retrieval, tools, citations | the Voice Live model | | Voice Live's job | speech in, speech out (and the avatar) | the whole conversation | | Turns in the thread | ordinary turns — tool cards, citations, persisted | live transcript, session-scoped | | First word of the reply | as fast as your agent | sub-second |

relay keeps the answer yours. Voice Live transcribes what the user said, the transcript goes through the composer exactly as if it had been typed, your agent answers with its full tool and retrieval surface, and the reply is spoken back in the Azure voice you configured. Under the hood that is turn_detection.create_response: false plus pre_generated_assistant_message — both documented Voice Live features, not tricks.

realtime hands the conversation to the Voice Live model for the lowest possible latency. It can still call your client-side tools: the schemas in clientToolRegistry are sent with the session, and a call is dispatched through the same registry the typed chat uses, so ui_navigate and your page_* tools work by voice with no second declaration.

createVoiceModule({
  mode: "realtime",
  connection: { proxyUrl: VOICE_PROXY_URL },
  session: { instructions: "You are a helpful assistant. Keep answers to one or two sentences." },
});

One thing to know about realtime and history

assistant-ui merges realtime transcripts into the thread while the session is live and clears them on disconnect (disconnectVoice() empties _voiceMessages). They are never handed to a ThreadHistoryAdapter. That is the framework's design for realtime voice, not a gap here — and it is why relay is the default. Pass onTranscript if you want to persist them yourself.

An avatar

Configure one on the session and the panel renders itself once the video track arrives:

import { sessionConfig } from "@iloveagents/foundry-voice-live-react";

createVoiceModule({
  connection: { proxyUrl: VOICE_PROXY_URL },
  session: sessionConfig()
    .hdVoice("en-US-Ava:DragonHDLatestNeural")
    .avatar("lisa", "casual-sitting", { codec: "h264" })
    .build(),
});

In relay mode the avatar lip-syncs your own agent's answer, because the words are sent as a pre-generated assistant message rather than generated by the Voice Live model.

Options that matter in a room

• halfDuplex: true — mute the microphone while the assistant speaks, so on laptop speakers it cannot hear itself and answer its own voice. Costs barge-in; right on speakerphones, wrong with headsets. It never overrides a mute the user set themselves.
• chromaKey — tune the avatar's green-screen removal. The default cuts a tighter edge than the SDK's, which left a green fringe on the silhouette in live runs. Tighter, not softer: the subject sits closer to the key colour than it looks, so widening the blend band does not soften the edge — it turns the whole figure translucent.
• stage: false — keep the thread visible during a call instead of replacing it.

You need a proxy

A browser cannot set an Authorization header on a WebSocket, so the credential has to live somewhere else. Run the SDK's proxy and point proxyUrl at it:

docker run -p 8080:8080 \
  -e FOUNDRY_RESOURCE_NAME=your-foundry-resource \
  -e FOUNDRY_API_KEY="…" \
  -e ALLOWED_ORIGINS="https://your-app.example.com" \
  ghcr.io/iloveagents/foundry-voice-live-proxy:latest

connection accepts everything @iloveagents/foundry-voice-live-react does, including a per-user MSAL token and Foundry Agent Service. An apiKey in the browser is for local development only.

The default model (gpt-realtime) is available in East US 2 and Sweden Central.

Without the shell

If you compose the runtime yourself, render FoundryVoice anywhere inside AGUIRuntimeProvider:

<AGUIRuntimeProvider>
  <FoundryVoice connection={{ proxyUrl: VOICE_PROXY_URL }} />
  <ChatContent />
</AGUIRuntimeProvider>

Pass chrome: false to keep the composer untouched and place VoiceMicButton / VoiceStatusStrip wherever you want them.

How it fits together

assistant-ui 0.15 already models realtime voice: useLocalRuntime accepts a RealtimeVoiceAdapter, the thread runtime merges the transcripts it emits into thread.messages, and useVoiceControls() / useVoiceState() / useVoiceVolume() expose the session. This package implements that one interface over Voice Live — which is why the mic button is ordinary assistant-ui code and would work for any other voice backend registered the same way.

foundry-web-ui contributes only the seams: registerVoiceAdapter() and a composerActions chat slot. Neither mentions Voice Live.

API

| Export | What it is | | -------------------------------------------------------- | ------------------------------------------------------------------------------- | | createVoiceModule(config) | a bootstrapShell module | | FoundryVoice | the same thing as a component, for shell-less apps | | installVoice(config) | { bridge, Host, dispose } for full control | | VoiceMicButton, VoiceStatusStrip, VoiceAvatarPanel | the chrome, for chrome: false | | VoiceBridge | the protocol layer — no components or hooks; driven by VoiceTransportControls | | buildVoiceSession, toVoiceLiveTools | mode → Voice Live session; registry → function tools | | SentenceStream, SpeechQueue, toSpeakableText | the markdown-to-speech pipeline | | VoiceStage, VoiceVisualizer | the in-call surface and its theme-aware canvas | | HalfDuplexGate | when the mic closes/opens around assistant speech (pure, tested) |

License

MIT