@microsoft/voice-widget

The headless core of the Voice Agent Widget SDK — createVoiceAgent: session lifecycle, state, audio, and provider orchestration, with no UI. Depends only on @microsoft/voice-widget-core-client — no provider-specific code.

This package is the BYO-UI escape hatch: import it directly when you want full brand control and will build your own UI. If you want the supported, ready-made UI, use @microsoft/voice-widget-ui (or the one-line @microsoft/voice-widget-embed embed), which are built on this core.

Exports

• createVoiceAgent(options) — creates a headless controller that creates a provider from the core-client registry, optionally acquires provider-specific session material via authEndpoint / getSession, and manages the connection. Exposes state (status / mode / muted / error), a subscribe/getState store, and start() / stop() / setMuted() / sendText() / audio-level access.
• VoiceAgentCore.registerClientTool(name, handler) — dynamically register a client-tool handler after mount. Requires features.dynamicClientTools on the provider. Returns an idempotent unregister. Duplicate names throw — unregister first to replace. name is case-sensitive and must match the server-declared schema exactly — a mismatch is the usual reason a tool never fires (it surfaces via onUnhandledClientToolCall).
• VoiceAgentOptions.onUnhandledClientToolCall — notification-only callback fired when the agent calls a name with no registered handler. Cannot fulfill the call; the adapter returns a standard error output. Listen to observe missing registrations.
• VoiceAgentOptions.onRawEvent — escape hatch: every raw provider event, unmapped and untyped. For debugging/telemetry and provider-specific fields the normalized events do not carry. High frequency (the Voice Live adapter fires it per data-channel message) and not part of the normalized contract — keep the handler cheap and don't build product behavior on it.
• VoiceAgentOptions.onTelemetryEvent — structured, correlatable telemetry: one typed VoiceAgentTelemetryEvent per lifecycle/transport event (attempt started, grant resolved, broker request/response, session-id resolved, status change, disconnect, failure, retirement), each wrapped in a widgetInstanceId / correlationId / sessionId envelope for tracing a session across the widget and your broker. Unlike onRawEvent this is a stable contract and is low-frequency. Fault-isolated — a throw is reported once, then suppressed — so it can't break a call. See docs/telemetry.md.
• VoiceAgentOptions.telemetryConsole — when true, telemetry events are also written to console.debug. Default false; never auto-enabled. A local-dev sink only — it adds an output destination and changes no event, schema, or behavior.
• Types: VoiceAgentState, VoiceAgentError, VoiceAgentOptions, VoiceAgentCore, VoiceAgentTelemetryEvent.

Client tools run untrusted input. A handler's arguments are filled in by the model and can be prompt-injected — validate them before any sensitive action (e.g. allow only http:/https: URLs before navigating; never eval or inject a model-supplied string as HTML/JS). Its return value is sent to the model and may be spoken aloud, so return only end-user-safe values — no PII or internal error detail. See Security - handler inputs and outputs.

Usage (bring your own UI)

import { createVoiceAgent } from "@microsoft/voice-widget";
import "@microsoft/voice-widget-provider-voicelive"; // self-registers the "voicelive" provider

const core = createVoiceAgent({
  provider: "voicelive",
  config: { targetType: "model", model: "gpt-realtime" },
  authEndpoint: "https://your-broker.example.com/session",
});

// Bind to your framework — e.g. React's useSyncExternalStore:
// const state = useSyncExternalStore(core.subscribe, core.getState);
// …render YOUR buttons / panel / visualizer from `state`, calling
// core.start() / core.stop() / core.setMuted() / core.sendText("Hello").

This package bundles no provider. Any registered one works — swap the import and the provider name. See Writing a provider.

sendText(text) requires a connected provider with capabilities.supportsText, rejects blank input, and emits the local user turn through onTranscript after the provider accepts it.

stop() releases provider media resources and, for authEndpoint sessions, closes the broker control session through POST {authEndpoint}/end. A passive disconnect/error is terminal rather than auto-reconnected; call start() again to create a fresh session with a new SDP negotiation. Server-side conversation state is not resumed; automatic reconnect and resume are not supported.

For a first-class React binding (<VoiceAgent/> + useVoiceAgent()) over this core, see @microsoft/voice-widget-react.

Scope: model & agent targets over WebRTC via the Media Gateway.