@runtypelabs/voice

Browser voice calls to Runtype, with a framework-independent client, React hook, and Persona adapter. The dashboard uses the same client as custom applications. The core has no runtime dependencies; React and Persona are optional peers.

React

pnpm add @runtypelabs/voice

'use client'

import { useVoiceClient } from '@runtypelabs/voice/react'

export function AgentVoice({ agentId, clientToken }: { agentId: string; clientToken: string }) {
  const voice = useVoiceClient({ agentId, clientToken })
  const inCall = voice.status !== 'idle' && voice.status !== 'error'
  return (
    <div>
      <p role="status">{voice.error ?? voice.status}</p>
      <button onClick={() => (inCall ? voice.endCall() : void voice.startCall())}>
        {inCall ? 'End call' : 'Start call'}
      </button>
      {inCall && <button onClick={voice.toggleMute}>{voice.isMuted ? 'Unmute' : 'Mute'}</button>}
      {voice.canCancel && <button onClick={voice.cancelResponse}>Stop response</button>}
      {voice.transcript.map((entry, index) => (
        <p key={index}>
          {entry.role}: {entry.content}
        </p>
      ))}
    </div>
  )
}

Pass apiUrl: 'https://api.runtype-staging.com' for staging or a full HTTP(S) / WebSocket base URL for another environment. The default is https://api.runtype.com. Proxy path prefixes are preserved: https://example.com/api connects through wss://example.com/api/ws/agents/{agentId}/voice. The proxy must forward WebSocket upgrades and the Sec-WebSocket-Protocol header to the API. Use a browser client token authorized for the agent and embedding origin. Do not pass a server API key. Tokens use the WebSocket subprotocol, never the URL. clientToken can also be a function returning a token or a promise; it is called once per new call. startCall(tokenOverride) overrides that call's token.

The hook ends the call on unmount and when agentId, apiUrl, artifacts, or sessionId selects a different session. Copying the session ID reported by onSession back into the hook preserves the active call, including after session renewal. The snapshot it spreads carries artifacts and session alongside transcript. Changing clientToken affects the next call without interrupting the current one. Startup and connection failures populate error and status: 'error'.

Plain JavaScript

import { VoiceClient } from '@runtypelabs/voice'

const voice = new VoiceClient({ agentId, clientToken })
const unsubscribe = voice.subscribe(() => renderVoiceState(voice.getSnapshot()))
startButton.onclick = () => {
  void voice.startCall()
}
stopButton.onclick = () => voice.cancelResponse()

// Call when the owning view is removed.
function dispose() {
  unsubscribe()
  voice.endCall()
}

Construction and module imports are safe during server rendering. Starting a call requires a browser secure context, microphone permission, WebSocket, AudioContext, and AudioWorklet. Initiate calls from a user click. A restrictive CSP must allow the configured WebSocket origin and the blob AudioWorklet module. This is a browser package, not a React Native audio implementation.

Persona

The adapter implements Persona 4.22's VoiceProvider API and delivers transcripts through onTranscript, avoiding duplicate text dispatches. Use it as the custom provider in the widget's config:

import { createPersonaVoiceProvider } from '@runtypelabs/voice/persona'

const config = {
  voiceRecognition: {
    enabled: true,
    provider: {
      type: 'custom' as const,
      custom: () => createPersonaVoiceProvider({ agentId, clientToken }),
    },
  },
}

Artifacts

Set artifacts: true to declare the capability. The client then connects with clientCapabilities=partial_transcript,artifacts, the server sends an additive artifact message for every artifact the agent produces, and the client assembles them into snapshot.artifacts. Leaving it off keeps an existing embed's wire byte-identical.

{
  "type": "artifact",
  "turnId": "turn-3",
  "executionId": "exec_123",
  "event": {
    "type": "artifact_start",
    "id": "art_1",
    "artifactType": "markdown",
    "title": "game.html",
    "file": { "path": "game.html", "mimeType": "text/html", "language": "html" }
  }
}

event is a unified artifact_start, artifact_delta, artifact_update, or artifact_complete frame. A start opens a streaming record, deltas append to its content, an update carries the component payload, and a complete settles it. Artifacts of a turn the caller stopped or cancelled are dropped, the same suppression the transcript applies. Ending the call keeps completed artifacts and drops half-streamed ones. Artifacts are never spoken: file bodies arrive only through these frames.

The Persona adapter adds onArtifact, which fires each time a record changes, and bindVoiceArtifactsToPersona streams those records into an initialized widget through its upsertArtifact handle:

import { bindVoiceArtifactsToPersona, createPersonaVoiceProvider } from '@runtypelabs/voice/persona'

const provider = createPersonaVoiceProvider({ agentId, clientToken, artifacts: true })
const widget = initAgentWidget({ ...config, features: { artifacts: { enabled: true } } })
const release = bindVoiceArtifactsToPersona(provider, widget)

The artifact id is the upsert id, so every frame updates the same record in place, and only the settled record writes a transcript block. The pane opens once on the first artifact; pass { showOnFirst: false } to leave it closed. Persona's features.artifacts.enabled must be on, because showArtifacts() is a no-op otherwise. Call the returned function to stop delivery; the adapter's disconnect() releases artifact callbacks along with the rest.

Reusing a conversation

snapshot.session is { sessionId, conversationId } once the server reports them on session_config. It survives endCall, so the next startCall sends the same sessionId and reattaches the same conversation. Pass the sessionId option to name another client session instead, for example the one a Persona chat widget already opened. Call resetSession() between calls when the next caller should not inherit the conversation, as on a shared kiosk; it is a no-op while a call is open. An older server omits both ids and session stays null.

Widget release prerequisite: Persona 4.22.0's factory accepts this adapter, but its built-in microphone controls incorrectly route custom providers to browser dictation. Use a Persona release containing the custom voice controls fix before enabling this configuration. The adapter does not replace Persona's built-in provider automatically. Existing script-tag installations also need that widget release and a bundled adapter supplied by their host.

Interruption contract

The agent's saved mode arrives in session_config and remains authoritative:

The client requests voiceProtocol=runtype-browser-v1 when connecting. The API routes Cloudflare calls using this protocol to its PCM browser engine only when the operator's enable-voice-browser-engine-cloudflare gate is enabled. When disabled, the API rejects new v1 calls with HTTP 503 (VOICE_BROWSER_ENGINE_DISABLED) before creating a session. Operators can enable the gate for the organization or configure ElevenLabs voice. Browsers expose rejected WebSocket handshakes as a connection failure, without the HTTP response body. The legacy Cloudflare Durable Object uses a different protocol and remains available to compatible older clients. Existing calls keep their selected engine. ElevenLabs supports this client with either value of its portable-engine gate.

• none: speaking does not clear playback; canCancel is false.
• cancel: tap cancelResponse() to clear local playback immediately and send cancel, then speak again. Speaking during a reply does not interrupt it.
• barge-in: server speech detection sends audio_clear; manual cancellation is also available.

The microphone streams continuously during a call. In cancel mode, the client sends silence while a reply is being generated or played, resuming microphone audio after playback drains or the server acknowledges a manual cancellation. The server owns speech detection and turn-taking. The client does not infer interruptions from volume. After manual cancellation, old audio is discarded until the server acknowledges with audio_clear; late assistant transcripts are discarded across that same boundary. Pending Blob conversions and stale player events are invalidated. audio_end means synthesis completed, while speaking continues until playback drains.

Persona's explicit stopPlayback() and the client's matching method also stop playback in none mode. They discard the current reply's audio and captions locally through audio_end without sending a mode-disallowed cancellation or ending the call. Stopping playback does not stop the agent's execution: in none mode the turn keeps running and its artifacts (snapshot.artifacts, onArtifact) keep arriving, whether they began before or after the stop. Artifacts are dropped only for a turn that was actually cancelled (cancel / barge-in), interrupted by the server (audio_clear), or cut off by hanging up. Ordinary cancelResponse() still respects the agent's mode. Disconnecting the Persona adapter releases its callbacks; register callbacks again when reusing it. When a server supplies turnId on final transcripts, the adapter forwards it as Persona's optional fourth callback argument { turnId }. Untagged servers continue to use the ordered audio_clear cancellation boundary.

Requires the server cancellation lifecycle from core PR #8185. An older server without session_config leaves cancellation disabled; an older server that does not acknowledge cancel cannot support this cancellation protocol.

Development

pnpm --filter @runtypelabs/voice test
pnpm --filter @runtypelabs/voice typecheck
pnpm --filter @runtypelabs/voice build

Share a conversation with text

Initialize /v1/client/init before starting voice, including when voice is the first user interaction. Request durableRecovery: true for a token whose durable turn policy is enabled. Keep the returned visitor.token in your host's visitor store, scoped to that client token. The server returns this secret only when it mints the visitor; later init responses do not replace the stored value.

Pass the current text session ID and visitor credential to the voice client or Persona adapter. For example, after your host initializes the text session:

const voice = createPersonaVoiceProvider({
  agentId,
  clientToken,
  get sessionId() {
    return currentSessionId
  },
  visitorToken: () => visitorStore.get(clientToken),
  onSession: (session) => {
    currentSessionId = session.sessionId
    currentConversationId = session.conversationId
  },
})

Use the returned session ID for subsequent text requests and send the same credential in X-Visitor-Token. For Persona, use your integration's supported session initialization and credential storage APIs. Do not read Persona's internal local-storage keys. onSession reports IDs only; never log the visitor secret. A configured getter that returns no credential fails before microphone acquisition, so disable Start until initialization completes.

A valid visitor credential without a session ID creates a visitor-owned voice conversation. An expired session ID can locate an existing conversation when current visitor proof still authorizes it. An invalid explicit session, another visitor's conversation, or a different agent returns an error instead of starting a separate conversation. A legacy unowned record can be claimed only when its stored initialization provenance matches the presented visitor. Old voice records without that provenance require a fresh conversation; a public client token does not authorize claiming them.

Both portable browser engines support this contract. ElevenLabs requires its browser engine flag for shared sessions. Existing clients without visitor proof retain their legacy new-call behavior. Visitor proof travels in WebSocket headers, never URLs, and the selected protocol remains runtype.bearer. Disconnecting still cancels the active voice turn; conversation reuse does not resume unfinished voice work.