@tinyweb-agent-framework/realtime

Voice / realtime call surface for tinyweb-agent-framework. Primary provider: Google Gemini Live API. Backed by plan 0014 and task 0003.

Text-mode agents (HTTP + OpenAI Chat Completions) keep working unchanged. This package is an additive, side-channel protocol — it never replaces or wraps the text loop in @tinyweb-agent-framework/agent.

Status

v1 MVP. Shipped:

• auth/ephemeral-token — server-side helper to mint a short-lived Gemini Live token so the API key never leaves the host backend.
• provider/gemini — typed config builder (geminiLive({...})).
• tools/bridge — convert framework tools to Gemini functionDeclarations; client-side handler dispatch.
• client/session — open WSS to Gemini, mic capture via AudioWorkletProcessor, audio playback scheduling, tool call dispatch, barge-in detection. AsyncIterable RealtimeEvent stream.

Not yet shipped (see plan 0014 §12):

• Server-to-server relay (/server).
• Built-in tool realtime.clientSafe flag for the tool catalogue.
• Transcript persistence into the agent session store.
• Cassette tests for the WS audio stream.

Quickstart

1. Mint an ephemeral token (server)

// Bun route handler / Next.js Route Handler / Express endpoint.
import { geminiLive } from "@tinyweb-agent-framework/realtime/provider"
import { mintEphemeralToken } from "@tinyweb-agent-framework/realtime/auth"

const config = geminiLive({
  model: "gemini-3.1-flash-live-preview",
  voice: "Puck",
  instructions: "You are a friendly bilingual assistant. Default to Vietnamese.",
})

const token = await mintEphemeralToken({
  apiKey: process.env.GOOGLE_API_KEY!,
  config,
  ttlMs: 30 * 60_000,
})

// Return `{ token, expiresAt, wsEndpoint, config }` to the browser.
return Response.json(token)

2. Open the session (browser)

import { connectRealtimeSession } from "@tinyweb-agent-framework/realtime/client"

const audioContext = new AudioContext()
const micStream = await navigator.mediaDevices.getUserMedia({ audio: true })

const session = await connectRealtimeSession({
  tokenEndpoint: "/api/realtime/token",
  micStream,
  audioContext,
  toolHandlers: new Map([
    ["time_now", async () => ({ iso: new Date().toISOString() })],
  ]),
})

for await (const ev of session.events) {
  if (ev.type === "user-transcript-delta") setUserText((s) => s + ev.text)
  if (ev.type === "agent-transcript-delta") setAgentText((s) => s + ev.text)
  if (ev.type === "tool-call") console.log("agent asked for", ev.name, ev.args)
  if (ev.type === "barge-in") console.log("user interrupted")
  if (ev.type === "closed") break
}

await session.close()
micStream.getTracks().forEach((t) => t.stop())

3. Use with @tinyweb-agent-framework/chat-ui

The chat UI ships a built-in <RealtimeCallButton /> and call overlay. Pass a realtime prop to <TinywebChat />:

<TinywebChat
  endpoint="/api/chat"
  realtime={{
    tokenEndpoint: "/api/realtime/token",
    config: geminiLive({ instructions: "..." }),
  }}
/>

See examples/next-chat for an end-to-end wiring.

Subpath exports

| Import | Use from | |---|---| | @tinyweb-agent-framework/realtime | Types + provider builders + tool bridge — server or browser. | | @tinyweb-agent-framework/realtime/auth | mintEphemeralToken — server only. Holds your Gemini API key. | | @tinyweb-agent-framework/realtime/provider | Provider builders + types. | | @tinyweb-agent-framework/realtime/tools | toGeminiFunctionDeclarations, buildToolHandlerMap. | | @tinyweb-agent-framework/realtime/client | connectRealtimeSession and PCM/worklet utilities — browser only. |

Audio formats

| Direction | Format | Source | |---|---|---| | Browser → Gemini | PCM16 LE mono 16 kHz | mic via AudioWorkletProcessor (resample + Int16) | | Gemini → Browser | PCM16 LE mono 24 kHz | scheduled on local AudioContext, resampled if needed |

The worklet source is bundled inline as a string literal (./src/client/worklet.ts) so consumers don't need a separate public asset URL.

Security

• Never import mintEphemeralToken from a browser bundle. Subpath ./auth is server-only.
• Tokens have a hard 60 min ceiling; default 30 min.
• The token endpoint should be guarded by the same auth as your chat endpoint (cookie / Bearer).
• Tool handlers in toolHandlers run in the browser under the user's privileges — never trust them with server secrets.

See also

• plan 0014 — realtime voice overview
• task 0003 — realtime call
• Gemini Live API: https://ai.google.dev/gemini-api/docs/live-api