Microsoft Teams Bridge for OpenAI Realtime Agents

@komaa/openai-msteams-bridge puts an OpenAI Realtime voice agent (gpt-realtime) on a real Microsoft Teams call. The hosted StandIn media bridge (standin.komaa.com) joins the Teams call and dials into this bridge over an HMAC-authenticated WebSocket; the bridge opens one OpenAI Realtime session per call and relays between them.

Microsoft Teams call
       |
       v
StandIn media bridge      (hosted; joins the call)
       |   HMAC WebSocket, PCM 16 kHz
       v
this bridge               (you run it)
       |   WebSocket, PCM 24 kHz
       v
OpenAI Realtime API       (speech-to-speech: STT + LLM + TTS + turn-taking)

The relay core is copy-only at the 16 kHz wire rate. The Realtime API only speaks PCM at 24 kHz, so the provider adapter converts 16k/24k at its boundary with a built-in dependency-free resampler - no codecs, no re-encoding, one ws dependency.

Features

• Realtime voice, end to end - the caller talks to a gpt-realtime agent and hears it reply. Turn-taking, VAD and interruption are the model's own (server-side, server_vad or semantic_vad); the bridge adds a relay hop and a linear resample.
• Barge-in done right - when the caller interrupts, the Realtime API cancels its in-flight response; the bridge mirrors the cut to the Teams side (assistant.cancel) and ghost-drops in-flight audio deltas from the cancelled response, so no stale audio plays after the cut.
• Per-call personalization - caller name, tenant and call direction are appended to the session instructions at call start; an optional deterministic opening line rides the first response.
• Vision on demand - a look function tool lets the agent see the caller's camera or screen-share: describe-then-answer via any OpenAI-compatible vision endpoint, or native Realtime image input (recording-gated). See Vision.
• Agent function tools - end_call, express (avatar emotion), show_image (image on the bot's video tile, SSRF-guarded), look. Registered automatically on every session; behavior is implemented by the bridge.
• Extensible tools - register your own function tools (lookup_order, transfer_call, ...) that the bridge executes in-process, and/or pass remote MCP servers that the Realtime API executes server-side (including OpenAI's built-in connectors). See Extending the agent's tools.
• Two call governors - a StandIn-side cutoff the bridge speaks a goodbye for, and a bridge-side MAX_CALL_MINUTES hard cap with a deterministic TTS goodbye (OPENAI_TTS_VOICE).
• Observability - GET /healthz for liveness and GET /metrics (Prometheus text format): calls, durations, rejects, relay/drop counters.
• Hardened transport - replay-proof HMAC upgrade, single-use handshake guard, connection caps, payload caps, backpressure bounds, pre-start timeout, dead-peer detection, duplicate-call rejection, graceful SIGTERM drain, and an OPENAI_HOST allowlist so your API key can only be sent to OpenAI.
• Group-call awareness - participant counts, DTMF digits and active-speaker changes are fed to the agent as non-interrupting context items.

Install

Run it directly:

npx @komaa/openai-msteams-bridge

Or add it to your project:

npm install @komaa/openai-msteams-bridge

Node.js >= 20. One runtime dependency (ws).

Quick start

1. As a CLI (env-configured)

Set the two required variables, then run it:

export OPENAI_API_KEY=sk-...
export WORKER_SHARED_SECRET=...
npx @komaa/openai-msteams-bridge

Or keep them in a .env file (copy .env.example, which ships with the package) and load it:

node --env-file=.env node_modules/.bin/openai-msteams-bridge

Every option is an environment variable; see .env.example for the full reference.

2. As a library

import { loadConfig, startServer } from "@komaa/openai-msteams-bridge";

// env-configured, same variables as the CLI
startServer(loadConfig());

Signal handling is opt-in for embedders: the built-in SIGTERM/SIGINT drain ends every live call gracefully and then exits the process, so it is only wired when you ask for it (the CLI does). Enable it when the bridge owns the process:

startServer(loadConfig(), undefined, undefined, { handleSignals: true });

With a custom vision hook (path-2 look: your model, your prompt, the raw frame never leaves your process):

import { loadConfig, startServer, type VisionDescriber } from "@komaa/openai-msteams-bridge";

const describe: VisionDescriber = async (frame, question) => {
  // frame: { source: "camera" | "screenshare", mime, dataBase64, width, height, participantName }
  return "a short description the agent relays aloud";
};

startServer(loadConfig(), undefined, describe);

3. Point StandIn at it

In your StandIn dashboard, set the bridge URL (e.g. wss://bridge.example.com/voice/msteams/stream) and copy the shared secret into WORKER_SHARED_SECRET. StandIn appends /{callId} and signs each upgrade with X-StandIn-Timestamp / X-StandIn-Signature (legacy X-OpenClawTeamsBridge-* names are still accepted).

How a call flows

1. StandIn joins the Teams call and opens wss://your-bridge/{callId} with the HMAC handshake.
2. On session.start, the bridge opens one OpenAI Realtime WebSocket and configures the session: PCM 24 kHz both ways, your instructions plus caller context, the four bridge tools, VAD with interrupt_response.
3. Caller audio (16 kHz frames) is upsampled to 24 kHz and appended to the input buffer; agent audio deltas (24 kHz) are downsampled to 16 kHz frames for the Teams side.
4. Caller speech triggers server-side interruption; the bridge mirrors it with assistant.cancel and drops in-flight deltas from the cancelled response.
5. Function calls (end_call, express, show_image, look) are mapped to StandIn capabilities and answered with function_call_output.
6. Either side can end the call; both governors speak a goodbye first.

Audio and the 16k/24k boundary

The StandIn wire protocol is base64 PCM 16 kHz, 16-bit, mono. The OpenAI Realtime API accepts and emits PCM at 24 kHz only. The bridge converts inside the provider adapter (a clean 2:3 ratio): upsampling is linear interpolation; downsampling first applies a windowed-sinc low-pass at the 8 kHz target Nyquist so 8-12 kHz content cannot alias into the audible band, then decimates. The relay core, tests and wire contract all stay at 16 kHz. This is a real (if mild) resample, unlike the strictly transcode-free ElevenLabs sibling - telephony-adequate for voice, not bit-transparent.

Extending the agent's tools

The four built-in tools are just the default registry. Two extension points, both merged into every session's tools array:

1. Custom function tools (the bridge executes them). Register a name, a JSON schema and a handler; when the model calls the tool, the handler runs in your process and its returned string goes back as the tool output. A thrown error becomes an error output the model can recover from.

import { loadConfig, startServer, type CustomTool } from "@komaa/openai-msteams-bridge";

const tools: CustomTool[] = [{
  name: "lookup_order",
  description: "Look up the status of a customer order by its order number.",
  parameters: {
    type: "object",
    properties: { orderNumber: { type: "string", description: "e.g. KO-1234" } },
    required: ["orderNumber"],
  },
  async handler({ orderNumber }, ctx) {
    ctx.log.info(`lookup_order ${String(orderNumber)}`);
    return await myBackend.orderStatus(String(orderNumber)); // the agent speaks this
  },
}];

startServer(loadConfig(), undefined, undefined, { tools });

Keep handlers fast (the caller is waiting on the answer) and enforce your own timeout for slow backends. Names must not collide with the built-ins.

2. Remote MCP servers (the Realtime API executes them). Pass MCP tool entries and OpenAI dials the server and runs the tools server-side - the bridge does nothing at call time. Works with any remote MCP server and with OpenAI's built-in connectors. Configure via env (OPENAI_MCP_SERVERS, a JSON array) or programmatically:

startServer(loadConfig(), undefined, undefined, {
  mcpServers: [{
    server_label: "calendar",
    server_url: "https://mcp.example.com/sse",
    allowed_tools: ["list_events", "create_event"],
    authorization: process.env.MCP_CALENDAR_TOKEN,
  }],
});

require_approval defaults to "never": a live voice call has no approval UI, and a pending approval would stall the turn. Only list servers and tools you trust with that autonomy, and prefer allowed_tools allowlists.

Vision (look tool)

When the agent calls look (optional source = camera|screenshare, optional question), the bridge grabs the latest buffered frame and answers one of two ways:

1. Path 2 (preferred, if VISION_API_URL/VISION_MODEL are set): the frame goes to your OpenAI-compatible vision endpoint and the description comes back as the tool result. The raw frame never leaves the bridge (only the text description does), and it works regardless of recording state. Note that the description becomes Realtime conversation content, retained per your OpenAI data controls.
2. Path 1 (fallback): the frame is attached to the live Realtime conversation as native image input (gpt-realtime is multimodal). Because this shares the raw frame with the conversation, it is refused unless Teams recording is active.

The other tools: end_call, express ({emotion}), show_image ({dataBase64, mime} or {url}, jpeg/png; URLs are SSRF-guarded - public hosts only, at most one redirect hop and the redirect target is re-validated through the same guard, bounded time and size).

Call governors

Two governors can end a call gracefully; both speak before hanging up:

• StandIn-side: when a tier limit is reached, StandIn sends assistant.say with the goodbye text; the bridge speaks it (exact text via the TTS endpoint when OPENAI_TTS_VOICE is set, otherwise the agent is asked to say it) and the call is torn down.
• Bridge-side (MAX_CALL_MINUTES > 0): the bridge arms a timer at call start. On expiry it flushes playback, speaks GOODBYE_TEXT, waits for the audio to play out (real TTS duration, or GOODBYE_GRACE_MS when unknown, always hard-bounded), then ends the call with reason time-limit. Use this when the billing limit lives with you, since OpenAI knows nothing about your budget.

Both goodbye paths are backstopped: if whichever side is supposed to hang up never does, the bridge force-ends the call after the goodbye grace plus a hard cap, so a call can never sit open with the agent muted.

The 60-minute ceiling: the OpenAI Realtime API caps a session at 60 minutes server-side. A call that reaches it drops with reason agent-disconnected (no goodbye) - there is no session migration. Set MAX_CALL_MINUTES below 60 so the bridge governor always fires first; the bridge logs a startup warning when it would not.

Disconnects and reconnects

If the worker socket drops mid-call (network, StandIn redeploy), the bridge tears the call down: the Realtime session is closed (it is billed per minute) and the callId is freed. There is no mid-call re-attach: a StandIn retry with the same callId after teardown is a fresh call with a fresh Realtime session and no conversation memory; a retry arriving while the old session is still live is rejected with 409 so one call can never pay for two agent sessions. If the OpenAI socket drops instead, the bridge ends the Teams call with session.end(agent-disconnected).

Privacy / recording gate

StandIn reports the Teams recording state (recording.status). The bridge never logs transcripts unless LOG_TRANSCRIPTS=true and recording is active (user transcripts additionally need OPENAI_TRANSCRIPTION_MODEL). Video frames are buffered in memory only and dropped at teardown; path-1 frame attaches are recording-gated. In all cases caller audio transits OpenAI's cloud and is retained per your OpenAI data controls - disclose in a spoken OPENAI_FIRST_MESSAGE that an AI is on the call where required.

Repository layout

src/
  server.ts     HTTP + WS upgrade, HMAC validation, connection guards, session registry
  session.ts    per-call relay: StandIn WS <-> OpenAI Realtime WS, tools, governors
  openai.ts     Realtime socket, session config builders, goodbye TTS, rate conversion boundary
  resample.ts   dependency-free PCM16 16k/24k linear resampler
  protocol.ts   wire message types (JSON, camelCase, discriminated on "type")
  hmac.ts       HMAC-SHA256("{timestampMs}.{callId}") hex, constant-time verify
  ssrf.ts       public-URL guard for agent-supplied fetches
  vision.ts     path-2 describe-then-answer vision hook
  config.ts     env config (fail-loud numeric parsing)
examples/       runnable example projects
website/        docs site (Astro Starlight), deployed to GitHub Pages
test/           node:test suites (run with tsx)

Documentation

• Docs site: komaa-com.github.io/openai-msteams-bridge - getting started, architecture, configuration and library API reference, wire protocol, tool extensibility, troubleshooting.
• Example project: examples/basic-bridge/ - a runnable embedding with a custom vision hook and a custom tool.

Siblings

Same wire contract, different agent platforms - the StandIn side is interchangeable:

• @komaa/elevenlabs-msteams-bridge (ElevenLabs Agents, Node) and elevenlabs-msteams-bridge (Python)
• @komaa/livekit-msteams-bridge (LiveKit Agents, Node) and livekit-msteams-bridge (Python)

StandIn (the hosted service): standin.komaa.com · docs.komaa.com

Contributing

PRs welcome - see CONTRIBUTING.md for local setup and conventions.

License

MIT