gpt-live

A native macOS voice companion plus a reusable Node.js library, CLI, and browser bridge for gpt-realtime-2.1. It supports live speech, streaming transcripts, server-side WebSockets, browser WebRTC session minting, typed tools, and injectable Codex or API-key auth.

The package name is historical: this targets GPT Realtime 2.1, not the original GPT-2 text model.

What is included

• RealtimeClient: a server-side WebSocket client for text, PCM16 audio, transcripts, interruption, raw events, reasoning, and function tools.
• RealtimeTranscriptionClient: a dedicated server-side streaming speech-to-text client for gpt-realtime-whisper, with item-aware deltas and manual audio commits.
• ToolRegistry: register tools once and reuse them in WebSocket clients and browser sessions.
• RealtimeSessionService: mint short-lived ek_... credentials without exposing the real credential to a browser.
• TranscriptionSessionService: mint short-lived transcription credentials with the same auth, timeout, and secret-handling infrastructure.
• createRealtimeHandler / createRealtimeServer: embed the browser bridge in another Node project or run the included voice UI.
• createCodexAuthProvider: read a Codex auth.json file on every connection, so refreshed credentials are picked up without restarting.
• gpt-live (with gpt-realtime / gptrt compatibility aliases): serve, chat, and auth commands.
• Native SwiftUI/AppKit companion: an animated menu-bar orb and a floating, compact top-right conversation panel. It defaults to gpt-live-1-boulder-alpha over bidirectional WebRTC, with gpt-realtime-2.1 available from the menu.

Imports are side-effect free. Importing the package does not read auth, create directories, start a server, or enable shell execution.

Install and build

pnpm install
pnpm build

Install from npm:

pnpm add gpt-live
# or: npm install gpt-live

The Node package is ESM-only and requires Node.js 22 or newer. The native app requires macOS 13 or newer plus Xcode Command Line Tools (xcode-select --install) for its one-time local build.

Native macOS conversation

On macOS, chat now opens the native menu-bar experience instead of printing responses and writing WAV files:

npx gpt-live chat

On first run, Swift builds and signs a small local application at ~/Library/Application Support/GPT-Live/GPT-Live.app. Unchanged launches reuse that exact signed bundle instead of rebuilding and replacing it. macOS asks for microphone access once. The app then:

• places an animated multicolor voice orb in the status bar;
• opens a glass conversation panel at the top-right of the active display;
• streams microphone PCM audio directly to the Realtime WebSocket when GPT Realtime is selected;
• streams echo-cancelled microphone and speaker audio over WebRTC when GPT Live is selected (the default);
• waits through natural speech pauses before completing your turn;
• pauses microphone upload during assistant playback to prevent echo loops;
• plays assistant audio as it arrives, without creating WAV files;
• displays live user and assistant transcripts;
• keeps conversation context only in memory during the current app run and forwards it across panel reconnects or service switches; chat is never saved and does not survive an app relaunch;
• handles GPT Live client delegations natively, including live web search, desktop-window discovery, exact-window/named-app/full-screen visual inspection, and Bash tool calls, then returns tool output to the voice session;
• lets you mute, interrupt, clear, retry, hide, or quit; typed messages are available when GPT Realtime is selected;
• can run Bash commands in a configured working directory;
• pauses for an explicit Approve/Reject decision before any command that may edit files or system state;
• disconnects and releases the microphone whenever the panel is hidden.

Only one app instance may run at a time, preventing duplicate microphone listeners and overlapping Realtime connections.

The first visual screen inspection asks for macOS Screen Recording permission. If macOS requests it, enable GPT-Live under System Settings → Privacy & Security → Screen & System Audio Recording, then restart the app. Captures are resized for model input and their temporary local files are deleted immediately after encoding. The current capture remains in memory so the image appears when its tool card is expanded, but screen pixels and chat messages are never written to saved conversation history.

The local build auto-selects a stable Apple Development or Developer ID Application identity from your keychain when available, so Screen Recording permission survives source updates and rebuilds. Override with GPT_LIVE_CODESIGN_IDENTITY if needed. Without a stable identity the build falls back to an ad-hoc signature, which changes on every rebuild and forces macOS to ask for Screen Recording again. The first launch after switching to a stable signature needs one new permission grant.

The conversation panel can be resized by dragging any edge or corner; transcript text reflows between compact and roomy layouts. Click the status-bar orb to reopen or hide the panel. Right-click the orb to toggle between GPT Live 1 and GPT Realtime, select a voice for the active service, or choose Test Screen Sharing… to perform a capture without asking the assistant. The test opens the transcript and places the captured image in an expandable tool card. Menu-selected voices persist across launches; explicit CLI or environment settings still take precedence. Configuration is passed through the same command:

npx gpt-live chat \
  --service gpt-live \
  --live-model gpt-live-1-boulder-alpha \
  --live-voice cove \
  --auth /path/to/auth.json \
  --model gpt-realtime-2.1 \
  --voice marin \
  --shell full \
  --cwd ~/Documents \
  --instructions "Keep responses brief."

The native macOS app enables Bash in full mode by default. A narrow set of read-only inspection commands runs immediately. Commands outside that set, including commands containing shell operators, scripts, and commands that may write files, are shown in the app and do not run until you choose Approve and Run. Use --shell readonly to deny those commands completely or --shell none to disable Bash. The app runs approved commands with /bin/bash -lc in the directory selected by --cwd (default: ~/Documents). Each command that actually runs is recorded as a BASH entry in the chat transcript together with its standard output, standard error, and nonzero exit status.

This capability is part of the macOS target. iOS applications cannot launch /bin/bash under the iOS application sandbox.

The previous terminal workflow remains available explicitly:

npx gpt-live terminal
npx gpt-live terminal --no-audio

Use npx gpt-live mac --build-only to prepare the application bundle without opening it.

CLI

# Validate ~/.codex/auth.json without printing the token
pnpm auth:check

# Browser voice app at http://127.0.0.1:8787
pnpm start

# Experimental GPT-Live Boulder app at http://127.0.0.1:8787/gpt-live

# Explicit auth, workspace, and unrestricted shell tool
pnpm start -- --auth /path/to/auth.json --cwd ./workspace --shell full

# Native macOS menu-bar conversation
pnpm chat

# Legacy terminal REPL
tsx src/bin.ts terminal --no-audio

# Installed package equivalents
gptrt serve --shell readonly --cwd ./files
gptrt terminal --no-audio
gptrt auth --auth /path/to/auth.json

serve binds to 127.0.0.1 by default. Its shell tool defaults to readonly; use --shell none to disable it or --shell full only for a trusted local workflow. Passing --terminal or --no-audio to chat selects the terminal REPL; --shell and --cwd now configure the native macOS app directly.

Run gptrt help for all flags. You can pass a prompt with --instructions or --instructions-file ./files/me.md.

Experimental GPT-Live browser demo

After pnpm start, open http://127.0.0.1:8787/gpt-live. This page connects gpt-live-1-boulder-alpha over WebRTC using the Codex/ChatGPT quicksilver=v2 call-creation route. The local Node server reads Codex auth from ~/.codex/auth.json; the browser receives only the SDP answer and never receives the access token.

GPT-Live Boulder and this backend route are experimental rather than part of the public Realtime API, so their availability and wire contract can change. The demo supports live microphone/speaker audio, transcripts, and two tools through GPT-Live client delegation: run_shell and web_search. serve enables Bash in readonly mode by default and exposes a live tool inventory at GET /gpt-live/tools. A server-side Realtime delegation agent selects the appropriate function tool, translates spoken shell work into a structured run_shell call, and uses web_search for current public information. Tool inputs and results appear in the transcript and the concise result returns with delegation.context.append. Use --shell full only in a trusted local workspace, or --shell none to disable Bash. Keep the server bound to its default localhost address.

Import in a Node project

Text session with automatic custom tools

import {
  RealtimeClient,
  ToolRegistry,
  createCodexAuthProvider,
} from "gpt-live";

const tools = new ToolRegistry().register<{ orderId: string }, unknown>({
  type: "function",
  name: "get_order",
  description: "Look up an order by id.",
  parameters: {
    type: "object",
    properties: { orderId: { type: "string" } },
    required: ["orderId"],
    additionalProperties: false,
  },
  execute: async ({ orderId }) => database.orders.find(orderId),
});

const client = new RealtimeClient({
  auth: createCodexAuthProvider({ path: "/path/to/auth.json" }),
  model: "gpt-realtime-2.1",
  reasoningEffort: "low",
  instructions: "Be concise and verify orders with the tool.",
  tools,
});

client.on("text.delta", (text) => process.stdout.write(text));
client.on("tool.result", (call, result) => console.log(call.name, result));
client.on("server.error", console.error);
client.on("client.error", console.error);

await client.connect();
client.sendUserText("Where is order A-123?");

When a registered function is called, RealtimeClient parses its arguments, awaits the handler, sends function_call_output, and requests the next model response. Set autoExecuteTools: false to handle tool.call yourself, then use sendToolOutput() and createResponse().

Remote MCP servers and built-in connectors

Function tools run in your application. MCP tools and built-in connectors run inside the Realtime service and can be combined with the same local registry:

import { RealtimeClient, defineMcpTool } from "gpt-live";

const docs = defineMcpTool({
  type: "mcp",
  server_label: "openai_docs",
  server_url: "https://developers.openai.com/mcp",
  allowed_tools: ["search_openai_docs", "fetch_openai_doc"],
  require_approval: "never",
});

const client = new RealtimeClient({ auth, tools, sessionTools: [docs] });
client.on("mcp.event", (event) => console.log(event.type));
client.on("mcp.approval", (request) => {
  // Apply your own policy/UI before approving an action.
  client.respondToMcpApproval(String(request.id), true);
});

For a managed connector, use connector_id instead of server_url and supply the end user's OAuth token in authorization. Keep allowed_tools narrow. defineMcpTool() rejects ambiguous URL/connector and authorization shapes. You can also attach MCP tools to one turn with client.createResponse({ tools: [docs] }).

Pass auth directly

import { RealtimeClient, createApiKeyAuth } from "gpt-live";

const withApiKey = new RealtimeClient({
  auth: createApiKeyAuth(process.env.OPENAI_API_KEY!),
});

const withCustomProvider = new RealtimeClient({
  auth: async () => ({
    accessToken: await secrets.get("openai-token"),
    accountId: "optional-chatgpt-account-id",
  }),
});

The auth provider is invoked on each connection or session mint. Credentials stay server-side. Never put an API key or Codex access token in browser code.

Embed browser WebRTC support

import {
  RealtimeSessionService,
  createRealtimeHandler,
  createCodexAuthProvider,
} from "gpt-live";

const sessions = new RealtimeSessionService({
  auth: createCodexAuthProvider(),
  defaultInstructions: "You are the voice assistant for Acme.",
  tools,
  sessionTools: [docs], // optional hosted MCP servers/connectors
  safetyIdentifier: hashInternalUserId(user.id),
});

const handler = createRealtimeHandler({
  sessions,
  tools,
  publicDir: false, // expose API routes only in an existing server
});

// Mount `handler` in a Node HTTP server, or call:
// createRealtimeServer({ sessions, tools }).listen(8787, "127.0.0.1");

The handler provides:

• GET /session?model=...&voice=... — mint an ephemeral browser credential.
• GET /transcription-session?language=en&delay=low — mint an ephemeral gpt-realtime-whisper transcription credential.
• GET /models — return the configured model catalog.
• POST /gpt-live/session?voice=cove&instructions=... — proxy an SDP offer to the experimental GPT-Live Boulder call-creation route.
• GET /gpt-live/tools — list the delegation agent's enabled tools and count.
• POST /gpt-live/delegate — run a natural-language task through the server-side tool delegation agent.
• POST /tools/:name — execute a registered server-side tool.
• Static UI routes when publicDir is enabled.

If your framework uses Fetch Request/Response objects rather than Node's IncomingMessage/ServerResponse, call RealtimeSessionService.mint() and ToolRegistry.execute() directly from framework-native routes.

Audio and low-level control

Realtime PCM audio is 24 kHz, mono, signed 16-bit. In an audio client:

const client = new RealtimeClient({ audio: true, voice: "marin", auth });
client.on("audio.delta", (pcmBuffer) => playback.write(pcmBuffer));
client.on("transcript.delta", (delta, role) => console.log(role, delta));

await client.connect();
client.appendAudio(pcmChunk);
client.commitAudio();       // manual turn mode only
client.createResponse();
client.cancelResponse();    // interruption

Use sendRaw() as an escape hatch for newer client events while keeping the rest of the package abstractions.

Streaming transcription with GPT-Realtime-Whisper

Use the dedicated transcription client when you need live text without an assistant response:

import {
  RealtimeTranscriptionClient,
  createCodexAuthProvider,
} from "gpt-live";

const client = new RealtimeTranscriptionClient({
  auth: createCodexAuthProvider(),
  language: "en",
  delay: "low", // minimal, low, medium, high, or xhigh
  include: ["item.input_audio_transcription.logprobs"],
});

client.on("transcript.delta", ({ itemId, delta }) => {
  process.stdout.write(`[${itemId}] ${delta}`);
});
client.on("transcript.completed", ({ itemId, transcript }) => {
  console.log(`\n[${itemId}] ${transcript}`);
});
client.on("server.error", console.error);
client.on("client.error", console.error);

await client.connect();
client.appendAudio(pcm16Mono24kChunk);
client.commitAudio();

connect() resolves only after the service accepts the transcription session update. GPT-Realtime-Whisper requires 24 kHz mono PCM, does not support transcription prompts or server turn detection, and therefore requires commitAudio() at each boundary. These constraints are validated before a network request. Completion ordering across different turns is not guaranteed, so delta and completion events expose itemId and contentIndex.

For browser or mobile WebRTC clients, mint an ephemeral secret server-side:

import { TranscriptionSessionService } from "gpt-live";

const sessions = new TranscriptionSessionService({
  auth: createCodexAuthProvider(),
  defaultLanguage: "en",
  defaultDelay: "low",
});

const secret = await sessions.mint();
// Return only `secret.value` and non-sensitive session metadata to the client.

Security and auth notes

• Codex auth loading is read-only; this package never refreshes or writes the Codex credential store. Run codex login when the access token expires.
• ChatGPT/Codex OAuth credentials working with an endpoint does not make that flow a stable public authentication contract. For production applications, prefer an OpenAI API key or another officially supported server credential.
• The browser receives only a short-lived ephemeral secret.
• Shell execution is opt-in at library level. Even readonly is a convenience allowlist, not a strong security sandbox; do not expose the demo server to an untrusted network.
• For applications with end users, set a stable, privacy-preserving safetyIdentifier on the session service/client.

Project layout

| Path | Responsibility | | --- | --- | | src/index.ts | Stable public exports | | src/auth.ts | Injectable auth and read-only Codex loader | | src/tools.ts | Tool definitions, handlers, and registry | | src/realtime.ts | Server WebSocket client | | src/socket.ts | Shared authenticated WebSocket transport | | src/session.ts | Ephemeral client-secret service | | src/transcription.ts | GPT-Realtime-Whisper client and session service | | src/server.ts | Embeddable Node HTTP routes and static UI | | src/exec.ts | Explicitly-created optional shell tool | | src/bin.ts | CLI entry point | | src/macos.ts | Native companion build/launch bridge | | macos/GPTRealtimeMac | Swift menu-bar app, Realtime audio, and panel UI | | examples/custom-tools.ts | Custom-tool integration example | | test/library.test.ts | Auth, registry, and session contract tests |

Verify

pnpm typecheck
pnpm test
pnpm build
pnpm pack --dry-run
swift test --package-path macos/GPTRealtimeMac

The Realtime GA flow uses /v1/realtime, /v1/realtime/client_secrets, and /v1/realtime/calls. The browser uses WebRTC; the Node client uses WebSocket. See OpenAI's current Realtime overview, WebSocket guide, and Realtime tools/MCP guide.