pk-speak

A conversational assistant for Pi / pi-mono, reachable over voice, phone, and browser remote.

pk-speak runs a persistent conversational assistant that can see session and background-agent state and read the workspace on every turn, but never mutates anything — launching an agent, archiving a session, or running a command outside a small read-only allowlist — without your explicit approval. Voice (/mono), Telegram (/phone), and the browser/Android remote (/remote) are channels into that same assistant, not separate products. It gives you:

• a conversational assistant that reads subagent state and proposes commands for approval
• spoken assistant replies with multiple TTS backends
• the always-listening PK wake phrase flow as a way to start a conversation
• Telegram text and voice turns from your phone
• a local HTTP control API
• a built-in mobile web app at /app/
• a Unified Remote control surface

What To Use

These are all input channels to the same conversational assistant. Pick the one that fits where you are:

1. Local desktop voice: use /speak on
2. Hands-free on the same machine: use /mono on (say PK to start a conversation)
3. Remote from your phone with the least friction: use /phone on
4. Remote from your phone with QR setup: use /pk-remote, then scan the QR from the Android phone

Documentation

• Getting Started — install, five voice paths, first session, TTS ladder, phone setup, session routing, Gemini Live
• Architecture — subsystem map, data flow, provider model, HTTP API surface, Android/PWA clients, environment variables
• Troubleshooting — 12 common problem areas with causes and fixes, plus a 5-command quick diagnostic checklist
• Session Operations — named sessions, wake routing, compact PK1/PK2 lanes
• Remote Operating Guide — phone setup, Android app, web app, Tailscale/tunnel configuration
• Validation Checklist — full phone-focused pass/fail run sheet
• Codebase Map — file-level tour of the TypeScript source

Install

Install the Pi extension package inside Pi:

pi npm i pi-pk-speak

Reload Pi after install.

For the standalone desktop/phone bootstrap package, install or run pk-speak from npm, then run:

pi-speak-pk

That opens the first-time setup CLI. It chooses the coding backend (codex, claude, or pi), configures the voice/TTS provider, creates the gateway pairing token, and asks whether to include Android setup. After that, use:

pk-speak doctor
pk-speak speak "Build finished"
pk-speak wrap -- codex
pk-speak tray
pk-speak mobile

pk-speak speak is the direct sag-like TTS command for hooks and CLI agents. pk-speak wrap runs any CLI command and speaks start/finish notices without requiring the phone gateway. pk-speak tray starts the Windows tray plus gateway. pk-speak gateway starts the headless gateway directly. pk-speak mobile prints the Android download/setup QR.

Package split details are in docs/PACKAGE_SPLIT.md.

Quick Start

1. Make Pi Speak Locally

/speak on
/speak test
/speak status

If you do nothing else, auto provider selection will try available backends in this order:

1. legacy via speak11
2. gemini
3. elevenlabs
4. openai
5. edge

If an earlier auto-selected backend fails at synthesis time, Pi now falls through to the next available provider instead of stopping on the first failure.

2. Enable The Always-Listening Wake Phrase

/mono on

Say:

PK

Pi will open a short voice-input window, play a short listening cue, and update the mono status so you can tell it is actively listening. Say PK again within the timeout to keep it alive. Default keep-alive is 15 seconds.

Wake matching now has a sensitivity preset. Use PI_SPEAK_WAKE_SENSITIVITY=low|medium|high to make activation stricter or more forgiving. medium is the default.

3. Remote In From Your Phone With Telegram

/phone setup
/phone token <bot-token>
/phone on
/phone code

Then in Telegram:

1. Open your bot
2. Send /link <code>
3. Send text or voice notes to Pi

This is the easiest remote path. It works well when you want reliability more than low latency.

/phone setup prints the running-session setup steps. If the token is not already in the environment, paste it with /phone token <bot-token>; the extension saves it, starts the bridge, and prints the /link code. PI_SPEAK_TELEGRAM_BOT_TOKEN can still point to an existing bot you already control.

Pi Speak has two distinct phone-facing gateway surfaces:

• The Telegram gateway in phone-bridge.ts owns bot polling, /link pairing, and Telegram text/voice session control.
• The Pi Speak mobile-app gateway in control-server.ts owns the authenticated HTTP API, browser/PWA, Android app, and APK download.

They remain separate transports and pairing models even when the persistent pk-speak gateway daemon hosts both against the same ompk / Agent Hub runtime. To configure the persistent Telegram surface rather than an in-terminal Pi session:

pk-speak phone token <bot-token>
pk-speak phone code

The Telegram surface stores its bot credential and pairing state in the local setup profile and starts its poller automatically when configured. The mobile-app surface continues to use its own HTTP token and machine profile. Use pk-speak phone status, off, or unpair to inspect or revoke only the Telegram bridge.

4. Remote In From Your Phone With The Built-In Web App

/pk-remote
/remote setup
/remote setup bluetooth

/pk-remote is the shortest path. It starts the remote API if needed, chooses a setup URL in this order, and prints a QR code for the phone setup page:

1. PI_SPEAK_PUBLIC_BASE_URL
2. detected Tailscale IPv4 address
3. detected local LAN IPv4 address
4. configured fallback

Scan the QR from the Android phone to open the setup page. From there you can download the bundled APK, open the native pi-speak://setup link, and save the machine URL, token, profile name, connection mode, and Codex route metadata. If you want the browser app instead, open one of the printed browser URLs:

http://localhost:8767/app/
https://<tailnet-host>/app/
https://<tunnel-domain>/app/

The web app:

• records your microphone in the browser
• sends audio to /v1/turn/voice
• shows the transcript
• plays the returned reply audio
• stores the remote token in the current browser session by default
• can explicitly remember the token on that device if you enable it in Settings

The web app also has a Workspace tab for reviewing files and choosing where the agent runs:

• browse the directory tree from the workspace root, stepping into folders and back up to the parent
• tap a file to view its contents in a read-only viewer (preview is capped at the first 512 KB of large files, and binary files show a notice instead of bytes)
• tap Use this folder to set the agent working directory, which becomes the launch path / cwd sent with text and voice turns

By default the Workspace tab is rooted at the agent working directory (so the file viewer can't read files elsewhere on the machine). Set PI_SPEAK_WORKSPACE_ROOT on the gateway to widen or relocate that root — point it at a specific directory, or set it to fs to browse the whole drive/filesystem. The browser is confined to that root; paths outside it (and symlinks resolving outside it) are rejected.

/remote setup prints the same QR and links as /pk-remote. Use /remote setup bluetooth or /pk-remote bluetooth when the phone is paired over Bluetooth networking/PAN.

For real phone use, prefer an HTTPS URL through Tailscale Serve or a tunnel. If the phone is paired over Bluetooth networking/PAN instead, use /remote setup bluetooth; the Android app treats that as a Bluetooth local-link profile and does not require Tailscale.

The native Android app matches the web remote's control surface: session dashboard with rename / wake-alias / archive / remove, an OPS pane with the routing target picker (/v1/route), PK1/PK2 route slots (/v1/sessions/slots), discovered agents (/v1/agents), a live session-event feed (/v1/events), and a workspace file browser with a read-only viewer (/v1/workspace/file).

The Agent Hub tab's Tasks pane is a portal onto the same oh-my-pk background lanes, but hierarchical and actionable instead of flat and read-only: it lists each lane's subagents (GET /v1/herdr/agents), lets you send a message straight into a lane (POST /v1/herdr/agent/:id/chat) with a live transcript stream (GET /v1/herdr/stream/:id), archive a lane with a two-step confirm (POST /v1/herdr/agent/:id/kill), and launch a brand-new task anywhere with a free-form prompt/model/provider (POST /v1/sessions/launch) instead of only the fixed "Launch Hub" / "Launch Colab" presets. The e-ink (Boox) build exposes the same launcher and per-lane chat/archive controls, minus the live transcript stream (EPD panels ghost badly under frequent partial redraws, so lane detail there refreshes on the same periodic poll as the rest of the Hub peek).

When the active gateway is reachable over Tailscale, Android prefers advertised 100.64.0.0/10 base URLs and exposes a Warp / psmux control card in Discovery. That card calls /v1/warp to list open psmux sessions, tabs, and panes, /v1/warp/tab to open a native Warp tab via warp://action/new_tab?path=..., /v1/warp/tab-config to open a saved Warp Tab Config via warp://tab_config/<name>, and can create a detached psmux session or tab from the phone. Set PI_SPEAK_WARP_REMOTE_BASE_URL on the gateway to show the deployed Warp remote-control relay URL in the Android card.

Optional Windows tray:

/remote tray on

Right-click the tray icon and choose Show setup QR code. Scanning the QR opens the Android app with this computer's Tailscale endpoint, token, and saved machine profile metadata. Set PI_SPEAK_TRAY=1 to start the tray automatically with /remote on.

NPM-installed tray/service path:

npx -p pk-speak pi-speak-tray

Or, after global install:

pi-speak-tray --install-startup

The tray keeps the headless gateway running in the background, restarts it if it exits, and exposes setup, APK download, status, settings, restart, and web remote actions from the tray menu.

Speak From Any CLI

Use pk-speak speak when you want any command, hook, or coding agent to talk without starting the full phone gateway:

pk-speak speak "Tests passed"
git status --short | pk-speak speak --provider edge
pk-speak speak --provider gemini "Gemini TTS is ready."
pk-speak speak --provider sag "I need approval on this command."
pk-speak speak --no-play --output reply.mp3 "Saved an audio artifact."

It reads text from arguments or stdin, uses the saved setup profile, and supports auto, edge, gemini, elevenlabs, openai, sag, and legacy providers. Use --dry-run to inspect what would be spoken without synthesizing audio.

Wrap Any CLI Agent

Use pk-speak wrap when you want lifecycle voice notices around an arbitrary coding-agent CLI:

pk-speak wrap -- codex
pk-speak wrap --label "Claude Code" -- claude
pk-speak wrap --provider sag -- npm test
pk-speak wrap --capture -- npm test
pk-speak wrap --no-speak -- node -e "console.log('ok')"

The wrapper preserves the child command's terminal and exit code. It speaks the start and finish/failure notices around the command instead of capturing and parsing all output, so interactive CLIs keep their normal TTY behavior. Use --shell only when you intentionally need shell syntax or shell built-ins.

Use --capture for non-interactive commands when you want pk-speak to mirror output and classify important events. Capture mode prints and speaks compact notices for patterns such as approval prompts, input prompts, test failures, and errors. Keep capture off for fully interactive CLIs unless you are comfortable with stdout/stderr being piped instead of attached directly to the terminal.

Gemini Live Smoke Test

Use this before wiring a real-time session into the phone UI:

set PI_SPEAK_GEMINI_BACKEND=vertex
set PI_SPEAK_VERTEX_API_KEY=<optional-vertex-api-key>
set GOOGLE_CLOUD_PROJECT=<your-gcloud-project>
set GOOGLE_CLOUD_LOCATION=us-central1
gcloud auth application-default login
pi-speak-gemini-live-smoke --modality audio

To run the tray/headless gateway through ElevenLabs voice, backed by Vertex AI Gemini text reasoning:

REM Set ELEVENLABS_API_KEY in your shell before launch
set PI_SPEAK_GEMINI_BACKEND=vertex
set PI_SPEAK_VERTEX_API_KEY=<optional-vertex-api-key>
set GOOGLE_CLOUD_PROJECT=<your-gcloud-project>
set GOOGLE_CLOUD_LOCATION=us-central1
gcloud auth application-default login
set AGENT_PROVIDER=elevenlabs
pi-speak-gateway

This is the recommended high-quality voice stack. It uses ElevenLabs for reply audio and Vertex AI for Gemini reasoning so Google Cloud billing/credits apply through your Cloud project. Set REDACTED_ELEVENLABS_HISTORY_LINE when quality matters more than credit use.

To run the tray/headless gateway through Gemini Live instead:

set AGENT_PROVIDER=gemini-live
set PI_SPEAK_GEMINI_BACKEND=vertex
set GOOGLE_CLOUD_PROJECT=<your-gcloud-project>
set GOOGLE_CLOUD_LOCATION=us-central1
pi-speak-gateway

Optional environment:

• PI_SPEAK_GEMINI_BACKEND=vertex|developer-api selects Vertex AI or direct Gemini Developer API
• PI_SPEAK_VERTEX_API_KEY uses a Vertex AI API key instead of Application Default Credentials
• GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION configure Vertex AI
• PI_SPEAK_GEMINI_LIVE_MODEL selects the Live model
• PI_SPEAK_GEMINI_LIVE_MODALITY=audio|text selects response mode
• PI_SPEAK_GEMINI_API_VERSION=v1beta|v1alpha selects the Gemini API version
• PI_SPEAK_ELEVENLABS_MODEL_ID selects the ElevenLabs speech model
• PI_SPEAK_ELEVENLABS_VOICE_ID selects the ElevenLabs voice

Keep Gemini, Vertex, and ElevenLabs credentials server-side. Do not put them in the Android app or browser app.

The Conversational Assistant

The real-time gateway (realtime-gateway.ts) runs a Gemini Live session that acts as a conversational assistant, not a voice command executor. The assistant:

• can see all subagent state (list_agents, get_agent, read_transcript) and the workspace (list_workspace, read_workspace_file)
• interviews you to scope ambiguous requests before acting
• asks for explicit approval before any command that mutates a subagent, terminal, or file, using the propose_command tool, which returns a confirmation token and a human-readable description and does not execute until you approve
• keeps replies short and conversational

Mutating actions (execute_terminal_command, launch_agent, chat_agent, kill_agent, archive_session) are staged as proposals through the same approval registry used by the terminal and Agent Hub gateways. The assistant proposes; you approve; only then does it run.

Voice (/mono, /speak), Telegram (/phone), and the mobile web/Android remote (/remote, /pk-remote) are all input channels to this same assistant. The wake phrase PK is how you start a conversation hands-free.

Main Commands

/voice

One easy toggle for the whole local voice layer, with a persistent status-bar indicator (voice:tts, voice:stt, voice:combo, voice:realtime).

/voice            # cycle: off → tts → stt → combo → realtime → off
/voice tts        # spoken replies only (listener off)
/voice stt        # listening only (PK wake on, no spoken replies)
/voice combo      # turn-based loop: listen + speak (half-duplex, not realtime)
/voice realtime   # full-duplex Live agent on /v1/live (opens desktop orb for terminal users)
/voice off        # everything off (same end state as /pk-speak stop)
/voice status     # show each switch: TTS, STT, realtime readiness

combo is the turn-based speak/transcribe loop — it is not a realtime emotive voice agent. realtime is the full-duplex Live path (Gemini Live by default; optional OpenAI-Realtime/HF S2S via PI_SPEAK_LIVE_BACKEND). Selecting it stands the local TTS/wake loop down so the two audio paths never fight, checks credentials, ensures the gateway serves /v1/live, and for local operators opens the desktop orb at /orb/ (Edge app mode) rather than the full remote chrome. Phones and the PWA use /app/?mode=live or the Android Live button against the same socket. The chosen mode persists across sessions.

/speak

Turns spoken replies on or off, selects the TTS backend, or enables agent-driven speech.

Common examples:

/speak on
/speak agent
/speak off
/speak stop
/speak status
/speak test
/speak providers
/speak provider edge
/speak provider gemini
/speak provider openai
/speak provider elevenlabs
/speak rewrite on
/speak rewrite off

Behavior:

• Pi keeps the complete response on screen; the final terminal text is never automatically sent to TTS.
• /speak on injects the bundled speech command so the coding agent can give short, timely acknowledgements while it works and a concise outcome when useful.
• Speech commands run quiet, start playback without blocking the agent, and must discuss results rather than recite output.
• /speak agent remains a compatibility alias for explicit agent-driven speech behavior.
• /speak stop interrupts playback without disabling speech mode.
• /pk-speak stop hard-stops speech, persists mode off, and writes the root voice-disable sentinel; /pk-speak on is the authoritative re-enable path.

For MCP clients, use pk-speak-mcp (desktop package) or pi-speak-mcp (extension package). The server runs the bundled dispatcher with bounded request text and stderr, and cancels child speech on request cancellation, timeout, transport close, or process shutdown.

/voice

Selects the whole voice layer with one command instead of coordinating TTS, wake listening, and realtime audio separately.

/voice tts
/voice stt
/voice combo
/voice realtime
/voice off
/voice status

• tts enables layered spoken progress without reading the final terminal reply.
• stt enables wake-word input without spoken replies; combo enables both turn-based input and speech.
• realtime disables the local turn-based audio paths, starts the Gemini Live gateway when configured, and waits for a phone or web live client at /v1/live.
• Bare /voice cycles through off → tts → stt → combo → realtime.

/pk-speak

Hard-stop for voice chatter. Unlike /speak stop (interrupt current playback only), /pk-speak stop disables speech mode and stops the wake listener.

/pk-speak stop
/pk-speak status
/pk-speak on

Aliases for stop: off, quiet, silence, shush.

/mono

Controls the wake-word listener — the hands-free way to start a conversation with the assistant.

/mono on
/mono off
/mono status

Behavior:

• waits for the wake phrase PK by default
• activates voice input for a short window
• keeps the existing /mono flow intact with a faster-whisper wake detector
• supports PK <session-name> to route into a named session when the target name is spoken clearly
• keeps short numeric routes deterministic:
  • PK one, PK 1, and PK1 belong to the same 1 family
  • PK two, PK 2, and PK2 belong to the same 2 family
  • 1 stays distinct from 2
  • multi-word names like PK to Google stay literal and are not coerced into 2

/phone

Controls the Telegram bridge.

/phone on
/phone off
/phone status
/phone setup
/phone token <bot-token>
/phone code
/phone unpair

Behavior:

• text messages become Pi turns
• voice notes are transcribed, then sent to Pi
• replies can be delivered as text plus generated audio

/remote

Controls the HTTP API and mobile web app.

/remote on
/remote off
/remote status
/remote token
/remote setup
/remote setup bluetooth
/remote tray on
/remote tray off
/remote tray status

Behavior:

• starts the HTTP server
• serves the mobile app from /app/
• serves the phone setup page from /setup
• serves the bundled Android APK from /download/pi-speak.apk
• exposes remote-control endpoints
• generates a token if one is not already configured
• prints one-step setup URLs for the browser app and native Android app

/sess

Named sessions, wake aliases, and routing summaries for the assistant. In realtime voice, get_session_info/list_sessions report observed state, and switch_session selects one exact or unambiguous target for that live connection only. The assistant can then call send_session_message, resume_session, launch_agent, kill_agent, revive_agent, or archive_session; every mutation opens an operator approval card before execution. get_realtime_capabilities reports the features actually available on the selected Gemini or OpenAI-Realtime/HF backend.

/sess
/sess new bugfix
/sess switch bugfix
/sess name active-work
/sess rename bugfix voice-bugfix
/sess wake one
/sess wake clear one
/sess alias add bugfix one
/sess alias remove one
/sess edit bugfix
/sess remove bugfix
/sess confirm remove bugfix
/sess slots
/sess export
/sess ui
/sess ui open

This matters because PK bugfix can route voice input to that named session, while compact routes like PK one / PK1 and PK two / PK2 can stay stable and distinct.

/sess with no args shows the current session, ready sessions, aliases, store path, a compact 1 vs 2 lane summary, and inline state for known sessions.

Use /sess slots when you want the explicit compact-route view for PK one / PK1 and PK two / PK2.

Use /sess ui for inline guidance without opening another terminal. Use /sess ui open only when you explicitly want the older terminal pane; repeat launches reuse the existing pane instead of creating more terminal windows. The pane mirrors the /sess dashboard, refreshes within one second of external mutations, supports focus movement with arrow keys, tab, or j / k, shows the compact PK1/PK2 route lanes plus a focused-session footer, and adds keybindings [r] rename, [a] alias, [x] remove, and [q] quit.

For operator details, see:

• docs/VOICE_SESSION_BRIDGE.md
• docs/SESSION_OPERATIONS.md
• docs/CODEBASE_MAP.md

Architecture

The conversational assistant is the center; voice, wake-word, Telegram, and the mobile remote are input/output channels around it. There are seven main subsystems:

1. index.ts The extension entrypoint. Registers commands, persists state, owns wake-word routing, and coordinates TTS, STT, Telegram, and HTTP control.

2. realtime-gateway.ts The conversational assistant core. Runs a Gemini Live session with read-only subagent/workspace tools and a propose_command approval flow. Voice, phone, and remote turns all reach this assistant.

3. tts.ts Multi-provider speech synthesis (the assistant's voice). Supports legacy, edge, gemini, openai, elevenlabs, sag, higgs, stable-audio, and auto.

4. stt.ts, moonshine-stt.ts, and the listener workers Uploaded-audio transcription keeps the existing provider chain as the default. PI_SPEAK_REMOTE_STT_BACKEND=existing|moonshine|auto selects the completed-utterance backend policy; auto tries the existing chain first and activates the persistent Moonshine worker only after a classified transient/unavailable failure. PI_SPEAK_REMOTE_STT_PROVIDER=auto|local|openai|elevenlabs|google remains unchanged and applies only inside the existing leg.

5. listener/listener.py The always-on two-tier listener (wake-phrase detection is how you start a conversation with the assistant):

   • Tier 1: faster-whisper tiny for wake-phrase detection
   • Tier 2: faster-whisper for actual speech transcription

6. phone-bridge.ts Telegram transport for remote text and voice notes (a channel to reach the assistant from your phone).

7. control-server.ts Local HTTP API, audio artifact serving, and the built-in mobile app host.

Conversational Assistant Mode

realtime-gateway.ts runs the live-voice conversational assistant. It can call read-only tools on every turn — list_sessions, get_session_info, list_agent_hub_agents, get_agent_hub_agent, browse_workspace, read_workspace_file — to see real session, background-agent, and workspace state before answering.

Anything that mutates state goes through operator approval first:

• execute_terminal_command outside the read-only allowlist
• launch_agent when it actually launches an agent (not just opening the hub)
• archive_session (archive or recover)

The assistant calls the tool normally; if the action needs approval, the client shows what is about to happen and waits for an explicit approve/reject before anything runs. Nothing is claimed as done until a real tool result confirms it.

Remote Paths

Best Overall: Built-In Mobile Web App

Use this when you want:

• browser mic capture
• browser audio playback
• one-tap remote use from Android
• compatibility with Tailscale or an HTTPS tunnel

Start it:

/remote on

Open:

https://<your-url>/app/

Best Zero-Friction Fallback: Telegram

Use this when you want:

• the least setup
• reliable remote turns
• simple text plus voice note interaction

Start it:

/phone on

Best Button Grid: Unified Remote

Use this when you want:

• fast buttons for mono, speak, provider changes, and phone pairing
• a control surface on the phone

Do not use this as your main audio path. It is a controller, not a real voice transport.

Mobile Web App

The mobile app is built into the extension and served from:

/app/

Capabilities:

• record a voice turn with the browser microphone
• send typed fallback text
• request spoken replies on each turn
• autoplay returned audio when the browser allows it
• keep the token in session storage by default
• optionally remember the token on that device
• install as a PWA on Android

Token onboarding options:

1. Paste the token in the Settings panel
2. Open the app once with:

/app/?token=YOUR_TOKEN

The app will save the token into the current browser session and clean the URL immediately.

Secure-origin rules:

• localhost works
• HTTPS works
• random plain HTTP hostnames usually will not allow browser microphone access

That is why Tailscale Serve or an HTTPS tunnel is the right remote path.

Native Android can also use Bluetooth networking/PAN. Pair the phone with the desktop, start /remote on, run /remote setup bluetooth, then open the native setup link or select the built-in Bluetooth / local link profile and adjust the base URL to the desktop Bluetooth adapter IP if needed. Set PI_SPEAK_BLUETOOTH_BASE_URL before launching Pi Speak if you want /remote setup bluetooth to print a known adapter URL instead of the default http://192.168.44.1:8767/.

HTTP API

Start it with:

/remote on

Default bind:

host: 0.0.0.0
port: 8767

Public Routes

These are available before auth because they serve the built-in app:

GET /
GET /app/
GET /app/index.html
GET /app/app.webmanifest
GET /app/sw.js
GET /app/icon.svg

Control Routes

GET  /v1/health
GET  /v1/status
GET  /v1/diagnostics
GET  /v1/route
POST /v1/route

GET  /v1/workspace?path=<absolute-path>
GET  /v1/workspace/file?path=<absolute-path>

GET  /v1/mono/status
POST /v1/mono/on
POST /v1/mono/off

GET  /v1/speak/status
GET  /v1/speak/providers
POST /v1/speak/on
POST /v1/speak/off
POST /v1/speak/stop
POST /v1/speak/test
POST /v1/speak/provider/:provider
POST /v1/speak/rewrite/:onOrOff

GET  /v1/phone/status
POST /v1/phone/on
POST /v1/phone/off
POST /v1/phone/code
POST /v1/phone/unpair

GET  /v1/turn/text?text=hello&audio=1
POST /v1/turn/text
POST /v1/turn/voice

GET  /v1/audio/:id

Auth

Local bypass applies only to true localhost requests:

• localhost
• 127.0.0.1
• ::1

Remote clients must send one of:

• Authorization: Bearer <token>
• X-Pi-Speak-Token: <token>

Query-string token auth is reserved for:

• /app/?token=... bootstrap onboarding
• /v1/audio/:id?token=... reply-audio playback in the browser

Remote control and turn endpoints should use headers, not query-string auth.

Hardening Defaults

The production-oriented defaults are:

• same-origin CORS unless PI_SPEAK_HTTP_ALLOWED_ORIGINS is set
• request body limit for text turns: 64 KB
• request body limit for voice turns: 25 MB
• lightweight in-memory rate limits for non-local traffic
• background cleanup of expired reply-audio artifacts
• authenticated diagnostics at /v1/diagnostics, including a compact summary block for queue state, phone linkage, mono state, current session/target, and active error sources
• queue/backpressure for remote turns so Pi returns a deterministic busy response instead of piling up unlimited work
• synchronous remote turns fail fast when the current Pi session is already mid-turn, instead of hanging the HTTP request against the same active session
• mutating control routes require POST, leaving GET read-only for fetch-safe status endpoints
• outbound provider calls share a default 30s timeout via PI_SPEAK_OUTBOUND_TIMEOUT_MS

Inspect the active token with:

/remote token

Example Requests

Text turn:

curl -X POST http://127.0.0.1:8767/v1/turn/text ^
  -H "Content-Type: application/json" ^
  -d "{\"text\":\"Summarize the repo\",\"audio\":true}"

Voice turn:

curl -X POST "https://<your-host>/v1/turn/voice?audio=1" ^
  -H "Authorization: Bearer <token>" ^
  -H "Content-Type: audio/webm" ^
  --data-binary "@voice.webm"

Unified Remote

Bundled remote source:

unified-remote/Pi Speak

Install path:

C:\ProgramData\Unified Remote\Remotes\Custom\Pi Speak

What it is good at:

• toggling mono
• toggling speak
• switching providers
• requesting the Telegram pair code
• sending short text turns

What it is not good at:

• full remote voice capture
• browser-style audio playback
• low-latency conversational audio

Environment Variables

Core

AGENT_PROVIDER=pi|codex|claude|elevenlabs|gemini|gemini-live
CODEX_BIN=codex
CLAUDE_BIN=claude
PI_BIN=pi
AGENT_MODEL=
PI_SPEAK_EXECUTION_ROUTER_MODE=auto|pi|codex|claude
AGENT_CWD=
AGENT_WORKSPACE=
PI_SPEAK_TTS_PROVIDER=auto|legacy|edge|gemini|openai|elevenlabs|sag|higgs|stable-audio
PI_SPEAK_REWRITE_ENABLED=true|false
PI_SPEAK_WAKE_PHRASE=PK
PI_SPEAK_MONO_ACTIVITY_TIMEOUT=15
PI_SPEAK_WAKE_SENSITIVITY=low|medium|high
PI_SPEAK_WAKE_FUZZY_ENABLED=true|false              # optional override
PI_SPEAK_WAKE_FUZZY_MAX_DISTANCE=0|1|2              # optional override
PI_SPEAK_WAKE_COMPACT_PREFIX_ENABLED=true|false     # optional override

If PI_SPEAK_EXECUTION_ROUTER_MODE is unset, explicit AGENT_PROVIDER=pi, AGENT_PROVIDER=codex, or AGENT_PROVIDER=claude controls which backend remote turns dispatch to. Set the router mode to auto when you want the conversation router to choose from the reduced task, while phone/app-selected provider overrides still win for that turn.

The gateway provider contract currently treats codex and claude as resumable CLI session providers. Recent Codex and Claude sessions can show resume commands in the mobile/session dashboard, while pi remains a direct turn provider without stored-session resume support.

Rewrite

OPENROUTER_API_KEY=...
PI_SPEAK_REWRITE_MODEL=openai/gpt-oss-20b:nitro
PI_SPEAK_OPENROUTER_URL=https://openrouter.ai/api/v1/chat/completions

OpenAI

# Dedicated key for audio TTS (avoids consuming the general LLM key)
PI_SPEAK_OPENAI_KEY=...
# Legacy fallback
VOICE_TOOLS_OPENAI_KEY=...
PI_SPEAK_OPENAI_TTS_MODEL=gpt-4o-mini-tts
PI_SPEAK_OPENAI_VOICE=alloy
PI_SPEAK_REMOTE_OPENAI_STT_MODEL=whisper-1
PI_SPEAK_OPENAI_BASE_URL=https://api.openai.com/v1

ElevenLabs

# ELEVENLABS_API_KEY is read from your environment
PI_SPEAK_ELEVENLABS_VOICE_ID=<voice-id>
PI_SPEAK_ELEVENLABS_MODEL_ID=eleven_flash_v2_5

Vertex AI Gemini

PI_SPEAK_GEMINI_BACKEND=vertex
PI_SPEAK_VERTEX_API_KEY=<optional-vertex-api-key>
GOOGLE_CLOUD_PROJECT=<your-gcloud-project>
GOOGLE_CLOUD_LOCATION=us-central1
PI_SPEAK_GEMINI_TEXT_MODEL=gemini-3.5-flash
PI_SPEAK_GEMINI_LIVE_MODEL=gemini-3.1-flash-live-preview
PI_SPEAK_TTS_PROVIDER=gemini
PI_SPEAK_GEMINI_TTS_MODEL=gemini-3.1-flash-tts-preview
PI_SPEAK_GEMINI_TTS_VOICE=Kore

Run gcloud auth application-default login on the machine hosting the tray/gateway, or set PI_SPEAK_VERTEX_API_KEY to a Vertex AI API key. Enable the Vertex AI API on the Cloud project.

Google Cloud Speech-to-Text

google is Google Cloud Speech-to-Text v2 using Google Cloud ADC from gcloud auth application-default login, not Gemini TTS. PI_SPEAK_VERTEX_API_KEY does not authenticate Speech STT. One-time setup on the host:

gcloud auth application-default login
gcloud services enable speech.googleapis.com --project <your-gcloud-project>

Project resolution order: GOOGLE_CLOUD_PROJECT, GCLOUD_PROJECT, PI_SPEAK_VERTEX_PROJECT, then ADC discovery.

GOOGLE_CLOUD_PROJECT=<your-gcloud-project>
# optional:
PI_SPEAK_GOOGLE_STT_LOCATION=global
PI_SPEAK_GOOGLE_STT_MODEL=chirp_3
PI_SPEAK_STT_LANGUAGE=en-US
PI_SPEAK_REMOTE_STT_PROVIDER=google

PI_SPEAK_STT_LANGUAGE feeds both Google and ElevenLabs STT. Defaults differ: Google uses en-US; ElevenLabs uses en.

Moonshine local STT fallback

Moonshine Voice is an optional on-device backend for uploaded voice turns. It does not replace Gemini/OpenAI full-duplex /v1/live; it shares the existing completed-utterance STT boundary used by /v1/turn/voice, Telegram voice notes, and CLI audio transcription.

Install the pinned optional Python binding:

python -m pip install -r listener/requirements-moonshine.txt
python -m moonshine_voice.download --stt --language en --model-arch 1

Select a policy:

PI_SPEAK_REMOTE_STT_BACKEND=existing   # default; current provider behavior is unchanged
PI_SPEAK_REMOTE_STT_BACKEND=moonshine  # always use Moonshine
PI_SPEAK_REMOTE_STT_BACKEND=auto       # existing first; Moonshine only on a safe backend failure

auto never switches because a transcript is empty, uncertain, or merely slow. It performs at most one fallback attempt for the completed utterance, never mid-utterance, and preserves both failures if Moonshine also fails. Set PI_SPEAK_STT_TELEMETRY=off to suppress the structured selection/fallback log lines.

Moonshine settings:

MOONSHINE_VOICE_CACHE=<optional-cache-directory>
PI_SPEAK_MOONSHINE_MODEL_PATH=<optional-pre-provisioned-base-en-directory>

Without an explicit model path, Moonshine downloads the English base-en model into its cache on first use. For deterministic offline startup, pre-provision that directory and set PI_SPEAK_MOONSHINE_MODEL_PATH. Audio decoding reuses the installed faster-whisper/PyAV path, so WAV, WebM/Ogg, MP3, and MP4 uploads share the same 16 kHz mono decode behavior. Moonshine model downloads are not content-pinned by the upstream helper; pre-provision and checksum model files when reproducible packaging is required.

Moonshine's code and English models are MIT licensed. Non-English models use the Moonshine Community License and require a separate commercial/distribution review. The public Windows wheel is x86-64; Windows ARM64 is not currently distributed on PyPI.

Edge TTS

PI_SPEAK_EDGE_VOICE=en-US-AriaNeural
PI_SPEAK_EDGE_LANG=en-US
PI_SPEAK_EDGE_RATE=1
PI_SPEAK_EDGE_TIMEOUT_MS=15000

Legacy / Local Python

PI_SPEAK_SPEAK11_PATH=...
PI_SPEAK_PYTHON=...
WHISPER_MODEL=tiny
WHISPER_DEVICE=cpu
WHISPER_COMPUTE=int8
PI_SPEAK_REMOTE_WHISPER_MODEL=base
PI_SPEAK_REMOTE_STT_PROVIDER=auto|local|openai|elevenlabs|google

PI_SPEAK_PYTHON and PI_SPEAK_SPEAK11_PATH are now the first-class override path for local Python audio setups. When they are unset, Pi scans the normal Windows user-site Python*/Scripts locations before falling back to PATH resolution.

Telegram

PI_SPEAK_TELEGRAM_BOT_TOKEN=...
TELEGRAM_BOT_TOKEN=...
PI_SPEAK_PHONE_WAIT_TIMEOUT_MS=180000

HTTP Remote

PI_SPEAK_HTTP_HOST=0.0.0.0
PI_SPEAK_HTTP_PORT=8767
PI_SPEAK_HTTP_TOKEN=...
PI_SPEAK_HTTP_AUDIO_TTL_MS=600000
PI_SPEAK_HTTP_AUDIO_CLEANUP_MS=30000
PI_SPEAK_HTTP_ALLOWED_ORIGINS=https://your-tailnet-host,https://your-tunnel-host
PI_SPEAK_HTTP_TIMEOUT_MS=180000
PI_SPEAK_HTTP_TEXT_BODY_LIMIT_BYTES=65536
PI_SPEAK_HTTP_VOICE_BODY_LIMIT_BYTES=26214400
PI_SPEAK_HTTP_RATE_LIMIT_WINDOW_MS=60000
PI_SPEAK_HTTP_RATE_LIMIT_CONTROL=20
PI_SPEAK_HTTP_RATE_LIMIT_VOICE=6
PI_SPEAK_OUTBOUND_TIMEOUT_MS=30000

Troubleshooting

The phone web app opens, but the mic does not work

You are probably not on a secure origin.

Use one of:

• http://localhost:8767/app/
• Tailscale Serve over HTTPS
• Cloudflare Tunnel over HTTPS

/mono on starts, but voice transcription fails

You likely do not have the Python audio stack installed. The local listener depends on:

• numpy
• sounddevice
• faster_whisper

Remote voice turns fail

Check these in order:

1. /remote status
2. /v1/diagnostics
3. /remote token
4. PI_SPEAK_REMOTE_STT_PROVIDER
5. OpenAI key or local whisper setup
6. For google STT: Google Cloud ADC (gcloud auth application-default login), Speech-to-Text API enabled, and a resolvable project (GOOGLE_CLOUD_PROJECT / GCLOUD_PROJECT / PI_SPEAK_VERTEX_PROJECT, then ADC discovery). PI_SPEAK_VERTEX_API_KEY does not authenticate Speech STT.

Speech is using the wrong provider

Check:

/speak status
/speak providers
/speak provider gemini

Telegram pairing is stuck

Use:

/phone code
/phone unpair

Then link again with the fresh code.

Testing

Synthetic /v1/live text smoke: node ./dist/scripts/synthetic-live-smoke.js --dry-run (or omit --dry-run against a running gateway). TTS/STT provider latency (after npm run build): node dist/scripts/benchmark-tts.js --dry-run --text "hello" and node dist/scripts/benchmark-stt.js --dry-run --audio-file <wav> --providers google. Dry-run prints the plan only (no JSON, no provider calls). Default STT providers stay local openai elevenlabs.

Run the automated production-readiness checks with:

npm test

Current automated coverage includes:

• non-local auth enforcement
• localhost auth bypass
• body-size rejection
• voice content-type rejection
• rate limiting
• audio artifact expiry
• Telegram link + text-turn handling
• PWA token persistence rules
• remote queue backpressure behavior
• runtime path resolution for local Python / speak11
• explicit listener shutdown signaling with force-kill fallback

Manual Smoke Checklist

Before treating a machine as production-ready, verify:

1. /mono on
2. local wake phrase: say PK
3. /phone on then /phone code, then complete a Telegram text turn and voice-note turn
4. /remote on, open /app/, complete a text turn and voice turn, and confirm reply audio playback
5. over Tailscale or your HTTPS tunnel, confirm non-local requests fail without the token and succeed with it

For a full phone-focused run sheet with pass/fail capture fields, use docs/REMOTE_VALIDATION_CHECKLIST.md. For a compact operator worksheet, use docs/REMOTE_VALIDATION_RUN_SHEET.md.

Files You Will Care About

• index.ts
• realtime-gateway.ts
• tts.ts
• stt.ts
• phone-bridge.ts
• control-server.ts
• listener/listener.py
• web/remote/index.html
• docs/CODEBASE_MAP.md

Release Notes

See CHANGELOG.md.