COS Glasses Server

Self-hosted AI heads-up display for Even G2 smart glasses. Runs on your Mac, talks to your local Claude Code, Codex, or Cursor Agent CLI, and pushes answers, voice transcription, and notes to the lens. Your data never leaves your machine, and no API key is pasted into the phone for chat.

Quick start

npx --yes @gotcos/glasses-server@latest

For the optional COS Control macOS menu bar app, run the same non-mutating readiness check it uses before guided installation:

npx --yes @gotcos/glasses-server@latest --prepare-only

COS Control then installs the same npm package as a launchd-managed runtime. The original foreground command remains supported and unchanged.

Normal server start checks Node, finds your CLI, checks voice and image processing, writes ~/.cos-glasses/.env, and starts the server on 0.0.0.0:3141. Optional Whisper and Kokoro models are provisioned when their local services start; --prepare-only intentionally does not download or install optional models, write COS configuration, or start a listener. It may invoke an installed agent CLI's read-only version/auth probe, and that CLI may maintain its own user cache. On boot the server prints an API token — paste that into the COS Glasses app. Only one COS Glasses server may run on a Mac at a time; a second npx or source runner exits before opening ports or touching shared conversation/media state. Version 6.6.0 also gives that server a durable identity and boot-scoped display replay, allowing build 188+ to reconnect after a Tailscale, Wi-Fi, or process interruption without silently losing completed replies.

Requirements

• Node.js 20.11+ — https://nodejs.org
• Claude Code CLI (Opus/Fable/Sonnet). Claude Desktop alone does not install the terminal command. Install it on one line with npm install -g @anthropic-ai/claude-code (never with sudo), then run claude and finish the browser sign-in or Codex CLI (GPT Frontier/Balanced) — https://developers.openai.com/codex/, then codex login
• Optional: Cursor Agent CLI for Composer 2.5 Fast and Grok 4.5 Fast. Ensure agent is on PATH, run agent login, and verify agent models lists composer-2.5-fast and cursor-grok-4.5-high-fast. COS exposes the Cursor slots only after both models resolve; it never silently substitutes Claude or Codex.
• Even G2 glasses + the COS Glasses app from the Even Hub
• brew install whisper-cpp for free local voice (the launcher can download the model)
• Optional: brew install [email protected] ffmpeg espeak-ng for local Kokoro spoken replies on Apple silicon (Python 3.11-3.12 is supported). ffmpeg also enables phone/output image attachments; text chat remains available without these optional dependencies.
• Optional: Tailscale so your phone reaches your Mac from anywhere

No provider API key is needed for chat when using signed-in CLIs. Usage is billed to the corresponding Claude, Codex, or Cursor subscription. Pick a provider per query, or set a default with COS_G2_DEFAULT_MODEL (opus|fable|sonnet|codex-frontier|codex-balanced|cursor-grok|cursor-composer). Claude tier aliases and the two GPT slots resolve dynamically, so new model releases do not require a new glasses package. GPT discovery refreshes every 15 minutes and retains its last-known-good catalog through transient failures. Cursor discovery also refreshes every 15 minutes and retains its last-known-good catalog through transient failures. Cursor Agent mode can edit files and run shell commands in the selected workspace. Choose Ask mode when you want a non-editing answer; clients that omit the execution mode default to Ask. Existing COS_CODEX_MODEL / COS_CODEX_REASONING_EFFORT settings remain supported on the migrated Frontier slot; leave them blank for auto-latest. Codex runs sandboxed read-only by default. Set COS_CODEX_SANDBOX=workspace-write for workdir writes + outbound network (sandbox_workspace_write.network_access=true is passed by the managed server and should also be set in ~/.codex/config.toml for interactive Codex). Claude is the most permissive provider by default. It runs with --dangerously-skip-permissions, so a glasses query on the Claude/Opus path can run shell commands and read, edit, and write files on this Mac without prompting you. That is what makes the glasses useful for real work, and it has been the behavior for some time — but as of 6.18.3 the model is also correctly told it has those tools, so you will see it use them more readily than before. Set COS_CLAUDE_TRUST_MODE=allowlist to remove Claude's permission bypass and restrict it to COS's explicit per-query tool allowlist; undeclared tools then fail closed without prompting. Only the exact value allowlist restricts anything — any other value logs a warning and stays trusted.

Connect your phone (the one gotcha)

The glasses app runs on your iPhone and must reach this server on your Mac.

1. The launcher binds 0.0.0.0 (all interfaces) for you.
2. Same WiFi (simplest): find your Mac's LAN IP (System Settings > Wi-Fi > Details), and in the COS Glasses app enter http://192.168.x.x:3141.
3. From anywhere: install Tailscale on the Mac + iPhone (same account), note the Mac's 100.x address, and enter http://100.x.x.x:3141.
4. Either way, paste the API token the server printed at boot.

To restrict the server to localhost only, set BIND_HOST=127.0.0.1 in ~/.cos-glasses/.env. The built-in IP allowlist blocks public-internet traffic regardless. Its mesh range is the exact Tailscale/CGNAT allocation (100.64.0.0/10), not all of 100.0.0.0/8; RFC1918 LAN ranges remain supported.

What it does

• Ask anything, get a streamed answer on the lens (/api/query, /v1/chat/completions)
• With COS Glasses build 204+, server-owned durable queries are on by default: accepted work survives phone backgrounding, WebView reloads, and network handoffs, then reattaches without duplicate work or duplicate replies
• Choose Opus, Fable, Sonnet, GPT Frontier, GPT Balanced, Composer 2.5 Fast, or Grok 4.5 Fast. Cursor slots fail closed when the local CLI or concrete model is unavailable; optional redacted tool activity streams only to the authenticated query that requested it
• Message History + cross-day "reference message N" — your chats are archived by day and every message keeps a permanent number you can recall (/api/archive, /api/message/:num)
• Recent/history responses preserve validated photo references. Recovery uses exact session + global-message + message-era identity without exposing storage paths. Ambiguous pre-version historical refs fail closed; unversioned refs are recovered only inside the active era when created and associated after its boundary.
• Send phone photos with queued prompts, and review assistant-selected generated, research, or explicitly used email images in Messages and on the G2 lens
• Recover long voice prompts after phone, network, or server interruptions. Audio chunks are saved before transcription and retained locally for 72 hours. On compatible app builds, their warm transcript also appears live while speaking; final HQ transcription remains authoritative.
• Live voice capture + transcription during meetings
• With COS Glasses build 209+ and server 6.11.0+, meetings continue recording locally through a network interruption. Reconnecting reconciles the exact chunks already stored by the Mac, uploads only missing audio, and finalizes through an idempotent save receipt without duplicating the meeting.
• Since 6.19.0, meeting audio whose save never lands is quarantined for 72 hours (COS_UNSAVED_AUDIO_RETENTION_HOURS) instead of being cleaned up, surfaces on /api/health as unsaved_captures, and can be recovered into a durable meeting scribe with one authenticated call (POST /api/meeting/orphans/:sessionId/recover; list via GET /api/meeting/orphans).
• Local whisper.cpp transcription (free and local-only by default). OpenAI Whisper fallback is optional and requires both the exact COS_OPENAI_WHISPER_FALLBACK=1 opt-in and a configured key; a key alone never uploads audio.
• HQ prompt dictation is requested by default; the phone's Fast mode switch opts into turbo. Server 6.16.0 reports whether full local large-v3 actually ran, and compatible companions alert once if an HQ request used Fast or Cloud instead of silently claiming HQ.
• Local-first spoken reply playback through Kokoro on Apple silicon. The first use creates a private Python environment and downloads its model without blocking the API. Selecting Local fails closed; local_first can fall back to OpenAI TTS only when a key and budget are available. /api/health reports the independent tts_local state and current engine.
• Tasks / calendar / people context if you run the COS Starter Kit (COS_SCRIPTS_DIR); otherwise it is glasses + AI only
• Welcome weather on the glasses home screen via authenticated GET /api/welcome-context?lat=&lon=. The phone supplies GPS (Even Hub location permission); this server only proxies Open-Meteo. Without phone coords the route uses last-known process coords, then optional COS_WEATHER_DEFAULT_*, otherwise omits weather. Optional nextEvent appears when COS_SCRIPTS_DIR calendar data is available.

Configuration

Config lives at ~/.cos-glasses/.env (created on first run). Every key is optional except an installed CLI. Highlights: BIND_HOST, PORT, COS_API_TOKEN (auto if unset), COS_OPENAI_WHISPER_FALLBACK=1 plus OPENAI_API_KEY (explicit cloud transcription/TTS fallback), COS_TTS_ENGINE (local_first or openai_primary), COS_TTS_KOKORO_VOICE (local voice id), COS_TTS_LOCAL_DISABLE=1 (disable the sidecar), and COS_TTS_PRONUNCIATIONS_JSON (optional local/cloud pronunciation overrides), COS_EXTRA_TOOLS (comma-separated mcp__server__tool or mcp__server__* selectors shared by full and lightweight Claude paths), COS_CLAUDE_MCP_CONFIG (optional absolute config path when .mcp.json is not in the managed CLI working directory), COS_CURSOR_AGENT_BIN (optional absolute Cursor agent binary), COS_CURSOR_PERSIST_SESSIONS=0 (disable Cursor session resume), COS_WEATHER_DEFAULT_LAT / COS_WEATHER_DEFAULT_LON / COS_WEATHER_DEFAULT_CITY (optional home fallback when phone GPS is denied), COS_SCRIPTS_DIR (full pipeline), COS_DURABLE_QUERY_JOBS=0 (optional machine-wide rollback for build 204+ server-owned query recovery), and COS_MEDIA_ROOT (optional image-store location; default ~/.cos-glasses/data/media). Your name + transcription vocabulary live in ~/.cos-glasses/.cos-profile.json (see .cos-profile.example.json). Factory example values are ignored; add the real names, companies, acronyms, and specialist terms you say often. Guided Setup writes a safe empty profile instead of biasing Whisper toward placeholder text. Telegram activity export is disabled by default even when a private COS pipeline contains .telegram_config.json; enable it only with the explicit COS_TELEGRAM_NOTIFICATIONS=1 opt-in.

Speaker diarization (opt-in)

Without a voiceprint model this server does not classify speakers at all — it passes through whatever label the client sends (Unknown when the client sends nothing; the COS companion sends its own wearer/Ext labels). Named per-speaker diarization needs a ~26 MB voiceprint model that is deliberately not shipped in the npm package, so it is a bolt-on:

mkdir -p ~/.cos-glasses/models
# put 3dspeaker_speech_eres2net_sv_en_voxceleb_16k.onnx there, then restart

The server searches, in order: COS_SPEAKER_MODEL_PATH (explicit full path to the .onnx), ~/.cos-glasses/models/, then a bundled server/models/ copy (source checkouts only). Use the data home, not the installed package — anything inside the package is destroyed by the next update, while ~/.cos-glasses/ survives.

Use an absolute path if you set COS_SPEAKER_MODEL_PATH — a relative one resolves against the working directory, which under the managed LaunchAgent is the installed package.

Verify with /api/health → speaker_id:

| Value | Meaning | |---|---| | active | model loaded, diarization running | | unavailable | no model found — labels come from the client | | error | a model is present but the runtime rejected it (see the startup log) |

The model is read once at startup, so restart after adding it. A corrupt or mismatched .onnx is screened and probed in a child process first, so a bad download disables diarization instead of taking the server down — but it does mean a wrong file fails silently apart from that log line.

The wearer's label comes from owner_speaker_label in ~/.cos-glasses/.cos-profile.json (default Me); set it to match the profile name you enrol under. Train voices via /api/voice/enroll?name=… (the default name is owner_speaker_label); profiles persist in ~/.cos-glasses/data/voice-profiles.json.

Upgrading from an older server: before this release the enrollment default was hardcoded to MU, so an existing install may hold a profile under that name while owner_speaker_label resolves to Me. /api/voice/status will then report enrolled: false. Set owner_speaker_label to MU to keep the existing voiceprints rather than re-enrolling, which would split the same voice across two profiles.

HQ dictation

Prompt dictation defaults to HQ. The phone owns the preference: Fast mode OFF requests HQ, and Fast mode ON requests turbo. The Mac performs all decoding; the phone does not run Whisper.

For the recommended Balanced setup, run:

npx --yes @gotcos/glasses-server@latest --setup-transcription --transcription-tier balanced

That keeps three jobs separate: Small.en supplies provisional prompt words on the lens, Large-v3-Turbo commits the authoritative live transcript, and Large-v3 polishes saved prompts and meetings. Small.en never writes the recovery ledger and receives no decoder-bias prompt. If its sidecar is missing or unhealthy, preview falls back to Turbo without changing final quality. Set COS_WHISPER_PREVIEW_MODEL=turbo to keep one live model, or off to disable provisional peeks. Existing installs that only update the server remain on Turbo until Guided Setup opts them into Small.en.

Max is an opt-in tier for powerful Macs:

npx --yes @gotcos/glasses-server@latest --setup-transcription --transcription-tier max

Max keeps Turbo resident in the isolated preview sidecar for low-latency provisional words, while Large-v3 remains authoritative for live commit and saved-work polish. Canonical transcription has strict GPU priority: a cosmetic preview is dropped or aborted instead of competing with a committed decode. If Large-v3 is missing, health reports the downgrade and the server falls back to Turbo rather than making transcription unavailable. COS Control is the supported owner of the machine-wide tier; the per-lane environment variables remain advanced overrides.

Server 6.21.8 adds a default-off meeting-completion canary. With COS_MEETING_PROGRESSIVE_HQ=1, sealed meeting windows can be polished ahead of Stop/save on a CPU-only, single-flight lane; finalization reuses only matching audio/model/context checkpoints. Balanced is capped at two background threads for fanless M1/M2 MacBook Airs, while Max defaults to six and remains capped by available CPUs. COS_MEETING_EARLY_SYNC=1 separately gives the Operations sync pipeline a stable meeting identity before HQ completes. Either switch can be disabled without changing canonical live transcription or raw meeting audio.

The first server start downloads the real-time turbo model. True HQ additionally requires the full ggml-large-v3.bin model (about 3.1 GB):

mkdir -p "$HOME/.local/share/whisper-models"
curl -fL --progress-bar \
  https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3.bin \
  -o "$HOME/.local/share/whisper-models/ggml-large-v3.bin.partial"
mv "$HOME/.local/share/whisper-models/ggml-large-v3.bin.partial" \
  "$HOME/.local/share/whisper-models/ggml-large-v3.bin"

Restart the server, then confirm capabilities.transcription.hq.hqAvailable: true at /api/health. The response does not expose local paths. If the CLI or model is unavailable, dictation stays usable on Fast and reports the downgrade truthfully. Set COS_HQ_SPECULATIVE_WARM=0 to disable background HQ warm immediately; set COS_BATCH_LARGE_V3=0 to explicitly use turbo. Interactive HQ uses beam 2 by default (COS_HQ_BEAM_INTERACTIVE); meeting batch remains beam 5.

Run from source

git clone https://github.com/ukaoma/cos-glasses-server.git
cd cos-glasses-server
npm install
BIND_HOST=0.0.0.0 npm run start:server

Troubleshooting

• Claude Desktop is installed but COS says Claude Code is missing — Desktop and the terminal CLI are separate. Run npm install -g @anthropic-ai/claude-code on one line without sudo, then run claude and complete sign-in. Verify with claude --version before starting COS again.
• npm reports EACCES or a root-owned cache — never run COS or npm with sudo, and do not recursively change system ownership. Use a private COS cache instead: npm_config_cache="$HOME/.cos-glasses/npm-cache" npx --yes @gotcos/glasses-server@latest. Version 6.12.2+ never runs a second install from inside npm's temporary cache.
• Phone can't connect — check BIND_HOST=0.0.0.0, the same Tailscale account on both devices, and the correct 100.x IP + token.
• Safari connects but the app does not — confirm npx --yes @gotcos/glasses-server@latest is 6.6.0+, then use the app's server reconnect/edit control to verify the current URL and token. Do not run a second source or npx server alongside it.
• AI queries fail — run claude auth status, codex login status, or agent status for the selected provider, then authenticate with claude auth login, codex login, or agent login when signed out.
• Composer or Grok is missing — update to server 6.16.1+, confirm agent is discoverable on the service PATH, and run agent models. /api/health must report features.cursor: true; authenticated /api/models must include both cursor-composer and cursor-grok. Missing models fail closed instead of falling through to another provider.
• Voice getting billed? — voice is local-only by default in 6.12.0+. Confirm /api/health reports capabilities.transcription.mode: "local-only". Remove COS_OPENAI_WHISPER_FALLBACK (or set it to 0) to disable an earlier opt-in.
• Local voice unavailable? — install whisper-cpp, restart the server, and confirm /api/health reports features.whisper: true. A typed retryable 503 keeps compatible prompt/meeting audio available for retry instead of silently sending it to OpenAI.
• Local spoken replies unavailable? — on Apple silicon, install [email protected] ffmpeg espeak-ng, restart the server, and wait for the first-run Kokoro model download. Confirm /api/health reports tts_local.ready: true. If Python lives outside the normal Homebrew paths, set its absolute 3.11 or 3.12 path as COS_TTS_BOOTSTRAP_PYTHON in ~/.cos-glasses/.env. Selecting Local never falls back to cloud; set COS_TTS_ENGINE=openai_primary only when OpenAI playback is intentionally configured.
• Photos unavailable? — install ffmpeg, restart the server, and confirm /api/health reports features.mediaProcessingReady: true.
• Prompt recovery unavailable? — update with npx --yes @gotcos/glasses-server@latest, then confirm /api/health reports features.promptRecovery: true.
• Durable query recovery unavailable? — build 204+ requires server 6.10.0+. Restart once, then confirm /api/health reports features.durableQueryJobs: true, protocol 1, and state ready. To roll back, set COS_DURABLE_QUERY_JOBS=0; accepted jobs still drain while new prompts use legacy streaming.
• Offline meeting recovery unavailable? — build 209+ requires server 6.11.0+. Restart once, then confirm /api/health reports features.localFirstMeetings: true and capabilities.localFirstMeetings.protocolVersion: 1. Older app builds keep using their existing live-transcription and meeting-save paths.

License

MIT. Learn more at gotcos.com.