shared-whisper-server

One shared, on-demand Whisper Large v3 Turbo instance for macOS (Apple Silicon / MLX) serving several desktop apps over HTTP — instead of every app loading its own multi-gigabyte STT model.

Built for (but not limited to) Handy, screenpipe and Electron meeting assistants.

How it works

Handy ──────────── POST /transcribe (raw WAV) ──────┐
screenpipe ─────── POST /v1/audio/transcriptions ───┤        ┌───────────────────────────┐
meeting-assistant  POST /v1/audio/transcriptions ───┼──────▶ │ Node supervisor :8737      │
                                                    │        │  • spawns MLX worker       │
                                                    │        │    on first request        │
                                                    │        │  • shortest-audio-first    │
                                                    │        │    queue (PTT ahead of     │
                                                    │        │    long batch chunks)      │
                                                    │        │  • unloads worker after    │
                                                    │        │    10 min idle (TTL)       │
                                                    │        └──────────┬────────────────┘
                                                    │                   │ JSON over stdin/stdout
                                                    │        ┌──────────▼────────────────┐
                                                    │        │ Python worker (mlx-whisper)│
                                                    │        │ large-v3-turbo, Metal GPU  │
                                                    │        │ ~1.8 GB RSS while loaded   │
                                                    │        └───────────────────────────┘

The supervisor is a tiny always-on launchd agent (~30 MB). The heavyweight MLX worker starts on the first transcription request (~10–15 s cold start) and is unloaded after ttlSeconds of idle — so the ~2 GB of model memory is only resident while you actually transcribe.

Inference is intentionally single-threaded and single-process — MLX GPU streams are thread-scoped (multi-threaded Python servers crash with There is no Stream(gpu, N) in current thread), and a second worker process was measured to slow everything down through Metal contention rather than add capacity. Latency fairness comes from scheduling instead:

• long audio is transcribed in 15 s segments, each a separate queue op, so short requests run between the segments of a long batch chunk (worst-case wait ≈ one segment, not one chunk);
• the queue is aged shortest-first: effective weight = audio length − 1 s of credit per second waited, so push-to-talk beats batch chunks instantly yet can't be starved by an endless stream of realtime chunks;
• context is carried across segments via the Whisper prompt, so segmentation does not degrade transcription quality.

Measured on an Apple M5 under a 2-minute worst-case mixed load (continuous 1.5 s realtime chunks + 45 s batch chunks every 30 s + push-to-talk every ~10 s), 0 errors, worker RSS stable at ~1.8 GB:

| client | p50 | p95 | max | |---|---|---|---| | realtime 1.5 s chunks | 0.96 s | 2.7 s | 2.7 s | | push-to-talk ~2 s | 2.0 s | 3.2 s | 3.2 s | | batch 45 s chunks | 11.8 s | 12.2 s | 12.2 s (client timeout 30 s) |

Install / use

npx -y shared-whisper-server ensure    # install if needed, start, wait until healthy
npx shared-whisper-server status
npx shared-whisper-server stop
npx shared-whisper-server uninstall

ensure is idempotent and safe to call from apps on startup. Everything lives in ~/Library/Application Support/SharedWhisper (standalone uv, Python 3.12 venv with mlx-whisper, HF model cache, logs) plus one LaunchAgent ai.sharedwhisper.server.

HTTP API (127.0.0.1:8737)

| Route | Description | |---|---| | GET /health | Service + backend status, always available without auth | | GET /v1/models | Lists the pinned model (OpenAI shape) | | POST /v1/audio/transcriptions | OpenAI-compatible multipart: file, language, prompt, response_format = json | text | verbose_json. The model field is accepted and ignored — the shared instance always runs the configured model | | POST /transcribe | Raw WAV body → {"text": "..."} (Handy remote-server contract) |

Audio in any common format/sample rate is accepted (decoded with libsndfile, resampled to 16 kHz; no ffmpeg dependency).

Configuration

~/Library/Application Support/SharedWhisper/config.json:

{
  "port": 8737,
  "model": "mlx-community/whisper-large-v3-turbo",
  "ttlSeconds": 600,
  "token": ""
}

token enables Authorization: Bearer checking (off by default; the server only binds to 127.0.0.1). Env overrides: SWS_PORT, SWS_MODEL, SWS_TTL_SECONDS, SWS_TOKEN, SHARED_WHISPER_HOME.

Tests

npm test                  # unit: contracts, auth, TTL, concurrency, priority (fake worker)
npm run test:integration  # against the real installed service (SWS_INTEGRATION=1)

Requirements

macOS 14+ on Apple Silicon, Node.js ≥ 18. Python is not required system-wide — a managed 3.12 toolchain is installed via uv into the shared dir.

Known limitations

• Preemption granularity is one 15 s segment: a request that is already running finishes its current segment before the next job is picked (worst-case extra wait ≈ 1.5–2 s of inference).
• Batch chunks stretch under sustained realtime load (they yield between segments) — by design; they stay well under typical client timeouts.
• No streaming endpoint yet — pseudo-realtime clients should send short chunks to the batch endpoint.