audio-transcribe

Speech-to-text for pi using an oMLX server running a Qwen3-ASR model. Give pi an audio file; it transcribes, validates the speech, and persists a JSON transcript sidecar beside the original.

Built and verified against oMLX 0.6.4 on http://192.168.31.152:8123 with Qwen3-ASR-1.7B-8bit.

What you get

| Surface | Trigger | |---|---| | transcribe_audio tool | The model calls it when you reference an audio file | | /transcribe command | You invoke it directly | | input hook | Audio paths pasted into chat are auto-transcribed before the model sees them |

Per file: pre-flight → ffprobe → ASR → cross-check pass → validation → atomic JSON sidecar (audio.m4a → audio.transcript.json).

Usage

/transcribe                                             # interactive controls
/transcribe "/Users/me/Downloads/Medidas da gaveta.m4a"
/transcribe ./recordings --recursive --lang pt --force
/transcribe --status                                    # scriptable diagnostics

The bare command opens a responsive control panel in TUI mode. It manages the input hook, validation pass, overwrite policy, language/model hints, server, timeout, file-size limit, diagnostics, and reset. It can also launch a direct transcription. Outside TUI mode the bare command emits status. Existing nested forms remain the stable scripting interface.

Flags: --lang <iso>, --model <id>, --no-crosscheck, --force, --recursive.

Pasting a path into chat also just works, including the backslash-escaped form pi inserts on interactive paste:

> what did I say in /Users/me/Downloads/Medidas\ da\ gaveta.m4a ?

Configuration

~/.pi/agent/audio-transcribe.json (defaults shown):

{
  "baseUrl": "http://192.168.31.152:8123/v1",
  "language": null,
  "model": null,
  "timeoutMs": 300000,
  "crossCheck": true,
  "autoDetect": true,
  "maxBytes": 0,
  "force": false
}

The file is created on first load, written atomically with owner-only permissions, and a corrupt copy is preserved before defaults recover. The panel intentionally hides API-key contents; add optional "apiKey": "..." by hand or use an environment variable.

Precedence: defaults < config file < env (OMLX_BASE_URL, OMLX_API_KEY, AUDIO_TRANSCRIBE_{MODEL,LANGUAGE,TIMEOUT_MS,CROSSCHECK,AUTODETECT,MAX_BYTES}). language: "auto" or null means send no hint and let the server detect. Active environment overrides are called out in the panel instead of silently masking the saved file value.

Validation

Every transcript gets a deterministic verdict — ok / review / no_speech / failed — written into the sidecar with the individual checks.

Structural: speech present, repetition/hallucination loops (single-word stutter and multi-word), UTF-8/garble, duration coverage, words-per-second plausibility, language label vs content heuristics.

Cross-check: a second ASR pass using the language the server itself reported, compared by word-bigram similarity. Stable audio reproduces byte-for-byte, so divergence is a real instability signal rather than noise.

Measured on a real 102.9 s Portuguese recording: verdict ok, confidence 1.0, cross-check 100 %, ~4.4 s total (two passes).

Sidecar format (pi-audio-transcript/v1)

{
  "schema": "pi-audio-transcript/v1",
  "audio": { "path": "...", "bytes": 1679661, "sha256": "...", "mime": "audio/mp4",
             "durationSeconds": 102.867729, "sampleRateHz": 48000, "channels": 1, "codec": "aac" },
  "transcript": { "text": "Vou gravar aqui as medidas...", "words": 56, "characters": 337, "language": "pt" },
  "validation": { "verdict": "ok", "confidence": 1, "checks": [...], "recommendations": [...],
                  "crossCheck": { "performed": true, "similarity": 1 }, "measurements": [...] },
  "provider": { "name": "oMLX", "baseUrl": "...", "model": "Qwen3-ASR-1.7B-8bit", "engine": "audio_stt" }
}

sha256 lets a downstream consumer tell a stale sidecar from a fresh one if the audio changes. Writes are atomic (temp + rename), so an interrupted run never leaves a half-written transcript that looks authoritative.

validation.measurements is additive derived data — spelled-out numbers like "vinte seis centímetros" are also exposed as { "value": 26, "unit": "cm" }. The transcript.text is never rewritten, so nothing you dictated is silently altered.

Verified server behaviour (this is what shapes the design)

These were measured against the live box, not assumed:

• duration in the response is inference latency, not audio length. A 102.87 s clip reported duration: 1.72 — a 60× error. Real duration comes from ffprobe, falling back to segments[].end (which was accurate to 3 decimal places). The sidecar labels this explicitly.
• Unsupported input returns HTTP 500 unsupported file format, not a 4xx. Files are therefore pre-flight-checked client-side so the message is actionable and no upload is wasted.
• word_timestamps=true and response_format are accepted but ignored. This extension makes no claim about word-level timing.
• language casing is inconsistent — "Portuguese" on auto-detect vs "portuguese" when hinted. Always normalized.
• Silence is honest: text: "", language: null, segments[].language: "None" → surfaced as no_speech, not an error.
• Formats confirmed working: wav, mp3, m4a, aac, opus (WhatsApp voice notes).

Known limitations

• Auto-detect was wrong on synthetic audio. A TTS-generated Portuguese clip came back labelled English with a mangled transcript. On real recorded speech auto-detect was correct and stable, so the default stays auto-detect; the language check plus cross-check are the safety net, and a mismatch emits a language=pt recommendation rather than silently overriding you.
• Heuristic language detection is shallow — tuned for pt/en/es plus CJK and Cyrillic scripts. Portuguese and Spanish share markers, so es/pt can be confused on short clips.
• No diarization and no word timestamps — the server does not provide them.
• extractMeasurements reads literal phrasing. "vinte seis centímetros ponto dois milímetros" becomes 26 cm + 2 mm, not necessarily the 26.2 cm you may have meant. Treat measurements as hints, not resolved values.
• Passing mentions of audio paths get transcribed. The input hook stats every candidate and skips missing/directory/non-audio, but a real path mentioned in passing will be transcribed. Set autoDetect: false if that is unwanted.
• Existing sidecars are skipped unless force is set, so re-transcribing after a config change needs --force.

Install

Published to npm as @maheidem/pi-audio-transcribe:

pi install npm:@maheidem/[email protected]

or install from a local path / add the path to packages in ~/.pi/agent/settings.json, then /reload.

Requires ffprobe (ffmpeg) on PATH for ground-truth duration; without it the extension still works and falls back to segments[].end.

Development

npm install
npm test          # 68 offline contracts in isolated HOME directories
npm run typecheck

Architecture: pipeline.ts holds all transcription logic and deliberately imports no pi, which is what makes the whole flow testable against a fixture server that reproduces the real oMLX quirks. settings.ts owns canonical persistent actions, ui/audio-panel.ts derives the view, and index.ts remains the thin Pi adapter (tool / command / input hook). omlx.ts is the HTTP client, validate.ts the pure validation functions, paths.ts path parsing, and sidecar.ts atomic persistence + ffprobe.