@molecule/api-ai-speech-llm

Language-model speech-to-text provider for molecule.dev.

Implements transcribe from @molecule/api-ai-speech with any @molecule/api-ai chat model that accepts audio input, so an app with no speech service still gets batch transcription through the model it already pays for — hosted or self-hosted.

Auto-generated, AI-first package reference for the molecule.dev ecosystem. It is written to be read by coding agents as much as by people, and is generated from this package's source — edit src/index.ts JSDoc, not this file.

Quick Start

import { setProvider as setAi } from '@molecule/api-ai'
import { provider as google } from '@molecule/api-ai-google'
import { requireProvider, setProvider } from '@molecule/api-ai-speech'
import { createProvider } from '@molecule/api-ai-speech-llm'

setAi(google) // the app's AI provider — its model must accept audio
setProvider(createProvider()) // or createProvider({ ai, model }) for another

const { text } = await requireProvider().transcribe!({
  audio: new Uint8Array(recordingBytes),
  filename: 'memo.webm',
  language: 'en',
})

Type

provider

Installation

npm install @molecule/api-ai-speech-llm @molecule/api-ai @molecule/api-ai-speech

API

Interfaces

LlmSpeechConfig

Options for {@link createProvider}. ai falls back to the bonded ai provider; every other field has a prompt-level default.

interface LlmSpeechConfig {
  /**
   * The AI provider to transcribe with. Defaults to the provider bonded as
   * `ai` (`@molecule/api-ai` `requireProvider()`), whose model must accept
   * audio input.
   */
  ai?: AIProvider
  /** The model id to pass to the provider. Omit to use the provider's default. */
  model?: string
  /** Default language hint when a `transcribe` call passes none. */
  language?: string
  /** Extra instructions, e.g. "Medical dictation; keep drug names verbatim". Appended to the system prompt. */
  instructions?: string
  /** Output token ceiling. Defaults to a length derived from the audio size. */
  maxOutputTokens?: number
}

Functions

systemPrompt(options)

The system prompt for one transcription pass.

function systemPrompt(options: {
  diarize?: boolean
  maxSpeakers?: number
  instructions?: string
}): string

• options — Diarization request and extra caller instructions.
• options.diarize — Ask the model to label speakers line by line.
• options.maxSpeakers — Upper bound on speakers to label.
• options.instructions — Extra caller instructions, appended last.

Returns: The system prompt.

userPrompt(language, context)

The text part of the user message (the audio block is sent alongside it).

function userPrompt(language?: string, context?: string): string

• language — Optional language hint.
• context — Optional context / spelling hints.

Returns: The user-side instruction.

audioMediaType(filename)

Picks the audio MIME type from a filename.

function audioMediaType(filename: string | undefined): string

• filename — Filename hint (defaults to WAV when absent or unknown).

Returns: The MIME type.

parseSpeakerLines(transcript)

Parses speaker_<n>: text lines into labelled segments.

function parseSpeakerLines(
  transcript: string,
): { segments: TranscriptionSegment[]; text: string } | null

• transcript — The model's diarized answer.

Returns: The segments (start = end = 0 — the model gives no timings) and the transcript with labels removed, or null when no line carried a label.

createProvider(config)

Creates a language-model speech provider.

function createProvider(config?: LlmSpeechConfig): AISpeechProvider

• config — Which AI provider/model to use and how to prompt it.

Returns: An AISpeechProvider backed by an audio-capable chat model.

Constants

provider

The provider implementation.

const provider: AISpeechProvider

Core Interface

Implements @molecule/api-ai-speech interface.

Bond Wiring

Setup function to register this provider with the core interface:

import { setProvider } from '@molecule/api-ai-speech'
import { provider } from '@molecule/api-ai-speech-llm'

export function setupAiSpeechLlm(): void {
  setProvider(provider)
}

Injection Notes

Requirements

Peer dependencies:

• @molecule/api-ai >=1.3.0
• @molecule/api-ai-speech >=1.1.0

Runtime Dependencies

• @molecule/api-ai

• @molecule/api-ai-speech

• The model must accept audio input. Most chat models do not — a text-only model fails the request or invents a transcript. Do not bond this provider for a non-audio model; check the model catalog first and pass one with createProvider({ model }).

• Batch only. No transcribeStream, no diarize(), no translate, no TTS. Feature-detect per the core. responseFormat: 'srt' | 'vtt' throws — a chat model returns no timings — and no result carries words or duration.

• diarize: true is best-effort text labelling, NOT acoustic diarization — treat it as unreliable. The model is asked to prefix lines with speaker_<n>:; those become segments[].speaker, and every segment has start = end = 0 because there are no timings. Use @molecule/api-ai-speech-nemo-speech when "who said what" matters.

• A model is a transcriber that may "improve" the speaker (fix grammar, drop fillers, translate) despite the prompt. Spot-check a new model before trusting it in bulk; markdown fences are stripped, other chatter lands in text verbatim.

• The filename extension picks the audio MIME type (.webm → audio/webm, .m4a → audio/mp4, unknown → audio/wav); whether the provider accepts that container is up to the chat bond and model.

• Uses the AI provider bonded as ai unless you pass one. Spend is that provider's token spend — audio bills as input tokens.

E2E Tests

Integration checklist — drive the real UI (live preview, no mocks), adapt each item to this app's actual screens/flows, and check every box off one by one. A box you can't check is an integration bug to fix — not a skip. The wired provider implements a subset (TTS, STT, or both) — feature-detect and run only the direction(s) it actually exposes:

• [ ] TTS: each read-aloud / narration / voice-note flow the app defines returns real PLAYABLE audio — the response carries an audio Content-Type (e.g. audio/mpeg) and non-trivial bytes (not a 0-byte file, not a JSON-encoded blob), and the UI's <audio>/player actually plays it. The spoken audio reflects the input text you sent.
• [ ] TTS: synthesizing two different strings yields two audibly different clips (distinct byte payloads) — proves the text drives synthesis and you are not hearing a cached/placeholder file.
• [ ] TTS: any exposed voice/format/speed option takes effect — switching voice (voice/voiceId) or output format (responseFormat/ outputFormat) changes the returned audio (audible voice, Content-Type, or file extension); an unknown voice/format surfaces a visible error rather than silently falling back.
• [ ] STT: transcribing a KNOWN spoken clip returns text that MATCHES the words spoken (not empty, not garbage) and renders in the UI; translating a non-English clip returns English text. Any timestamps the UI shows line up with the audio.
• [ ] STT: a silent / empty / too-short clip is handled gracefully — an empty or "no speech detected" result shown in the UI, never a crash or a spinner that hangs forever.
• [ ] The audio artifact is served from the app's OWN origin (a stored upload or a route the app serves), NOT a raw expiring provider URL — copy the audio src, reload it later, and confirm it still plays.
• [ ] A provider error (bad key, quota, unsupported format/voice, or a capability the wired provider lacks) surfaces as a readable message in the UI, not an unhandled crash or a blank/broken player.
• [ ] Synthesis/transcription runs SERVER-SIDE and authorized: the provider key never reaches the browser (check the network tab / client bundle), and no unauthenticated or unbounded endpoint lets a caller fire arbitrary synthesize/transcribe calls — each is billed per character / per minute of audio, so an open endpoint is a cost-DoS vector.