@codefusion-cc/speech

Streaming text to speech behind one interface. An app opens an utterance, writes text as a model produces it and gets PCM16 audio back through a callback; which provider speaks is one line of configuration, and no app talks to a provider directly. It runs in Workers: fetch, WebSocket, WebCrypto, no node: imports, no dependencies.

Audio is always mono, PCM16 little-endian, at the rate each chunk states (AudioChunk.sampleRate: 24000 for Soniox and for the OpenRouter models measured so far, 44100 or others when a model answers so). A chunk holds whole samples only, so it can be played or written to a WAV without care for where the provider cut its stream.

import { sonioxTextToSpeech } from '@codefusion-cc/speech/soniox'
import { openRouterTextToSpeech } from '@codefusion-cc/speech/openrouter'
import type { TextToSpeech } from '@codefusion-cc/speech'

// Switching the provider is this line and nothing else.
const speech: TextToSpeech = env.SPEECH === 'soniox'
  ? sonioxTextToSpeech({ apiKey: env.SONIOX_KEY, language: 'pl' })
  : openRouterTextToSpeech({ apiKey: env.OPENROUTER_KEY, model: 'elevenlabs/eleven-flash-v2.5', voice: 'Rachel' })

const controller = new AbortController()
const utterance = speech.open({
  language: 'pl',
  signal: controller.signal, // aborting is the same as utterance.cancel()
  onAudio: ({ pcm, sampleRate }) => socket.send(pcm), // play at sampleRate
})

for await (const token of answerFromModel) utterance.write(token) // pieces of any size
utterance.end()

const result = await utterance.done // never rejects
if (result.ok) {
  console.log(result.status, result.spoken.firstAudioMs, result.spoken.audioSeconds, result.cost.usd)
} else {
  console.log(result.error.code, result.error.providerMessage) // 'rate_limited', 'voice_unknown', 'timeout', ...
}

// Barge-in: utterance.cancel() stops the provider, no further onAudio call happens, done settles as 'cancelled'.

Entry points

| Entry | For | Has | | --- | --- | --- | | . | Workers, servers, the page | The contract (TextToSpeech, Utterance, AudioChunk, SpeechResult, SpeechFailure, Voice, SpeechLimits), speakText, createSpeechSplitter / splitForSpeech, createSpeechCleaner / cleanForSpeech, createPcmAligner, pcm16ToWav, createUtterance (for writing an adapter) | | ./soniox | Workers, servers | sonioxTextToSpeech: one WebSocket per utterance, text streamed as it comes | | ./openrouter | Workers, servers | openRouterTextToSpeech: POST /audio/speech, one request per sentence, the model is a parameter | | ./testing | Apps' tests | createFakeTextToSpeech, createFakeOpenRouter, createFakeSoniox, tonePcm16, and everything from . |

Speech to text is not here yet; it will arrive as ./stt entries without changing these.

What a caller can rely on

• Failures are results. done resolves to { ok: false, error } with a code: unauthorized, payment_required, forbidden, rate_limited (with retryAfterMs when the provider said), timeout, unavailable, voice_unknown, text_refused, limit, closed, protocol, invalid, rejected. The provider's own status, code and message are kept in status, providerCode, providerMessage, with the key removed. Only a mistake in the app throws (a missing key or model, a limit that is not a positive integer).
• Cancel is final. cancel() (or aborting the signal) tells the provider to stop, sends no later request, makes no further onAudio call and settles done as cancelled. Before first audio, mid-stream and after the end all work; after the end it does nothing.
• Text arrives in any pieces. The same text written whole, by sentence or by character produces the same requests in the same order. Emoji are removed; text with nothing to say (empty, whitespace, symbols) is refused as text_refused when the utterance ends, without a request.
• Limits are parameters: maxCharacters (5000), firstAudioTimeoutMs (10000, from the first text written; for OpenRouter from the first request), idleTimeoutMs (15000, the wait for the caller's next write or end), stallTimeoutMs (10000, silence from the provider once the text has ended). While text is still being written, only cancel() (and OpenRouter's requestTimeoutMs) watch a provider that went quiet.
• What was spoken and what it cost: result.spoken has characters, requests, audio bytes and seconds, chunk count, firstAudioMs and the sample rate, also after a failure or a cancel. result.cost is { usd, basis }; OpenRouter also gives cost.later.resolve({ timeoutMs?, delayMs?, signal? }), which reads the provider's own figure from its generation record. The record took 9 to 12 seconds to appear in real calls, so it asks again, less and less often (from 1 s up to 5 s), until it is there or timeoutMs (30 s) has passed, and then returns an error result. Aborting signal ends it at once.
• voices(language?) lists { id, name, languages }. An unsupported language is an ok result with no voices.

Providers

Soniox keeps one WebSocket per utterance, so text goes out as the model writes it and the provider decides how to phrase it. Its price is per second of generated speech, so cost.usd is exact from spoken.audioSeconds. The socket opens with open(), so firstAudioMs includes the handshake. Soniox ends a stream that gets no text for a few seconds (request_timeout); the adapter reports it as timeout, and idleTimeoutMs should stay below that when text can stall. Three streams run at once per project and a stream lasts two minutes at most (limit).

OpenRouter takes whole texts, so the adapter cuts the streaming text into sentences (createSpeechSplitter) and sends one request per piece, in order, as the previous one finishes. A sentence needs the text after it to be known as ended, so a sentence end waits lookaheadMs (150) for the next word and is spoken anyway when the text stalls. The audio rate comes from the response's Content-Type, and a model that answers with anything but mono PCM fails as protocol. pricePerCharacterUsd turns the characters sent into an estimate, without those of a request the provider answered with an error status: it made no audio for them and bills none.

The splitter

import { splitForSpeech } from '@codefusion-cc/speech'

splitForSpeech('Dr Nowak zapłacił 3,5 zł za np. bilet. Było 22°C.', { maxLength: 400 })
// ['Dr Nowak zapłacił 3,5 zł za np. bilet. ', 'Było 22°C.']

Pieces joined equal the input, no piece is longer than maxLength, and a number is never cut from its unit nor an abbreviation from the word it introduces. A sentence longer than the limit is cut at a comma, else at a space.

Markdown

A model writes for a screen. createSpeechCleaner takes the answer as it streams and gives back what a voice should say, so it goes in front of the splitter or of write:

import { createSpeechCleaner } from '@codefusion-cc/speech'

const cleaner = createSpeechCleaner()
for await (const delta of answer) utterance.write(cleaner.push(delta))
utterance.write(cleaner.flush())
utterance.end()

| Written | Said | | --- | --- | | **bold**, *italic*, ~~struck~~, `code` | the words, without the marks | | [the docs](https://example.com/docs), ![a cat](cat.png) | the docs, a cat | | https://www.example.com/a/b?c=d | example.com | | # Heading, - item, 1. step | Heading., item., 1. step.: each ends like a sentence, so the voice pauses | | a table row | its cells with commas between them | | a fenced block of code, a rule (---) | nothing |

2 * 3, snake_case, ~5 and a < b stay as they are. A mark is said or dropped only once the text after it shows what it is, so a few characters may wait in pending; a bracket or a backtick that never closes is let go after 200 characters or at the end of its line. However the text is cut into parts, the result is that of cleanForSpeech on the whole text.

Tests

npx vitest run packages/speech runs the Node tests; *.workers.test.ts run in workerd (npm run test:workers). A real call to each provider is node packages/speech/scripts/real-calls.ts --vars <env file> --out .review/audio (costs a fraction of a cent, not part of CI).