@andrewhoyle/voice (TypeScript SDK)

Typed fetch-based client for the Imperium Voice gateway, plus a React voice picker and a MindTale compatibility shim. Runs anywhere with a global fetch — Node ≥18, edge runtimes, Deno, browsers.

The full server-side REST contract for consuming apps (endpoints, auth, error codes, learning-loop feedback) ships with this package: GATEWAY-INTEGRATION.md.

npm install @andrewhoyle/voice

Quick start

import { VoiceClient, QuotaError } from "@andrewhoyle/voice";

const voice = new VoiceClient({ baseUrl: "https://voice.imperium.example", apiKey });

// One clause, content-addressed + cached (live voice-switching):
const clip = await voice.segment(
  "Once upon a time, in a kingdom by the sea.",
  { provider: "elevenlabs", elevenlabs_voice_id: "pNInz6obpgDQGcFmaJgB" },
  { contentType: "story" },
);
console.log(clip.audioUrl, clip.durationMs, clip.cached);

// A whole lesson:
const lesson = await voice.synthesize("black-holes-101", [
  { title: "Intro", text: "Welcome." },
  { text: "Black holes bend light." },
]);

Every non-2xx response throws a typed error (all extend VoiceError): AuthError (401), PolicyError (403), BadRequestError (400/422), QuotaError (429), StorageUnavailableError (503), ServerError (5xx), and TransportError (no response). ApiError carries .status and .detail.

try {
  await voice.segment(text, { provider: "elevenlabs" });
} catch (e) {
  if (e instanceof QuotaError) { /* back off / upgrade */ }
}

React voice picker

import { useState } from "react";
import { VoicePicker, DEFAULT_VOICES, type VoiceOption } from "@andrewhoyle/voice/react";

function Narrator() {
  const [voice, setVoice] = useState<VoiceOption>(DEFAULT_VOICES[0]!);
  return (
    <>
      <VoicePicker value={voice.id} onChange={setVoice} />
      <button onClick={() => voice && client.segment(text, voice.voiceConfig)}>Play</button>
    </>
  );
}

VoicePicker is data-driven: pass your own voices prop (from your catalogue) or use the bundled DEFAULT_VOICES. onChange hands back the full option — read option.voiceConfig and pass it straight to a client call. React is an optional peer dependency, isolated to this subpath so non-React consumers don't pull it in.

Voice clones

Create a clone (needs the clones:write scope and a consent reference), then voice it by clone_id on a Chatterbox config — the gateway resolves the stored sample server-side, so the raw URL never travels on the synthesis request:

const clone = await voice.createClone({ name: "Narrator", sampleUrl, consentRef: "consent-2026-01" });
await voice.segment("Read this in my voice.", {
  provider: "chatterbox",
  voice_name: "chatterbox_default",
  clone_id: clone.id,
});

MindTale migration

The gateway's response shape is identical to the MindTale Lesson row, so the compat shim lets existing call sites switch transport with minimal change:

import { VoiceClient } from "@andrewhoyle/voice";
import { createMindTaleEngine } from "@andrewhoyle/voice/mindtale";

const engine = createMindTaleEngine(new VoiceClient({ baseUrl, apiKey }));
const audio = await engine.generateLessonAudio({ slug, sections, voiceConfig });
// audio.audioUrl / audio.durationSeconds / audio.sectionTimestamps — unchanged

The shim defaults textFormat to "raw" (MindTale fed unprocessed prose and let the worker normalise), unlike the base client's "normalized" default.

Speech-to-text

// Sync transcription — upload a Blob/bytes, or pass { url }:
const t = await voice.transcribe(fileBlob, { diarize: true, wordTimestamps: true });
console.log(t.text, t.language, t.speakerCount);

// Long audio → async job (+ optional signed webhook):
const job = await voice.createTranscriptionJob("https://files/meeting.mp3", {
  diarize: true, webhookUrl: "https://me/cb", idempotencyKey: "meeting-42",
});
const done = await voice.getTranscriptionJob(job.id);

// Real-time (browser) — the key rides the apikey.<KEY> subprotocol, never the URL:
const session = voice.transcribeStream(
  { onTranscript: (m) => console.log(m.type, m.transcript) },
  { config: { sample_rate: 16000, diarize: true } },
);
session.sendAudio(pcmChunk);
session.finish();

Streaming TTS + discovery

for await (const chunk of await voice.synthesizeStream("Once upon a time.")) {
  player.feed(chunk);              // low time-to-first-audio
}
const voices = await voice.listVoices();   // VoiceInfo[] (each with a ready voice_config)
const models = await voice.listModels();   // ModelInfo[] (features + cost_per_minute)

The React picker can load entitled voices live: const { voices } = useVoices(client).

Live transcription — listen() (device-first)

@andrewhoyle/voice/universal adds live speech-to-text that is device-first: when the browser has Web Speech it uses the browser-native recogniser — free, low-latency, interim results, and with no round-trip through our gateway. Only when Web Speech is unavailable does it fall back to the gateway (over a short-lived token — the long-lived key never enters the page). It's a drop-in replacement for hand-rolled webkitSpeechRecognition, with the gateway fallback as a strict upgrade over a typing-only fallback.

Residency note — Web Speech is not guaranteed on-device. The browser controls where recognition happens: some browsers process locally, but Chrome streams audio to Google and Safari to Apple. So listen()'s device tier is "no transfer through our infra and no change versus the webkitSpeechRecognition an app already hand-rolls" — not an offline guarantee. If you need a hard on-device/UK-residency guarantee, that comes from the gateway tier pointed at a UK/EU STT provider, not from Web Speech. Confirm behaviour per target browser before making an "offline" claim to users.

import { Voice } from "@andrewhoyle/voice/universal";

const voice = new Voice({ client /* optional: enables the gateway fallback */ });
const session = await voice.listen({
  lang: "en-GB",
  onPartial: (t) => setDraft(t),     // interim
  onFinal: (t) => setValue(t),       // committed
});
console.log(session.tier);           // "device" on Chrome/Edge; "gateway" otherwise
// …
session.stop();                      // graceful (deliver final, then end)

React:

import { useListen } from "@andrewhoyle/voice/react";

const { listening, partial, final, start, stop } = useListen(voice, { lang: "en-GB" });
<button onMouseDown={start} onMouseUp={stop}>{listening ? "…" : "🎤"}</button>
<input value={final || partial} />

Force a tier with preferTier: "device" | "gateway" (default is device-first). The gateway fallback needs the gateway to allow your origin (CORS_ALLOWED_ORIGINS) and the tenant to hold the transcribe:stream scope.

The gateway tier captures mic audio as anti-aliased 16 kHz PCM16 (a Kaiser windowed-sinc resampler runs in an AudioWorklet — design record: ADR-0019 in the platform repo) in fixed 20 ms frames, and stop() drains the buffered tail before finishing so the last word isn't clipped. Bundler note: the worklet embeds SDK classes via Function.toString(), so do not enable property mangling (e.g. terser mangle.properties) on @andrewhoyle/voice — plain minification/renaming is fine.

Develop

pnpm install
pnpm typecheck   # tsc --noEmit
pnpm test        # vitest
pnpm build       # emit dist/

Streaming sessions: reconnection & keepalive

Live WS sessions (transcribeStream, converse) are stateful — audio position and transcript context live server-side — so reconnection is your responsibility: on onClose/onError, open a fresh session and resume from your own application state (they cannot be resumed transparently). sendAudio is safe to call immediately: frames sent while the socket is still connecting queue and flush in order once it opens. For long idle gaps (e.g. push-to-talk pauses), keep the session warm by continuing to stream silence frames, or send a small audio frame periodically as a keepalive — idle sockets are subject to intermediary timeouts.