@dannyyoo/korean-tts

v1.2.8

Published

a month ago

Korean phonology to IPA monophthong converter and audio utilities for Kokoro-82M TTS WASM

0High
0Medium
0Low

Korean Kokoro TTS — Phonology Engine & WASM Playground

A lightweight, zero-backend WebAssembly (WASM) and WebGPU speech synthesis engine and testing playground for running Kokoro-82M TTS on Korean sentences.

🎮 Live Interactive Demo: https://dyoo.github.io/korean-tts/

This package can be used as an npm library in your own web applications/PWAs, or run locally as an interactive playground.

Library Usage (npm)

1. Installation

npm install @dannyyoo/korean-tts kokoro-js

2. Safari & Web Worker Compatibility

ReadableStream Polyfill: In Safari/WebKit (macOS & iOS) and dedicated Web Worker environments, ReadableStream lacks native async iterator ([Symbol.asyncIterator]) support. Because underlying phonemizer dependencies decompress phonetic dictionaries at module evaluation, import and call polyfillReadableStreamAsyncIterator() at the top of your worker or main script:
```
import { polyfillReadableStreamAsyncIterator } from "@dannyyoo/korean-tts";
polyfillReadableStreamAsyncIterator();
```
Web Worker Threading: Safari's DedicatedWorkerGlobalScope does not allow nested worker spawning (new Worker() inside a Worker). When running Kokoro in a dedicated Web Worker, enforce single-threaded WASM execution before initializing:
```
import { env } from "@huggingface/transformers";
if (env.backends?.onnx?.wasm) {
  env.backends.onnx.wasm.numThreads = 1;
  env.backends.onnx.wasm.proxy = false;
}
```

3. High-Level `KoreanSpeaker` Example

KoreanSpeaker manages model downloading, caching, voice selection, Hangul-to-IPA phonology conversion, audio synthesis, structured cancellation, and offline cache maintenance in one unified interface:

import { KoreanSpeaker } from "@dannyyoo/korean-tts";

// 1. Initialize speaker instance
const speaker = new KoreanSpeaker({
  device: "wasm", // "wasm" or "webgpu"
  dtype: "q8",    // "q8" (~86MB), "fp32", "fp16", or "q4"
});

// 2. Load model with progress tracking (auto-cached in browser CacheStorage)
await speaker.load({
  progressCallback: (p) => {
    console.log(`Downloading: ${p.file} (${p.progress}%)`);
  },
});

// 3. Get supported voices (Japanese / Mandarin CJK voices tuned for syllable timing)
const voices = speaker.getVoices();
// [{ id: "jf_nezumi", name: "Nezumi", traits: "...", ... }, ...]

// 4. Synthesize speech from Korean text (or raw IPA) with cancelable task handle
const task = speaker.synthesize({
  text: "안녕하세요! 반갑습니다.",
  voice: "jf_nezumi", // Default: jf_nezumi
  speed: 1.0,
  onProgress: (ev) => {
    console.log(`Stage: ${ev.stage} (${Math.round(ev.progress * 100)}%)`);
  },
});

// Cancel anytime if needed:
// task.cancel("User navigated away");

const result = await task;

// Access metrics & outputs
console.log(`Generated in ${result.genTimeMs}ms (${result.rtf.toFixed(2)}x RTF)`);
console.log(`IPA Payload: ${result.ipa}`);

// 5. Playback or download WAV
const wavBlob = result.toWavBlob();
const audioUrl = result.toAudioUrl();

// 6. Direct one-line speak & play
await speaker.speak({ text: "오늘도 좋은 하루 되세요!" });

// 7. Inspect or clear offline storage (PWA ready)
const storage = await speaker.getStorageInfo();
console.log(`Storage: ${storage.modelSizeFormatted} (Offline Cached: ${storage.isCached})`);

// Delete cached model from disk and release RAM when user opts out
// await speaker.clearStorage();

API Reference (`korean-speaker`)

`KoreanSpeaker` Methods

| Method | Parameters | Returns | Description | | :--- | :--- | :--- | :--- | | constructor(options?) | SpeakerInitOptions? | KoreanSpeaker | Creates a new speaker instance with default or custom backend config. | | load(options?) | SpeakerInitOptions? | Promise<void> | Downloads and initializes the ONNX model into WASM or WebGPU. | | isLoaded() | — | boolean | Returns true if the model is initialized and ready for synthesis. | | getBackend() | — | { device, dtype, modelId } | Returns active hardware device, precision, and model repo. | | getVoices() | — | VoiceConfig[] | Returns all available voices with language, gender, and trait metadata. | | textToIpa(text) | koreanText: string | string | Converts Korean text into normalized, phonetically assimilated IPA. | | getVoiceVector(name) | voiceName: string | Promise<Float32Array> | Fetches and caches voice style embedding vector from CDN. | | preloadVoices(names) | voiceNames: string[] | Promise<void> | Preloads multiple voice style vectors into memory. | | synthesize(input) | SynthesisInput | SynthesisTask | Starts synthesis and returns a cancelable SynthesisTask handle. | | getActiveTasks() | — | SynthesisTask[] | Returns a snapshot of all active/in-flight synthesis tasks. | | cancelCurrent(reason?) | reason?: string | void | Cancels the most recently initiated synthesis task. | | cancelAll(reason?) | reason?: string | void | Cancels all active and in-flight synthesis tasks. | | speak(input) | SynthesisInput | Promise<{ result, audio }> | Synthesizes and immediately begins audio playback via HTMLAudioElement. | | getStorageInfo() | — | Promise<StorageInfo> | Inspects browser CacheStorage for offline model size and origin quota. | | clearStorage() | — | Promise<boolean> | Deletes model weights from CacheStorage and frees WebAssembly RAM. | | dispose() | — | void | Releases in-memory model instances, active tasks, and cached style vectors. |

Exported Types & Interfaces

SynthesisTask: Structured PromiseLike task handle supporting .cancel(reason?), .onProgress(cb), .stage, .isCancelled, and .isSettled.
SynthesisCancelledError: Error thrown when a task is aborted (err.name === 'AbortError', err.isCancelled === true).
SpeakerInitOptions: Configuration for model loading (modelId, dtype, device, progressCallback, requestPersistence).
SynthesisInput: Discriminated union { text: string; voice?: string; speed?: number; onProgress?: cb } | { ipa: string; voice?: string; speed?: number; onProgress?: cb }.
SynthesisResult: Synthesis outputs (audio: Float32Array, sampleRate, durationSec, genTimeMs, rtf, ipa, voice, speed, toWavBlob(), toAudioUrl(), createAudioElement()).
SpeakerProgress: Discriminated union of progress lifecycle events (SpeakerInitiateProgress, SpeakerDownloadProgress, SpeakerChunkProgress, SpeakerDoneProgress, SpeakerReadyProgress).
SpeakerProgressStatus: "initiate" | "download" | "progress" | "done" | "ready".
SpeakerProgressCallback: (progress: SpeakerProgress) => void.
StorageInfo: Offline cache inspection metrics (isCached, modelSizeBytes, modelSizeFormatted, totalUsageBytes, totalUsageFormatted, persisted).

Low-Level Phonology & Audio Utilities

You can also import individual building blocks:

import {
  koreanToIpa,
  koreanToPronunciation,
  decomposeHangul,
  composeHangul,
  numberToNativeKorean,
  createWavBlob,
  Visualizer,
} from "@dannyyoo/korean-tts";

// 1. Phonetic Hangul pronunciation according to Standard Korean rules (표준 발음법)
const pron = koreanToPronunciation("국밥"); // -> "국빱"
const thankYou = koreanToPronunciation("감사합니다"); // -> "감사함니다"
const liaison = koreanToPronunciation("한국어"); // -> "한구거"

// 2. Phonetic Hangul-to-IPA transcription with assimilation rules
const ipa = koreanToIpa("감사합니다"); // -> "kamsahamnita"

// 3. Native Korean number conversion (순우리말 수사)
const count = numberToNativeKorean(20, true); // -> "스무" (e.g. "스무 살")

// 4. Convert raw Float32Array PCM samples to WAV Blob
const wavBlob = createWavBlob(float32Array, 24000);

Korean G2P & IPA Algorithm Architecture

The speech synthesis pipeline converts raw Korean text into phonetically transcribed, Kokoro-compatible IPA monophthongs through a 4-stage pipeline:

┌─────────────────────────┐
│     Raw Korean Text     │  "국밥 2개 주세요! 50% 할인되나요?"
└────────────┬────────────┘
             │ 1. Normalization & Tokenization
┌────────────▼────────────┐
│   Normalized Hangul     │  "국밥 두 개 주세요! 오십 퍼센트 할인되나요?"
└────────────┬────────────┘
             │ 2. Unicode Jamo Decomposition (초성 / 중성 / 종성)
┌────────────▼────────────┐
│   Decomposed Syllables  │  [{ᄀ, ᅮ, ᆨ}, {ᄇ, ᅡ, ᆸ}, ...]
└────────────┬────────────┘
             │ 3. Multi-Pass Phonology Engine (표준 발음법)
┌────────────▼────────────┐
│ Phonetic Hangul Pron.   │  "국빱 두 개 주세요! 오십 퍼센트 할인되나요?"
└────────────┬────────────┘
             │ 4. Allophonic Kokoro IPA Transcription
┌────────────▼────────────┐
│       Output IPA        │  "kuk̚p͈ap̚ tu ɡe ʨusejo! oɕip̚ pʰʌsɛntʰɯ haɾindwenajo?"
└─────────────────────────┘

Stage 1: Text & Number Normalization (`normalizeKoreanText`)

Raw inputs often contain digits, currency, dates, percentages, and acronyms that must be converted to spoken Korean words before phonetic transcription:

Native Korean Counting Units (순우리말 수사):
- Matches numbers 1–99 before counting classifiers (개, 명, 살, 마리, 잔, 권, 장, 번, etc.) and transforms them into pure Korean attributive forms:
  - 1개 → 한 개, 2명 → 두 명, 3살 → 세 살, 4마리 → 네 마리, 20살 → 스무 살, 21명 → 스물한 명.
Clock Times & Hours:
- Hours use Native Korean, while minutes and seconds use Sino-Korean: 3시 30분 → 세시 삼십분, 12시 5분 → 열두시 오분.
Decimals, Percentages & Phone Numbers:
- Decimals: 3.14 → 삼 점 일사, 0.5 → 영 점 오
- Percentages: 99.9% → 구십구 점 구 퍼센트
- Phone numbers: 010-1234-5678 → 공일공 일이삼사 오육칠팔
- Ordinals: 1번째 → 첫 번째, 2번째 → 두 번째, 3번째 → 세 번째
Sino-Korean Currency & Dates:
- 24,500원 → 이만 사천오백원, 2026년 8월 15일 → 이천이십육년 팔월 십오일.
English Acronyms & Letters:
- AI 모델 → 에이아이 모델, TTS → 티티에스, OK → 오케이.
Standalone Jamo Normalization (단독 자모 발음):
- Standalone Vowels: Mapped to canonical zero-onset syllables (ㅗ → 오 [o], ㅏ → 아 [a], ㅜ → 우 [u], ㅣ → 이 [i], ㅐ → 애 [ɛ]).
- Standalone Consonants: Vocalized with phonetic base vowel ㅡ (ㄱ → 그 [kɯ], ㄴ → 느 [nɯ], ㄷ → 드 [tɯ], ㅅ → 스 [sɯ], ㅋ → 크 [kʰɯ], ㄲ → 끄 [k͈ɯ], ㅇ → 응 [ɯŋ]).

Stage 2: Syllabic Jamo Decomposition (`decomposeHangul`)

Hangul syllables in the Unicode range 0xAC00–0xD7A3 (and standalone compatibility Jamos 0x3131–0x318E / 0x1100–0x11FF) are decomposed arithmetically into their 19 Initial Consonants (초성), 21 Vowels (중성), and 28 Final Codas (종성):

$$\text{offset} = \text{charCode} - \text{0xAC00}$$ $$\text{choIdx} = \lfloor \text{offset} / 588 \rfloor, \quad \text{jungIdx} = \lfloor (\text{offset} / 28) \bmod 21 \rfloor, \quad \text{jongIdx} = \text{offset} \bmod 28$$

Stage 3: Multi-Pass Phonological Transformation (`applyPhonologicalRules`)

Applies the official Standard Korean Pronunciation Rules (국립국어원 표준 발음법) across syllable boundaries:

Palatalization (구개음화 — 제17항):
- ㄷ, ㅌ, ㄾ before ㅣ or j-glides become ㅈ, ㅊ:
- 굳이 → [구지], 같이 → [가치], 핥이다 → [할치다], 닫히다 → [다치다].
Aspiration & ㅎ-Elision (격음화 및 ㅎ 탈락 — 제12항):
- Obstruent + ㅎ or ㅎ + obstruent fuse into aspirated consonants (ㅋ, ㅌ, ㅍ, ㅊ): 축하 → [추카], 좋다 → [조타], 맞히다 → [마치다].
- ㅎ between vowels/sonorants drops: 좋아 → [조아], 많이 → [마니], 싫어 → [시러].
Liaison (연음법칙 — 제13항, 제14항):
- Single and compound codas move to empty onset (ㅇ) of the following syllable: 한국어 → [한구거], 값이 → [갑씨], 닭을 → [달글], 삶이 → [살미].
Liquid Lateralization & Nasalization (유음화 및 ㄹ의 비음화 — 제19항, 제20항):
- ㄴ + ㄹ and ㄹ + ㄴ become lateral geminate ㄹㄹ: 신라 → [실라], 난로 → [날로], 설날 → [설랄].
- ㅁ, ㅇ + ㄹ → ㅁ, ㅇ + ㄴ: 종로 → [종노], 대통령 → [대통녕], 침략 → [침냑].
- ㄱ, ㅂ + ㄹ → ㅇ, ㅁ + ㄴ (Mutual assimilation): 국립 → [궁닙], 독립 → [동닙], 협력 → [혐녁].
Nasalization (비음화 — 제18항):
- Stops (ㄱ, ㄷ, ㅂ) before nasals (ㄴ, ㅁ) become nasals (ㅇ, ㄴ, ㅁ): 국물 → [궁물], 감사합니다 → [감사함니다], 있는 → [인는].
Tensification / Glottalization (경음화 / 된소리되기 — 제23항~제26항):
- Post-Obstruent (제23항): 국밥 → [국빱], 학교 → [학꾜], 있다 → [읻따], 잡지 → [잡찌].
- Special ㄺ + ㄱ (제25항): 맑게 → [말께], 읽고 → [일꼬].
- Predicate Stems ending in ㄴ, ㅁ (제24항): 신다 → [신따], 앉다 → [안따], 젊다 → [점따], 삼다 → [삼따].
- Sino-Korean ㄹ Coda (제26항): Hanja roots ending in ㄹ tensify subsequent ㄷ, ㅅ, ㅈ: 갈등 → [갈뜽], 발전 → [발쩐], 물질 → [물찔], 실수 → [실쑤], 활동 → [활똥], 열정 → [열쩡].
Coda Neutralization (자음군 단순화 & 음절 끝소리 규칙 — 제8항~제11항):
- Final codas in isolation or before consonants reduce to the 7 stop archetypes (ㄱ, ㄴ, ㄷ, ㄹ, ㅁ, ㅂ, ㅇ): 닭 → [닥], 값 → [갑], 삶 → [삼], 여덟 → [여덜], 꽃 → [꼳].

Stage 4: Allophonic Kokoro-Targeted IPA Transcription (`convertKoreanToSpeechText`)

Converts the assimilated syllable tokens into accurate International Phonetic Alphabet (IPA) representations optimized for Kokoro-82M CJK acoustic models:

Alveolo-palatalization ([ɕ, ɕ͈]):
- ㅅ, ㅆ preceding /i/ or /j/ glides are transcribed as alveolo-palatal [ɕ, ɕ͈]:
  - 시간 → ɕiɡan (instead of sikan)
  - 신라 → ɕilla
  - 시작 → ɕiʥak̚
  - 씨앗 → ɕ͈iat̚
Intervocalic & Post-Sonorant Voicing ([ɡ, d, b, ʥ]):
- Plain stops and affricates (ㄱ, ㄷ, ㅂ, ㅈ) become voiced between sonorants (vowels and ㄴ, ㄹ, ㅁ, ㅇ):
  - 아버지 → abʌʥi
  - 친구 → tʃʰinɡu
  - 한국어 → hanɡuɡʌ
  - 감사합니다 → kamsahamnida
Lateral Gemination ([ll]):
- Consecutive ㄹ sounds are represented as true alveolar lateral geminates ([ll]):
  - 설날 → sʌllal
  - 빨리 → p͈alli
Unreleased Stop Codas ([k̚, t̚, p̚]):
- Syllable-final stops are marked as unreleased: 국밥 → kuk̚p͈ap̚.
Vowel Hiatus & Zero-Onset Boundaries ([ˌ]):
- Open syllables ending in a vowel followed by a zero-onset syllable (ㅇ) receive a secondary stress syllable foot marker ˌ (Token ID 161) to create a crisp, micro-beat syllable transition while preventing diphthong collapse:
  - 내일 → nɛˌil
  - 오이 → oˌi
  - 아이 → aˌi
  - 좋은 → ʨoˌɯn
Expressive Question Intonation ([↗?]):
- Question sentences ending in ? are augmented with Kokoro's native rising pitch contour operator (↗ — Token ID 175), sweeping fundamental frequency ($F_0$) upward on the final syllable for authentic spoken Korean interrogative delivery:
  - 이거 뭐예요? → iɡʌ mwʌˌjeˌjo↗?
  - 밥 먹었어? → pap̚ mʌɡʌs͈ʌ↗?
Single-Word / Isolated Syllable Duration Closure ([.]):
- Unpunctuated isolated vocabulary items receive declarative sentence-final boundary closure (.) prior to tokenization. This prevents neural duration predictors from treating single syllables as unfinished floating phrases, ensuring crisp $\sim 200\text{ms}$ pronunciations rather than drawn-out, drone-like vowels:
  - 넋 → nʌk̚.
  - 가 → ka.
  - 물 → mul.

Kokoro-82M 115-Token Phoneme Architecture & IPA Compatibility

Kokoro-82M is a neural Text-to-Speech model with an internal 115-token phoneme vocabulary (comprising ASCII letters, selected IPA extensions, punctuation, and Japanese/Chinese phonetic tokens). Unlike standard NLP tokenizers with thousands of subwords, Kokoro processes text strictly at the phoneme level.

Characters not present in Kokoro's 115-token vocabulary are silently dropped by the tokenizer. Understanding this mapping is essential for natural Korean synthesis.

Complete 115-Token Vocabulary Breakdown

Kokoro's vocabulary consists of 115 valid tokens indexed across ID 0 to 177:

Korean Hangul to Kokoro IPA Phoneme Mapping

Key Phonetic Gaps & Solutions

1. The Intervocalic Affricate Gap (`d͡ʑ` $\rightarrow$ `d` Regression)

The Problem: In standard linguistic literature, the voiced intervocalic allophone of ㅈ is written as [d͡ʑ]. However, neither the tie bar \u0361 (͡) nor the curly-tail z \u0291 (ʑ) exists in Kokoro's 115-token vocabulary. When d͡ʑ was passed to the tokenizer, it silently stripped ͡ and ʑ, leaving only d (alveolar plosive /d/).
Result:
- 휴지 (hyu-ji) became ['h', 'j', 'u', 'd', 'i'] $\rightarrow$ synthesized as "휴디" (hyudi).
- 타조 (ta-jo) became ['t', 'ʰ', 'a', 'd', 'o'] $\rightarrow$ synthesized as "타도" (tado).
- 된장 (doen-jang) became ['t', 'w', 'e', 'n', 'd', 'a', 'ŋ'] $\rightarrow$ synthesized as "된당" (doendang).
- 아버지 (a-beo-ji) became ['a', 'b', 'ʌ', 'd', 'i'] $\rightarrow$ synthesized as "아버디" (abeodi).
The Solution: Kokoro contains the dedicated CJK voiced alveolo-palatal affricate token ʥ (Token 19) (the same token used by Misaki for Japanese ジ and voiced affricates). Mapping voiced ㅈ to ʥ produces natural affricate voicing: hjuʥi, tʰaʥo, twenʥaŋ, abʌʥi.

2. The Aspirated Affricate Tokenization (`ㅊ` $\rightarrow$ `tʃʰ`)

The Problem: When ㅊ was previously mapped to alveolo-palatal ʨʰ (\u02A8 + \u02B0), the token sequence [ʨ, ʰ] represented an out-of-distribution phoneme combination for Kokoro (as ʨ in Kokoro was trained on unaspirated Mandarin/Japanese data). In multi-syllable reduplications like 차차 (ʨʰaʨʰa), the neural acoustic model split the phonemes into a high-palatal glide and disconnected breath puff, mispronouncing it as "ye ha chul" and severely stretching the initial syllable (4.67s).
The Solution: Mapping ㅊ to tʃʰ (Tokens 62 + 131 + 162) maps to Kokoro's native aspirated postalveolar affricate representation (as generated by eSpeak for "ch"). This produces crisp, natural articulation across initial and medial syllables (차 → tʃʰa, 차차 → tʃʰatʃʰa, 친구 → tʃʰinɡu, 초코 → tʃʰokʰo), reducing syllable duration to a natural ~1.8s.

3. Tension / Glottalization Diacritic Gap (`\u0348` / `͈`)

The Problem: The IPA tension mark \u0348 (͈) is not in Kokoro's vocabulary.
Acoustic Behavior: When k͈a (까) or s͈a (싸) is passed, the tokenizer strips ͈ and feeds k / s. In Kokoro, Asian voice models (e.g. zf_xiaobei, jf_nezumi) naturally articulate unvoiced initial stops k, t, p with high vocal tract tension compared to intervocalic voiced stops ɡ, d, b, ʥ.

4. Unreleased Coda Diacritic Gap (`\u031a` / `̚`)

The Problem: The IPA unreleased stop mark \u031a (̚ as in k̚, t̚, p̚) is not in Kokoro's vocabulary.
Acoustic Behavior: The tokenizer strips ̚ and tokens become k, t, p. Because these tokens reside in syllable coda position before a boundary or subsequent onset, Kokoro's acoustic model naturally decays them without release bursts.

5. Vowel Hiatus & Zero-Onset Syllable Transition (`내일`, `아이`, `오이`)

The Problem: When a vowel-final syllable is followed by an ㅇ-onset syllable (e.g. 내일 $\rightarrow$ 내 + 일 $\rightarrow$ nɛil), direct concatenation of ɛ + i without boundary markers causes multilingual acoustic models to fuse the adjacent vowels into a single English-like diphthong (e.g. pronouncing 내일 as the 1-syllable English word "nail" /neɪl/). Full punctuation marks like . introduce an unnaturally long sentence-level pause (~250ms).
The Solution: The engine automatically detects vowel hiatus across zero-onset boundaries (prev.jongIdx === 0 && s.choIdx === 11) and inserts a secondary stress syllable foot marker ˌ (Token ID 161 / \u02CC). This creates a crisp, natural 2-syllable beat without dead silence:
- 내일 → nɛˌil (0.82s vs 2.08s with .)
- 오이 → oˌi
- 아이 → aˌi
- 좋은 → ʨoˌɯn

Unit Testing & Verification

The engine is covered by 227 automated unit tests across 22 suites:

npm run test
# or directly with Node:
node --experimental-strip-types --test test/**/*.test.ts

Running the Interactive Demo Playground

# 1. Install dependencies
npm install

# 2. Run unit tests
npm run test

# 3. Start dev server
npm run dev
# Open http://localhost:5173

# 4. Build library package (ESM + CJS + .d.ts)
npm run build:lib

# 5. Build demo web app
npm run build:demo

References & Standards

Kokoro-82M (hexgrad / Hugging Face): Open-weight 82M parameter multi-lingual neural TTS model.
Kokoro-82M-v1.0-ONNX (onnx-community): WebAssembly / WebGPU ONNX weights for in-browser client-side inference.
국립국어원 표준어 규정 — 제2부 표준 발음법 (Wikisource Full Text): Complete official statutory text of the Standard Korean Pronunciation Rules (Articles 1–30, covering liaison, palatalization, aspiration, nasalization, liquid assimilation, tensification, and coda reduction).
National Institute of Korean Language Portal (국립국어원 한국어 어문 규범): Official NIKL regulations and commentary.
Wikipedia: Korean Phonology: Comprehensive linguistic overview of the sound system of modern Korean.

Development & Attribution

This project and its Korean phonology G2P engine were designed and pair-programmed by Danny Yoo in collaboration with Antigravity (Google DeepMind).

License

This project is licensed under the MIT License. See LICENSE for details.

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme