@yujinapp/nac3-kikoe

Local, speaker-dependent voice-command recognition for atypical speech.

nac3-kikoe recognizes a small set of personal voice commands by template matching ("mathematical proximity"). The user enrolls a few samples per command; the package extracts a numeric fingerprint (a sequence of MFCC feature vectors) from each sample and stores only those numbers. At runtime an incoming utterance is compared against the stored templates with a length-normalized DTW (Dynamic Time Warping) distance and a rejection threshold. Because every speaker enrolls their own samples, the engine adapts to non-standard pronunciation without any cloud model.

It is decoupled: the host app supplies storage and (optionally) an external recognizer. Yuemail is adopter number one.

What it is (and is not)

• It is a tiny, dependency-free MFCC + DTW matcher tuned for a handful of per-user commands.
• It is not a general speech-to-text system. For free-form dictation, pair it with the host's own recognizer (see Layer A).

The three layers

• Layer A - Vocabulary bias. buildBiasVocabulary(commands, contacts) produces a deduplicated, normalized word list that the host feeds to its RecognizerAdapter.setBiasVocabulary(...). This nudges an external recognizer toward known commands and contact names.
• Layer B - Correction memory. CorrectionMemory records "heard -> meant" pairs (case/whitespace-insensitive) so repeated misrecognitions get fixed deterministically.
• Layer C - Project Euphonia / personalized Google enrollment. OUT OF SCOPE for this package. If the host integrates Google's personalized speech models, that enrollment is the host's responsibility.

Privacy guarantee

Raw audio (PCM) is consumed transiently during feature extraction and is then discarded. Only numeric fingerprints ({ frames: number[][] }) and command metadata are ever handed to the StorageAdapter. The test suite asserts that no object passed to storage.save(...) contains a pcm key anywhere.

Install / build

npm install
npm run typecheck
npm run test
npm run build

Requires Node 22+. ESM only. Zero runtime dependencies.

Wiring StorageAdapter + RecognizerAdapter

import {
  Trainer,
  Matcher,
  buildBiasVocabulary,
  type StorageAdapter,
  type RecognizerAdapter,
  type CommandTemplate,
} from "@yujinapp/nac3-kikoe";

// 1. Persistence: the host decides where templates live (file, DB, etc.).
const storage: StorageAdapter = {
  async load() {
    return loadTemplatesFromDisk(); // returns CommandTemplate[]
  },
  async save(templates: CommandTemplate[]) {
    await saveTemplatesToDisk(templates); // numeric only, no audio
  },
};

// 2. Optional external recognizer (bring your own voice provider).
const recognizer: RecognizerAdapter = {
  setBiasVocabulary(words: string[]) {
    myCloudRecognizer.setHints(words);
  },
};

Quickstart

import { Trainer, Matcher, InMemoryStorage } from "@yujinapp/nac3-kikoe";

const storage = new InMemoryStorage();
const trainer = new Trainer(storage, {
  matcher: new Matcher({ threshold: 8, margin: 0.5 }),
});

// Enroll a few samples per command (PCM is discarded after extraction).
await trainer.enroll("leer bandeja", [
  { pcm: sample1, sampleRate: 16000 },
  { pcm: sample2, sampleRate: 16000 },
  { pcm: sample3, sampleRate: 16000 },
]);

// Recalibrate just this command, or all commands when omitted.
await trainer.train("leer bandeja");

// Recognize a new utterance.
const result = await trainer.recognize({ pcm: liveAudio, sampleRate: 16000 });
if (result.accepted) {
  console.log("command:", result.command, "distance:", result.distance);
} else {
  console.log("rejected; ranked:", result.ranked);
}

Metrics + router (v0.2.0)

Each command earns two numbers that mature with real use, both smoothed so a fresh command starts "green from inexperience" at 0.5:

• confidence - of the times the local engine fired this command, how often did the person let it stand (vs. correct/cancel it)? The source of truth is the user. This is the command's reputation. Moved by observeOutcome.
• effectiveness - of the measured utterances, how often did the local lane agree with the cloud path (Google STT + brain)? The cloud is a second opinion, not a supreme judge: for severe atypical speech the cloud is the one that fails, so a low effectiveness validates the local lane rather than condemning it. Moved by observeCloud.

The router turns those numbers + a mode (the cost "perilla") into a verdict:

| Mode (perilla) | Confident local hit | Doubtful / untrained | | --------------- | ------------------------------ | -------------------- | | on_doubt (a) | fire local, no cloud | escalate to cloud | | learning (b) | fire local and shadow-measure vs cloud until the command graduates, then behaves like on_doubt | escalate to cloud | | always (c) | fire local and always measure vs cloud | escalate to cloud |

The router never executes anything: it returns { preferLocal, runCloud } and the host app decides. A host with zero enrolled commands always routes to the cloud, so the trainer is invisible until the person trains it.

import { KikoeEngine, InMemoryKikoeStorage } from "@yujinapp/nac3-kikoe";

const engine = new KikoeEngine(new InMemoryKikoeStorage("learning"), {
  now: () => Date.now(),            // clock seam (the package never calls Date.now)
  router: { minConfidence: 0.6 },
});

await engine.enroll("leer bandeja", [{ pcm, sampleRate: 16000 }]);
const { match, route } = await engine.recognize({ pcm: live, sampleRate: 16000 });
if (route.preferLocal) runCommand(match.command);
if (route.runCloud)    { const cloud = await cloudResolve(live);
                         await engine.observeCloud(match.command!, cloud === match.command); }
// later, when the user accepts or corrects:
await engine.observeOutcome("leer bandeja", true);
const rows = await engine.listMetrics(); // -> CommandMetric[] for the UI table

Browser hosts extract features client-side (extractFeatures) and call enrollFingerprints / recognizeFeatures, so the audio never leaves the device at all.

NAC3 verbs

See nac3.manifest.json. The exposed verbs are:

| Verb | Policy | Side effects | | ------------------------------------- | ------------ | ------------ | | yujin.voicetrainer.enroll | confirm | filesystem | | yujin.voicetrainer.train | confirm | filesystem | | yujin.voicetrainer.recognize | unrestricted | - | | yujin.voicetrainer.list | unrestricted | - | | yujin.voicetrainer.forget | destructive | filesystem | | yujin.voicetrainer.set-vocabulary | confirm | - | | yujin.voicetrainer.correct | confirm | filesystem | | yujin.voicetrainer.metrics | unrestricted | - | | yujin.voicetrainer.observe-outcome | unrestricted | filesystem | | yujin.voicetrainer.observe-cloud | unrestricted | filesystem | | yujin.voicetrainer.set-mode | confirm | filesystem |

The host application must wire StorageAdapter/KikoeStorageAdapter + (optionally) a RecognizerAdapter; raw audio is never persisted.