echo-voice

Client-side voice for Echo VoiceFlow: the caller owns the turn. Silero VAD (with an automatic energy fallback), push-to-talk / auto / hybrid turn control, local barge-in, reply waveform, and mute — a small package your integrators just install. Vanilla core + a React hook.

Built for the GrayPorter/Porter feedback: "nothing but the shopper should decide when their turn is over." This package makes the client end the turn (record- and-send or client VAD), and the Echo server buffers audio and waits for its commit — see endpointing below.

Install

npm install echo-voice          # onnxruntime-web + react are optional peers
# or, no build step:
<script src="https://cdn.jsdelivr.net/npm/echo-voice/dist/echo-voice.iife.js"></script>

Quick start (vanilla)

import { EchoVoice } from "echo-voice";

const voice = EchoVoice.connect({
  workflow: "ai-guru-voice",
  apiKey: "vfk_…",              // or ticket: "…"
  turn: "hybrid",               // "auto" | "manual" | "hybrid"
  endpointing: "client",        // this package owns the turn (default)
  vad: { engine: "auto", silenceMs: 700, minSpeechMs: 250 },
  metadata: { user_id: "u_42" },
});

voice.on("transcript", ({ text, final }) => …);
voice.on("reply",      ({ text, delta, final }) => …);
voice.on("waveform",   ({ bars, rms }) => drawEqualizer(bars));  // AI reply audio
voice.on("listening",  ({ on }) => …);   // local VAD: user speaking
voice.on("speaking",   ({ on }) => …);   // AI speaking

await voice.start();            // opens mic + loads the VAD

// push-to-talk (manual/hybrid):
btn.onmousedown = () => voice.startTalking();
btn.onmouseup   = () => voice.stopTalking();   // ends + commits the turn
voice.send();                   // hybrid: force-send now
voice.mute(); voice.unmute();

React

import { useEchoVoice } from "echo-voice/react";

function Mic() {
  const v = useEchoVoice({ workflow: "ai-guru-voice", ticket, turn: "hybrid" });
  return (
    <>
      <button onPointerDown={v.startTalking} onPointerUp={v.stopTalking}>Talk</button>
      <p>{v.transcript}</p><p>{v.reply}</p>
      <Equalizer bars={v.waveform} muted={v.muted} onMute={() => v.setMuted(!v.muted)} />
    </>
  );
}

Turn modes

• manual — push-to-talk / record-and-send. startTalking() … stopTalking(). Nothing but the user ends the turn. VAD is used only for barge-in.
• auto — client VAD ends the turn after silenceMs of silence. Utterances shorter than minSpeechMs are dropped as noise (no empty turns).
• hybrid — auto VAD plus a manual send() override, so a mid-sentence pause never fires but the user can force-send.

Local barge-in (default on): the moment the user starts talking over the AI, the package cancels the turn + ducks playback locally — faster than a server hop.

Switch live, without reconnecting — a "hands-free" toggle is one call:

voice.setTurnMode("manual");  // pure push-to-talk: the VAD never opens a turn
voice.setTurnMode("auto");    // hands-free
voice.on("turnmode", ({ mode }) => …);

In manual the VAD is ignored for turn-taking entirely, so ambient noise (or the agent's own audio on speakers) can't flip the client into listening.

Signals — what the visitor is doing

Context is what the visitor has; a signal is what they are doing — dwelling, a discount firing, abandoning a cart. Signals are not conversation: the server stores them apart from the transcript, so telemetry never lands in history as something the visitor said.

// ambient (default) — state only, never speaks on its own
voice.sendSignal("page_view", { page: "/p/x100vi" });

// trigger — may make the agent speak FIRST
voice.sendSignal("dwell", { seconds: 20 }, { kind: "trigger" });
voice.sendSignal("discount", { percent: 10 }, { kind: "trigger", force: true });

// spoke | cooldown | busy | max | disabled | recorded
voice.on("signal", ({ name, outcome }) => …);

Your site decides when to fire; the flow decides how often a trigger may be acted on (proactive, proactive_cooldown_s, proactive_max, and per-signal instructions in events_json on the LLM node). Hard rails are not configurable: never over a running turn, never while muted, never after the call ended. Payloads are treated strictly as data — a signal can never instruct the model.

VAD

vad.engine: "auto" (default) loads Silero (ONNX via onnxruntime-web, from a CDN) and automatically falls back to a tiny energy gate if the model/wasm can't load. Force one with "silero" / "energy". Tune: silenceMs, minSpeechMs, preRollMs, positiveThreshold/negativeThreshold (Silero) or energyThreshold (energy), sileroModelUrl, ortBaseUrl.

For the smallest bundle, keep onnxruntime-web out of your app and let the CDN build load it lazily — the core stays tiny; Silero downloads on first start().

Whispered speech

Silero and RNNoise both key on voicing — the glottal pulse. A whisper has none, so it reads as shaped turbulent noise and scores ~0.2-0.4 where modal speech scores >0.9. No threshold setting fixes this: lowering the voiced gate far enough to admit a whisper also admits every fan in the room, because the number being lowered is not a whisper/noise boundary.

So whispers get their own opening path, scored on what actually distinguishes whispered speech from noise — see the term list below. Every term must pass; none can be traded for another.

const voice = new EchoVoice({
  apiKey, workflow: "front-desk",
  vad: { noiseProfile: "whisper" },
  // Optional but recommended for whispers: the browser's own noise
  // suppression is tuned to preserve VOICED speech and can attenuate a
  // whisper before the VAD ever sees it.
  audioConstraints: { noiseSuppression: false, voiceIsolation: false },
});
voice.on("whisper", ({ score }) => showQuietCue(score));

The whisper profile turns the detector on; it is off in quiet/normal/ noisy. Tune with whisperThreshold (default 0.30, chosen by sweeping measured score distributions for the highest value that still admits real whispers while every noise negative fails), whisperFrames (default 4 — qualifying frames to open, deliberately longer than the voiced startFrames because a whisper is a weaker signal; a short gap of sub-threshold frames is tolerated inside a run, since real speech dips between syllables) and whisperFloorRms.

Properties worth knowing:

• It only ever adds an opening. The voiced path keeps its full strictness, so enabling this cannot make normal speech harder to hear.
• Barge-in stays on the voiced path. At whisper amplitudes, residual echo is indistinguishable from a whisper, so allowing whisper barge-in would make the agent interrupt itself.
• RNNoise is not consulted on this path — it vetoes every whisper by construction. It still guards the voiced path.
• Mixed utterances are one turn. Once open, whisper frames sustain the turn like voiced ones, so a sentence that starts whispered and ends aloud (or the reverse) doesn't split.

Detection is layered so that each term rejects something the others cannot: formant structure (a vocal tract, vs paper or a keyboard), spectral flatness (peaky speech vs flat room tone), syllabic modulation (speech envelopes move; hiss does not), brightness relative to the room, and an autocorrelation veto that rejects anything with pitch — that last one is what keeps ordinary speech on the voiced path where it belongs.

Measured over 30 independent trials per condition:

| condition | opens a turn | |---|---| | whisper | 30/30 | | whisper, quiet (0.25x level) | 30/30 | | ordinary voiced speech | 0/30 | | room tone (three levels) | 0/30 | | mic self-noise, digital silence | 0/30 |

Thresholds are relative, not absolute: amplitude gates track the room's noise floor and the brightness test tracks the capture chain's own baseline. This matters more than it sounds — the same voice measured through Safari and Chrome produced a 9.7x difference in absolute band ratio (Chrome attenuates the high band), while formant and modulation scores were identical. Any fixed threshold works on one browser and fails on the other.

Still synthetic-signal validation for the negatives; verify with real voices in the target room before shipping.

Gain on the wire

A whisper reaches the microphone 10-20x quieter than ordinary speech, and STT is trained overwhelmingly on normal-level speech — so a correctly detected whisper can still transcribe poorly. whisperGain scales whisper-opened turns toward a normal level on the way out:

vad: { noiseProfile: "whisper", whisperGain: true }

Off by default: this package otherwise never modifies wire audio, so boosting is something you opt into. It is linear gain only — no compression, EQ or denoising, since those add the spectral artifacts that measurably worsen STT. The gain is capped by each frame's own peak, so it cannot clip, and it ramps per sample so a whisper-to-speech transition has no step discontinuity. It adds no latency: the multiply happens inside the Float32 -> PCM16 conversion that runs on every frame anyway, with no lookahead.

Tune with whisperGainTarget (RMS to normalize toward, default 0.04) and whisperGainMax (ceiling, default 6). The whispergain event reports the gain applied to each frame.

Whether boosting improves transcription for your STT provider is unverified — measure it before relying on it.

React: const { whispering } = useEchoVoice(...) is true while the current turn was opened by the whisper path — useful for a "heard you" cue when there is no audible level to render.

Endpointing (how it works with the server)

With endpointing: "client" the package connects with ?endpointing=client. The Echo server then buffers incoming audio and never auto-commits on silence — it responds only when the package sends {"type":"commit"} (at push-to-talk release, or client-VAD end). Barge-in sends {"type":"cancel"}. Set endpointing: "server" to revert to Echo's own server-side VAD (the default for everyone not using this package — nothing existing changes).

Events

status · session · resumed · transcript · reply · audio · turn · node · ended · context · output · speaking · listening · vad · whisper · whispergain · waveform · level · muted · committed · signal · turnmode · phase · error · close. voice.onAny((ev, data) => …) for all of them.

Resume + sessions

Pass session: priorSessionId to resume (rehydrates history; resumed event). voice.sessionId is the id to store against your user. See the platform's Integrate tab / docs/voiceflow-sessions.md for listing + per-user history.

Build / dev

npm i && npm run build     # → dist/ (esm, cjs, iife/CDN, .d.ts)
open demo/index.html       # live demo (paste a ticket)

Architecture

Mic (16kHz PCM) ─┬─► Transport ──WS──► Echo (buffers until commit)
                 └─► VAD (Silero|energy) ─► TurnController ─► commit / cancel / barge-in
Echo ──tts.chunk──► Player (gapless) ─► gain(mute) ─► analyser(waveform) ─► speakers

Files: core.ts (glue) · transport.ts (WS protocol) · audio/mic.ts · audio/player.ts · vad/{energy,silero,index}.ts · turn.ts · react/.