narrator-avatar

React component for 3D talking avatars with lip-sync, Deepgram or Google TTS, content-aware hand gestures, and pause/resume. Built on @met4citizen/talkinghead.

Features

• 3D avatars – Ready Player Me–compatible GLB models (full-body)
• Lip-sync – Word-level sync with Deepgram or Google TTS (English bundled; other languages need extra setup)
• TTS – Deepgram (streaming) or Google Cloud Text-to-Speech
• Gestures – Content-aware hand gestures: handup, index, ok, thumbup, side, shrug
• Playback – Speak, Pause, Resume, Stop (mid-phrase resume in accurate mode)
• Speech reliability – Prefetches upcoming phrases, retries TTS, resumes AudioContext on tab focus (reduces lag and silent dropouts)
• Avatar Studio controls – Visual presets, skin gloss/pores/warmth, eye contact, and mood
• Accessibility – Subtitle callback for closed captions

Table of contents

• Install
• Usage
• Props
• Ref API
• Environment variables
• TypeScript
• License

Install

npm install narrator-avatar

Peer dependencies (React 18+) are installed automatically. The package bundles TalkingHead, Three.js, and English lip-sync—no import maps or Vite config required.

Usage

import { useRef } from 'react';
import NarratorAvatar from 'narrator-avatar';

function MyPage() {
  const avatarRef = useRef(null);

  return (
    <div style={{ width: '400px', height: '500px' }}>
      <NarratorAvatar
        ref={avatarRef}
        avatarUrl="/avatars/brunette.glb"
        avatarBody="F"
        ttsService="deepgram"
        ttsVoice="aura-2-aurora-en"
        ttsApiKey={import.meta.env.VITE_DEEPGRAM_API_KEY}
        accurateLipSync={true}
        speechRate={0.9}
        visualPreset="beauty"
        skinGloss={1.5}
        skinPores={2}
        onReady={() => {}}
        onSpeechStart={() => {}}
        onSpeechEnd={() => {}}
        onSubtitle={() => {}}
      />
      <button onClick={() => avatarRef.current?.speakText('Hello! How are you?')}>
        Speak
      </button>
      <button onClick={() => avatarRef.current?.pauseSpeaking()}>Pause</button>
      <button onClick={() => avatarRef.current?.resumeSpeaking()}>Resume</button>
      <button onClick={() => avatarRef.current?.stopSpeaking()}>Stop</button>
      <button onClick={() => avatarRef.current?.makeEyeContact(2000)}>
        Eye contact
      </button>
    </div>
  );
}

Props

| Prop | Description | |------|-------------| | avatarUrl | URL to GLB model (e.g. /avatars/brunette.glb) | | avatarBody | 'M' or 'F' for posture | | cameraView | Camera framing: 'full', 'mid', 'upper', 'head' (default 'mid') | | cameraRotateEnable | Allow mouse drag to rotate view (default false). Set true to enable. | | cameraZoomEnable | Allow mouse wheel to zoom (default false). Set true to enable. | | cameraPanEnable | Allow mouse to pan (default false) | | ttsService | 'google' or 'deepgram' | | ttsVoice | Deepgram: e.g. aura-2-mars-en, aura-2-aurora-en. Google: e.g. en-GB-Standard-A | | ttsApiKey | API key (or set VITE_DEEPGRAM_API_KEY / VITE_GOOGLE_TTS_API_KEY) | | lipsyncModules | Array of language codes (default ['en']) | | lipsyncLang | Lip-sync language (default 'en') | | visualPreset | Lighting preset: 'cinematic', 'beauty', 'studio', 'sunset', 'broadcast' | | visualQuality | 'auto' (default): ultra on desktop, balanced on mobile/slow network. 'ultra' | 'balanced' | 'performance' | | lazyMount | true | false | 'auto' (default): defer WebGL init until visible on mobile | | skinGloss | Skin sheen/sweat intensity from 0 to 2 | | skinPores | Procedural pore/normal intensity from 0 to 2 | | skinWarmth | Warm skin tone blend from 0 to 2 | | eyeContactIntensity | Eye openness/contact strength from 0 to 2 | | modelFPS, modelPixelRatio | Render cadence and resolution multiplier. TalkingHead also multiplies modelPixelRatio by devicePixelRatio, so use values near 1 (not 4). Mobile is capped automatically. | | dracoEnabled, dracoDecoderPath | Enable Draco-compressed avatar loading | | modelDynamicBones | TalkingHead dynamic-bone config for rigged hair/body parts | | update | Per-frame callback (dt, talkingHead) | | accurateLipSync | true = REST per phrase, best lip-sync + pause/resume. Default true for stable tutor playback | | speechRate | e.g. 0.9 for 10% slower (pitch-preserving) | | phraseGapMs | Base teacher breath between speech units (default 560). Adapts longer after ? / long beats. Complete sentences are not glued together so breaths can land. | | speechGestures | Content-aware hand gestures (default true) | | speechGestureStyle | 'tutor' (default): sparse LMS instructor gestures with cooldown. 'expressive': legacy busier mapping | | getGestureForPhrase | (phrase) => { name, dur?, mirror?, eyeContactMs? } \| null to override per phrase | | avatarMode | 'auto' (default): voice-only on mobile (~22KB entry, no GLB/THREE); full 3D lazy-loaded on desktop. 'voice' / 'full' to force | | preloadText | Lesson text to warm into the TTS cache when the avatar is ready (phrase 0 first, then ahead window) | | onPreloadReady | Fires when phrase 0 is cached — Speak can start without waiting on Deepgram | | pauseWhenHidden | true (default): pause on tab hide / phone call; resume on return. Leave/unmount always stops | | subtitleMaxWords | Cap word-follow captions to last N words (0 = full phrase). Reduces cut-off / layout shake | | onReady, onError, onSpeechStart, onSpeechEnd | Lifecycle callbacks | | onSubtitle | Word-follow caption callback as speech plays (text: string) => void |

Ref API

| Method / property | Description | |-------------------|-------------| | speakText(text, options?) | Speak text via TTS | | preloadSpeech(text) | Prefetch first phrases into TTS cache (call when lesson text is known) | | pauseSpeaking() | Pause mid-phrase when accurateLipSync is true | | resumeSpeaking() | Resume from the same point in the current phrase | | stopSpeaking() | Stop and clear | | makeEyeContact(durationMs?) | Ask the avatar to hold stronger eye contact | | setMood(mood) | Change TalkingHead mood | | setLighting(options) | Pass lighting options to TalkingHead | | setView(view, options?) | Change camera view | | playGesture(name, dur?, mirror?, ms?) | Play a built-in gesture | | playAnimation(url, onprogress?, dur?, ndx?, scale?) | Play a Mixamo/RPM FBX animation | | playPose(url, onprogress?, dur?, ndx?, scale?) | Play a Mixamo/RPM FBX pose | | stopAnimation(), stopPose() | Stop active animation or pose | | setMixerGain(speech, background?, fadeSecs?) | Adjust speech/background audio gain | | playBackgroundAudio(url), stopBackgroundAudio() | Control background audio | | isReady | Whether the avatar has finished loading | | isSpeaking | Whether the avatar is currently speaking |

Environment variables

| Variable | Use | |----------|-----| | VITE_DEEPGRAM_API_KEY | Deepgram TTS (or pass ttsApiKey prop) | | VITE_GOOGLE_TTS_API_KEY | Google TTS when ttsService="google" (or pass ttsApiKey) |

TypeScript

The package is JavaScript. For TypeScript, add a declaration file (e.g. src/narrator-avatar.d.ts) that declares the component props and ref type, or use the component with // @ts-expect-error if you prefer.

Performance (web and mobile, one GLB per avatar)

Use one model URL per avatar (e.g. /avatars/tutor.glb). The component adapts at runtime — no -mobile / -web copies.

| Mechanism | Desktop | Mobile / Save-Data / low-end | |-----------|---------|------------------------------| | visualQuality="auto" | ultra — 60 FPS, studio lights, 4K shadows | Phones → balanced (20 FPS, capped DPR ≤~1.35, no shadows/studio lights). Low-RAM / 2G / Save-Data → performance (15 FPS, minimal lights) | | Pixel ratio | ~1.25 × DPR | Multiplier auto-capped so total device pixels stay ~1.15–1.35 | | Motion | Full idle/speaking head sway | Reduced modelMovementFactor + head move | | TTS preload | First 2–3 phrases | First phrase only; cache capped | | lazyMount="auto" | init immediately | init when scrolled into view | | Load timeout | 30s | 120s (large GLB on cellular) | | Tab hidden | animation paused | animation paused |

<NarratorAvatar
  avatarUrl="/avatars/test.glb"
  visualQuality="auto"
  lazyMount="auto"
/>

Force a lighter tier anytime: visualQuality="performance".

App tips: mount only one avatar on narrow viewports; enable gzip/brotli (and Draco if you ship compressed GLBs) on your CDN.

Keep modelPixelRatio near 0.45–0.75 on mobile if you override — TalkingHead multiplies it by devicePixelRatio.

Next.js: dedupe React in next.config (see narrator-avatar-test/next.config.ts).

License

MIT