@thattobi/narrator-avatar

React component for 3D talking avatars with lip-sync, Deepgram or Google TTS, content-aware hand gestures, and pause/resume. Built on @met4citizen/talkinghead.

Start here

1. Install @thattobi/narrator-avatar in a React 18+ app: npm install @thattobi/narrator-avatar.
2. Put your compatible .glb in your app's public assets, for example public/avatars/teacher.glb. The npm package does not include an avatar model.
3. Render <NarratorAvatar avatarMode="full" avatarUrl="/avatars/teacher.glb" ... /> in a container with an explicit width and height.
4. For speech, configure Deepgram through a same-origin server endpoint (recommended), or use a restricted test key locally. See TTS credentials and server proxy.
5. Call avatarRef.current?.speakText('Hello!') from a user click so the browser can unlock audio.

For a ready-to-run local demo, see Local development (playground). For releasing a new version, see PUBLISHING.md.

Features

• 3D avatars – Ready Player Me–compatible GLB models (full-body)
• Lip-sync – Deepgram audio-driven mouth shapes when AudioWorklet is available; estimated word timing is the fallback (English bundled; other languages need extra setup)
• TTS – Deepgram (streaming) or Google Cloud Text-to-Speech
• Gestures – 8 built-in hand poses plus 25 teaching gestures and combined sequences
• Playback – Speak, Pause, Resume, Stop (mid-phrase resume in accurate mode)
• Speech reliability – Prefetches upcoming phrases, retries TTS, resumes AudioContext on tab focus (reduces lag and silent dropouts)
• Avatar Studio controls – Visual presets, skin gloss/pores/warmth, eye contact, and mood
• Accessibility – Subtitle callback for closed captions

Table of contents

• Local development (playground)
• Install
• Usage
• Props
• Ref API
• Gesture names
• TTS credentials and server proxy
• Environment variables
• TypeScript
• Troubleshooting
• Publishing
• License

Local development (playground)

Preview library changes with hot reload before publishing:

From the package root:

npm install
npm install --prefix playground
Copy-Item playground/.env.example playground/.env
# Add your Deepgram key to playground/.env, then:
npm run dev

On macOS/Linux, use cp playground/.env.example playground/.env instead of Copy-Item. Open http://localhost:5173. Select avatar.glb or male.glb, wait for the avatar to load, then click Speak. The playground's VITE_DEEPGRAM_API_KEY is local testing only: it is visible in the browser. Do not deploy this setup unchanged.

The playground imports live source from src/ (not dist/). Edit components under src/ and the Vite app at http://localhost:5173 will pick up changes.

Teacher experience

The playground defaults to your Avaturn avatar.glb and offers male.glb as the other local choice. Both models have the same teaching skeleton and all Oculus viseme targets required for lip-sync. MetaPerson is not loaded by the playground.

Teaching behavior (library tutor mode):

• Clearer phrase breaks and longer breaths after questions / key points
• Avatar-matched Deepgram voices: Thalia for avatar.glb, Arcas for male.glb
• Slightly slower speech and 700ms base phrase gaps
• Tutor gestures: deliberate hand poses with longer cooldowns and smooth transitions
• Audio-driven visemes from the actual speech waveform when AudioWorklet is available

Select the other GLB from the dropdown anytime.

Custom Ready Player Me avatar

Stock brunette.glb is fine for smoke tests. For a more personal tutor mesh:

1. Create a full-body avatar at readyplayer.me/avatar
2. Copy the avatar id from the share URL (24 hex characters)
3. Fetch it with TalkingHead morph targets into the playground:

cd playground
npm run avatar:fetch -- <rpm-avatar-id> teacher
# writes public/avatars/teacher.glb and updates public/avatars/manifest.json

4. Refresh the playground and choose teacher.glb in Local avatar

You can also drop any compatible .glb into playground/public/avatars/ and add its filename to manifest.json. Prefer local files — the Ready Player Me CDN is often blocked.

Install

npm install @thattobi/narrator-avatar

React 18+ and React DOM 18+ are peer dependencies; install them in your app if they are not already present. The package bundles its renderer and English lip-sync—no import maps or special Vite config required. Your app supplies the avatar .glb and TTS endpoint.

Usage

import { useRef } from 'react';
import NarratorAvatar from '@thattobi/narrator-avatar';

function MyPage() {
  const avatarRef = useRef(null);

  return (
    <div style={{ width: '400px', height: '500px' }}>
      <NarratorAvatar
        ref={avatarRef}
        avatarMode="full"
        avatarUrl="/avatars/teacher.glb"
        avatarBody="F"
        ttsService="deepgram"
        ttsVoice="aura-2-thalia-en"
        deepgramEndpoint="/api/deepgram/v1/speak"
        accurateLipSync={true}
        audioDrivenLipSync={true}
        speechRate={0.9}
        visualPreset="beauty"
        skinGloss={1.5}
        skinPores={2}
        onReady={() => {}}
        onSpeechStart={() => {}}
        onSpeechEnd={() => {}}
        onSubtitle={() => {}}
      />
      <button onClick={() => avatarRef.current?.speakText('Hello! How are you?')}>
        Speak
      </button>
      <button onClick={() => avatarRef.current?.pauseSpeaking()}>Pause</button>
      <button onClick={() => avatarRef.current?.resumeSpeaking()}>Resume</button>
      <button onClick={() => avatarRef.current?.stopSpeaking()}>Stop</button>
      <button onClick={() => avatarRef.current?.makeEyeContact(2000)}>
        Eye contact
      </button>
    </div>
  );
}

Place teacher.glb in your app's public/avatars/ directory. The example expects your server to handle /api/deepgram/v1/speak and attach its Deepgram key. Do not put a production key in this browser component. In Next.js, put "use client"; at the top of the component file.

Props

| Prop | Description | |------|-------------| | avatarUrl | URL to GLB model (e.g. /avatars/brunette.glb) | | avatarBody | 'M' or 'F' for posture | | cameraView | Camera framing: 'full', 'mid', 'upper', 'head' (default 'mid') | | cameraDistance, cameraY | Optional distance and vertical offsets for precise framing; the playground uses them for a head-to-waist view | | cameraRotateEnable | Allow mouse drag to rotate view (default false). Set true to enable. | | cameraZoomEnable | Allow mouse wheel to zoom (default false). Set true to enable. | | cameraPanEnable | Allow mouse to pan (default false) | | ttsService | 'google' or 'deepgram' | | ttsVoice | Deepgram: e.g. aura-2-mars-en, aura-2-aurora-en. Google: e.g. en-GB-Standard-A | | ttsApiKey | Provider key. Avoid using this in a browser: expose a server-side TTS proxy instead. | | deepgramEndpoint | Deepgram REST endpoint. Set this to a same-origin server-side proxy in production. | | lipsyncModules | Array of language codes (default ['en']) | | lipsyncLang | Lip-sync language (default 'en') | | visualPreset | Lighting preset: 'cinematic', 'beauty', 'studio', 'sunset', 'broadcast' | | visualQuality | 'auto' (default): ultra on desktop, balanced on mobile/slow network. 'ultra' | 'balanced' | 'performance' | | lazyMount | true | false | 'auto' (default): defer WebGL init until visible on mobile | | skinGloss | Skin sheen/sweat intensity from 0 to 2 | | skinPores | Procedural pore/normal intensity from 0 to 2 | | skinWarmth | Warm skin tone blend from 0 to 2 | | eyeContactIntensity | Eye openness/contact strength from 0 to 2 | | modelFPS, modelPixelRatio | Render cadence and resolution multiplier. TalkingHead also multiplies modelPixelRatio by devicePixelRatio, so use values near 1 (not 4). Mobile is capped automatically. | | dracoEnabled, dracoDecoderPath | Enable Draco-compressed avatar loading | | modelDynamicBones | TalkingHead dynamic-bone config for rigged hair/body parts | | update | Per-frame callback (dt, talkingHead) | | accurateLipSync | true = REST per phrase for stable playback and mid-phrase pause/resume. Default true | | audioDrivenLipSync | true (default) = drive Deepgram mouth shapes from the audio waveform through HeadAudio; falls back to estimated word visemes if AudioWorklet cannot load | | speechRate | For Deepgram Aura-2, 0.7–1.5; sent as Deepgram's native speed control, preserving the voice's natural pitch and prosody. | | phraseGapMs | Base teacher breath between speech units (default 560). Adapts longer after ? / long beats. Complete sentences are not glued together so breaths can land. | | speechGestures | Content-aware hand gestures (default true) | | teachingAnimations | Optional full-body FBX URLs keyed by the semantic teaching beats. Use this to replace a compact hand pose with a retargeted motion-capture clip. | | speechGestureStyle | 'tutor' (default): purposeful LMS gestures. 'kids': high-energy hands + face emoji reactions for children. 'expressive': legacy busier keyword mapping | | getGestureForPhrase | (phrase) => { name, dur?, mirror?, eyeContactMs? } \| null to override per phrase | | avatarMode | 'auto' (default): voice-only on mobile (~22KB entry, no GLB/THREE); full 3D lazy-loaded on desktop. 'voice' / 'full' to force | | preloadText | Opt-in lesson text to send to TTS and warm into the cache when the avatar is ready (phrase 0 first, then ahead window); this makes a real provider request before Speak | | onPreloadReady | Fires when phrase 0 is cached — Speak can start without waiting on Deepgram | | pauseWhenHidden | true (default): pause on tab hide / phone call; resume on return. Leave/unmount always stops | | subtitleMaxWords | Cap word-follow captions to last N words (0 = full phrase). Reduces cut-off / layout shake | | onReady, onError, onSpeechStart, onSpeechEnd | Lifecycle callbacks | | onSubtitle | Word-follow caption callback as speech plays (text: string) => void |

Ref API

| Method / property | Description | |-------------------|-------------| | speakText(text, options?) | Speak text via TTS | | preloadSpeech(text) | Prefetch first phrases into TTS cache (call when lesson text is known) | | pauseSpeaking() | Pause mid-phrase when accurateLipSync is true | | resumeSpeaking() | Resume from the same point in the current phrase | | stopSpeaking() | Stop and clear | | makeEyeContact(durationMs?) | Ask the avatar to hold stronger eye contact | | setMood(mood) | Change TalkingHead mood | | setLighting(options) | Pass lighting options to TalkingHead | | setView(view, options?) | Change camera view | | playGesture(name, dur?, mirror?, ms?) | Play a built-in gesture | | playTeachingGesture(name, options?) | Play a teaching gesture or a combined gesture sequence (see below) | | playAnimation(url, onprogress?, dur?, ndx?, scale?) | Play a Mixamo/RPM FBX animation | | playPose(url, onprogress?, dur?, ndx?, scale?) | Play a Mixamo/RPM FBX pose | | stopAnimation(), stopPose() | Stop active animation or pose | | setMixerGain(speech, background?, fadeSecs?) | Adjust speech/background audio gain | | playBackgroundAudio(url), stopBackgroundAudio() | Control background audio | | isReady | Whether the avatar has finished loading | | isSpeaking | Whether the avatar is currently speaking |

Gesture names

Use avatarRef.current?.playGesture(name, dur?, mirror?, ms?) for a built-in pose. dur is in seconds, ms is the transition time in milliseconds, and mirror switches hands.

| Built-in pose names | |---------------------| | handup, index, ok, thumbup, thumbdown, side, shrug, namaste |

Use avatarRef.current?.playTeachingGesture(name, { dur?, mirror?, blendMs? }) for a semantic teaching beat. It returns false if the full avatar is not ready or the name is unknown.

| Teaching beats | Extra reactions | Combined sequences | |----------------|-----------------|--------------------| | greet, invite, explain, count, emphasize, celebrate, encourage, think, caution, thanks, recap, transition | wave, bow, nod, disagree, present, whiteboard | welcomeSequence, boardExplainSequence, guidedQuestionSequence, celebrateSequence, gentleCorrectionSequence, recapSequence, goodbyeSequence |

avatarRef.current?.playGesture('thumbup', 2, false, 850);
avatarRef.current?.playTeachingGesture('explain');
avatarRef.current?.playTeachingGesture('boardExplainSequence', { blendMs: 850 });

These are two different methods: pass greet to playTeachingGesture, not playGesture. Set avatarMode="full" if gestures must also appear on phones; auto switches Deepgram users to voice-only on mobile.

Teaching gestures and sequences

The playground uses TalkingHead's built-in hand and face gestures with a tuned head-to-waist camera view. Its teaching beats are greet, invite, explain, count, emphasize, celebrate, encourage, think, caution, thanks, recap, and transition. The extra reactions wave, bow, nod, disagree, present, and whiteboard use the same built-in rig-safe gesture system.

For a single teaching performance rather than an isolated pose, call one of the built-in sequences: welcomeSequence, boardExplainSequence, guidedQuestionSequence, celebrateSequence, gentleCorrectionSequence, recapSequence, or goodbyeSequence. Each sequence combines hand poses, facial reactions, and eye contact. For example:

avatarRef.current?.playTeachingGesture('boardExplainSequence');

The playground uses only the built-in rig-safe gestures. An unrelated FBX motion pack previously bent an avatar's arms because its shoulder rest pose did not match. The playground does not include a full-body motion pack.

The optional teachingAnimations prop is for FBX clips that you have already retargeted and tested against your avatar. For example:

<NarratorAvatar
  teachingAnimations={{
    greet: '/animations/wave.fbx',
    explain: '/animations/explain.fbx',
    count: '/animations/counting.fbx',
    celebrate: '/animations/celebrate.fbx',
    think: '/animations/thinking.fbx',
  }}
/>

Without a compatible mapping, playTeachingGesture('explain') uses the built-in gesture. Test any custom animation against the exact avatar before adding it to the mapping.

TTS credentials and server proxy

Do not put Deepgram or Google credentials in VITE_* variables or pass a production secret through ttsApiKey: Vite embeds those values in the browser bundle. Route TTS through your server instead, enforce authentication and rate limits there, and keep provider credentials only on that server.

For Deepgram REST speech, set deepgramEndpoint="/api/deepgram/v1/speak", leave ttsApiKey unset, and keep accurateLipSync={true}. Your server endpoint should accept POST with JSON { "text": "..." } and the model, speed, encoding, container, and sample_rate query parameters, then forward the request to https://api.deepgram.com/v1/speak with Authorization: Token <server-side key>. Return the audio response with its content type. Reject unauthenticated or excessive requests before forwarding. The browser will not send the provider key in this setup.

The direct Deepgram WebSocket mode (accurateLipSync={false}) still needs browser-side credentials; use the REST proxy mode for production. The same-origin proxy works in both full-avatar and mobile voice-only modes.

The component retains ttsApiKey only for local experimentation and restricted test keys. It is not a secure production credential boundary.

Environment variables

| Variable | Use | |----------|-----| | VITE_DEEPGRAM_API_KEY | Local development only; never deploy a provider secret with this prefix. | | VITE_GOOGLE_TTS_API_KEY | Local development only; never deploy a provider secret with this prefix. |

TypeScript

Type declarations ship with the npm package; no separate declaration file is needed:

import { useRef } from 'react';
import NarratorAvatar, { type NarratorAvatarRef } from '@thattobi/narrator-avatar';

const avatarRef = useRef<NarratorAvatarRef>(null);

Troubleshooting

| Symptom | Check | |---------|-------| | Avatar does not appear | Confirm avatarUrl returns your .glb, the parent has a nonzero width and height, and the model has a TalkingHead-compatible rig/visemes. The npm package does not ship a GLB. | | Speak produces no audio | Wait for onReady, call speakText from a click/tap, and verify your Deepgram key (local only) or authenticated same-origin proxy endpoint. | | CORS / failed to fetch | Use a same-origin server endpoint for production. The Vite proxy in the playground is for development only. | | Gesture has no visible effect | Use playGesture for the eight built-in names, playTeachingGesture for teaching names, and avatarMode="full" on mobile. | | Audio-driven mouth movement falls back | AudioWorklet must load in a secure context (https:// or localhost) and be allowed by your Content Security Policy. Speech still works with estimated visemes when it cannot load. |

Performance (web and mobile, one GLB per avatar)

Use one model URL per avatar (e.g. /avatars/tutor.glb). The component adapts at runtime — no -mobile / -web copies.

| Mechanism | Desktop | Mobile / Save-Data / low-end | |-----------|---------|------------------------------| | visualQuality="auto" | ultra — 60 FPS, studio lights, 4K shadows | Phones → balanced (20 FPS, capped DPR ≤~1.35, no shadows/studio lights). Low-RAM / 2G / Save-Data → performance (15 FPS, minimal lights) | | Pixel ratio | ~1.25 × DPR | Multiplier auto-capped so total device pixels stay ~1.15–1.35 | | Motion | Full idle/speaking head sway | Reduced modelMovementFactor + head move | | TTS preload | First 2–3 phrases | First phrase only; cache capped | | lazyMount="auto" | init immediately | init when scrolled into view | | Load timeout | 30s | 120s (large GLB on cellular) | | Tab hidden | animation paused | animation paused |

<NarratorAvatar
  avatarUrl="/avatars/test.glb"
  visualQuality="auto"
  lazyMount="auto"
/>

Force a lighter tier anytime: visualQuality="performance".

App tips: mount only one avatar on narrow viewports; enable gzip/brotli (and Draco if you ship compressed GLBs) on your CDN.

The package loader also supports Meshopt-compressed GLBs. The playground includes avatar-optimized.glb and male-optimized.glb alongside the originals for comparison; keep the rig, joint names, and facial morph targets when optimizing your own models. visualQuality controls rendering cost, not the number of bytes downloaded. Changing a Deepgram ttsVoice now selects the next request's voice without reloading WebGL. Deepgram REST requests have a 30-second per-attempt timeout and retry only network errors, timeouts, rate limits, and server failures. On connections reporting Save-Data or 2G/3G, REST requests use smaller 48 kbps MP3 audio, decoded back to PCM for the same audio-driven mouth movement; other connections keep WAV quality.

Keep modelPixelRatio near 0.45–0.75 on mobile if you override — TalkingHead multiplies it by devicePixelRatio.

Next.js: render the component from a client component ("use client";). Keep the Deepgram key in a server route, never in NEXT_PUBLIC_* variables.

Publishing

Maintainers: follow PUBLISHING.md for the exact build, tarball inspection, version bump, npm login, and publish commands.

License

MIT