talking-head-studio

Make any GLB model talk — on the web and on React Native — with phoneme-accurate, audio-aligned lip-sync. With or without blend shapes.

The point: lip-sync that's driven by the audio, not guessed

Most avatar libraries flap a jaw open in proportion to audio loudness. That reads as "mouth moving," not "speaking." talking-head-studio is built around a different model: a viseme schedule — a timed list of mouth shapes derived from the actual synthesized speech — drives morph targets on the model.

TTS server  ──▶  AgentVisemePayload          ──▶  scheduleVisemes()  ──▶  morph drive
(word-aligned     { cues: [{ viseme, startMs,      (this library,         (Three.js
 phonemes)          endMs }], durationMs }          web + native)          morph targets)

The wire format is AgentVisemePayload: per-phoneme cues using the 9-shape Rhubarb vocabulary (A–H, X), each with a start/end time in milliseconds. The library maps those onto Oculus viseme morphs and schedules them against the audio clock, so the mouth hits each shape when that sound is actually heard.

This pairs directly with a TTS server that emits viseme timings from real word alignment (we built Qwen3-TTS for exactly this — it serves AgentVisemePayload over an SSE endpoint). But the format is open: emit cues from any source and the renderer consumes them identically.

Four lip-sync tiers — every model works

The model decides the fidelity; you don't have to pre-process anything.

| Your model has… | Method | Quality | |---|---|---| | Oculus viseme morphs | Direct morph drive (MorphTargetBackend) | Excellent | | ARKit blend shapes (52 AUs) | remapArkitToOculus() → morph drive | Good | | Only jawOpen / mouthOpen | Amplitude fallback | Acceptable | | No face rig at all | Gaussian splat backend (roadmap — not yet built) | Excellent |

If a model has no viseme morphs, scheduled cues still fall back to the jaw/amplitude path automatically — you never get a frozen face.

Two renderers, one contract

The same AgentVisemePayload / FaceControl contract drives both render paths, so you write your voice pipeline once:

• Web — an isolated <iframe> running met4citizen TalkingHead as the rig (TalkingHead.web.tsx). Drop it into any React / Next / Vite app.
• React Native — a native WebGPU renderer (WgpuAvatar, via react-native-wgpu + react-three-fiber). No WebView, no postMessage latency, morphs driven on the GPU.

Capabilities differ slightly between the two — see the capability matrix.

Install

# React Native / Expo WebView path
npm install talking-head-studio react-native-webview

# React Native / Expo native WebGPU path
npx expo install react-native-wgpu @react-three/fiber three three-stdlib expo-asset

# Web (React, Next.js, Vite)
npm install talking-head-studio

three, @react-three/fiber, and the platform packages are peer dependencies — bring your own versions. react-native-webview is only required for the WebView renderer. Native WebGPU uses react-native-wgpu and must run in a native build, not Expo Go.

React Native / Expo WebGPU setup

Native WebGPU needs the React Native new architecture and the WebGPU build of Three.js. The example app in example/ has the full working config; these are the important parts:

// app.json
{
  "expo": {
    "newArchEnabled": true,
    "plugins": ["expo-asset"]
  }
}

// metro.config.js
const path = require('path');
const { getDefaultConfig } = require('expo/metro-config');

const config = getDefaultConfig(__dirname);
const nodeModules = path.resolve(__dirname, 'node_modules');
const threeWebgpu = path.resolve(nodeModules, 'three/build/three.webgpu.js');

config.resolver.assetExts.push('glb');
config.resolver.extraNodeModules = {
  three: threeWebgpu,
};

module.exports = config;

Build and launch a native app so WebGPUModule is linked:

npx expo prebuild --platform android --no-install
npx expo run:android

Expo Go cannot load the native WebGPU module.

Quick start

Web / React Native component

import { useRef } from 'react';
import { TalkingHead, type TalkingHeadRef } from 'talking-head-studio';

export default function Avatar() {
  const ref = useRef<TalkingHeadRef>(null);

  return (
    <TalkingHead
      ref={ref}
      avatarUrl="https://example.com/your-model.glb"
      mood="happy"
      cameraView="upper"
      style={{ width: 400, height: 600 }}
      onReady={() => {
        // Drive the mouth from a viseme schedule (e.g. from your TTS server)
        ref.current?.scheduleVisemes({
          cues: [
            { viseme: 'A', startMs: 0, endMs: 90 },
            { viseme: 'E', startMs: 90, endMs: 170 },
            { viseme: 'X', startMs: 170, endMs: 220 },
          ],
          durationMs: 220,
          audioStartedAtMs: Date.now(),
        });
      }}
    />
  );
}

Native WebGPU (React Native, no WebView)

import { WgpuAvatar, type WgpuAvatarRef } from 'talking-head-studio/wgpu';

const ref = useRef<WgpuAvatarRef>(null);

<WgpuAvatar
  ref={ref}
  avatarUrl="https://example.com/your-model.glb"
  mood="neutral"
  style={{ flex: 1 }}
/>;
// ref.current?.scheduleVisemes(payload) — same contract as the web component

TalkingHead component — props & ref

Props

| Prop | Type | Default | Description | |------|------|---------|-------------| | avatarUrl | string | required | Any .glb. Rigged or not. | | authToken | string \| null | null | Bearer token for authenticated GLB URLs. | | mood | TalkingHeadMood | 'neutral' | neutral \| happy \| sad \| angry \| fear \| disgust \| love \| sleep \| excited \| thinking \| concerned \| surprised | | cameraView | 'head' \| 'upper' \| 'full' | 'upper' | Framing preset. | | cameraDistance | number | -0.5 | Zoom offset. Negative = closer. | | hairColor | string | — | Hex color. Applied to materials named hair, fur. | | skinColor | string | — | Applied to skin, body, face. | | eyeColor | string | — | Applied to eye, iris. | | accessories | TalkingHeadAccessory[] | [] | Bone-attached GLB items. | | onReady | () => void | — | Fired when fully loaded. | | onError | (msg: string) => void | — | Fired on load failure. | | style | ViewStyle / CSSProperties | — | Container style. |

Ref methods

// Lip-sync
ref.current?.scheduleVisemes(payload); // AgentVisemePayload → full timed lip-sync schedule
ref.current?.clearVisemes();
ref.current?.sendAmplitude(0.7);       // amplitude 0..1 → jaw (fallback / no schedule)

// Expression & appearance
ref.current?.setMood('excited');
ref.current?.setHairColor('#ff0000');
ref.current?.setSkinColor('#8d5524');
ref.current?.setEyeColor('#2e86de');
ref.current?.setAccessories([...]);

// Body — procedural motions, gestures, poses, animation clips
ref.current?.dispatchMotion('groove');                       // looping procedural motion
ref.current?.stopMotion();
ref.current?.playGesture('thumbup');                         // upstream hand gesture
ref.current?.playPose('oneknee');                            // upstream pose template
ref.current?.playAnimation('/animations/wave.glb', { dur: 2 });
ref.current?.lookAt(120, 80, 500);                           // turn toward viewport coords

The motion vocabulary (groove, wave, nod, idle, attack, defend, celebrate, plus every upstream gesture/pose name) is exported as typed constants — MOTION_KEYS, TALKINGHEAD_GESTURES, TALKINGHEAD_POSES, and the isMotionKey() guard — from both the package root and talking-head-studio/contract.

Runtime capability matrix

Both renderers share one API; where native can't match the WebView's upstream rig, it falls back to a procedural approximation rather than failing. This table is the honest gap list.

| Feature | Web (iframe) | Native (WGPU) | Notes | |---|:---:|:---:|---| | Viseme schedules (scheduleVisemes) | ✅ | ✅ | Both consume AgentVisemePayload. | | Amplitude jaw fallback (sendAmplitude) | ✅ | ⚠️ | Web drives jaw from amplitude; native exposes the method for API parity. | | Core procedural motions (groove, attack, defend) | ✅ | ✅ | Shared MOTION_DEFS source of truth. | | Gesture names (thumbup, shrug, …) | ✅ | ⚠️ | Web delegates to TalkingHead; native uses procedural approximations. | | Pose names (oneknee, kneel, sitting, …) | ✅ | ⚠️ | Web delegates to TalkingHead; native uses static procedural poses. | | Full mood vocabulary | ✅ | ✅ | All 8 upstream moods + friendly aliases. | | External animation clips (playAnimation) | ✅ | ⚠️ | Web delegates to TalkingHead; native plays GLB clips via AnimationMixer. | | Gaze (lookAt) | ✅ | ❌ | Native eye/head-gaze bridge is future work. | | Listening / mic-reactive mouth | ⚠️ | ❌ | Web can route host-provided audio; native bridge not implemented. |

Self-hosting the runtime assets

By default the web iframe pulls the TalkingHead rig, three.js, and the HeadAudio model from public CDNs (jsDelivr, gstatic). To run fully self-hosted — no external CDN — vendor those files and point the renderer at your own origin:

import { buildAvatarHtml } from 'talking-head-studio/html';

const html = buildAvatarHtml({
  avatarUrl: 'https://your-cdn/model.glb',
  vendorBaseUrl: 'https://your-cdn/vendor', // serves three.module.js, talkinghead.mjs, etc.
  // ...
});

vendorBaseUrl replaces every CDN reference; dracoDecoderUrl overrides the DRACO decoder location independently.

FaceControl — the lower-level contract

If you're writing a custom backend or a game-engine integration, FaceControl is the single value that flows between a voice pipeline and any avatar backend.

import type { FaceControl, ExpressionState, HeadPose, EyeGaze } from 'talking-head-studio';

type HeadPose = { yaw: number; pitch: number; roll: number };  // each -1..1
type EyeGaze = { x: number; y: number };                       // each -1..1

type ExpressionState = {
  jawOpen: number; mouthSmile: number; mouthFunnel: number; mouthPucker: number;
  mouthWide: number; upperLipRaise: number; lowerLipDepress: number; cheekRaise: number;
  blinkLeft: number; blinkRight: number; browInnerUp: number;
  browDownLeft: number; browDownRight: number;
  eyeGazeLeft: EyeGaze; eyeGazeRight: EyeGaze;
}; // all weights 0..1 unless noted

Drive it from a viseme schedule:

import { useFaceControlsFromVisemes } from 'talking-head-studio';

const faceControl = useFaceControlsFromVisemes(schedule); // rAF-sampled FaceControl

Or implement a backend against it:

import type { AvatarBackend, AvatarRenderTarget, FaceControl } from 'talking-head-studio';

class MyBackend implements AvatarBackend {
  initialize() {}
  attach(target: AvatarRenderTarget) {}
  setControl(control: FaceControl) {}
  renderFrame() {}
  dispose() {}
}

MorphTargetBackend — the built-in Three.js adapter

The concrete AvatarBackend for GLB-with-morphs. Hand it a loaded scene; it discovers morph targets, builds a lookup cache, and drives them from FaceControl.

import { MorphTargetBackend, createNeutralExpression } from 'talking-head-studio';

const backend = new MorphTargetBackend(gltf.scene, {
  mood: 'neutral',
  expressionScale: 1.0,
  calibration: {
    neutral: { pose: { yaw: 0, pitch: 0, roll: 0 }, expr: createNeutralExpression() },
    ranges: { jawOpen: { min: 0, max: 0.85 } }, // clamp jaw for this model
    gazeLimits: { x: { min: -0.6, max: 0.6 } },
  },
});

backend.setControl(faceControl);
backend.renderFrame();
console.log(backend.availableChannels); // what this model actually supports

ARKit → Oculus remap (no ML, no artist work)

import { remapArkitToOculus, getArkitWeightsForViseme } from 'talking-head-studio';

remapArkitToOculus({ jawOpen: 0.7, mouthLowerDownLeft: 0.4 }); // → { aa: 0.68, oh: 0.12, ... }
getArkitWeightsForViseme('ou');                                // → { mouthPucker: 0.9, ... }

The full ARKIT_TO_OCULUS coefficient table is exported for building your own bake pipeline.

Accessories

Any GLB attached to any skeleton bone, placeable at runtime.

interface TalkingHeadAccessory {
  id: string;
  url: string;
  bone: string;                       // 'Head' | 'Spine' | 'RightHand' | ...
  position: [number, number, number];
  rotation: [number, number, number]; // Euler, radians
  scale: number;
}

Common Mixamo bones: Head, Neck, Spine, Spine1, Spine2, LeftHand, RightHand, LeftFoot, RightFoot, Hips. The 3D editor (talking-head-studio/editor, web only) provides a gizmo for live placement.

Subpath exports

| Import | Description | |------|-------------| | talking-head-studio | Avatar component + FaceControl contracts + motion constants | | talking-head-studio/contract | Stable type-only entrypoint — visemes, FaceControl, backends, motion | | talking-head-studio/html | buildAvatarHtml() for self-hosted / custom iframe embedding | | talking-head-studio/wgpu | React Native WebGPU renderer (WgpuAvatar) | | talking-head-studio/editor | R3F 3D editor with placement gizmo (web only) | | talking-head-studio/appearance | Material color system for any GLB | | talking-head-studio/voice | Audio recording + WAV conversion hooks | | talking-head-studio/sketchfab | Sketchfab search + download hooks | | talking-head-studio/api | Studio API client (avatar CRUD, voice profiles) | | talking-head-studio/wardrobe | Accessory + outfit state management |

Workspace packages (packages/avatar-creator, packages/agent-avatar) ship an embeddable creator widget and a LiveKit + MCP agent integration.

Roadmap

Status legend: ✅ shipped · 🔜 in progress · 🧪 designed, not yet built

Shipped today

• ✅ FaceControl face-control space (pose + expression + gaze) and AvatarBackend interface
• ✅ MorphTargetBackend — GLB morph discovery + mood layering
• ✅ ARKit → Oculus analytical remap with full coefficient table
• ✅ AgentVisemePayload viseme schedule format + scheduleVisemes on both renderers
• ✅ Shared procedural motion engine (web + native WGPU), gestures, poses, animation clips
• ✅ Self-hosting via buildAvatarHtml({ vendorBaseUrl })
• ✅ packages/avatar-creator, packages/agent-avatar

In progress

• 🔜 Native (WGPU) gaze bridge (lookAt) and mic-reactive listening
• 🔜 GLB schema walker — report morph coverage, bones, LODs, viseme tier for any model

Designed, not yet built

• 🧪 GaussianBackend — Gaussian-splat renderer + FLAME per-viseme delta transfer, so a model with no face rig still gets excellent lip-sync. This is the zero-prerequisite path.
• 🧪 FLAME viseme transfer pipeline (companion backend) — bake Oculus visemes into a GLB that lacks them
• 🧪 Unity / Unreal SDKs implementing the same AvatarBackend contract
• 🧪 Avatar marketplace + RPM import tooling (CatalogItem / AvatarAsset types exist; backend and store do not)

Contributing

git clone https://github.com/sitebay/talking-head-studio.git
cd talking-head-studio
npm install
npm run typecheck   # must be clean
npm test

Monorepo with packages/* as npm workspaces; the main library is the root package. The publish gate (prepublishOnly) runs lint, typecheck, tests, and metadata checks.

Credits & license

Built on met4citizen/TalkingHead (rig + gestures/poses on the web path) and Three.js. MIT licensed.