talking-head-studio

Drop a talking, lip-syncing 3D avatar into any React app -- native or web -- in under five minutes.

Why this?

• Truly cross-platform. One component, two renderers. React Native gets a WebView; React on web gets an iframe with srcdoc. Same API, same props, same ref.
• Bring any GLB. Rigged models with ARKit/Oculus blend shapes get full phoneme-based lip-sync via HeadAudio. Non-rigged models still work -- they get a static viewer with amplitude-driven jaw animation as a fallback. No vendor lock-in to a single avatar format.
• Built for LLM voice pipelines. Wire sendAmplitude to LiveKit, Web Audio, ElevenLabs, or any audio source. The avatar speaks when your AI speaks.
• Accessory system. Attach hats, glasses, backpacks, or any GLB to any bone at runtime. Position, rotate, and scale each piece independently.

Table of Contents

• Installation
• Quick Start
• Subpath Exports
• Props
• Ref API
• Accessories
• Color Customization
• Voice Pipeline Integration
• GLB Compatibility
• Plain React / Next.js
• MotionEngine (Upcoming)
• Contributing
• Credits
• License

Installation

React Native / Expo

npm install talking-head-studio react-native-webview

react-native-webview is a peer dependency. If you are using Expo, it is available as a built-in package.

Web only (React, Next.js, Vite)

npm install talking-head-studio

No WebView dependency needed. The package ships a .web.tsx entry point that renders via <iframe srcdoc> automatically when bundled for web targets.

Quick Start

import { useRef } from 'react';
import { TalkingHead, type TalkingHeadRef } from 'talking-head-studio';

export default function Avatar() {
  const ref = useRef<TalkingHeadRef>(null);

  return (
    <TalkingHead
      ref={ref}
      avatarUrl="https://models.readyplayer.me/your-model.glb"
      mood="happy"
      cameraView="upper"
      hairColor="#1a1a2e"
      skinColor="#e0a370"
      accessories={[
        {
          id: 'sunglasses',
          url: 'https://example.com/sunglasses.glb',
          bone: 'Head',
          position: [0, 0.08, 0.12],
          rotation: [0, 0, 0],
          scale: 1.0,
        },
      ]}
      style={{ width: 400, height: 600 }}
      onReady={() => console.log('Avatar loaded')}
      onError={(msg) => console.error('Load failed:', msg)}
    />
  );
}

Subpath Exports

The package ships five independent entry points. Import only what you need — each subpath has its own optional peer dependencies.

talking-head-studio — Live talking avatar

import { TalkingHead } from 'talking-head-studio';
// Peer deps: react, react-native (optional), react-native-webview (optional)

talking-head-studio/editor — 3D editor with gizmo (web)

R3F-based canvas with PivotControls gizmo for placing accessories on an avatar. Web only.

import { AvatarCanvas } from 'talking-head-studio/editor';
// Peer deps: @react-three/fiber, @react-three/drei, three

talking-head-studio/appearance — Material color system

Apply skin/hair/eye colors to any GLB avatar. Works in both the live view and the 3D editor.

import { applyAppearanceToObject3D, type AvatarAppearance } from 'talking-head-studio/appearance';
// No extra peer deps

talking-head-studio/voice — Audio recording hooks

Headless hooks for recording voice samples (WebM→WAV conversion included). Backend-agnostic — send audio wherever you want (Qwen3-TTS, ElevenLabs, Groq, etc).

import { useAudioRecording, useAudioPlayer } from 'talking-head-studio/voice';
// No extra peer deps (browser APIs only)

talking-head-studio/sketchfab — Sketchfab search & download

Headless hooks and utilities for searching and downloading GLB models from Sketchfab. Bring your own UI and API key.

import { useSketchfabSearch, ACCESSORY_CATEGORIES, downloadModel } from 'talking-head-studio/sketchfab';
// No extra peer deps

Props

| Prop | Type | Default | Description | |------|------|---------|-------------| | avatarUrl | string | required | URL to any .glb model. Rigged or non-rigged. | | authToken | string \| null | null | Bearer token sent when fetching the model URL. CDN URLs are excluded automatically. | | mood | TalkingHeadMood | 'neutral' | Avatar expression. See Moods below. | | cameraView | 'head' \| 'upper' \| 'full' | 'upper' | Camera framing preset. | | cameraDistance | number | -0.5 | Camera zoom offset. Negative values zoom in. | | hairColor | string | -- | CSS color applied to materials whose name contains hair or fur. | | skinColor | string | -- | CSS color applied to materials whose name contains skin, body, or face. | | eyeColor | string | -- | CSS color applied to materials whose name contains eye or iris. | | accessories | TalkingHeadAccessory[] | [] | Array of GLB items to attach to bones. See Accessories. | | onReady | () => void | -- | Fires once the avatar and scene are fully loaded. | | onError | (message: string) => void | -- | Fires on load failure. | | style | ViewStyle | -- | Container style (works on both native and web). |

Moods

The mood prop accepts one of:

neutral | happy | sad | angry | excited | thinking | concerned | surprised

Mood can be changed at any time via props or the ref API. On rigged models, mood maps to blend shape expressions. On non-rigged models, mood is a no-op.

Ref API

Access runtime controls through a React ref. Every method is safe to call at any time -- calls made before the avatar is ready are silently dropped.

const ref = useRef<TalkingHeadRef>(null);

// Drive lip-sync from an audio amplitude value (0..1)
ref.current?.sendAmplitude(0.7);

// Change expression
ref.current?.setMood('excited');

// Change colors at runtime
ref.current?.setHairColor('#ff0000');
ref.current?.setSkinColor('#8d5524');
ref.current?.setEyeColor('#2e86de');

// Swap accessories without re-mounting the component
ref.current?.setAccessories([
  {
    id: 'crown',
    url: 'https://example.com/crown.glb',
    bone: 'Head',
    position: [0, 0.22, 0],
    rotation: [0, 0, 0],
    scale: 0.8,
  },
]);

Ref Methods

| Method | Signature | Description | |--------|-----------|-------------| | sendAmplitude | (amplitude: number) => void | Feed audio amplitude (0 to 1) for jaw animation. | | setMood | (mood: TalkingHeadMood) => void | Change avatar expression at runtime. | | setHairColor | (color: string) => void | Update hair material color. | | setSkinColor | (color: string) => void | Update skin material color. | | setEyeColor | (color: string) => void | Update eye/iris material color. | | setAccessories | (accessories: TalkingHeadAccessory[]) => void | Replace the entire accessory set. Handles loading, diffing, and cleanup automatically. |

Accessories

Attach any GLB model to any bone on the avatar skeleton. The system handles loading, disposal, and transform updates.

Accessory shape

interface TalkingHeadAccessory {
  id: string;                        // Unique identifier for diffing
  url: string;                       // URL to a .glb file
  bone: string;                      // Target bone name (e.g. "Head", "RightHand", "Spine")
  position: [number, number, number]; // Offset from the bone origin
  rotation: [number, number, number]; // Euler rotation in radians
  scale: number;                      // Uniform scale factor
}

Example: hat + glasses + backpack

<TalkingHead
  avatarUrl="https://example.com/avatar.glb"
  accessories={[
    {
      id: 'cowboy-hat',
      url: '/models/cowboy-hat.glb',
      bone: 'Head',
      position: [0, 0.18, 0],
      rotation: [0, 0, 0],
      scale: 1.2,
    },
    {
      id: 'aviators',
      url: '/models/aviator-glasses.glb',
      bone: 'Head',
      position: [0, 0.06, 0.11],
      rotation: [0, 0, 0],
      scale: 1.0,
    },
    {
      id: 'backpack',
      url: '/models/backpack.glb',
      bone: 'Spine1',
      position: [0, 0, -0.15],
      rotation: [0, Math.PI, 0],
      scale: 0.9,
    },
  ]}
/>

Common bone names

Mixamo-rigged models typically expose these bones:

Head, Neck, Spine, Spine1, Spine2,
LeftShoulder, LeftArm, LeftForeArm, LeftHand,
RightShoulder, RightArm, RightForeArm, RightHand,
LeftUpLeg, LeftLeg, LeftFoot,
RightUpLeg, RightLeg, RightFoot

Bone matching is flexible -- if an exact match is not found, the component tries a prefix match (useful for Sketchfab exports like Head_5). If no bone matches, the accessory falls back to the scene root.

Runtime accessory swaps

// Remove all accessories
ref.current?.setAccessories([]);

// Swap glasses for a monocle
ref.current?.setAccessories([
  { id: 'monocle', url: '/models/monocle.glb', bone: 'Head', position: [0.03, 0.07, 0.11], rotation: [0, 0, 0], scale: 0.6 },
]);

Accessories that were previously loaded but are absent from the new array are automatically disposed (geometry, materials, textures).

Color Customization

Colors can be set via props (applied on initial load) or via the ref API (applied at runtime without reloading the model).

The system matches material names against known keywords:

| Target | Material name keywords | |--------|----------------------| | Hair | hair, fur | | Skin | skin, body, face | | Eyes | eye, iris |

// Via props
<TalkingHead hairColor="#2d1b00" skinColor="#f0c8a0" eyeColor="#3d6b4f" />

// Via ref (runtime)
ref.current?.setHairColor('#ff4500');
ref.current?.setSkinColor('#c68642');
ref.current?.setEyeColor('#1abc9c');

This works on both rigged and non-rigged models -- any GLB with appropriately named materials will respond to color changes.

Voice Pipeline Integration

The component is designed to sit at the end of a voice pipeline. Feed it audio amplitude and it handles the rest.

Primary: HeadAudio phoneme lip-sync

On rigged models in browser contexts with Web Audio available, HeadAudio provides phoneme-level lip-sync automatically. Audio elements in the page are intercepted and routed through the lip-sync engine -- no wiring required on your end.

Fallback: amplitude-driven jaw

When phoneme-level lip-sync is unavailable (React Native WebView, non-rigged models, or missing blend shapes), sendAmplitude drives jaw movement directly via morph targets.

LiveKit integration

import { useDataChannel } from '@livekit/components-react';

function AvatarWithLiveKit() {
  const ref = useRef<TalkingHeadRef>(null);

  useDataChannel('agent_speaking', (data) => {
    if (data.amplitude !== undefined) {
      ref.current?.sendAmplitude(data.amplitude);
    }
  });

  return <TalkingHead ref={ref} avatarUrl="..." />;
}

Web Audio analyser

const audioCtx = new AudioContext();
const analyser = audioCtx.createAnalyser();
const buf = new Uint8Array(analyser.frequencyBinCount);

// Connect your audio source to the analyser
source.connect(analyser);

// Poll amplitude and feed the avatar
const interval = setInterval(() => {
  analyser.getByteFrequencyData(buf);
  const amplitude = buf.reduce((a, b) => a + b, 0) / buf.length / 255;
  ref.current?.sendAmplitude(amplitude);
}, 50);

Any audio source

The only contract is a number between 0 and 1, called at roughly 20 Hz. This works with ElevenLabs, OpenAI Realtime, Deepgram, Whisper, or any other TTS/STT pipeline.

GLB Compatibility

Rigged models (full feature set)

For the complete experience -- phoneme lip-sync, expressions, moods, gestures -- your GLB should have:

• A Mixamo-compatible armature (the component expects standard bone names)
• ARKit blend shapes and/or Oculus viseme blend shapes for lip-sync
• Standard Three.js-compatible GLB format

Models from Ready Player Me, Avaturn, or any Mixamo-rigged source work out of the box.

Non-rigged models (static fallback)

Any valid GLB loads successfully. Non-rigged models get:

• Auto-framing and centering in the viewport
• Orbit controls for rotation
• Embedded animation playback (walk cycles, idle loops, etc.)
• Amplitude-driven jaw via morph targets (if the model has jawOpen, mouthOpen, or viseme_aa blend shapes)
• Color customization (if materials are named appropriately)
• Accessory attachment (falls back to scene root if no bones exist)

Upstream documentation

For detailed model authoring guidance, see the TalkingHead documentation.

Plain React / Next.js

Despite the package name, this works on the web without React Native installed. The react-native and react-native-webview peer dependencies are both marked optional.

On web, the component renders an <iframe> with srcdoc containing the full Three.js scene. No WebView, no native modules, no build plugins.

// Works in any React 18+ web app
import { TalkingHead } from 'talking-head-studio';

export default function Page() {
  return (
    <TalkingHead
      avatarUrl="/models/avatar.glb"
      mood="happy"
      style={{ width: 600, height: 800 }}
    />
  );
}

Bundlers that support the react-native platform field (Metro, Expo) resolve TalkingHead.tsx (WebView). Standard web bundlers (webpack, Vite, esbuild) resolve TalkingHead.web.tsx (iframe). The API is identical.

MotionEngine (Upcoming)

MotionEngine integration is in development. This will add real-time body tracking and gesture replay to the avatar, driven by webcam or motion capture data.

Stay tuned.

Contributing

Contributions are welcome. Please open an issue to discuss your idea before submitting a pull request.

git clone https://github.com/sitebay/react-native-talking-head.git
cd react-native-talking-head
npm install
npm run typecheck
npm test

Credits

This project builds on excellent open-source work:

• met4citizen/TalkingHead -- The 3D avatar engine powering model loading, rigging, and expression systems.
• met4citizen/HeadAudio -- Phoneme-based lip-sync from audio streams using AudioWorklet.
• lhupyn/motion-engine -- Real-time body motion tracking (upcoming integration).
• Three.js -- 3D rendering, loaded via CDN at runtime.

License

MIT