use-lipsync

React hook for text-driven lip sync with ARKit morph targets. Supports English + Indian languages (Hindi, Tamil, Telugu, Kannada, Malayalam).

Install

npm install use-lipsync

Usage

import { useRef, useState } from "react";
import { useLipSync } from "use-lipsync";

function Avatar() {
  const audioRef = useRef(null);
  const [isPlaying, setIsPlaying] = useState(false);
  const [morphValues, setMorphValues] = useState({ jawOpen: 0, mouthFunnel: 0, mouthSmileLeft: 0, mouthSmileRight: 0 });

  const handleMorphUpdate = ({ jawOpen, mouthFunnel, mouthSmileLeft }) => {
    // Apply these morph targets to your 3D model
  };

  useLipSync(audioRef, isPlaying, handleMorphUpdate, "வணக்கம் hello");

  return (
    <audio
      ref={audioRef}
      src="speech.mp3"
      onPlay={() => setIsPlaying(true)}
      onEnded={() => setIsPlaying(false)}
    />
  );
}

API

useLipSync(audioRef, isPlaying, onMorphUpdate, spokenText)

| Param | Type | Description | |-------|------|-------------| | audioRef | RefObject<HTMLAudioElement> | Ref to the audio element | | isPlaying | boolean | Whether audio is currently playing | | onMorphUpdate | (morphs) => void | Callback with { jawOpen, mouthFunnel, mouthSmileLeft } | | spokenText | string | The text being spoken (supports Indic scripts) |

Supported Languages

• English (passthrough)
• Hindi (Devanagari)
• Tamil
• Telugu
• Kannada
• Malayalam