@busyexplore/zotonic-lip-sync

A React hook that turns an audio file into lip-sync viseme data — analyzed client-side with pitch detection snapped to a C-major scale, then mapped to Preston Blair viseme names. Processing state and results are also broadcast as events, which you can subscribe to from anywhere in your app via the re-exported useWatchCommandEmitter.

Features

• 🎙️ Client-side audio analysis — no server round trip
• 👄 Preston Blair viseme output — frame-indexed cues from LipSyncAnalyzer
• 📡 Broadcasts processing events (start, update, error, ready) so distant components can react without prop drilling
• ⚛️ Single hook API — useZotonicLipSync(fps)

Installation

npm install @busyexplore/zotonic-lip-sync

yarn add @busyexplore/zotonic-lip-sync

pnpm add @busyexplore/zotonic-lip-sync

Quick start

import { useZotonicLipSync } from "@busyexplore/zotonic-lip-sync";

function LipSyncPlayer() {
  const { loadFile } = useZotonicLipSync(30);

  async function handleFileChange(e) {
    const file = e.target.files[0];
    if (!file) return;

    const result = await loadFile(file);
    console.log(result.visemes); // frame-indexed Preston Blair cues
  }

  return <input type="file" accept="audio/*" onChange={handleFileChange} />;
}

API

useZotonicLipSync(fps = 24)

React hook. fps sets the frame rate used by the analyzer (can be changed later via setFPS). Fires ACTION_READY shortly after mount. Returns:

| Key | Type | Description | |---|---|---| | loadFile | (file: File \| Blob \| string \| Request \| Buffer \| ArrayBuffer \| ArrayBufferView) => Promise<LipSyncResult \| {}> | Analyzes audio with LipSyncAnalyzer. Accepts a File/Blob, a URL string or Request (fetched internally), a Node Buffer, a raw ArrayBuffer, or any typed array. | | setFPS | (fps: number) => void | Sets the frame rate used for the next loadFile call. | | MOUTH_TYPE | object | LipSyncAnalyzer.VISEME_TO_MOUTH — a lookup table mapping each Preston Blair viseme name to itself ({ MBP: "MBP", E: "E", AI: "AI", O: "O", U: "U", FV: "FV", L: "L", WQ: "WQ", rest: "rest" }). | | useWatchCommandEmitter | hook | Re-exported so other components can subscribe to lip-sync events. See Listening for events. | | LIP_SYNC_TOOL | object | Command/action name constants (see below). |

Not available in this version: loadAudioUrl/loadAudioFromFile, getShapeAtTime, setLowAccuracyMode. loadFile only accepts a File/Blob, and there's no built-in helper for looking up the shape at a given playback time — you'll need to search result.visemes yourself.

LipSyncResult

This is LipSyncAnalyzer.processAudio()'s return value, passed through unchanged:

{
  fps: number;         // frame rate used for analysis
  duration: number;    // audio duration, in seconds
  frames: number;      // audio duration, in frames (duration * fps, rounded up)
  visemes: VisemeCue[];// [{ start, end, note, viseme, mouth }] — start/end in frame numbers
}

viseme/mouth are Preston Blair names: "MBP" | "E" | "AI" | "O" | "U" | "FV" | "L" | "WQ" | "rest".

On error, loadFile resolves to {} (empty object) rather than a LipSyncResult — check for a missing visemes key, or listen for ACTION_ERROR, to detect failures.

LIP_SYNC_TOOL

Command/action constants, exported both from the hook's return value and as a named export:

export const LIP_SYNC_TOOL = {
  ACTION_START_PROCESSING: "START_PROCESSING",
  ACTION_STOP_PROCESSING: "STOP_PROCESSING",
  ACTION_UPDATE: "update",
  ACTION_READY: "ready",
  ACTION_ERROR: "ERROR",
  LIP_SYNC_COMMAND: "LIP_SYNC",
};

| Action | Emitted when | |---|---| | ACTION_START_PROCESSING | loadFile begins processing | | ACTION_UPDATE | A LipSyncResult is ready — event data is the full result object | | ACTION_ERROR | Analysis failed — event data has a message | | ACTION_READY | The hook has mounted and is ready to load audio |

Note: ACTION_STOP_PROCESSING is defined in LIP_SYNC_TOOL but is not currently emitted anywhere in loadFile — don't rely on it to detect completion; use ACTION_UPDATE or the resolved promise instead.

All events are emitted under LIP_SYNC_COMMAND, so you can listen for everything lip-sync related in one place.

Listening for events

Because useWatchCommandEmitter is re-exported from this package, any component can subscribe to lip-sync events without receiving props from the component that calls loadFile:

import { useZotonicLipSync } from "@busyexplore/zotonic-lip-sync";

function LipSyncStatus() {
  const { LIP_SYNC_TOOL, useWatchCommandEmitter } = useZotonicLipSync();

  useWatchCommandEmitter((command, action, data) => {
    if (command !== LIP_SYNC_TOOL.LIP_SYNC_COMMAND) return;

    switch (action) {
      case LIP_SYNC_TOOL.ACTION_START_PROCESSING:
        console.log("Analyzing audio…");
        break;
      case LIP_SYNC_TOOL.ACTION_UPDATE:
        console.log("Got visemes:", data.visemes);
        break;
      case LIP_SYNC_TOOL.ACTION_ERROR:
        console.error(data.message);
        break;
    }
  });

  return null;
}

This is useful for surfacing a global loading spinner or toast from a component that isn't the one calling loadFile.

If you pass an inline arrow function as shown above, it's a new reference every render, so the listener unsubscribes/resubscribes each render — harmless, but wrap it in useCallback if you want to avoid the churn.

Behavior notes

• setFPS only affects future loadFile calls — it doesn't retroactively convert an already-loaded result.
• loadFile does not throw on failure — a failed load resolves to {} rather than rejecting; listen for ACTION_ERROR to get the failure message.
• Events are global, not scoped to the calling component — every subscriber in your app receives every emitted command, so filter by command (and action) in your listener.
• Pitch detection is C-major-scale-snapped — LipSyncAnalyzer detects pitch via autocorrelation and snaps it to the nearest note in a C-major scale before mapping to a viseme. This is a heuristic, not phonetic recognition — it will not always match true mouth shapes for actual speech content, particularly on noisy or polyphonic audio.

Happy syncing! 👄🎵