avatar-engine

The Talking-Avatar Engine — the spine of GBFolio's diegetic avatar and R3F Studio's Pillar 2.

audio → viseme engine (wawa-derived, 9 visemes) → OVR-15 → 150+ Reallusion CC5 morphs + jaw/tongue bones, with emotion (priority 10) → lipsync (20) → blink/eyes (30) → manual (40) layering, ActorCore gesture selection, procedural idle/blink/breathing, and a swappable TTS adapter behind speak().

Quick start

import { createTalkingAvatar } from 'avatar-engine';
import { AuraTTSAdapter } from 'avatar-engine/tts';

const avatar = createTalkingAvatar({
  scene: gltf.scene,               // or glb: '/models/avatar.glb'
  animations: gltf.animations,
  tts: new AuraTTSAdapter({ endpoint: '/api/tts', voice: 'aura-2-thalia-en' }),
});

// render loop (R3F: useFrame, or use avatar-engine/react)
avatar.update(delta);

const result = await avatar.speak('Let me walk you through the methodology.');
// result.visemes is a bake-ready track; result.emotion drove the face.

React/R3F: import { useTalkingAvatar } from 'avatar-engine/react'.

Voice providers

| Provider | Kind | Status | |---|---|---| | AuraTTSAdapter | audio (analyser path, max fidelity) | Launch — needs your server-side proxy to Cloudflare Workers AI (@cf/deepgram/aura-2); token stays server-side | | WebSpeechTTSAdapter | live (timed track + boundary re-sync) | Dev fallback, zero deps, works offline | | VoxCPM2 (Toronto DO GPU droplet) | audio | Graduate — implement as another TTSAdapter, no upstream changes |

Default when no adapter passed: Web Speech fallback chain.

Layout

• src/types.ts — the Step-0 contract (see .prism/shared/contracts/ENGINE-API.md)
• src/core/ — viseme map, analyser lipsync, facial layers/emotions, ActorCore gestures, master controller, sentiment
• src/tts/ — adapter boundary, Aura-2, Web Speech, text→viseme track
• src/react/ — optional R3F hooks (only place react appears)
• reference/ — the original engine reference files, verbatim (provenance)

Notes

• Ships as TS source (exports point at src/); Vite/esbuild consume it directly. Add tsup before publishing to npm.
• ENGINE-CHANGE comments mark the only deviations from the reference implementation (deterministic blink envelope, viseme tap for bake-recording, timed-drive hook, external-end for live TTS, mic-mode entry points).
• Mic mode uses createMediaElementSource on a muted element with srcObject — verify per-browser during P2 dev; if Safari balks, switch wawa connection to createMediaStreamSource.