@animated-waffle/avatar

VRM avatar renderer plus renderer-neutral performance coordination for Narya Animated Waffle talking Agents.

This is the low-level package. If you only want to embed a published Agent with its avatar, use @animated-waffle/react — it wraps this package in a single useWaffle hook. Reach for @animated-waffle/avatar directly when the application picks its own model, drives the stage, or authors expression contracts.

npm install @animated-waffle/avatar three @pixiv/three-vrm

three and @pixiv/three-vrm are peer dependencies so the application owns one three.js instance.

Two layers, one package

Performance coordination is renderer-neutral and pulls in no three.js. It owns signal timing and face ownership, not rendering:

• live microphone, tab, WebRTC, or file audio drives the avatar without ASR;
• semantic emotion and vocal-action cues share the audio playout clock;
• out-of-order and stale streams are rejected;
• the local HeadAudio viseme classifier is the mouth fallback;
• the face coordinator assigns one owner to articulation, mouth affect, brows, lids, gaze, blink, cheeks/nose and authored effects.

import { createAvatarPerformanceCoordinator } from "@animated-waffle/avatar";

const coordinator = createAvatarPerformanceCoordinator({
  timing: "live-input",
  // Override these URLs if the host serves HeadAudio assets elsewhere.
  // audioFallbackFactory: createBrowserHeadAudioFallback({ modelUrl, workletUrl }),
});
coordinator.setStage(stage);
coordinator.setAudioTrack(sourceTrack); // immediate local mouth + VAD

Managed-runtime emotion and vocal-action cues arrive through handleEvent. handleAgentStartedSpeaking / handleAgentStoppedSpeaking anchor the cue timeline for managed Agent playback.

The VRM renderer implements the AvatarStageController boundary the coordinator consumes. three.js enters the module graph only through the dynamic import inside createStageController, so a coordination-only consumer never pays for the renderer chunk:

import { createStageController, MODELS, resolvePresentationFraming }
  from "@animated-waffle/avatar";

const stage = await createStageController(container);
stage.setFraming(resolvePresentationFraming(MODELS.reina));
await stage.loadModel(MODELS.reina);
coordinator.setStage(stage);

The package also exports the authored expression contracts each built-in character resolves against, the 55-tag performance vocabulary, and transcript tag segmentation.

Avatar sources

Source inspection validates the upload shell (name, size, MIME, URL) and normalizes metadata. It does not parse VRM binary contents — the application injects an AvatarLoader backed by its renderer to do that. After loading, pass the runtime expression manager to inspectVrmRigCapabilities to verify actual rig expression support.

HeadAudio assets

The local viseme fallback loads its worklet and model at runtime from DEFAULT_HEAD_AUDIO_WORKLET_URL and DEFAULT_HEAD_AUDIO_MODEL_URL (/vendor/headaudio/… by default). Serve those files from your app, or pass your own URLs to createBrowserHeadAudioFallback.