@animated-waffle/avatar

VRM avatar renderer plus renderer-neutral performance coordination for Narya Animated Waffle talking Agents.

This is the advanced, low-level package. If you only want to embed a published Agent with its avatar, use @animated-waffle/react — it wraps this package in a single useWaffle hook. Reach for @animated-waffle/avatar directly only when the application intentionally owns its renderer or transport lifecycle, picks its own model, or authors expression contracts.

npm install @animated-waffle/avatar three @pixiv/three-vrm

three and @pixiv/three-vrm are peer dependencies so the application owns one three.js instance.

Two layers, one package

Performance coordination is renderer-neutral and pulls in no three.js. It owns signal timing and face ownership, not rendering:

• live microphone, tab, WebRTC, or file audio drives the avatar without ASR;
• semantic emotion and vocal-action cues use the reported speaking lifecycle and a browser monotonic timer; they are approximate, not media-clock synced;
• out-of-order and stale streams are rejected;
• the local HeadAudio viseme classifier is the mouth fallback;
• the face coordinator assigns one owner to articulation, mouth affect, brows, lids, gaze, blink, cheeks/nose and authored effects.

import { createAvatarPerformanceCoordinator } from "@animated-waffle/avatar";

const coordinator = createAvatarPerformanceCoordinator({
  timing: "live-input",
  // Override these URLs if the host serves HeadAudio assets elsewhere.
  // audioFallbackFactory: createBrowserHeadAudioFallback({ modelUrl, workletUrl }),
});
coordinator.setStage(stage);
coordinator.setAudioTrack(sourceTrack); // immediate local mouth + VAD

Managed-runtime emotion and vocal-action cues arrive through handleEvent. handleAgentStartedSpeaking / handleAgentStoppedSpeaking anchor the cue timeline approximately for managed Agent playback. The local HeadAudio mouth still analyzes the actual remote audio track.

The VRM renderer implements the AvatarStageController boundary the coordinator consumes. three.js enters the module graph only through the dynamic import inside createStageController, so a coordination-only consumer never pays for the renderer chunk:

import { createStageController, MODELS, resolvePresentationFraming }
  from "@animated-waffle/avatar";

const stage = await createStageController(container);
stage.setFraming(resolvePresentationFraming(MODELS.reina));
await stage.loadModel(MODELS.reina);
coordinator.setStage(stage);

StageController.destroy() means permanent host/window teardown. Never call it for voice disconnect, reconnect, or session replacement. Likewise, AvatarPerformanceCoordinator, direct audio-track wiring, and setStage are advanced lifecycle APIs; the declarative React API owns those bindings for the normal managed-Agent path.

The package also exports the authored expression contracts each built-in character resolves against, the 55-tag performance vocabulary, and transcript tag segmentation.

Avatar sources

Source inspection validates the upload shell (name, size, MIME, URL) and normalizes metadata. It does not parse VRM binary contents — the application injects an AvatarLoader backed by its renderer to do that. After loading, pass the runtime expression manager to inspectVrmRigCapabilities to verify actual rig expression support.

HeadAudio assets

The local viseme fallback loads its worklet and model at runtime from DEFAULT_HEAD_AUDIO_WORKLET_URL and DEFAULT_HEAD_AUDIO_MODEL_URL (/vendor/headaudio/… by default). Serve those files from your app, or pass your own URLs to createBrowserHeadAudioFallback.