@fano/vrm-controller

A thin, tree-shakeable wrapper around @pixiv/three-vrm that simplifies loading and controlling VRM avatars in Three.js: one call to load a model, a facade for facial expression + lip-sync, a Mixamo animation retargeter, and a procedural animation system that generates body motion at runtime — all built on top of the standard VRM humanoid rig, so it works across VRoid, VRChat, and generic models.

three and @pixiv/three-vrm are peer dependencies — you bring your own versions. React / @react-three/fiber and solid-js are optional peer deps, only needed if you use the /r3f or /solid entry points.

Install

bun install @fano/vrm-controller three @pixiv/three-vrm

Quick start

import { loadVRM } from "@fano/vrm-controller";

const avatar = await loadVRM("/avatar.vrm", { scene });

const clock = new THREE.Clock();
function animate() {
  requestAnimationFrame(animate);
  avatar.update(clock.getDelta()); // flushes expressions, spring bones, look-at…
  renderer.render(scene, camera);
}
animate();

That's the whole core loop. Everything below — face control, lip-sync, Mixamo animation, procedural animation — plugs into avatar without touching this loop again.

What's in the box

| | What it does | Section | | --- | --- | --- | | loadVRM | Load a model, and the one per-frame seam everything else hooks into | 1 | | createVRMFace | Expressions, blinking, eye gaze, mouth | 2 | | face.viseme / speak | Vowel-based lip-sync | 3 | | loadMixamoAnimation | Retarget a Mixamo .fbx onto any VRM → THREE.AnimationClip | 4 | | createVRMBody | Read/write individual joints by VRMHumanBoneName | 5 | | createProceduralAnimator | Generate motion at runtime — idle, look-at, gestures; bake it to a clip | 6 | | createVRM / createFrameLoop | Solid bindings — reactive loading, disposal, and the frame loop | 7 |

Every piece is optional and independently importable — pull in the loader alone if that's all you need.

Three entry points

| Import path | Consumes | Use it for | | --- | --- | --- | | @fano/vrm-controller | three + @pixiv/three-vrm | Loading, face control, Mixamo animation, procedural animation | | @fano/vrm-controller/r3f | + react + @react-three/fiber | <Vrm url="..." /> and useVRM() in React Three Fiber | | @fano/vrm-controller/solid | + solid-js | createVRM() and a frame loop for Solid — section 7 |

The framework entries add lifecycle, not features: everything in the table above is framework-agnostic and comes from the root entry either way.

1. The loader — loadVRM

const avatar = await loadVRM(url, {
  scene,                 // optional: adds vrm.scene to this Object3D for you
  onProgress(p) {},       // { loaded, total, progress }
  vrmLoadOptions: {},     // forwarded to VRMLoaderPlugin
});

loadVRM wraps GLTFLoader + VRMLoaderPlugin and resolves to a VRMAvatar:

type VRMAvatar = {
  vrm: VRM;                                   // the raw @pixiv/three-vrm instance
  scene: THREE.Group;                         // === vrm.scene
  registerUpdate(fn: (delta: number) => void): () => void;
  update(delta: number): void;
  dispose(): void;
};

The animation-engine seam. registerUpdate(fn) adds a per-frame callback that runs every frame before vrm.update(delta). avatar.update(delta) runs every registered callback (in registration order), then vrm.update(delta) last — always last, so spring bones and look-at react to the final pose. This is the one seam everything else in this package hooks into:

const stop = avatar.registerUpdate((delta) => {
  face.update(delta); // auto-blink FSM
  mixer.update(delta); // Mixamo AnimationMixer
});

Because a Mixamo clip only ever carries skeletal tracks (never expression tracks), and the face controllers only ever touch expression weights / raw morphs, the two never fight over the same data — they just both run before vrm.update flushes everything to the mesh.

dispose() removes vrm.scene from its parent and disposes every geometry, material, and texture found by traversing it (see disposeVRM in loader-utils.ts if you're adding a feature that allocates GPU resources — extend disposal there).

2. Face control — createVRMFace

import { loadVRM, createVRMFace } from "@fano/vrm-controller";

const avatar = await loadVRM("/avatar.vrm", { scene });
const face = createVRMFace(avatar.vrm);
avatar.registerUpdate((delta) => face.update(delta)); // advances auto-blink

face.emotion.setExclusive("happy");
face.viseme.set("aa", 0.8);
face.eye.autoBlink();
face.eye.gaze(camera);
face.brow.set("angry", 1);

VRMFace is one facade with one controller per concern, so you never mix unrelated blendshapes through a single generic call:

| Accessor | Concern | Backed by | | --- | --- | --- | | face.emotion | happy / angry / sad / relaxed / surprised / neutral | standard expression → VRoid Fcl_ALL/BRW/EYE/MTH fallback | | face.viseme | vowels aa ih ou ee oh (lip-sync) | standard vowel expression → fuzzy mouth-morph fallback | | face.eye | blink / wink + gaze (look-at) | standard blink/look expressions + procedural VRMLookAt | | face.brow | eyebrows (no VRM preset exists) | raw Fcl_BRW_* / generic brow* morphs | | face.expression | any standard preset or custom name | VRMExpressionController (escape hatch) | | face.morph | any blendshape by name / regex | VRMFaceController (escape hatch) |

Every concern controller prefers the portable VRM standard expression system (vrm.expressionManager) and falls back to a universal fuzzy morph matcher when a model only ships raw blendshapes — so the exact same code drives VRoid, VRChat, and generic rigs without you branching on rig type. Weights are only recorded when you call set(); they're applied to the mesh on the next vrm.update(delta), same as everything else that flows through avatar.update.

Two lower-level engines sit underneath the facade, if you want to bypass it:

• createVRMExpressionController(vrm) — the standard expression system directly: set/get/has/list, clamped 0..1, chainable, portable across VRM 0.x/1.0. Degrades to no-ops when the model has no expressionManager.
• createVRMFaceController(vrm) — raw morph targets by exact name or regex pattern: setMorph, setMorphByPatterns, matchMorph, findMorphTargets, plus VRoid Fcl_* group getters. Use this for morphs the expression system doesn't manage (brows have no standard preset at all).

See docs/face-example/ and docs/expression-example/ for the full API reference and a live demo of each layer.

3. Vowels & visemes (lip-sync)

Lip-sync is face.viseme — the VRM standard defines exactly five vowel presets, and this package's fuzzy fallback resolves the same five letters across non-standard rigs too:

face.viseme.set("aa", 0.8);              // single vowel, 0..1
face.viseme.setAll({ aa: 0.6, ih: 0.1 }); // blend; zeroes every other viseme
face.viseme.get("aa");                    // current weight
face.viseme.reset();                      // silence

| Viseme | Sound | Standard VRM preset | Fuzzy fallbacks tried (in order) | | --- | --- | --- | --- | | aa | "ah" (open) | aa | Fcl_MTH_A, MTH_A, mouth_A, viseme_A, vrc.v_*.A | | ih | "ee/ih" | ih | Fcl_MTH_I, MTH_I, … | | ou | "oo" | ou | Fcl_MTH_U, MTH_U, … | | ee | "eh" | ee | Fcl_MTH_E, MTH_E, … | | oh | "oh" | oh | Fcl_MTH_O, MTH_O, … |

The fallback patterns are end-anchored on purpose — an unanchored MTH_A would also match Fcl_MTH_Angry, so viseme.set("aa", 1) would silently also drive the angry-mouth morph. If you're driving lip-sync from an audio analyser or a TTS viseme stream, feed the five weights straight into setAll() every frame; nothing here manages timing/interpolation for you.

For a model with no standard expressions at all, face.viseme.list() tells you which of the five this specific model can actually show, so you can build UI (or skip visemes it doesn't support) instead of guessing.

4. Mixamo animation retargeting

Mixamo animations are authored against Mixamo's own skeleton — bone names like mixamorigHips, and Mixamo's own rest pose — so a downloaded .fbx clip can't play on a VRM directly. This package resolves that with a retargeter that targets vrm.humanoid's normalized bone rig: every VRM exposes this rig at an identical, identity rest pose, which is what lets one retargeting formula work across any VRM model regardless of its actual raw skeleton proportions or orientation.

import { loadVRM, loadMixamoAnimation } from "@fano/vrm-controller";

const avatar = await loadVRM("/avatar.vrm", { scene });
const mixer = new THREE.AnimationMixer(avatar.vrm.scene);

const clip = await loadMixamoAnimation("/walk.fbx", avatar.vrm);
mixer.clipAction(clip).play();

avatar.registerUpdate((delta) => mixer.update(delta)); // before vrm.update

vrm.update(delta) (called by avatar.update) already copies the animated normalized-bone pose onto the model's real bones every frame (humanoid.autoUpdateHumanBones), so nothing else is needed to see it move.

Two layers, same as face control:

• loadMixamoAnimation(url, vrm, options?) — convenience, end-to-end: fetches and parses the .fbx with three's FBXLoader, picks a clip (options.animationName, or the first one), and retargets it.
• retargetMixamoToVRM(vrm, mixamoRoot, clip, options?) — the engine itself, for callers who already have the FBX loaded some other way (your own FBXLoader instance, a cache, drei's useFBX). mixamoRoot must still be in its bind/rest pose when this is called — read it immediately after loading, before any mixer evaluates a clip against it.

Bone matching resolves Mixamo's standard mixamorigHips-style names, plus common renamed variants (mixamorig1Hips / mixamorig2Hips from multi-character scenes, mixamorig:Hips from some re-exports). Pass boneMap to override or extend the mapping for a non-standard rig:

await loadMixamoAnimation("/walk.fbx", avatar.vrm, {
  boneMap: { Root: "hips" },
  hipsPositionScale: 0.9, // override the auto-computed hips scale, if it looks off
});

No playback/crossfade controller is included on purpose — loadMixamoAnimation hands you back a plain THREE.AnimationClip and stops there, so the full three.js animation API (crossfading, weights, time-scaling, additive blending) is yours to use directly:

const idleAction = mixer.clipAction(idleClip);
const walkAction = mixer.clipAction(walkClip);

idleAction.play();
walkAction.reset().play();
idleAction.crossFadeTo(walkAction, 0.3, false); // blend idle -> walk over 0.3s

See docs/mixamo-example/ for the full API reference and a live demo.

5. Body posing — createVRMBody

Direct control over every joint of the humanoid skeleton — the "body" counterpart to createVRMFace. Like the Mixamo retargeter, it targets vrm.humanoid's normalized bone rig, so posing code is portable across models:

import { loadVRM, createVRMBody } from "@fano/vrm-controller";

const avatar = await loadVRM("/avatar.vrm", { scene });
const body = createVRMBody(avatar.vrm);

body.listBones();                          // every joint this model actually has
body.setEulerDegrees("leftUpperArm", { z: -45 }); // raise the left arm
body.getRotation("head");                   // read a joint back (quaternion)

const pose = body.getPose(); // snapshot every joint, relative to rest
body.resetPose();            // back to rest (T-pose)
body.setPose(pose);          // restore the snapshot

Like expression weights, a write is only recorded immediately — it reaches the mesh on the next vrm.update(delta) (already called for you by avatar.update), since that's what copies the normalized bones VRMBody writes to onto the model's actual raw bones. There's no separate update() method on VRMBody itself to remember to call, though — unlike face, body posing has no time-based state (no equivalent of the auto-blink FSM).

setEulerDegrees takes degrees (not raw quaternion components) and merges partial input against the joint's current rotation, so three independent X/Y/Z sliders can each drive one axis without clobbering the other two. Like any Euler-angle scheme, driving the middle axis to exactly ±90° hits gimbal lock — the outer two axes' sum survives, not each individually. Not every VRM exposes every joint (only 15 are mandated by the VRM spec — fingers, shoulders, toes, eyes, and jaw are all optional), so build UI from listBones() rather than assuming a fixed set.

resetBoneRotation(name) is always quaternion.identity(), never a per-model lookup — because the normalized rig rests at identity by construction, the same fact that makes Mixamo retargeting work.

6. Procedural animation — motion without a .fbx

Everything above either poses the avatar once or replays motion somebody else authored. This generates motion at runtime, from composable signal functions:

import {
  loadVRM,
  createIdleAnimator,
  createLookAtAnimator,
  createGestureAnimator,
  createProceduralAnimator,
} from "@fano/vrm-controller";

const avatar = await loadVRM("/avatar.vrm", { scene });

const gestures = createGestureAnimator({ wave: WAVE_KEYFRAMES });
const idle = createIdleAnimator();               // breathing, sway, micro-wander
const look = createLookAtAnimator({ target: camera }); // head/neck/chest turn

const anim = createProceduralAnimator(avatar.vrm, [gestures, idle, look]);
avatar.registerUpdate((delta) => anim.update(delta)); // the same seam as always

gestures.play("wave");

One rig, one owner. A Mixamo AnimationMixer and a procedural animator both write vrm.humanoid's normalized rig, and the procedural animator writes every bone it owns on every frame — so whichever runs later in registerUpdate order overwrites the other. Use pre-authored clips or procedural animation, not both at once. Switching at runtime is fine: call anim.reset() first, which returns every bone the stack owns to rest before the mixer takes over. (Face control is unaffected either way — expression weights and bone poses never overlap.)

Four layers, each usable on its own:

signals      pure fns of time      sine / noise / spring / envelope
   ↓
channels     signal → bone+axis    channel("spine", "x", sine({ hz: .25, amp: 3 }))
   ↓
animators    { update(ctx) }       createAnimator(fn) — the imperative escape hatch
   ↓
behaviors    ready-made            idle / look-at / gesture

| Behavior | What it does | | --- | --- | | createIdleAnimator | Breathing, weight-shift sway, seeded micro-wander — at unrelated rates so the loop never becomes visible. Optional hip bob (off by default: it lifts the feet). | | createLookAtAnimator | Bone-level head/neck/chest turn toward a target, clamped and spring-smoothed. Composes with the eye-only face.eye.gaze() — eyes lead, head follows. | | createGestureAnimator | Named keyframe timelines (play/stop/loop/crossfade). A non-looping gesture releases itself so the idle takes back over. |

Animators write to a PoseSink, never to a VRMBody directly. That one indirection is what lets the same animator drive the live model or be sampled offline — which is what bakeAnimator does:

const clip = bakeAnimator(avatar.vrm, createIdleAnimator(), {
  duration: 8,
  fps: 30,
  loop: true, // last sample == first, so LoopRepeat has no seam
});

mixer.clipAction(clip).play(); // a plain THREE.AnimationClip, same as Mixamo's

Baking is reproducible — nothing in the signal library calls Math.random() (noise is seeded instead), the animator is reset before sampling, and the timestep is fixed. It also never touches the live model, so an avatar on screen doesn't twitch while a clip is generated.

Composition rule: within a layer, channels on the same bone sum their degrees and convert to a quaternion once; across layers, per-bone quaternions multiply in stack order. Every frame writes an absolute rotation derived from rest, so there's no drift and no gimbal-lock round-trip — and a bone nothing drives any more relaxes to rest instead of sticking.

One authoring convention, both spec versions. A VRM 1.0 rig faces +Z and a VRM 0.x rig faces -Z, so a pose authored for one is mirrored on the other — an arm raise becomes an arm drop. Everything here is authored in the VRM 1.0 convention (+Z forward, +X model-left) and converted at the write boundary, by the same rule retargetMixamoToVRM applies to Mixamo clips. So one gesture library works on both. Reach for createRigFrame(vrm) if you're computing directions yourself in a raw createAnimator callback.

See docs/procedural-example/ for the full API reference and a live demo that toggles each idle component, tracks the camera, fires gestures, and flips between the live animator and a baked clip of the same motion.

Putting it all together

docs/full-example/ combines all of the above — one VRM, driven by face control, lip-sync, Mixamo animation, and body posing at the same time, through the same registerUpdate seam:

const avatar = await loadVRM("/avatar.vrm", { scene });
const face = createVRMFace(avatar.vrm);
const body = createVRMBody(avatar.vrm);
const mixer = new THREE.AnimationMixer(avatar.vrm.scene);

avatar.registerUpdate((delta) => {
  face.update(delta); // auto-blink — owns expression weights
  mixer.update(delta); // Mixamo playback — owns bone poses
});

body.setEulerDegrees("head", { y: 15 }); // recorded now, same flush rule as everything else

body has no update() method of its own to call (unlike face.update(delta), which the auto-blink FSM needs) — but its writes still only reach the mesh on the next vrm.update(delta), exactly like face.expression.set(...): that's what copies the normalized bones body writes to onto the model's actual raw bones (humanoid.autoUpdateHumanBones). Since avatar.update(delta) already calls vrm.update every frame, this is invisible in practice — just don't expect a setEulerDegrees call to show up before the next render. Since expressions and bone poses never overlap, everything above runs independently without any coordination beyond "run before vrm.update." See docs/full-example/full-example.md for the live demo.

The one pairing that doesn't combine is Mixamo playback and procedural animation — both own the normalized bone rig, so pick one per model (see section 6). Everything else layers freely.

React Three Fiber

import { Vrm } from "@fano/vrm-controller/r3f";

<Canvas>
  <Vrm url="/avatar.vrm" onLoad={(avatar) => console.log(avatar.vrm)} />
</Canvas>;

<Vrm> loads the model, drives vrm.update(delta) via useFrame, and disposes on unmount/url change. useVRM(url) is the hook it's built on, if you want the loading state without the component.

7. Solid

import { createVRM, createFrameLoop } from "@fano/vrm-controller/solid";

const vrm = createVRM("/avatar.vrm", { scene });

createFrameLoop((delta) => {
  vrm.avatar()?.update(delta); // the same one-call-per-frame rule as everywhere
  renderer.render(scene, camera);
});

Unlike the R3F entry, this one depends on nothing but solid-js — there's no mature Solid renderer for three.js to lean on (solid-three still calls itself early development), so you keep your own renderer, scene, and camera, and these primitives manage what goes into that scene and when it goes away. The /solid entry ships no JSX, so no babel-preset-solid pass runs over dist/.

| Primitive | What it does | | --- | --- | | createVRM(url, options?) | Reactive load. Tracks a url signal, attaches to a scene, disposes on url change / unmount — including loads that land after the url changed again. Exposes avatar / loading / progress / error, plus resource for <Suspense>. | | createFrameLoop(cb, options?) | The rAF loop Solid doesn't come with. Seconds, not ms; clamps the delta so a backgrounded tab can't detonate spring bones; pausable; cancels on cleanup. | | createVRMUpdate(avatar, fn) | The reactive form of registerUpdate — re-registers when the avatar changes identity, so a model swap can't leave a driver bound to a disposed rig. | | Vrm | Declarative mount: <Show> / <For> become the lifecycle. Renders nothing to the DOM. |

Everything in sections 1–6 works from Solid without a Solid-specific wrapper, because none of it is framework-specific. Three ideas cover all of it:

// 1. A memo keyed on the avatar builds the controller — it rebuilds on a swap.
const face = createMemo(() => {
  const avatar = vrm.avatar();
  return avatar ? createVRMFace(avatar.vrm) : null;
});

// 2. createVRMUpdate registers whatever needs a frame.
createVRMUpdate(vrm.avatar, (delta) => face()?.update(delta));

// 3. State lives in signals; the model is a projection of it.
createEffect(() => face()?.emotion.setExclusive(mood()));

The third is the one that pays for itself: because the effect depends on both the controller and the choice, swapping models re-applies every control — the expression, the gesture that was playing, the joints you posed — with no resynchronisation code. Apply a control in a click handler instead and it silently vanishes on the next load.

The exception is the rig itself. A Mixamo AnimationMixer, a procedural animator, and manual createVRMBody posing all write the normalized bone rig (section 6's one rig, one owner rule), so that's a mode signal with a handover, not three independent toggles — face control is unaffected either way and runs in every mode.

See docs/solid-example/ for the full API reference, SSR notes, a recipe per feature — expressions, brows, visemes, gestures, retargeting, baking — and a live demo that runs all of it at once and lets you swap the model underneath it.

Development

bun install
bun run build          # tsup -> dist/ as esm + cjs + .d.ts
bun run dev             # same, in watch mode
bun run test            # vitest run
bun run test:watch      # vitest watch

Tests run in Node with no headless-GL setup. They fake vrm / vrm.humanoid (a plain object cast to VRM) but use real three instances for the math — quaternions, Euler conversions, keyframe tracks — so the rotation and retargeting logic is checked against three.js's own algebra rather than a reimplementation of it. The Solid tests stub loadVRM, requestAnimationFrame and window/document, but run real Solid reactivity — which is why vitest.config.ts pins resolve.conditions to ["browser", "development"]: under Node's own resolution solid-js serves its SSR build, where every reactive primitive is an inert stub and the tests would silently pass nothing. Nothing renders, so the live docs/*-example/*.html demos are how you actually see any of this on screen (bunx serve ., then open the demo you want).

License

This project is licensed under the Apache License 2.0 with additional attribution terms.

You may use, modify, and distribute this package for free, including in commercial projects, as long as proper credit is preserved:

Fano VRM Controller by Ayush Pradhan

See LICENSE and NOTICE for details.