@flyos/design-system-twin-face

In-browser, client-side, audio-driven 3D talking-face lip sync for the FlyOS agent-panel voice overlay. Zero backend or wire changes — it taps the audio the Twin is already speaking.

Deliberately NOT part of @flyos/design-system nor its Native Federation singleton. It carries heavy optional deps (three, onnxruntime-web) that must never be forced onto every remote. Do not merge it into the design system "for tidiness".

Pipeline

<audio> (AgentAudioPlaybackService)
   │  MediaElementAudioTap.connect() — createMediaElementSource, once, reconnect to destination
   ▼
AudioWorklet resample → 16 kHz mono
   ▼
Wav2ArkitDriver (onnxruntime-web, RMS-normalised) → 52 ARKit coeffs @30fps
   ▼
<fly-twin-face> (three.js WebGPU/WebGL) → facecap morph targets, by NAME
   ▲
VoicePhase → procedural life layer (blink / saccade / breath / head-sway)

ARKIT_52 is the swappable seam between asset, driver, and renderer.

Usage

<!-- Speaking mount (voice overlay) -->
<fly-twin-face
  [enabled]="twinFace() === 'character'"
  [phase]="voicePhase()"
  [audioElement]="playback.audioElement()"
  [appearance]="twinAppearance()"
  [reducedMotion]="prefersReducedMotion()"
  (fallback)="showOrb($event)" />

<!-- Non-speaking mount (a settings preview). `audioDriven=false` keeps the
     large on-demand wav2arkit weights from being fetched for a face that will
     never lip-sync; the procedural life layer still blinks and breathes. -->
<fly-twin-face
  [enabled]="open()"
  [audioDriven]="false"
  [appearance]="draft()"
  [phase]="'idle'" />

Appearance

TwinAppearance (lib/twin-appearance.ts) is the serialisable customization model — skin, eyes, gaze, proportions, skin detail, stage/render — plus the palettes and per-field ranges. It imports nothing from three, so a settings UI can render swatches and sliders without pulling the 3D bundle.

Always read it back through normalizeTwinAppearance: every field lands on a shader uniform or a node transform, and it clamps per-field so one bad value never discards the user's other choices. Applying an appearance is live — the scene writes uniforms and node transforms, so nothing remounts.

Accessories

Twenty-three GLBs under assets/twin-face/accessories/, generated by twin-face-lab/blender/ and gated against the shipped bytes by verify-accessories.mjs. They mount in four independent slots — a headwear pick does not clear the hair, a face cover does not clear the headwear, and that independence is what lets hijab + niqab and shayla + earrings compose:

| Slot | Field | Count | Colour | |---|---|---|---| | hair | hairStyle | 10 | hairColor | | headwear | headwear | 7 | headwearColor | | faceCover | faceCover | 2 | headwearColor (shared — a niqab is cut from the shayla it is worn with) | | earrings | earrings | 4 | none — every earring material is authored _fixed |

Four things about them that are easy to break and hard to see:

• COLOR_0 is not a colour. It carries the sway rig (weight/phase/lag), and the glTF spec says that attribute multiplies base colour — so GLTFLoader turns vertexColors on and every accessory renders as tie-dye until buildAccessoryMaterial turns it back off.
• The material must be TRANSFERRED, not copied. Material.copy knows nothing about maps, sheen or anisotropy, so the obvious implementation drops the whole bake and still renders — just flat. The allow-list in accessory-rig.ts is enumerated for that reason.
• The tint is gain-compensated. Base-colour textures are mid-grey luminance maps (hair measures 0.115 mean linear), so tint × map lands an order of magnitude below the swatch the user clicked. The gain is measured from the map at load, never tabulated — the hair albedo's mean moved 0.572 → 0.163 the day white-clipping was fixed, and a constant would have gone quietly wrong.
• Hair tucks. Every hair GLB carries one tucked morph target at influence 0, eased to 1 whenever a crown covering is worn. The seven coverings share one innermost envelope, so a single pose per style covers all seventy pairs.

gender narrows the four pickers (unspecified, the default, offers everything). It is a presentation field: nothing about the head mesh, skin or voice reads it. It carries one hard rule — a male wearer is offered no earrings at all, enforced in stylesForSlot and therefore in normalizeTwinAppearance, not merely hidden in the UI.

twin-accessory-catalogue.spec.ts reads the shipped GLBs and fails if the TypeScript catalogue and the Blender output disagree on ids, tucked or _fixed. Geometric invariants (sway weights, head-space bounds) stay in verify-accessories.mjs, which can decode accessors.

Contributing to the renderer

Import three ONLY from three/webgpu — never bare three, never three/tsl. three/webgpu re-exports the whole core and TSL; a second copy gives the TSL builder two incompatible node systems, it recurses until it throws Maximum call stack size exceeded, and a material whose build throws renders pure black. Enforced by lib/face/three-single-instance.spec.ts.

When a face looks wrong: read the browser console first, confirm the render loop is running (requestAnimationFrame doesn't fire without compositing — use FaceScene.renderOnce()), and only then measure pixels. Full playbook and the measured renderer facts: skills/twin-face.md.

Delivery

Assets are served same-origin (no CDN, strict CSP) from assets/twin-face/. The GLB ships in the package; the ~400 MB ONNX weights are never in git (.gitignore + .dockerignore) and arrive by exactly two routes, which must stay in step:

• Dev box — init.sh / init.ps1 (download, copy ORT wasm out of node_modules, generate the int8 fallback when python+onnxruntime exist).
• Image — the twin-model → twin-model-int8 stages in src/frontend/desktop-app/Dockerfile. The int8 build is generated there by tools/quantize-model.py, so moving that script breaks the image build.

quantize-model.py is the only tool left under tools/, and that is the reason. The accessory modelling studio that used to sit beside it (Blender Python, reference meshes, the head exporter and the GLB verifier) moved to twin-face-lab/blender/ — it generates assets rather than shipping in this package, and ng-package.json never published it in the first place.

The ORT .wasm/.mjs are copied from the same npm install as the app bundle — a mismatched pair fails at session init. onnxruntime-web needs 'wasm-unsafe-eval' in script-src (prod).

A11y

prefers-reduced-motion ⇒ static portrait; the transcript stays the accessible surface; the orb is always the fallback and the default presence.

Supported asset formats

The runtime loader is three.js GLTFLoader — it loads GLB/glTF only. Other formats you may encounter while sourcing a face asset:

• .blend — a Blender editor project, never a runtime format. Export to GLB before it can be used here.
• .fbx — loadable via three.js FBXLoader, but morph-target fidelity is heavier and flakier than glTF's. Convert to GLB; .fbx is not wired into v1.
• .vrm — glTF-based and loadable via @pixiv/three-vrm, but it drives VRM's own expression set (aa/ih/ou/ee/oh + emotion blends), not ARKit-52. Using a VRM avatar would need an ARKit-52 → VRM-expression adapter in front of the renderer. Not implemented; a viable future avatar-source option.

The rule that actually matters is not the container but the rig: any GLB with a named, driveable ARKit-52 blendshape set (see ARKIT_52 in twin-face.contracts.ts) drops in with zero code changes. A beautifully modeled GLB without matching morph-target names is not usable — this is what disqualified the Sketchfab test models during the Wave-0 spike.