mtr-facemesh

Contents

• Description
• Usage
• Features
• Authoring an eyeliner
• Authoring a glitter
• Demo
• WASM Compilation

Description

The current repository is the mtr-facemesh project. The module projects 3D masks (in this case make ups) on a face from a video element detected by an AI model.

Usage

In order to use the module, add the div with id="facemesh-holder" and set the width and height of the canvas to show.

<div id="facemesh-holder" style="width: 282px; height: 500px;"></div>

After that, you can create the FaceMesh object with the following.

import {FaceMesh, THREE} from './lib/mtr_facemesh.js';

...

const config = {
  divName: "facemesh-holder",
  modelConfigs: undefined,
  meshConfigs: [
    {
        alphaMapDir: './assets/imgs/face/mask_lips.jpg',
        ambientOcclusionMapDir: './assets/imgs/face/ambient_occlusion.png',
        normalMapDir: './assets/imgs/face/normal_texture_lips_3.png',
        color: 0xff0000,
        roughness: 0.25,
        metalness: 0.1,
        opacity: 0.3,
        clearcoat: 0.5,
        clearcoatRoughness: 1.0,
    }
  ],
  binaryDir: './lib',
  refineLandmarks: true,
  backend: 'webgl',
  flipCamera: true,
  verbose: true,
  showFPSPanel: true,
  showScreenLogs: true,
  loadedCallback,
  errorCallback,
  cameraStatusCallback,
  oneEuroFilterConfig: {
    mincutoff: 0.005,
    beta: 0.12,
    dcutoff: 1,
  },
  faceRetouchConfig: {
    retouchAlphaMask: './assets/imgs/face/facemesh_alpha_base_v4.jpg',
    opacity: 1.0,
    blurRadius: 10,
  },
};

faceMesh = new FaceMesh(config);

Where config is an object with properties:

• divName HTML div ID of the AR holder. Default is facemesh-holder.

• imageID HTML image element ID that will replace the camera.

• modelConfig object of configuration for the 3D model:

  • dir: 3D model file path. File must be .glb or .gltf. No default.
  • animationSpeed: 3D model animation speed. (0, 1) for slower animations and animationSpeed > 1 for faster animations. Defaults to 1.
  • scale: Sets how big the 3D model will be. Defaults to 1.
  • position: position of the model considering camera is at {0, 0, 0}. Defaults to global {0, 0,-distance}.
  • rotation: rotation of the model. Defaults to {0, 0, 0}.
  • loop: if the model should loop. Default is true.
  • showMannequin: show the head occluder to better position the object. Default is false.

• meshConfigs Array of configuration for each make up mesh:

  • legacy: if set to false the layer renders through the custom DynamicMaterial shader (video-derived relief normal and ambient occlusion, PMREM env map, frame light probe) instead of three's MeshPhysicalMaterial. Note this is opt-out: anything other than an explicit false uses the legacy material. Default is true.

  • region: render this layer on its own geometry, generated each frame from the landmarks, instead of on the shared face mesh. 'eyeLidUpper' and 'eyeLidLower' build a ribbon along the corresponding tracked lash line of both eyes, with a rectangular parametrisation of its own: u is normalised arc length along the lash line (0 at the inner canthus, eyeStrip.cornerU at the outer one, 1 at the wing tip) and v is the offset away from the aperture, 0 at the lash margin and 1 at the far edge of the stroke. Artwork is therefore authored as a flat strip — see demos/assets/imgs/face/eye_strip_uv_template.png — rather than as a sliver of the canonical face atlas.

    One region per layer. An upper and a lower liner are two meshConfigs entries, which keeps one entry = one mesh and lets each carry its own artwork and stroke width; a lower liner is normally thinner and gets no wing by default.

    Note the shape must reach the shader through alphaMapDir, not colorMapDir, if you want color to tint it. colorMapDir is multiplied as a vec4, so a map whose RGB is black inside the stroke — which is true of every eyeliner asset in this repo — multiplies color away. alphaMapDir is read on the green channel; tools/bake_eye_strip.py --emit mask and tools/make_liner_strip.py both write that form.

    This is what an eyeliner needs, because on the shared face mesh a liner inherits two defects that no amount of UV work removes. The arrayExpand table in src/face_mesh_geometry.ts moves each lid crease vertex to the midpoint toward the lash vertex under it while the UV table stays put, which compresses the artwork exactly 2x across the band; and the band's 3D area swings about 2.6x during a blink against a fixed UV area, which smears the artwork down the lid. Both live in a free scale factor that defining v against a pose-invariant length removes.

    Requires legacy: false: MeshPhysicalMaterial ignores the vertex color attribute unless vertexColors is set, which would silently drop the seam feather. The layer is skipped with a verbose note otherwise. No default.

  • eyeStrip: tuning for region: 'eyeLidUpper'. All fields optional.

    • columns: spine samples per eye, wing included. Sets the residual faceting: 48 gives 36 samples over the eye and about 3.5 deg of turn per segment, against 14.8 for the 8-segment landmark polyline. Default is 48.
    • rows: samples across the band. Default is 4.
    • cornerU: u of the outer canthus; everything above it is the wing. It is snapped to the nearest real column and the snapped value is readable back off the geometry. Default is 0.75.
    • strokeWidth: band width as a fraction of eye width (inner canthus to outer canthus). Eye width is pose-invariant, so this is a constant physical stroke: 0.11 is about 2.1 mm on an adult. Default is 0.11.
    • wingLength: wing length, also a fraction of eye width. Default is 0.2.
    • wingLift: wing direction, 0 straight out along the lash tangent, 1 fully up the lid. Default is 0.5.
    • lift: offset toward the camera along the surface normal, as a fraction of eye width. Parallax only — depth correctness comes from depthWrite: false plus render order. Default is 0.02.
    • foldOpenFrac: fraction of the authored stroke visible with the eye wide open. Default 1.0, i.e. off. Lid skin rolls under the supratarsal crease as the eye opens, so lowering this makes a blink reveal more of the stroke instead of stretching a fixed range over more skin. The direction is established; the magnitude is not, so it ships off — v = 1 is the band exposed at the canonical rest pose, which is the open-eye reference tools/bake_eye_strip.py targets, and a value below 1.0 hides art the bake placed. Tune it live in demos/controls against a real blinking face. Width and v range always scale together, so skin offset = v * strokeWidth * eyeWidth holds at every lid state. Leave it at 1.0 on 'eyeLidLower': the fold is an upper-lid phenomenon, the lower lid barely moves during a blink, and the aperture signal cannot tell the two apart.
    • foldAttackMs / foldDecayMs: time constants of the running open-eye reference that normalises the aperture across people — fast to learn a wider eye, slow to forget one, so a blink never drags it down. Defaults 150 and 8000.
    • seamFeather: alpha ramp at the ribbon's free boundaries, as a fraction of the band (for v = 1) and of the strip length (for the two ends). The lash margin at v = 0 is never feathered. Set to 0 if the artwork already fades at its edges. Default is 0.12.

  • colorMapDir: texture file path. File must be an image. No default.

  • alphaMapDir: alpha mask file path. File must be an image. No default.

  • ambientOcclusionMapDir: illumination mask file path. File must be an image. No default.

  • normalMapDir: normal map file path. File must be an image. No default.

  • color: make up color in hexadecimal. Default is 0xffffff.

  • metalness: how much the material is like a metal. Wood or stone use 0.0, metallic use 1.0. Default is 0.0.

  • opacity: float in the range of 0.0 - 1.0 indicating how transparent the material is. Default is 1.0.

  • clearcoat: use clear coat related properties to enable multilayer materials that have a thin translucent layer over the base layer. Default is 0.0.

  • roughness: represents the intensity of the clear coat layer, from 0.0 to 1.0. Default is 0.0.

  • clearcoatRoughness: roughness of the clear coat layer, from 0.0 to 1.0. Default is 0.0.

  • ior: index of refraction, which sets how reflective the layer is — F0 = pow2((ior - 1) / (ior + 1)), so the default 1.5 gives the usual 4% dielectric reflectance and raising it reflects more. Default is 1.5.

  • reflectivity: inert — the uniform exists but no shader line reads it. It is still accepted so existing configs keep working unchanged. Use ior for how reflective the layer is, and specularIntensity to scale it.

  • envMapDir: static HDR environment map file path for this layer, giving it image-based lighting. No default.

  • normalScale: strength of the combined video-derived and asset normal perturbation. Lower it toward 0 to flatten the relief and calm reflection noise at low roughness. legacy: false only. Default is 1.0.

  • frameLight: if set, the camera frame itself becomes the light source for this layer: the real per-pixel shading in the frame is fed to the material as irradiance and radiance, so roughness, metalness, IOR and clearcoat shape the measured light exactly as they shape environment-map light. legacy: false only. Merely being present switches the illumination on, so frameLight: {} is enough — every field below then takes its value from DEFAULT_FRAME_LIGHT_CONFIG (exported by the library), which is the single source of truth for these defaults. Omit frameLight entirely to leave the illumination off. Because the probe replaces the scene lights rather than supplementing them, the defaults are calibrated with the directional and ambient intensities at 0 (see autoLightIntensity below). Each field has exactly one job, listed below with what it acts on. Changing one should never alter another's territory.

    • 'irradiance' (diffuse): ambient level taken from the frame. The frame's own per-pixel shading, normalised by the measured region mean so it is independent of skin tone and exposure. This is the term that lights the layer; it adds to the ambient light, so zero the scene lights to see the probe alone. Default is 2.64.
    • 'gain' (specular): overall highlight brightness. A plain monotone multiplier over the whole highlight — it does not move or resize it. Default is 1.5.
    • 'hdr' (specular): highlight contrast, i.e. how much brighter the peak is than the rest of the lit region. 0.0 lights the region evenly. It shapes the brightness profile and never the extent. Default is 4.0.
    • 'region' (specular): how much of the make-up is highlighted, in units of the measured mean-to-p85 range. 1.0 means only pixels above the p85, which is already a very small core; useful values sit below that, and 0.0 highlights everything above the mean. Default is 0.6.
    • 'softness' (specular): edge softness of that region, in the same units. The region's centre does not move as this changes. Default is 0.35.
    • 'detail' (specular): amount of fine real specular sparkle — the high-frequency texture the broad region averages away. Added on top of the region rather than traded against it, so the two are independently visible. Default is 0.35.
    • 'radius': radius of the local average that separates the fine detail from the broad region, in x UV units (scaled by aspect in y). Too small and 'detail' is only sensor noise; this cannot reuse the shared blurred video texture, which is about 1.6 px because it feeds the relief normal. Requires gaussianSobelSmoothing to be negative (a blur). Default is 0.0135.
    • 'sheer' (albedo): how much of the real lip or skin chroma shows through the make-up colour. Hue only — the brightness already arrives as irradiance, so this does not double-count the shading. 0.0 is opaque. Default is 0.5.

    Only 'irradiance' and 'sheer' are calibrated values; the specular defaults are starting points. demos/controls exposes all of them live, plus a DEBUG view that replaces the layer's output with one intermediate term (the shading field, the region mask, the detail, the irradiance or the full radiance) so a single parameter can be checked in isolation.

• binaryDir Path to the face mesh binary files. No default.

• refineLandmarks Refine landmarks option. Default is true.

• backend Tensorflow backend: webgl, wasm or cpu. Default is webgl.

• flipCamera Flips the feedback camera. Default is false.

• verbose If set to true shows logs on the console. Default is false.

• loadedCallback Function to call after the AI model is loaded. No default.

• cameraStatusCallback Function to call after the video permission (no default):

  • 'requesting': when user is requested for the camera
  • 'hasStream': when user is granted access for the camera
  • 'failed': when something went wrong with the camera

• errorCallback Function to call after any error. No default.

• showFPSPanel If set to true shows the FPS on the main loop. Default is false.

• showScreenLogs If set to true shows the FPS for every step on the main loop. Default is false.

• oneEuroFilterConfig One Euro Filter parameters.

  • 'mincutoff': minimum cutoff frequency. Decreasing the minimum cutoff frequency decreases slow speed jitter. Default is 0.001.
  • 'beta': speed coefficient. Increasing the speed coefficient decreases speed lag. Default is 0.007.
  • 'dcutoff': cutoff frequency. Default is 1.

• faceRetouchConfig face retouch parameter (one object).

  • 'retouchAlphaMask': path to the alpha mask file. Only apply the retouch on the mask. No default.
  • 'blurRadius': Face retouch intensity. Default is 10.
  • 'color': Foundation color. Default is 0xffffff (white).
  • 'opacity': Foundation opacity. Default is 0.0 (transparent).

• faceRetouchConfigs face retouch parameters array (multiple face retouch objects). Each object of the array must contain:

  • 'retouchAlphaMask': path to the alpha mask file. Only apply the retouch on the mask. No default.
  • 'blurRadius': Face retouch intensity. Default is 10.
  • 'color': Foundation color. Default is 0xffffff (white).
  • 'opacity': Foundation opacity. Default is 0.0 (transparent).

• foundationMatchConfig If set, executes the foundation match algorithm after togglePhoto method.

  • 'option': tone algorithm option.
  • 'colors': array of foundation colors in RGB and its names.

• gaussianSobelSmoothing Smoothing applied to the video before it is used to derive the relief normal, the ambient occlusion and the frame light probe's specular residual. Negative is a blur, positive is a sharpen. The residual needs a blur, so keep this below 0 when using frameLight. Default is -2.0.

• frameLightProbeConfig Overrides for the frame light probe, the CPU luminance measurement that calibrates frameLight. Omit to take DEFAULT_FRAME_LIGHT_PROBE_CONFIG (exported by the library), which is the single source of truth for the defaults.

  • 'sampleSize': side of the square offscreen canvas read back. Default is 96.
  • 'updateInterval': frames between readbacks. Default is 3.
  • 'regionRungs': outer/inner landmark pairs defining the measured region. Defaults to the 22 lip rungs.

The scene's directional and ambient lights are switched on or off automatically at construction. They exist for the legacy lighting path, so they are on when any mesh is legacy (legacy !== false) or a modelConfig is present, and off — both intensities parked at 0 — when every layer is legacy: false and therefore lit by the frame light probe and/or an env map. They are shared by the whole scene, so a single legacy layer anywhere keeps them on for everything. Set autoLightIntensity to override.

While the flag is true, luminance re-derives both intensities from frame brightness every 60 frames. To hold your own values, clear the flag first inside loadedCallback, otherwise they are overwritten within a couple of seconds:

faceMesh.autoLightIntensity = false;
faceMesh.directionalLight.intensity = 0.0;
faceMesh.ambientLight.intensity = 0.0;

The method requestPermissions must be called after the model is loaded using the loadedCallback config property, since the face detector only starts predicting after the video is set. The method receives the camera configuration as a parameter, the same as seen at getUserMedia. Inside loadedCallback you must also add lights to the scene since the textures are very sensible to it. The following code is an example of one loadedCallback.

const loadedCallback = function () {
  scene = faceMesh.getScene();

  const hemiLight = new THREE.HemisphereLight(0xffffff, 0x080820, 0.5);
  scene.add(hemiLight);

  const ambientLight = new THREE.AmbientLight(0x404040, 0.1);
  scene.add(ambientLight);

  faceMesh.requestPermissions({
    facingMode: {ideal: 'user'},
    width: window.innerHeight,
    height: window.innerWidth,
  });
};

With these lines of code it's already possible to run the face mesh. If you want to personalize the threejs scene, you can do it by using the get methods faceMesh.getRenderer(), faceMesh.getScene(), faceMesh.getOrthographicCamera(), faceMesh.getPerspectiveCamera(), faceMesh.getGroup() and faceMesh.getMeshes(). For example, you can change material params with faceMesh.getMeshes()[0].material.color = new THREE.Color(0xff0000);.

Features

The AR object also include other functionalities.

Partial Render

The partial render works like a curtain for the threejs renderer and only renders what is from 0 to windowSlider.value * canvas.width. For this, you need to create a checkbox and a slider from 0 to 1 and then use them as follows.

const windowCheckbox = document.getElementById('window-checkbox');
const windowSlider = document.getElementById('window-slider');

windowCheckbox.oninput = function () {
  faceMesh.setWindow(windowCheckbox.checked);
  windowSlider.style.display = windowCheckbox.checked ? 'block' : 'none';
};

windowSlider.oninput = function () {
  faceMesh.setWindowFactor(windowSlider.value);
};

Photograph

The photograph functionality is simple and works like it is said. When the method togglePhoto is called, the animation loop freezes/unfreezes depending on the actual state. You only need to create on html and use the following code. The method also returns a promise of a canvas of the photograph.

const screenshotButton = document.getElementById('photo-button');

screenshotButton.onclick = async function () {
  const frozenVideoCanvas = await faceMesh.togglePhoto();
};

Share

The next feature complements the photograph feature. When the share method is called, the object takes a screenshot and then share it through the selected social media. For that, you only need to create a button to trigger the method.

const shareButton = document.getElementById('share-button');

shareButton.onclick = function () {
  faceMesh.share('screenshot');
};

Stop and Continue

The application can be stopped or continued after calling .stop() and .continue() methods. They are useful when you want to stop the application with a close button or start the application instantly after the user clicks a start button. The demo shows an example of usage.

const closeButton = document.getElementById('close-button');

var toggleCloseContinue = true;

closeButton.onclick = function () {
  if (toggleCloseContinue) {
    faceMesh.stop();
    closeButton.textContent = 'Continue';
  } else {
    faceMesh.continue();
    closeButton.textContent = 'Close';
  }

  toggleCloseContinue = !toggleCloseContinue;
};

Foundation Color

If faceRetouchConfig and/or faceRetouchConfigs are set, it is possible to set a color on the users face as if there is a foundation using .setFoundation method. The function receives two parameters, color and opacity. The color is a number array with 3 positions in RGB space, and the opacity is a floating point number between 0.0 and 1.0 that is the strength of the color. If there are multiple face retouch objects, you can set the color and opacity with the third parameter index. The following example will make up the face with red foundation.

faceMesh.setFoundation([255, 0, 0], 0.5, 0);

Light Exposure

Light exposure is a feature related to face color match since it detects the illumination on the users face. If the user face is under or over exposed, the faceMesh.exposure() will return -2 and 2 respectively.

<button id="exposure-log" style="position: absolute; left: 5%; top: 15%;">Loading</button>

function anim() {
  document.getElementById(
    'exposure-log'
  ).textContent = `${faceMesh.exposure()}`;
}
setInterval(anim, 500);

Face Color Match

To use it, call the method .getFaceClosestColor() to get the object of closest color. This feature matches the users face color to the closest color in foundationMatchConfig.colors and return an object with attributes foundationID, skinColor and distances.

• foundationID is the index of the best foundation ID in foundationMatchConfig.colors array;
• skinColor is the extracted color of the user skin in RGB space;
• distances This attribute is also an object with attributes distance and index. The index is the index to the foundationMatchConfig.colors array and the distance is the euclidean distance from foundationMatchConfig.colors[foundationID] to foundationMatchConfig.colors[distances[..].index]. This array is sorted in ascending order by the distance attribute.

FaceMesh(
  ...,
  foundationMatchConfig: {
    option: 1,
    colors: [
      {color: [203, 176, 145], name: '00'},
      {color: [169, 129, 83], name: '50'},
      {color: [77, 52, 48], name: '100'},
    ]
  }
)

window.afterScreenshot = function afterScreenshot() {
  // matches the face color with one of the 3 colors set in foundationMatchConfig
  const distances = faceMesh.getFaceClosestColor().distances;
  // second best color distance, and index
  console.log(distances[1].distance, distances[1].index);
  const foundationId = faceMesh.getFaceClosestColor().foundationID;
  console.log(window.foundations[foundationId], foundationId);
  // uses the setFoundation feature to set the matched color on faceRetouch as a foundation
  faceMesh.setFoundation(window.foundations[foundationId].color, 0.3, 0);

Using an image instead of camera

If faceMesh is created with imageID set, the video camera will be replaced with the image loaded on html with id=<imageID> and .requestPermissions() method will not be necessary to call. For example, an image tag is loaded with id "photo-id", and faceMesh is created with it replacing the camera.

<img id="photo-id" src="./any-image" />

faceMesh = new FaceMesh({
  imageID: "photo-id",
  ...
});

Authoring an eyeliner

A region: 'eyeLidUpper' / 'eyeLidLower' layer takes its artwork as a flat strip: x is arc length along the lash line, y is offset across the lid, and v = 0 — the lash margin — is the bottom row of the image. You do not have to draw in that space; if the art already exists in the canonical face-UV atlas, one command reprojects it.

There are two routes to a strip: the studio, in the browser, or the two Python tools at a prompt. They do the same arithmetic — tools/check_strip_bake.py diffs them — so pick whichever suits the person doing the work.

The studio (demos/eyeliner-studio/)

npx http-server .          # from the repo root, so /src is reachable
# then open http://localhost:8080/demos/eyeliner-studio/

Three panels: paint the strip on the left, watch it on a tracked face in the middle, tune the layer on the right. Bake → face pushes the drawing into the running material with no file round trip (and auto after each stroke does it for you), so a taper can be judged on a blinking eye rather than guessed at.

• Import takes either space. A wide image is loaded as a strip; a squarish one is treated as canonical-atlas art and reprojected on the way in — the same bake tools/bake_eye_strip.py does, using whatever strokeWidth, cornerU and wingLength the ribbon is currently set to. Both mask conventions are detected: an image with no transparent pixel is read as a mask (shape in RGB), anything else as art (shape in alpha).
• Generate a base stroke is tools/make_liner_strip.py in the page, for when there is no art to start from.
• The guides are read live off the geometry, and they are the point of doing this in a browser at all: v labelled in millimetres of skin, where the outer canthus falls, which bands the ribbon feathers away, and — the one that catches people — the foldOpenFrac line, above which nothing shows while the eye is open.
• Download strip PNG writes the same file the Python tools write, and copy config snippet gives you the meshConfigs entry with the strokeWidth the art was actually drawn against, which is the trap below made survivable.

The paint canvas is 1024×128 whatever it is displayed at, so the pointer maps straight to strip texels; zoom v only stretches the view, because a strip shown at its true 8:1 is unpaintable.

Bake existing atlas art into a strip

python3 tools/bake_eye_strip.py demos/assets/imgs/face/color_map_eye_liner.png \
  --emit mask --smooth-u 6 \
  --out demos/assets/imgs/face/eye_liner_upper_mask.png

Needs only numpy and Pillow. Then point a layer at it:

{
  legacy: false,                 // required with region
  region: 'eyeLidUpper',
  alphaMapDir: '../assets/imgs/face/eye_liner_upper_mask.png',
  color: 0x14100f,               // this is what tints it
  envMapDir: '../assets/imgs/face/studio2.hdr',
  eyeStrip: {strokeWidth: 0.11}, // MUST match the bake's --stroke
}

The flags that matter:

• --emit mask — almost always what you want. It writes the shape into RGB and makes the image opaque, which is the form alphaMapDir needs, because the shader reads alphaMap on the green channel. Without this you would reach for colorMapDir, which is multiplied as a vec4 — and every eyeliner asset in this repo is pure black inside the stroke, so it would multiply the layer's color away and the stroke would be black whatever you set. Use the default --emit color only for art whose RGB you actually want.
• --region lower — bakes onto the lower lash line instead. Must match the layer's region.
• --stroke F — where v = 1 lands, as a fraction of eye width. Default 0.11 (about 2.1 mm). It must equal the layer's eyeStrip.strokeWidth, or v means two different things in the art and in the geometry.
• --smooth-u 6 — optional. The atlas assets here carry a 1–13 texel stroke, so their edge is quantised to whole texels and that transfers as a ~70 µm wobble along the top edge. It is not artistic intent — a synthetic smooth source bakes with a measured 0.0 µm wobble. Leave it off (0) for a strictly faithful bake.
• --eye average|left|right — average (default) cancels left/right asymmetry in the source. The art in this repo is genuinely asymmetric: 15.6% of pixels differ by more than 32 alpha between the two eyes.

It prints where the ink landed, which is the number to sanity-check:

ink occupies v 0.000..0.656  (0.00..1.40 mm from the lash margin on a 19.4 mm eye)
ink occupies u 0.055..0.976   (outer canthus at u=0.745, wing beyond it)

If the stroke renders thinner than expected, that report is the first place to look — a thin band of ink inside a wide strokeWidth band renders thin, and raising strokeWidth only buys more empty band.

Or generate one parametrically

With no art to bake:

python3 tools/make_liner_strip.py --region lower \
  --out demos/assets/imgs/face/eye_liner_lower_mask.png

Also mask form. --start-u / --end-u bound the stroke along the lash line, --thick-inner / --thick-outer ramp its thickness, --edge softens the far edge. Note --end-u defaults to the outer canthus on the lower lid: with wingLength: 0, the lower-lid default, every column past the canthus collapses onto it, so ink painted out there is never rendered.

Checking the result

Load demos/controls/, set USE_STRIP_TEMPLATE = true in its config.js, and both ribbons render eye_strip_uv_template.png instead. The 32 ticks should be evenly spaced along the eye, the diagonal should read as one smooth curve, the band should hold its width through a blink, and the green bar should sit on the outer canthus. Check that before judging artwork — a geometry problem otherwise reads as an art problem.

Authoring a glitter

Glitter is a separate specular lobe, not a bumpier make-up layer. Any legacy: false layer can carry it — lipstick, blush, eye shadow, an eyeliner ribbon — by adding a glitter block to its meshConfigs entry. It is off unless asked for: intensity defaults to 0 and the shader skips the whole block.

Why it is not just a map

The three reasons a flake field cannot be expressed through the existing maps, all measurable in this tree:

• normalMapDir is blended 50/50 with the video-derived normal (DynamicMaterialShader.ts, mapN = mix(mapN1, mapN2, 0.5)), so authored flake normals arrive at half strength, and normalScale scales both together.
• There is one material.roughness per layer. A flake is a near-mirror (~0.06) suspended in a matte film (~0.8). One scalar cannot be both.
• Every sampler reads bare texture2D(map, tUV) — no UV transform anywhere — and the lips are only 1.02% of the UV square (78 of 880 triangles, 483x177 px at 2048²). Flake density would be capped by atlas resolution. glitter.scale is the one place in the library that tiles a map.

The studio (demos/glitter-studio/)

Pick a palette, shape the flakes, watch it on a tracked face, download the two PNGs. Nothing touches disk until you press Download.

npx http-server .          # from the repo ROOT
# then open http://localhost:8080/demos/glitter-studio/

The generator runs in the browser as glitter_gen.js, a dependency-free port of tools/make_glitter_map.py. The panel prints the command line that reproduces exactly what you downloaded, plus the glitter block to paste into a layer — the studio is for finding the numbers, the tool is for reproducing them in a build.

That duplication is only tolerable while the two agree, so python3 tools/check_glitter_gen.py diffs them. Change one side, change the other, re-run it. Unlike check_strip_bake.py the tolerance is zero: there is no supersample collapse here, just a fixed sequence of IEEE double operations both languages perform identically. Measured: all 13 cases byte-identical, and glitter_gen.js reproduces the shipped 1024² glitter_multi PNGs exactly.

Agreeing at all cost numpy. default_rng is PCG64 with no JS equivalent, so neither side uses its language's RNG — both run the same explicit mulberry32 and draw the same seven values per flake in the same order. Don't "simplify" the Python back to np.random; it would silently divorce the studio from the tool. The checker also compares the PRNG stream, the rounding and the HSV conversion head-on, because pixel diffing does not cover them: the only rint tie a real map produces is 127.5, where every rounding rule agrees, so a broken tie-break passes every image case.

The frame light is on a preset dropdown in both glitter demos, because it is the thing most likely to bury the flakes. The studio opens on a calm preset — its job is judging flakes, not judging a look — while demos/glitter opens on the validated tuned values; each has the other one click away, plus calibrated (demos/controls) and off.

If the illumination runs away, look at the read-only measured by the probe block before touching a slider. Every specular term in frame_light_radiance is (luma - mean) / max(hi - mean, 0.02), so the probe's two measurements are the input the sliders merely scale, and hdr and detail both multiply a value that is deliberately not clamped at the top. A hi sitting almost on top of mean means the camera is seeing a flat lip region and the 0.02 floor, not your slider, is deciding the result. probe running freezes the measurement, which is the only way to tell a jumpy reading apart from a bad parameter. Debug views 4 and 5 show the irradiance and radiance terms on their own.

One thing to know before judging density: --count is an absolute number of flakes, not a density. Flake positions scale with the tile while the radius stays in pixels, so the same count at 2048² spreads over four times the area at a quarter of the coverage. Preview size therefore defaults to the export size, and the panel turns amber when you make them differ.

Generate the two maps

They are written together and paired texel by texel, so they must always be used and swapped as a pair:

| file | RGB | A | |---|---|---| | <prefix>_normal.png | tangent-space flake normal, (128,128,255) between flakes | flake coverage | | <prefix>_color.png | that flake's base hue | that flake's film thickness, 0..1 |

python3 tools/make_glitter_map.py     --out-prefix demos/assets/imgs/face/glitter_festa     --size 1024 --count 6000 --radius-px 3.0 --slope 0.75     --palette '#ff3d8b,#ffd23d,#3dffd2,#8b3dff,#ff5f3d,#3dff7a'

--palette random spreads saturated hues around the wheel instead. The map tiles: flakes crossing an edge are redrawn on the opposite side, so a 1024 tile repeated a few times beats a 4096 one-shot at 1/16 the memory. Needs numpy + Pillow, nothing else.

Wire it to a layer

This is the look demos/glitter ships, tuned on a live camera rather than assembled from defaults — start here rather than from the defaults:

{
  legacy: false,
  alphaMapDir: '../assets/imgs/face/mask_lips.jpg',
  envMapDir: '../assets/imgs/face/studio2.hdr',

  // ior 1.0 puts the base's F0 at exactly 0, so it has NO dielectric specular
  // and all its shine comes through the metallic path. That is what stops the
  // film competing with the flakes for the same highlight — the pair of
  // `ior: 1.0` + `metalness: 0.72` go together.
  color: '#f66151', opacity: 0.37, roughness: 0.6,
  metalness: 0.72, ior: 1.0, clearcoat: 0, envMapIntensity: 0.68,
  frameLight: {
    irradiance: 2.64, gain: 1.16, hdr: 2.86, region: 0.66,
    softness: 0.44, detail: 0.93, radius: 0.021, sheer: 0.5,
  },

  glitter: {
    normalMapDir: '../assets/imgs/face/glitter_multi_normal.png',
    colorMapDir:  '../assets/imgs/face/glitter_multi_color.png',
    intensity: 4.0,
    coverage: 0.9,
    scale: 2.9,          // how many times the tile repeats over the layer's UV
    normalScale: 1.43,   // flake tilt, widened — the real "more sparkle" lever
    roughness: 0.47,     // the flakes' own, independent of the layer's
    metalness: 0.48,     // part-way from metallised film toward a pearl bead
    iridescence: 0.68,   // 0 = fixed hues only, 1 = + full thin-film travel
    thicknessRange: [180, 640],
    ambient: 0.8,
  },
}

The env-map orientation is not a meshConfigs field — it lives on the FaceMesh. That look also applies faceMesh.setEnvMapRotation(0.12, 0.12, -2.32) at startup; the roll does most of the work, swinging the studio HDR's key light so it rakes across the lips rather than hitting them head on, which is what gets flakes at different tilts to light up at different moments instead of all together.

Every field is live-tunable without a recompile — faceMesh.setGlitter(index, {...}) and faceMesh.setGlitterMaps(index, normalDir, colorDir). Unlike alphaMapDir / colorMapDir, the glitter samplers are compiled unconditionally, so a layer built without a glitter block can still be given one at runtime.

Two colour mechanisms, and how to tell them apart

• Base hue — the flake's own pigment, from the colour map's RGB. Fixed: a magenta flake stays magenta from every angle, its neighbour stays gold. This is what separates one flake from the next in a still frame.
• Angle shift — thin-film interference through the same Belcour & Barla evalIridescence the layer's own iridescence uses, with a per-flake thickness from the colour map's alpha. This is what makes a flake travel (magenta → gold) as the head turns, and because the thickness is drawn per flake, two flakes of the same base hue travel to different colours.

Sweep iridescence from 1 to 0 to see them separately: at 0 only the fixed hues remain.

Traps

• metalness defaults to 1, and it should. A real cosmetic glitter particle is metallised polyester, so its F0 is its colour, around 0.9. Built as a dielectric at ior 1.8 instead, a flake reflects 8% — measured about 11x too dim, and it vanished under the lipstick entirely. Drop it toward 0 only for a soft pearl.
• The layer's opacity fades the flakes, because the lobe is added to outgoingLight and blending multiplies that by the layer's alpha. That is also what keeps the flakes inside the lip mask, so it is not a bug — raise glitter.intensity on a sheer layer. On the demos/controls lipstick (opacity: 0.18) the flakes need about 6 where an opaque base needs 1.
• Off-palette flakes are expected, and they are the loudest ones. A red-only palette still renders some yellow/cyan/blue, because a flake reflects the environment: its colour on screen is its own tint times the colour of whatever it happens to be reflecting. A red flake's F0 is about (0.39, 0.06, 0.08) — the green and blue channels are not zero — so a flake catching a bright cool highlight in studio2.hdr can come out blue. That is correct PBR, the same reason a red mirror pointed at a blue lamp looks blue. What makes it conspicuous is that those flakes are also the brightest (measured 1.8x the rest overall, 2.8x for the env-map term alone), because they are the ones catching the brightest light — so 3-4% of flakes dominate what the eye picks out. Measured on a four-red palette, share of flakes far from red (hue 45-260): 3.4% with thin film off, 4.2% at iridescence: 0.62. Attribution with thin film off — env map 2.7%, ambient 4.3%, frame light 1.2%. Levers, in the order worth trying: iridescence toward 0 (it is what makes the gold and cyan specifically), saturation 1.5-2 (tightens mean hue error from 16.5 to 12.8 degrees by zeroing the off-hue channels of F0 so light cannot leak through them), a warmer env map or lower envMapIntensity, then frameLightDetail, which injects the camera's own residual chroma.
• On a light base, occlusion is the control — not intensity, not saturation. The lobe is added to the make-up, and adding light can brighten something already near white but cannot tint it. Measured on a white lipstick: the flake contributes ~13 levels over a base at ~83, so the composite is 86% base and the flake's own hue (saturation 0.75, matching the palette) is dragged down to the base's. occlusion lets the flake hide the film under it, which is what an opaque metallised particle really does. Measured colour lift over the whole lip: +0.007 additive → +0.027 at occlusion 1.0 → +0.037 with saturation: 2.0 on top. Expect flakes darker than the base in places — an opaque red flake on a white film is darker, except where it catches a highlight. Defaults to 0, so no look tuned before it existed moves.
• coverage thins flakes toward transparency; it does not remove them. It scales the map's alpha channel, so coverage: 0.08 makes every flake 92% transparent rather than leaving 8% of the flakes solid — and since occlusion is weighted by the same value, it also caps occlusion at 0.08. Compensating with a big intensity then fights the transparency instead of fixing it. For fewer, solid flakes lower the generator's --count and leave coverage near 1. Measured: coverage 1.0 + occlusion 1.0 at intensity 1.0 beat coverage 0.08 at intensity 7.18 (+0.027 against +0.014) at a seventh of the intensity.
• intensity saturates, so it is not the lever for "more sparkle". Measured on that same sheer base, total glitter energy goes 0.25M → 0.35M → 0.55M → 0.53M → 0.56M for intensity 1 → 3 → 6 → 10 → 16: it knees at ~6 and is flat after. The peak per-pixel delta is pinned at its maximum from intensity 1 onward, which is the explanation — lit flakes already clip to white, and raising intensity clips them harder rather than lighting the flakes that are facing nothing bright. For more sparkle use coverage or scale (more flakes) or normalScale (a wider spread of facing directions).
• A hot base hides them. With demos/controls' calibrated frame-light values (gain: 1.5, detail: 0.35) the lip goes near-white and the flakes disappear into it. demos/glitter deliberately runs a much quieter base.
• Both maps must change together. A normal map from one palette with a colour map from another gives every flake the wrong hue.
• Mipmapping does not kill the sparkle — this was expected to be the hard problem and it measurably is not. Sweeping scale 1 → 16 (which shrinks the flake footprint exactly as walking away from the camera does), total glitter energy stayed within 77-142% of the scale-1 value and the peak stayed flat. What changes is character: blobs at 1, distinct coloured specks at 4, fine even dust at 16, where sub-pixel flakes average their colours toward each other. Around 4 keeps the hues telling each other apart. Not measured: temporal flicker as the head moves — the test feed was a still image, so it says nothing about flakes popping between frames.

Checking the result

The glitter shares the frameLightDebug switch, and the demos/glitter GUI exposes all of it. Slots 6, 7 and 8 are the glitter's:

• 6 coverage — where the flakes are. It must sit still on the lip as the head moves; if it crawls, it is being sampled in video space instead of atlas space.
• 7 flake normals — a normal-map view. A flat lilac field means the tilt is not arriving.
• 8 glitter only — the lobe with the make-up removed. This is where you can see that only some flakes are lit at any pose, which is the twinkle.

To prove the flakes are lit by indirect light and not by a lamp: with an env map bound the shader already zeroes the directional light (directLight.color *= (1.0 - hasEnvMap)). Zero the ambient too and rotate the env map — the sparkle must move. Then clearEnvMap() and enable the frame light probe — the sparkle must survive, now driven by the real light in the room.

Demo

The demos folder have demos with all the features. The photo shows an example of using an image loaded on html and the webcam uses the user camera. The application loads a canvas and shows blush, lipsticks and eye shadows make ups on the face with the face retouch. It is also possible to render partially with the setWindow method, freeze the video like a photograph and share the photo.

The eyeliner-studio demo is the authoring bench for the eyelid ribbons — draw a strip, see it on a tracked face, tune the layer, download the PNG. See Authoring an eyeliner. Serve the repo root, not demos/: it reads src/geometry.ts for the canonical model.

The glitter-studio demo is the authoring bench for the flakes — pick a palette, shape them, see them on a tracked face, download the PNGs, and copy the command that reproduces them. See Authoring a glitter. Serve the repo root, not demos/.

The glitter demo is a temporary bench for the glitter lobe: one lipstick layer, a lil-gui panel for every glitter knob, a palette dropdown, and the env-map / frame-light switches needed to prove the flakes are lit by indirect light. See Authoring a glitter.

The controls demo is the parameter sandbox: it runs the webcam through the custom DynamicMaterial shader and exposes every knob live in a lil-gui panel — full PBR per layer, the video-derived normal scale, the Gaussian/Sobel smoothing, environment-map upload, and the frame light probe (with a read-only readout of what the illumination is currently measuring). Upload an image from the same page to A/B against a still instead of the camera.

WASM

1. Activate o EMSDK

   source /path/to/emsdk/emsdk_env.sh

2. Generate the Makefile with cmake command

   cd ./wasm/build/
   emcmake cmake .

3. Compile

   emmake make

4. The wasm and js files will be inside output directory