@boytecht/threeengine-virtual-try-on
v1.0.0
Published
ThreeEngine virtual try-on plugins - ring try-on on the webcam or a photo with MediaPipe hand tracking, finger occlusion and capture.
Maintainers
Readme
@boytecht/threeengine-virtual-try-on
Virtual try-on plugins for ThreeEngine.
RingTryOnPlugin - try rings on the webcam, a photo or a video with hand tracking by
MediaPipe (@mediapipe/tasks-vision HandLandmarker, Apache-2.0, bundled).
- The camera/photo is shown in an HTML element behind a transparent viewer canvas (not tonemapped), the front camera is
mirrored. Image mode (
tryOnImage) and user supplied videos (setVideoElement) are supported. - The ring is placed on the proximal phalanx of the selected finger (between the knuckle and the middle joint), the hole axis along the finger and the stone on the back of the hand. Depth and orientation come from the metric world landmarks, the position from the image landmarks.
- Automatic size from the finger, or an explicit ring size (US, EU or mm).
- Depth only occluders (capsules along the fingers and the palm) hide the back of the band.
- One Euro filters on position, size and rotation, confidence threshold, events (
handFound,handLost,poseUpdate...). captureImage()composites the camera frame and the rendered ring into a PNG.createDefaultRing()builds a procedural band with a brilliant cut stone (rendered as a diamond when theDiamondPluginis added).
npm install @boytecht/threeengine @boytecht/threeengine-virtual-try-onimport {ThreeViewer, DiamondPlugin} from '@boytecht/threeengine'
import {RingTryOnPlugin, createDefaultRing} from '@boytecht/threeengine-virtual-try-on'
const viewer = new ThreeViewer({canvas: document.getElementById('canvas'), rgbm: false, msaa: true})
viewer.addPluginSync(DiamondPlugin) // optional, for the stone
await viewer.setEnvironmentMap('https://samples.threepipe.org/minimal/studio_small_03_1k.hdr')
const tryOn = viewer.addPluginSync(RingTryOnPlugin)
await tryOn.setRing(createDefaultRing({innerDiameter: 17.3, stone: {diameter: 6.5}})) // or a glb url
tryOn.finger = 'ring'
tryOn.addEventListener('error', (e) => alert(e.message))
await tryOn.startCamera() // or: await tryOn.tryOnImage('hand.jpg')
const {blob} = await tryOn.captureImage()
tryOn.stop()Ring model convention
The hole axis (the finger direction, towards the fingertip) is +Z, the stone is at +Y, the hole centre is at the origin
and userData.ringInnerDiameter is the inner diameter in model units. Other conventions can be described with
userData.ringHoleAxis / userData.ringUpAxis ('+x', '-z', ...) and userData.ringCenter ([x, y, z]), or with the
plugin options ringHoleAxis, ringUpAxis and ringInnerDiameter. Without an inner diameter, the hole centre and
diameter are estimated from the geometry. positionOffset (mm), rotationOffset (degrees) and scaleMultiplier can be
used to calibrate a model.
MediaPipe files
The MediaPipe library is bundled, the wasm files and the model are loaded at runtime:
wasmUrl:https://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/wasm(must match the bundled version, seeMEDIAPIPE_TASKS_VISION_VERSION). To self-host, copynode_modules/@mediapipe/tasks-vision/wasm.modelUrl:https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task
Notes
- The camera needs a secure context (https or localhost). Errors (no camera, permission denied, camera in use) are
reported with the
errorevent,lastErrorandstatus. - With the default RGBM render targets, the transparent background is clipped with the GBuffer depth (a
GBufferPluginis added automatically) and anti-aliased edges are not blended with the camera image, usergbm: false(withmsaa: true) for smooth edges. LoadingScreenPluginshows its loading screen while the scene model root is empty (the try-on objects are not in it), set itsshowOnSceneEmptytofalse.- The metric size (
autoring size, depth) depends on the hand size estimated by MediaPipe and the camera field of view (fov); the ring always fits the finger on screen, but the estimated diameter in millimeters is only indicative.
Tests
Unit tests for the pose math (synthetic hands, axis conversion, mirroring, filters, ring sizes) run in node:
npx vitest run --config plugins/virtual-try-on/vitest.config.mjsLicense
Apache-2.0. See LICENSE and NOTICE. @mediapipe/tasks-vision is Apache-2.0 licensed (bundled), see
dist/dependencies.txt. The MediaPipe wasm files and the hand landmarker model are loaded at runtime from their
default locations (see the MediaPipe terms and privacy notice).
