mtr-sdk

Contents

• Description
• Usage
  • Configuration (SDKConfig)
  • Reporting load failures
• Features
• Public API reference
• Telemetry
• Project structure
• Demo
• Build

Description

The current repository is the MTR - SDK project. The system is responsible to retrieve information from mtr.center API and load the AR in a canvas.

Concretely, mtr-sdk is a single browser ES-module bundle (dist/mtr_sdk.js) that:

1. fetches the project catalogue from https://mtr.center/api/project/<projectId>/;
2. translates its slots into engine configs and renders the control UI (React, injected into the holder <div>);
3. starts one of two AR back ends, decided by the catalogue, never both:
   • Face Mesh — real-time virtual make-up via the bundled mtr-facemesh engine (MediaPipe/TensorFlow landmarks + three.js), when the project has make-up slots;
   • Model Viewer — @google/model-viewer for glTF products with WebXR / Scene Viewer / Quick Look, when the project has only a GLTF slot;
4. instruments the whole load with stage telemetry, phase watchdogs and webview workarounds (see Telemetry).

The bundle defines the global window.MTR (the MTR class). There is no npm import surface in practice: main points at the built bundle and consumers load it with a <script type="module"> tag.

Usage

In order to use the module, first you'll need the mtr-sdk script loaded in html with:

<script
  id="mtr-sdk"
  type="module"
  src="https://unpkg.com/mtr-sdk@3/dist/mtr_sdk.js"
></script>

Pin the major version. A bare unpkg.com/mtr-sdk URL resolves to whatever is on the npm latest dist-tag, so publishing a new major moves every page that uses it on the next page load — no deploy on the integrator's side, no opt-in, no staged rollout. Semver only protects consumers who declare a range in a package.json; a <script> tag declares nothing. Always pin at least the major.

Configuration (SDKConfig)

window.mtrObject = new MTR({
  projectId: 284,              // required — project id on mtr.center
  divName: "mtr-ar-holder",    // id of the holder <div>; default "mtr-ar-holder"
  verbose: false,              // console.log every step; default false
  loadedCallback: (infoReturn) => {},   // called when the AR is ready, with the item list
  errorCallback: (error, stage) => {},  // optional — terminal error + last stage reached
  loadingTimeout: 20000,       // optional — ms to the first camera frame; default 20000
  initTimeout: 60000,          // optional — ms to loadedCallback; default 60000
});

| field | type | default | notes | |---|---|---|---| | projectId | number | — | required | | divName | string | "mtr-ar-holder" | the element must already exist in the DOM | | verbose | boolean | false | | | loadedCallback | Function | — | receives infoReturn (see below) | | errorCallback | (error, stage) => void | — | without it a terminal error becomes an unhandled promise rejection | | loadingTimeout | number | 20000 | camera phase watchdog; <= 0 disables | | initTimeout | number | 60000 | init phase watchdog (API + ~15 MB of wasm/models + 3D); <= 0 disables |

MTR is a singleton: a second new MTR(...) returns the first instance and ignores the new config.

loadedCallback(infoReturn) receives, depending on the AR type:

• Face Mesh — one entry per material: {htmlId, meshConfigsId, materialId, name, ean}
• Model Viewer — one entry per model: {objectName, variantNames, variantHtmlIds}

The watchdogs warn, they do not abort: on timeout the SDK tracks a timeout event and calls errorCallback (once per phase — a phase that already reported a real error stays quiet), but the session continues if the phase finishes later.

Reporting load failures

The SDK's own telemetry can only report failures that happen after the bundle parses and runs. If the browser cannot parse it — too old for the bundle's syntax floor (see browserslist in package.json) — or the request never arrives, no SDK code executes, so errorCallback never fires and nothing reaches metrics.mtr.center. The failure is invisible from your side.

This snippet closes that gap. It is deliberately written in ES5 so that it parses on browsers that cannot parse the SDK itself, and it can go before or after the SDK tag.

<script>
(function () {
  var t0 = +new Date(), sent = false, TIMEOUT_MS = 30000;
  var ENDPOINT = 'https://metrics.mtr.center/sdk/event';
  function report(reason, detail) {
    if (sent || window.MTR) { return; } sent = true;
    var body = JSON.stringify({
      stage: 'sdk_load_failed',
      reason: reason,
      detail: String(detail || '').slice(0, 300),
      project_id: String(window.MTR_PROJECT_ID || ''),
      ms: (+new Date()) - t0,
      timestamp: new Date().toJSON(),
      URL: document.URL,
      in_iframe: window.location != window.parent.location,
      user_agent: navigator.userAgent
    });
    try {
      if (navigator.sendBeacon && window.Blob) {
        navigator.sendBeacon(ENDPOINT, new Blob([body], {type: 'application/json'}));
      } else {
        var x = new XMLHttpRequest();
        x.open('POST', ENDPOINT, true);
        x.setRequestHeader('Content-Type', 'application/json');
        x.send(body);
      }
    } catch (e) {}
  }
  // Capture phase on document: resource `error` events do not bubble, and this works
  // whether the snippet runs before or after the SDK tag. Matching on src rather than
  // on an element id keeps it working regardless of which id the tag uses.
  if (document.addEventListener) {
    document.addEventListener('error', function (e) {
      var t = e && e.target;
      if (t && t.tagName === 'SCRIPT' && String(t.src || '').indexOf('mtr_sdk') !== -1) {
        report('script_load_error', t.src);
      }
    }, true);
  }
  if (window.addEventListener) {
    window.addEventListener('error', function (e) {
      if (e && e.filename && e.filename.indexOf('mtr_sdk') !== -1) {
        report('parse_or_early_error', e.message);
      }
    });
  }
  setTimeout(function () { report('timeout_no_global', TIMEOUT_MS); }, TIMEOUT_MS);
})();
</script>

It covers three cases, and they fire through different channels — which is why both listeners are needed:

| failure | what fires | |---|---| | request fails / 404 / blocked | error on the script element (capture phase on document) | | module fails to parse | error on window, with filename set to the script URL | | anything else — loaded, no window.MTR | the timeout |

report() returns early if window.MTR already exists, so a runtime error from a healthy session is never reported as a load failure, and sent makes it report at most once. Set window.MTR_PROJECT_ID before the snippet if you want the report attributed to a project.

The payload matches the shape track() already posts to /sdk/event (stage, ms, timestamp, URL, in_iframe, user_agent, project_id) minus session_id, which does not exist yet at that point. Confirm the endpoint accepts it — sendBeacon with an application/json Blob triggers a CORS preflight, so /sdk/event must answer OPTIONS.

Features

• .switchModelViewerModel(slotName): you can use this function in case you want to change the 3D model in Model Viewer. The name of the slot will be available at mtr.center or infoReturn. Example:

  window.mtrObject.switchModelViewerModel(infoReturn[0].objectName);

• .mtrObject.modelViewerHolder.variantName: you can also change the variant of the model by changing the variantName.

  window.mtrObject.modelViewerHolder.variantName = infoReturn[0].variantNames[0];

• .switchFaceMesh(meshConfigsId, materialId): you can use this function when you want to change the mesh on the face without using the button click. The parameters are available on infoReturn. Example:

  window.mtrObject.switchFaceMesh(infoReturn[0].meshConfigsId, infoReturn[0].materialId);

• .stop(): this function stops the application and take off the control buttons from the holder specified. Use this function if you want to stop processing face mesh.

  window.mtrObject.stop();

• .continue(): this function continues the application and put the control buttons back on the holder specified. Use this function if you want to start the face mesh process again.

  window.mtrObject.continue();

• .shareScreenshot(fileName: string): this function takes a screenshot of the AR scene and shares it.

  window.mtrObject.shareScreenshot("MyFileName");

• .modelViewerHolder.activateAR(): after calling this on devices that support AR, the model is placed on the user environment. Works like a "View in your room" button.

  window.mtrObject.modelViewerHolder.activateAR();

• retornar no loadedCallback() um json do nome do material e do id sendo "mtr-item-" os botoes
• .startFaceMesh(width: number, height: number): call requestPermissions built in function to start the camera. width and height are the dimensions of the camera.
• .colorExtraction(colorType: string = "RGB"): returns a Promise of a string or array of 3 numbers depending on colorType parameter. If colorType is not set, the default is "RGB" and the return value will be an array of 3 positions representing the color RGB of the skin color extracted, otherwise the color returned will be a hexadecimal value in the format "#ABCDEF".

Face Mesh exclusive Features

The AR object also include other functionalities.

Partial Render

The partial render works like a curtain for the threejs renderer and only renders what is from 0 to windowSlider.value * canvas.width. For this, you need to create a checkbox and a slider from 0 to 1 and then use them as follows.

const windowCheckbox = document.getElementById('window-checkbox');
const windowSlider = document.getElementById('window-slider');

windowCheckbox.oninput = function () {
  window.mtrObject.faceMesh.setWindow(windowCheckbox.checked);
  windowSlider.style.display = windowCheckbox.checked ? 'block' : 'none';
};

windowSlider.oninput = function () {
  window.mtrObject.faceMesh.setWindowFactor(windowSlider.value);
};

Photograph

The photograph functionality is simple and works like it is said. When the method togglePhoto is called, the animation loop freezes/unfreezes depending on the actual state. You only need to create on html and use the following code. The method also returns a promise of a canvas of the photograph.

const screenshotButton = document.getElementById('photo-button');

screenshotButton.onclick = async function () {
  const frozenVideoCanvas = await window.mtrObject.faceMesh.togglePhoto();
};

Foundation Color

If foundationMatchConfig and faceRetouchConfig are set, it is possible to set a color on the users face as if there is a foundation using .setFoundation method. The function receives two parameters, color and opacity. The color is a number array with 3 positions in RGB space, and the opacity is a floating point number between 0.0 and 1.0 that is the strength of the color. The following example will make up the face with red foundation.

window.mtrObject.faceMesh.setFoundation([255, 0, 0], 0.5);

Using an image instead of camera

If faceMesh is created with an image with id="mtr-photo-img" set, the video camera will be replaced with the image loaded on html. For example, an image tag is loaded with id "mtr-photo-img", and faceMesh is created with it replacing the camera.

<img id="mtr-photo-img" src="./any-image">

Public API reference

Everything below is on the instance returned by new MTR(config) (conventionally window.mtrObject). window.MTR itself is the class.

Properties

| member | type | description | |---|---|---| | MTR.facemeshRevision (static) and .facemeshRevision | string | version of the mtr-facemesh engine compiled into this bundle, and the version whose CDN binaries are loaded. Static too, so window.MTR.facemeshRevision works from a console. | | .revision | string | SDK version (package.json#version, injected as __VERSION__) | | .faceMesh | FaceMesh \| null | the engine instance, null for Model Viewer projects | | .modelViewerHolder | ModelViewerElement \| null | the <model-viewer>, null for Face Mesh projects | | .arHolder | HTMLDivElement | the holder resolved from divName | | .projectDetail | any | raw JSON from /api/project/<id>/ | | .meshConfigs / .modelConfigs / .faceRetouchConfigs | arrays | configs derived from the catalogue slots | | .stage | string | last loading stage reached (see Telemetry) | | .sessionId | string | random id grouping this page load's telemetry | | .onDevice | boolean | mobile user-agent detection | | .useNewMediapipe | boolean | mirrors config.should_use_new_mediapipe from the API |

Methods

| method | description | |---|---| | startFaceMesh(width, height) | requests camera permission and starts the stream with aspectRatio = width/height. No-op when an #mtr-photo-img element exists (image mode). | | switchFaceMesh(meshConfigsId, materialId) | applies a make-up material (or foundation, when meshConfigsId is past the end of meshConfigs); returns its name, or "none". | | switchModelViewerModel(modelName) | swaps the glTF by slot name. | | colorExtraction(colorType = "RGB") | Promise<[r,g,b] \| "#RRGGBB"> — skin tone extracted by the engine; also posts the result to /sdk/color. | | shareScreenshot(fileName) | screenshot + navigator.share (engine share() for Face Mesh, toDataURL for Model Viewer). | | stop() | stops the AR, clears every timer/overlay and removes the control buttons from the holder. | | continue() | restarts the AR and re-attaches the control buttons. | | canRun() | false for Face Mesh on SamsungBrowser, true otherwise. | | detectMob() | user-agent mobile check. | | getGPU() | unmasked WebGL renderer string (used in telemetry). | | track(stage, extra?) | posts a telemetry event; never throws. |

Face-Mesh-only features reach through .faceMesh (the engine's own API): setWindow / setWindowFactor, togglePhoto, setFoundation, getMeshes, getFaceClosestColor, requestPermissions, share, stop / continue.

DOM the SDK creates

Inside the holder: #mtr-control-buttons (React root for #mtr-ui, z-index 100), #mtr-close-button (#mtr-close-button-icon, z-index 101), #mtr-model-viewer for Model Viewer projects, and #mtr-tap-overlay (z-index 99) when the camera video needs a user gesture. Per-item buttons are #mtr-item-<id>; slot rows are #mtr-<slot.html_id>. The camera <video> is appended to the holder 1×1 px and transparent (never display:none, which can stop the decode in some webviews).

Telemetry

Every load step is posted to https://metrics.mtr.center/sdk/event with session_id, project_id, stage, ms since init, timestamp, sdk_version, facemesh_version, URL, in_iframe, user_agent plus per-stage extras. Failures are swallowed — telemetry never breaks the experience.

Stages, in order: sdk_init (with gpu) → project_loaded → engine_created → engine_loaded → camera_requested → camera_stream → video_loadeddata → video_playing → first_frame. Off-path events: camera_failed, video_play_stalled, video_play_blocked, tap_overlay_clicked, error, timeout. error and timeout do not advance .stage, so the reported stage is the last healthy one.

Two other endpoints are called directly: POST https://metrics.mtr.center/sdk/session on init and POST https://metrics.mtr.center/sdk/color on colorExtraction().

The project catalogue comes from GET https://mtr.center/api/project/<projectId>/, and all slot assets (textures, masks, glTF, button images) are resolved as https://mtr.center + the path returned by the API. Engine binaries (wasm, MediaPipe models) load from https://cdn.jsdelivr.net/npm/mtr-facemesh@<facemeshVersion>/dist/lib.

There are no environment variables: every endpoint above is hard-coded in src/mtr_sdk.ts. The only build-time configuration is webpack's DefinePlugin (__VERSION__, __FACEMESH_VERSION__).

Project structure

src/mtr_sdk.ts          the MTR class — API call, AR selection, telemetry, timers, public methods
src/react_render.tsx    React control UI; also TRANSLATES catalogue slots into the config arrays
src/globals.d.ts        declares __VERSION__ / __FACEMESH_VERSION__
src/mtr-facemesh/       the engine, cloned from source (gitignored, pinned by facemesh.lock)
build/typecheck.js      tsc --noEmit, failing only on errors outside src/mtr-facemesh
build/check-es-version.js  parses dist/mtr_sdk.js with acorn at ES2020
build/append-licenses.js   webpack plugin adding banner-less deps to the LICENSE sidecar
webpack.config.js       bundling + three/engine version guards
dist/                   committed build output (mtr_sdk.js + mtr_sdk.js.LICENSE.txt)
demos/                  static HTML integration examples
facemesh.lock           engine commit this dist/ was built from

Published to npm: dist/, README.md, facemesh.lock (package.json#files).

Demo

The demos folder shows examples of usage. The 'facemesh-photo' folder shows an application that loads a canvas on facemesh-holder div and a MTR - Face Mesh scene. There is almost no difference from this Face Mesh demo to the Model Viewer one, since the main difference will be the projectID parameter.

| demo | project id | what it shows | |---|---|---| | demos/facemesh-webcam/ | 284 | webcam make-up, mobile/desktop layout switch, switchFaceMesh after load | | demos/facemesh-photo/ | 136 | same scene driven by <img id="mtr-photo-img"> instead of the camera | | demos/model-viewer/ | 161 | glTF product with <model-viewer> | | demos/iframe/ | 284 | the SDK running inside an iframe | | demos/host_iframe/ | — | a host page embedding demos/iframe (expects a server on port 8000) |

The demos load ../../dist/mtr_sdk.js, so run npm run compile first and serve the repo root over HTTP (ES modules and the camera will not work from file://), e.g. python3 -m http.server 8000, then open http://localhost:8000/demos/facemesh-webcam/. Each demo also carries a commented-out unpkg tag to test the published bundle instead.

Build

The SDK compiles mtr-facemesh from source, so clone it into src/mtr-facemesh/ (gitignored) at the commit pinned in facemesh.lock and install its dependencies:

git clone https://github.com/morethanrealio/mtr-facemesh.git src/mtr-facemesh
cd src/mtr-facemesh && git checkout $(cat ../../facemesh.lock) && npm install && cd ../..
npm install
npm run compile      # writes dist/mtr_sdk.js
npm run verify       # asserts the bundle still parses at the supported floor
npm run typecheck    # tsc --noEmit (ts-loader runs transpileOnly, so this is the only check)
npm run verify:es    # just the ES-floor parse check on dist/mtr_sdk.js
npm run watch        # development build, rebuilds dist/ on change

There is no test suite and no linter in this repo: npm run verify (typecheck + ES-floor parse check) is the whole gate, and there is no CI configuration — run it locally before publishing. dist/ is committed, so a build output change belongs in the commit.

• facemesh.lock holds the engine commit this dist/ was built from. The build warns when the checkout does not match it.
• The facemesh version (from src/mtr-facemesh/package.json) is injected as __FACEMESH_VERSION__ and builds the binaryDir CDN path (mtr-facemesh@<version>/dist/lib), so the wasm/model binaries always match the engine compiled into the bundle. Only build from a facemesh commit published to npm.
• The build fails if the engine's package.json#version disagrees with the revision in src/mtr_facemesh.ts. The two are maintained independently and have drifted before; that drift is how production came to run engine JS 5.2.6 against binaries from [email protected] for ~18 months. MTR.facemeshRevision now exposes the compiled engine version at runtime so the same skew cannot hide again.
• mtr-facemesh imports its .ts files with a .js extension. webpack.config.js resolves that with extensionAlias only inside src/mtr-facemesh/src (applied globally it breaks packages like lit).
• Some dependencies publish syntax newer than the browserslist floor (@mediapipe/tasks-vision uses ES2021 logical assignment; lit 3, via @google/model-viewer 4.x, uses ??= and ES2022 private fields). Those packages are transpiled down by a scoped babel-loader rule in webpack.config.js. Extend that rule's include list when npm run verify flags a new one — do not raise the floor by accident.
• three is aliased to a single copy. @google/model-viewer and the engine both want three ^0.182.0 but resolve to physically separate installs, which shipped three.js twice (~505 KB). The build fails if the two versions ever diverge, because the alias would otherwise compile the engine against a three it was never tested with.

Supported browsers

browserslist in package.json records the floor, and npm run verify enforces it against the built bundle. Raising it is a breaking change for integrators: the bundle is loaded as <script type="module">, so unsupported syntax fails at parse time, before any SDK code runs — errorCallback never fires and nothing reaches telemetry. See Reporting load failures.