MultiSet VPS WebXR

TypeScript SDK for integrating MultiSet's Visual Positioning System (VPS) into WebXR applications. Provides 6-DOF localization by matching camera frames against cloud-hosted maps, and object tracking by matching camera frames against registered 3D objects.

Contents

• Architecture
• Installation
• Requirements
• CORS Configuration
• Quick Start
  • VPS — Three.js
  • VPS — Vanilla / WebGL2
  • Object Tracking — Three.js
• Canvas Visibility During AR
• API Reference
  • MultisetClient
  • XRSessionManager
  • ThreeAdapter
• Placing Content at Map Coordinates
• Styling the AR Button
• Custom Start / Stop Button
• Type Definitions

Architecture

The SDK is split into two independent entry points:

| Entry point | Contents | Three.js required | |---|---|---| | @multisetai/vps/core | MultisetClient + XRSessionManager | No | | @multisetai/vps/three | ThreeAdapter | Yes (peer dep) |

XRSessionManager owns the full vanilla WebXR session lifecycle — frame loop, camera capture, localization, tracking-loss recovery — with zero Three.js dependency. ThreeAdapter wires it to a Three.js renderer and scene.

Installation

# Core only (no Three.js)
npm install @multisetai/vps

# With Three.js adapter
npm install @multisetai/vps three

Requirements

• HTTPS — WebXR requires a secure context (https:// or http://localhost).
• Android + ARCore — Chrome or Edge on an ARCore-capable Android device.
• Three.js ≥ 0.176.0 — only required when using @multisetai/vps/three.

iOS is not supported. This SDK requires the camera-access WebXR feature. Safari on iOS does not implement it.

CORS Configuration

The SDK makes direct browser-to-API requests, so your domain must be whitelisted in the MultiSet dashboard.

1. Open the MultiSet Dashboard
2. Go to Credentials → Settings → Domains
3. Click Add + and enter your origin (e.g. https://localhost:5173 for dev, https://your-app.com for prod)

Without this, the browser will block every API request with a CORS error.

Quick Start

VPS Localization — Three.js

import * as THREE from 'three';
import { MultisetClient, XRSessionManager } from '@multisetai/vps/core';
import { ThreeAdapter } from '@multisetai/vps/three';

// Check support before showing any AR UI
const supported = await ThreeAdapter.isSupported();
if (!supported) {
  console.warn('WebXR immersive-ar is not supported on this device.');
}

const client = new MultisetClient({
  clientId: 'CLIENT_ID',
  clientSecret: 'CLIENT_SECRET',
  code: 'MAP_OR_MAPSET_CODE',
  mapType: 'map',
});

await client.authorize();

const renderer = new THREE.WebGLRenderer({ antialias: true });
document.body.appendChild(renderer.domElement);

const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(70, window.innerWidth / window.innerHeight, 0.01, 1000);

const session = new XRSessionManager(renderer.getContext() as WebGL2RenderingContext, {
  client,
  overlayRoot: document.body,
  autoLocalize: true,
  confidenceCheck: true,
  confidenceThreshold: 0.5,
  onSessionStart: () => {
    // Hide the canvas during AR — the XR compositor owns the display.
    renderer.domElement.style.display = 'none';
  },
  onSessionEnd: () => {
    renderer.domElement.style.display = 'block';
  },
  onLocalizationResult: (result) => console.log('Localized:', result.localizeData.position),
  onLocalizationFailure: (reason) => console.warn('Failed:', reason),
  onError: (err) => console.error(err),
});

const adapter = new ThreeAdapter({ session, renderer, scene, camera, showMesh: true });
adapter.initialize(); // mounts the built-in START AR / STOP AR button

// Add your 3D content
scene.add(new THREE.Mesh(
  new THREE.BoxGeometry(0.1, 0.1, 0.1),
  new THREE.MeshBasicMaterial({ color: 0xff0077 })
));

VPS Localization — Without Three.js (WebGL2 / Vanilla)

In this mode the SDK manages the session lifecycle, camera capture, and localization. You are responsible for all rendering: each XR frame, draw your scene into event.baseLayer.framebuffer using the provided gl context. This approach works with any WebGL2-based renderer — Babylon.js, raw WebGL, or your own engine.

import { MultisetClient, XRSessionManager } from '@multisetai/vps/core';

const supported = await XRSessionManager.isSupported();
if (!supported) {
  console.warn('WebXR immersive-ar is not supported on this device.');
}

const client = new MultisetClient({ clientId: '...', clientSecret: '...', code: '...', mapType: 'map' });
await client.authorize();

// A WebGL2 context is required — WebXR renders into a GL framebuffer, not a 2D canvas.
const gl = document.querySelector('canvas')!.getContext('webgl2')!;

const session = new XRSessionManager(gl, {
  client,
  overlayRoot: document.body,
  autoLocalize: true,
  onLocalizationResult: (result) => console.log('Localized:', result.localizeData.position),
  onError: (err) => console.error(err),
});

// Wire your render loop — called every XR frame with the current pose and framebuffer.
session.setXRFrameHandler((event) => {
  // Bind event.baseLayer.framebuffer and render your scene using event.view for camera matrices.
});

document.body.appendChild(session.createButton());

Object Tracking — Three.js

Object tracking detects and poses registered 3D objects by matching a captured camera frame against the MultiSet cloud.

import * as THREE from 'three';
import { MultisetClient, XRSessionManager } from '@multisetai/vps/core';
import { ThreeAdapter } from '@multisetai/vps/three';

const client = new MultisetClient({
  clientId: 'CLIENT_ID',
  clientSecret: 'CLIENT_SECRET',
  mapType: 'object-tracking',
  code: ['YOUR_OBJECT_CODE'],
});

await client.authorize();

const renderer = new THREE.WebGLRenderer({ antialias: true });
document.body.appendChild(renderer.domElement);
const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(70, window.innerWidth / window.innerHeight, 0.01, 1000);

const session = new XRSessionManager(renderer.getContext() as WebGL2RenderingContext, {
  client,
  overlayRoot: document.body,
  autoTracking: true,          // detect once on session start
  confidenceCheck: true,
  confidenceThreshold: 0.5,
  onSessionStart: () => { renderer.domElement.style.display = 'none'; },
  onSessionEnd:   () => { renderer.domElement.style.display = 'block'; },
  onObjectTrackingSuccess: (result) => {
    console.log('Detected', result.objectCodes, 'at', result.position);
  },
  onObjectTrackingFailure: (reason) => console.warn('Tracking failed:', reason),
  onError: (err) => console.error(err),
});

const adapter = new ThreeAdapter({
  session,
  renderer,
  scene,
  camera,
  showObjectMeshes: true,       // load and display the 3D outline mesh
  onObjectMeshLoaded: (code) => console.log('Mesh loaded for', code),
});
adapter.initialize();

// Trigger tracking manually from a button
trackButton.addEventListener('click', () => {
  void adapter.trackObjects();
});

Canvas Visibility During AR

Important — the SDK renders the Three.js scene into the XR framebuffer, not the canvas element. During an active AR session the canvas element is not updated; it retains whatever was last drawn by the preview loop. If the canvas is visible during AR (e.g. as part of the WebXR DOM overlay) it will appear as a frozen image on top of the AR scene.

Always hide the canvas when the session starts and restore it when it ends:

onSessionStart: () => { renderer.domElement.style.display = 'none'; },
onSessionEnd:   () => { renderer.domElement.style.display = 'block'; },

API Reference

MultisetClient

Pure HTTP client for auth, localization, and object tracking. No WebXR or rendering concerns.

new MultisetClient(config: IMultisetClientConfig)

IMultisetClientConfig

VPS mode (mapType: 'map' or mapType: 'map-set')

| Parameter | Type | Description | |---|---|---| | clientId | string | Your MultiSet client ID | | clientSecret | string | Your MultiSet client secret | | mapType | 'map' \| 'map-set' | Whether code is a single map or a map set | | code | string | Map or map-set code | | endpoints? | Partial<IMultisetSdkEndpoints> | Override default API endpoints | | isRightHanded? | boolean | Handedness sent to the API. Default true | | convertToGeoCoordinates? | boolean | Request geographic coordinates in the response | | hintPosition? | string | Local-space position hint "x,y,z" | | hintRadius? | number \| string | Search radius in metres (1–100). Requires hintPosition or passGeoPose | | hintMapCodes? | string[] | Narrow candidates by map code. Only valid when mapType: 'map-set' | | passGeoPose? | boolean | Send a geoHint from the Geolocation API with each request | | use2DFiltering? | boolean | Skip altitude in geo filtering. Only valid when passGeoPose: true |

Object tracking mode (mapType: 'object-tracking')

| Parameter | Type | Description | |---|---|---| | clientId | string | Your MultiSet client ID | | clientSecret | string | Your MultiSet client secret | | mapType | 'object-tracking' | Enables object tracking mode. No map code required. | | code | string[] | Object codes to detect and track | | isRightHanded? | boolean | Handedness sent to the API. Default true | | endpoints? | Partial<IMultisetSdkEndpoints> | Override default API endpoints |

Methods

| Method | Returns | Description | |---|---|---| | authorize() | Promise<string> | Authenticate and cache an access token. Call before any other method. | | localizeWithFrame(frame, intrinsics) | Promise<ILocalizeAndMapDetails \| null> | Submit a captured frame for VPS localization. | | trackObject(frame, intrinsics) | Promise<IObjectTrackingResponse \| null> | Submit a captured frame for object detection. Uses code from the client config. Returns null when no object is detected. | | downloadObjectMesh(objectCode) | Promise<string \| null> | Fetch a signed download URL for the 3D mesh of an object. | | fetchMapDetails(mapCode) | Promise<IGetMapsDetailsResponse \| null> | Fetch map metadata by code (result is cached). |

XRSessionManager

Owns the WebXR session lifecycle — frame loop, camera capture, localization, object tracking, and tracking-loss recovery. Zero Three.js dependency.

import { XRSessionManager } from '@multisetai/vps/core';

new XRSessionManager(gl: WebGL2RenderingContext, options: IXRSessionOptions)

IXRSessionOptions

| Parameter | Type | Description | |---|---|---| | client | MultisetClient | Required. | | overlayRoot? | HTMLElement | Root element for the WebXR DOM overlay. | | autoLocalize? | boolean | Run one localization automatically when the session starts. | | relocalization? | boolean | Re-localize whenever tracking is lost and then recovered. | | confidenceCheck? | boolean | Only accept results with confidence >= confidenceThreshold. Applies to both VPS and object tracking. | | confidenceThreshold? | number | Minimum confidence (0.2–0.8). Default 0.5. | | poseTimeoutMs? | number | Max ms to wait for a valid viewer pose before failing. Default 10000. | | localizationTrackingTimeoutMs? | number | Deprecated — use poseTimeoutMs. Will be removed in v3. | | backgroundLocalization? | boolean | Periodically send localization/tracking requests in the background while the session is active. | | bgLocalizationInterval? | number | Interval in seconds between background attempts. Clamped to 10–180 s. Default 30 for VPS modes, 10 for object tracking. | | autoTracking? | boolean | Call trackObjects() once automatically when the session starts. Requires mapType: 'object-tracking' on the client. | | restartTracking? | boolean | Re-attempt tracking whenever XR tracking is lost and then recovered. | | trackingCaptureDelayMs? | number | Milliseconds to wait before capturing a frame when trackObjects() is called. Useful for camera stabilisation. Default 0. | | referenceSpaceType? | XRReferenceSpaceType | XR reference space type. Default 'local'. Use 'local-floor' for floor-relative tracking if the device supports it. | | framebufferScaleFactor? | number | XR framebuffer scale relative to native resolution. Values < 1 reduce GPU load; values > 1 supersample. | | onSessionStart? | () => void | Called when the AR session starts. | | onSessionEnd? | () => void | Called when the AR session ends. | | onLocalizationInit? | () => void | Called at the start of a VPS localization run. | | onLocalizationResult? | (result: ILocalizeAndMapDetails) => void | Called when VPS localization succeeds (and passes the confidence check, if enabled). | | onLocalizationSuccess? | (result: ILocalizeAndMapDetails) => void | Deprecated — use onLocalizationResult. If using ThreeAdapter, use its onLocalizationSuccess which also provides worldFromMap. Will be removed in v3. | | onLocalizationFailure? | (reason?: string) => void | Called when VPS localization fails or falls below the confidence threshold. | | onFrameCaptured? | (frame: IFrameCaptureEvent) => void | Called when a camera frame is captured for localization. | | onCameraIntrinsics? | (intrinsics: ICameraIntrinsicsEvent) => void | Called with camera intrinsic parameters for the captured frame. | | onPoseResult? | (pose: IPoseResultEvent) => void | Called with the raw pose result from the VPS backend. | | onObjectTrackingInit? | () => void | Called at the start of an object tracking run. | | onObjectTrackingRequested? | (frame: IFrameCaptureEvent, intrinsics: ICameraIntrinsicsEvent) => void | Called just before the tracking request is sent, with the captured frame and intrinsics. | | onObjectTrackingSuccess? | (result: IObjectTrackingResponse) => void | Called when object tracking succeeds and passes the confidence check (if enabled). | | onObjectTrackingFailure? | (reason?: string) => void | Called when object tracking fails or falls below the confidence threshold. | | onError? | (error: unknown) => void | Called when any error occurs. | | onContextLost? | () => void | Called when the WebGL context is lost. The active session is ended automatically. | | onContextRestored? | () => void | Called when the WebGL context is restored. The user may restart the session. |

Static methods

| Method | Returns | Description | |---|---|---| | XRSessionManager.isSupported() | Promise<boolean> | Returns true if the browser supports immersive-ar WebXR sessions. Use this to conditionally show AR UI before creating any objects. |

Methods

| Method | Returns | Description | |---|---|---| | createButton() | HTMLButtonElement | Create the built-in styled AR button. Shows START AR / STOP AR and toggles the session on click. | | startSession() | Promise<void> | Start an AR session programmatically. Must be called from within a user gesture handler (click/tap). | | stopSession() | void | Stop the active AR session. No-op if no session is running. | | localizeFrame() | Promise<ILocalizeAndMapDetails \| null> | Capture and localize one frame against the configured map. Requires an active session. | | trackObjects() | Promise<IObjectTrackingResponse \| null> | Capture one frame and run object detection. Requires an active session and mapType: 'object-tracking' on the client. | | isActive() | boolean | Whether an XR session is currently running. | | isLocalizing | boolean | Whether a localization or tracking run is currently in progress. | | getClient() | MultisetClient | Access the underlying MultisetClient. | | dispose() | void | End the session, clear background timers, remove context loss listeners, and release all resources. |

Adapter hooks

Used internally by ThreeAdapter. Only call these when building a custom renderer adapter.

| Method | Description | |---|---| | setXRFrameHandler(fn) | Called every XR frame with pose, view, viewport, and framebuffer info. | | setAdapterResultHandler(fn) | Called after a successful VPS localization with the result and tracker-space matrix. | | setAdapterObjectTrackingHandler(fn) | Called after a successful object tracking result with the result and tracker-space matrix. | | setAdapterSessionHandlers(onStart, onEnd) | Called on session start/end, before user callbacks. |

ThreeAdapter

Wires XRSessionManager to a Three.js renderer. Handles XR framebuffer binding, camera matrix sync, preview loop, resize, and optional map mesh / gizmo / object mesh display.

import { ThreeAdapter } from '@multisetai/vps/three';

new ThreeAdapter(options: IThreeAdapterOptions)

IThreeAdapterOptions

| Parameter | Type | Description | |---|---|---| | session | XRSessionManager | Required. | | renderer | THREE.WebGLRenderer | Required. | | scene | THREE.Scene | Required. | | camera | THREE.PerspectiveCamera | Required. | | showMesh? | boolean | Show the VPS map mesh after localization. Default false. | | showGizmo? | boolean | Show a transform gizmo after localization. Default true. | | showObjectMeshes? | boolean | Load and display a 3D outline mesh for each detected object. Default false. | | useDefaultButton? | boolean | Mount the built-in START AR / STOP AR button. Default true. Set to false to drive the session via startSession() / stopSession(). | | buttonContainer? | HTMLElement | Where to append the built-in button. Defaults to overlayRoot or document.body. | | onButtonCreated? | (button: HTMLButtonElement) => void | Called after the built-in button is created. | | onXRFrame? | (event: IXRFrameEvent) => void | Called every XR frame after camera matrices are synced, before the scene is rendered. Use this to update scene objects each frame. | | onLocalizationSuccess? | (result: ILocalizeAndMapDetails, worldFromMap: THREE.Matrix4) => void | Called immediately after a successful VPS localization. worldFromMap transforms map-space coordinates to Three.js world space — use it to place content at known map coordinates. | | onObjectMeshLoaded? | (objectCode: string) => void | Called when a detected object's 3D mesh has been loaded and placed in the scene. Only fires when showObjectMeshes: true. |

Static methods

| Method | Returns | Description | |---|---|---| | ThreeAdapter.isSupported() | Promise<boolean> | Returns true if the browser supports immersive-ar WebXR sessions. |

Methods

| Method | Returns | Description | |---|---|---| | initialize(buttonContainer?) | HTMLButtonElement \| null | Start the preview render loop, attach resize handler, and mount the built-in button. Returns null when useDefaultButton: false. | | isActive() | boolean | Whether an XR session is currently running. | | isLocalizing | boolean | Whether a localization or tracking run is currently in progress. | | startSession() | Promise<void> | Start an AR session. Must be called from within a user gesture handler. | | stopSession() | void | Stop the active AR session. No-op if no session is running. | | localizeFrame() | Promise<ILocalizeAndMapDetails \| null> | Capture and localize one frame. | | trackObjects() | Promise<IObjectTrackingResponse \| null> | Capture one frame and run object detection. Requires mapType: 'object-tracking' on the client. | | clearObjectMeshes() | void | Remove all object meshes from the scene that were placed by showObjectMeshes. | | dispose() | void | Stop loops, remove listeners, dispose Three.js resources, and end the session. |

Placing Content at Map Coordinates

After localization, the onLocalizationSuccess callback on ThreeAdapter provides a worldFromMap matrix that converts any point from VPS map space into Three.js world space. Use this to anchor scene objects to specific physical locations in the scanned map — independently of where the user started the AR session.

const adapter = new ThreeAdapter({
  session,
  renderer, scene, camera,
  onLocalizationSuccess: (result, worldFromMap) => {
    // mapPoint is a position you measured from the scanned map (in metres)
    const mapPoint = new THREE.Vector3(1.5, 0, -2.0);

    const marker = new THREE.Mesh(
      new THREE.SphereGeometry(0.05),
      new THREE.MeshBasicMaterial({ color: 0x00ff88 })
    );
    marker.position.copy(mapPoint.applyMatrix4(worldFromMap));
    scene.add(marker);
  },
});

Note — worldFromMap is recomputed on every successful localization. If you re-localize, update or re-add your objects so they stay in sync with the latest result.

Styling the AR Button

The built-in button ships with minimal inline styles. Use these CSS classes to override appearance from your own stylesheet:

| Class | When present | |---|---| | .multiset-ar-button | Always — use for base styles | | .multiset-ar-inactive | No active session (START AR state) | | .multiset-ar-active | During an active AR session (STOP AR state) |

.multiset-ar-button {
  font-family: 'Your Font', sans-serif;
  border-radius: 8px;
}

.multiset-ar-inactive {
  background: rgba(0, 0, 0, 0.5);
}

.multiset-ar-active {
  background: rgba(200, 0, 0, 0.6);
  border-color: #ff4444;
}

Custom Start / Stop Button

Set useDefaultButton: false to manage the session yourself:

const adapter = new ThreeAdapter({
  session,
  renderer, scene, camera,
  useDefaultButton: false,
});

adapter.initialize();

myButton.addEventListener('click', () => {
  if (adapter.isActive()) {
    adapter.stopSession();
  } else {
    // startSession() must be the first call inside the gesture handler —
    // any await before it consumes the browser's user activation token.
    void adapter.startSession();
  }
});

Type Definitions

import type {
  IMultisetClientConfig,
  IMultisetSdkEndpoints,
  IXRSessionOptions,
  IXRFrameEvent,
  IFrameCaptureEvent,
  ICameraIntrinsicsEvent,
  IPoseResultEvent,
  ILocalizeAndMapDetails,
  IObjectTrackingResponse,
  IGetMapsDetailsResponse,
  MapType,
} from '@multisetai/vps/core';

import type { IThreeAdapterOptions } from '@multisetai/vps/three';

IObjectTrackingResponse

Returned by trackObjects() and MultisetClient.trackObject() when an object is detected.

| Field | Type | Description | |---|---|---| | poseFound | boolean | Whether the API found a valid pose for the object. | | position | IPosition | Position of the detected object in tracker space { x, y, z }. | | rotation | IRotation | Orientation of the detected object as a quaternion { x, y, z, w }. | | confidence | number | Detection confidence score (0–1). | | objectCodes | string[] | The object code(s) that were matched. |

License