MultiSet VPS WebXR

TypeScript SDK for integrating MultiSet's Visual Positioning System (VPS) into WebXR applications. Provides 6-DOF localization by matching camera frames against cloud-hosted maps, and object tracking by matching camera frames against registered 3D objects.

Contents

• Architecture
• Installation
• Requirements
• CORS Configuration
• Quick Start
  • VPS — Three.js
  • VPS — Vanilla / WebGL2
  • Object Tracking — Three.js
  • Needle Engine
    • Unity Inspector Workflow
    • MapAnchor — zero-code object placement
• Canvas Visibility During AR
• API Reference
  • MultisetClient
  • XRSessionManager
  • ThreeAdapter
  • NeedleAdapter
  • MapAnchor
  • MultisetVPS
• Placing Content at Map Coordinates
• Styling the AR Button
• Custom Start / Stop Button
• Type Definitions
• Troubleshooting
  • AR button doesn't appear
  • MapAnchor not appearing after localization
  • Component start() never called

Architecture

The SDK is split into independent entry points so you only install what you need:

| Entry point | Contents | Peer deps | |---|---|---| | @multisetai/vps/core | MultisetClient + XRSessionManager | None | | @multisetai/vps/three | ThreeAdapter | three >=0.169.0 | | @multisetai/vps/needle | NeedleAdapter | @needle-tools/engine (brings its own three) |

XRSessionManager owns the full vanilla WebXR session lifecycle — frame loop, camera capture, localization, tracking-loss recovery — with zero Three.js dependency. ThreeAdapter and NeedleAdapter wire it to a renderer and scene.

Installation

# Core only (no Three.js)
npm install @multisetai/vps

# With Three.js adapter
npm install @multisetai/vps three

# With Needle Engine — no separate `three` install needed.
# Needle Engine ships its own three.js fork as a dependency.
npm install @multisetai/vps

Requirements

• HTTPS — WebXR requires a secure context (https:// or http://localhost).
• Android + ARCore — Chrome or Edge on an ARCore-capable Android device (Android 8+, Chrome 81+).
• Three.js ≥ 0.169.0 — only required when using @multisetai/vps/three. Compatible through the latest release (r184+).
• Needle Engine ≥ 5.0.0 — only required when using @multisetai/vps/needle. Needle ships its own three fork — do not install three separately.

iOS is not supported. This SDK requires the camera-access WebXR feature. Safari on iOS does not implement it.

CORS Configuration

The SDK makes direct browser-to-API requests, so your domain must be whitelisted in the MultiSet dashboard.

1. Open the MultiSet Dashboard
2. Go to Credentials → Settings → Domains
3. Click Add + and enter your origin (e.g. https://localhost:5173 for dev, https://your-app.com for prod)

Without this, the browser will block every API request with a CORS error.

Quick Start

VPS Localization — Three.js

import * as THREE from 'three';
import { MultisetClient, XRSessionManager } from '@multisetai/vps/core';
import { ThreeAdapter } from '@multisetai/vps/three';

// Check support before showing any AR UI
const supported = await ThreeAdapter.isSupported();
if (!supported) {
  console.warn('WebXR immersive-ar is not supported on this device.');
}

const client = new MultisetClient({
  clientId: 'CLIENT_ID',
  clientSecret: 'CLIENT_SECRET',
  code: 'MAP_OR_MAPSET_CODE',
  mapType: 'map',
});

await client.authorize();

const renderer = new THREE.WebGLRenderer({ antialias: true });
document.body.appendChild(renderer.domElement);

const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(70, window.innerWidth / window.innerHeight, 0.01, 1000);

const session = new XRSessionManager(renderer.getContext() as WebGL2RenderingContext, {
  client,
  overlayRoot: document.body,
  autoLocalize: true,
  confidenceCheck: true,
  confidenceThreshold: 0.5,
  onSessionStart: () => {
    // Hide the canvas during AR — the XR compositor owns the display.
    renderer.domElement.style.display = 'none';
  },
  onSessionEnd: () => {
    renderer.domElement.style.display = 'block';
  },
  onLocalizationResult: (result) => console.log('Localized:', result.localizeData.position),
  onLocalizationFailure: (reason) => console.warn('Failed:', reason),
  onError: (err) => console.error(err),
});

const adapter = new ThreeAdapter({ session, renderer, scene, camera, showMesh: true });
adapter.initialize(); // mounts the built-in START AR / STOP AR button

// Add your 3D content
scene.add(new THREE.Mesh(
  new THREE.BoxGeometry(0.1, 0.1, 0.1),
  new THREE.MeshBasicMaterial({ color: 0xff0077 })
));

VPS Localization — Without Three.js (WebGL2 / Vanilla)

In this mode the SDK manages the session lifecycle, camera capture, and localization. You are responsible for all rendering: each XR frame, draw your scene into event.baseLayer.framebuffer using the provided gl context. This approach works with any WebGL2-based renderer — Babylon.js, raw WebGL, or your own engine.

import { MultisetClient, XRSessionManager } from '@multisetai/vps/core';

const supported = await XRSessionManager.isSupported();
if (!supported) {
  console.warn('WebXR immersive-ar is not supported on this device.');
}

const client = new MultisetClient({ clientId: '...', clientSecret: '...', code: '...', mapType: 'map' });
await client.authorize();

// A WebGL2 context is required — WebXR renders into a GL framebuffer, not a 2D canvas.
const gl = document.querySelector('canvas')!.getContext('webgl2')!;

const session = new XRSessionManager(gl, {
  client,
  overlayRoot: document.body,
  autoLocalize: true,
  onLocalizationResult: (result) => console.log('Localized:', result.localizeData.position),
  onError: (err) => console.error(err),
});

// Wire your render loop — called every XR frame with the current pose and framebuffer.
session.setXRFrameHandler((event) => {
  // Bind event.baseLayer.framebuffer and render your scene using event.view for camera matrices.
});

document.body.appendChild(session.createButton());

Object Tracking — Three.js

Object tracking detects and poses registered 3D objects by matching a captured camera frame against the MultiSet cloud.

import * as THREE from 'three';
import { MultisetClient, XRSessionManager } from '@multisetai/vps/core';
import { ThreeAdapter } from '@multisetai/vps/three';

const client = new MultisetClient({
  clientId: 'CLIENT_ID',
  clientSecret: 'CLIENT_SECRET',
  mapType: 'object-tracking',
  code: ['YOUR_OBJECT_CODE'],
});

await client.authorize();

const renderer = new THREE.WebGLRenderer({ antialias: true });
document.body.appendChild(renderer.domElement);
const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(70, window.innerWidth / window.innerHeight, 0.01, 1000);

const session = new XRSessionManager(renderer.getContext() as WebGL2RenderingContext, {
  client,
  overlayRoot: document.body,
  autoTracking: true,          // detect once on session start
  confidenceCheck: true,
  confidenceThreshold: 0.5,
  onSessionStart: () => { renderer.domElement.style.display = 'none'; },
  onSessionEnd:   () => { renderer.domElement.style.display = 'block'; },
  onObjectTrackingSuccess: (result) => {
    console.log('Detected', result.objectCodes, 'at', result.position);
  },
  onObjectTrackingFailure: (reason) => console.warn('Tracking failed:', reason),
  onError: (err) => console.error(err),
});

const adapter = new ThreeAdapter({
  session,
  renderer,
  scene,
  camera,
  showObjectMeshes: true,       // load and display the 3D outline mesh
  onObjectMeshLoaded: (code) => console.log('Mesh loaded for', code),
});
adapter.initialize();

// Trigger tracking manually from a button
trackButton.addEventListener('click', () => {
  void adapter.trackObjects();
});

Needle Engine

NeedleAdapter is a Needle Engine Behaviour component. Add it to a GameObject via addNewComponent() from your own component's start(). The built-in START AR / STOP AR button mounts automatically — no initialize() call needed.

Important: Do not add a Needle WebXR component to the same scene. NeedleAdapter must own the WebXR session directly because it bypasses Three.js's built-in XR manager (renderer.xr). This is required to work around a Chrome/WebXR regression where enabling camera-access corrupts Three.js's internal texture state. Adding a Needle WebXR component would start a second session and conflict with this setup. Delete the WebXR component from the scene hierarchy before adding MultisetVPS.

Unity Inspector workflow

MultisetVPS and MapAnchor are ready-made components. Copy a few template files into your project, fill in credentials in the Inspector — no code required.

Why template files? Needle Engine's Unity codegen scans only your local src/scripts/ folder to generate C# stubs — it skips node_modules. Copying the templates into your project gives Unity the component definitions it needs, while all SDK utilities (NeedleAdapter, MultisetClient, etc.) are still imported from the package.

1. Install in your Needle web project:

   npm install @multisetai/vps

2. Copy the TypeScript templates into your Needle project's scripts folder (the folder Needle scans for components — web/src/scripts/ by default, but use whatever path your project uses):

   cp node_modules/@multisetai/vps/templates/MultisetVPS.ts <your-web-folder>/src/scripts/
   cp node_modules/@multisetai/vps/templates/MapAnchor.ts   <your-web-folder>/src/scripts/
   cp node_modules/@multisetai/vps/templates/MapType.ts     <your-web-folder>/src/scripts/

   MapType.ts must be present alongside MultisetVPS.ts — Needle's component compiler resolves the enum from this file to generate the Map Type dropdown. Without it the component will not appear in the Inspector.

   Needle Engine automatically picks up everything in your scripts folder and generates the matching C# stubs in Unity.

3. Copy the C# enum into your Unity project's Assets/ folder (one-time setup — this file is never overwritten by Needle). Run this from inside your web folder:

   cp node_modules/@multisetai/vps/templates/MapType.cs ../Assets/

   This gives you a Map Type dropdown in the Inspector (SingleMap / MapSet / ObjectTracking). Without it, Needle generates the field but Unity can't compile the enum reference.

4. In Unity, add a MultisetVPS component to any GameObject and fill in clientId, clientSecret, and mapCode in the Inspector.

5. Add MapAnchor to every GameObject you want placed in AR. Configure offset, matchOrientation, rotationOffset, hideUntilLocalized, and isRightHanded in the Inspector. MultisetVPS discovers all MapAnchor instances automatically at startup — no manual wiring needed.

6. Export to web — the START AR / STOP AR and CAPTURE FRAME buttons appear automatically.

For runtime-spawned objects (loaded from an API, instantiated mid-session), call adapter.registerAnchor(anchor) — see MapAnchor — zero-code object placement below.

Code-based setup (without Unity Inspector)

import { Behaviour, serializable, addNewComponent } from '@needle-tools/engine';
import { MultisetClient } from '@multisetai/vps/core';
import { NeedleAdapter } from '@multisetai/vps/needle';
import * as THREE from 'three';

export class MyARComponent extends Behaviour {
  async start() {
    const client = new MultisetClient({
      clientId: 'CLIENT_ID',
      clientSecret: 'CLIENT_SECRET',
      code: 'MAP_CODE',
      mapType: 'map',
    });

    const adapter = new NeedleAdapter({
      client,
      showMesh: true,
      sessionOptions: {
        autoLocalize: true,
        onSessionStart: () => console.log('AR started'),
        onSessionEnd:   () => console.log('AR ended'),
        onLocalizationFailure: (reason) => console.warn('Failed:', reason),
        onError: (err) => console.error(err),
      },
      onLocalizationSuccess: (result, worldFromMap) => {
        // Place content anchored to the scanned map
        const marker = new THREE.Mesh(
          new THREE.SphereGeometry(0.05),
          new THREE.MeshBasicMaterial({ color: 0x00ff88 })
        );
        marker.position.applyMatrix4(worldFromMap);
        this.context.scene.add(marker);
      },
    });

    // addNewComponent triggers awake() — button mounts automatically
    addNewComponent(this.gameObject, adapter);
    await client.authorize();
  }
}

MapAnchor — zero-code object placement (Unity)

MapAnchor is a Needle Engine component that anchors any GameObject to the VPS map origin after a successful localization. Add it to the object you want to place in AR from the Unity Inspector — no code required.

| Inspector field | Type | Description | |---|---|---| | Offset | Vector3 | Position offset from the map origin in metres. Enter Unity Inspector values directly — X is negated automatically unless Is Right Handed is enabled. | | Match Orientation | bool | Align the object's rotation to the map's orientation. | | Rotation Offset | Vector3 (degrees) | Additional rotation applied on top of the map orientation. Only used when Match Orientation is enabled. Enter Unity Inspector values — Y and Z are negated automatically unless Is Right Handed is enabled. | | Hide Until Localized | bool | Hide the object until the first successful localization. Re-hides it when the AR session ends. | | Is Right Handed | bool | Off by default. When off, Offset and Rotation Offset are treated as Unity (left-handed) values and converted automatically. Enable only if you are entering Three.js (right-handed) values directly. |

MultisetVPS (or your own component calling addNewComponent) automatically discovers all MapAnchor instances in the scene at startup and wires them to the adapter. You do not call any method manually.

Important: When hideUntilLocalized is true, Needle sets visible = false on the GameObject in awake(). An invisible GameObject is inactive in Needle — every component on it, not just MapAnchor, will have its start() skipped. Keep MapAnchor on the object you want placed in AR. Put any logic components (session listeners, custom UI) on a separate, always-visible GameObject.

For objects spawned at runtime (e.g. from API data or a prefab instantiated mid-session), register them explicitly so they receive localization events and are placed immediately if localization has already succeeded:

import { MapAnchor } from '@multisetai/vps/needle';

// Spawn a new object at runtime and anchor it to the map
const obj = instantiate(myPrefab);
const anchor = addNewComponent(obj, new MapAnchor());
anchor.hideUntilLocalized = true;
anchor.offset.set(0.5, 0, -1.0);

// Connect it — places immediately if localization already succeeded
this.adapter.registerAnchor(anchor);

Using useDefaultButton: false (custom button):

const adapter = new NeedleAdapter({
  client,
  useDefaultButton: false,
  sessionOptions: { autoLocalize: true },
});

addNewComponent(this.gameObject, adapter);
await client.authorize();

myButton.addEventListener('click', () => {
  if (adapter.isActive()) {
    adapter.stopSession();
  } else {
    void adapter.startSession();
  }
});

Canvas Visibility During AR

Important — the SDK renders the Three.js scene into the XR framebuffer, not the canvas element. During an active AR session the canvas element is not updated; it retains whatever was last drawn by the preview loop. If the canvas is visible during AR (e.g. as part of the WebXR DOM overlay) it will appear as a frozen image on top of the AR scene.

Always hide the canvas when the session starts and restore it when it ends:

onSessionStart: () => { renderer.domElement.style.display = 'none'; },
onSessionEnd:   () => { renderer.domElement.style.display = 'block'; },

API Reference

MultisetClient

Pure HTTP client for auth, localization, and object tracking. No WebXR or rendering concerns.

new MultisetClient(config: IMultisetClientConfig)

IMultisetClientConfig

VPS mode (mapType: 'map' or mapType: 'map-set')

| Parameter | Type | Description | |---|---|---| | clientId | string | Your MultiSet client ID | | clientSecret | string | Your MultiSet client secret | | mapType | 'map' \| 'map-set' | Whether code is a single map or a map set | | code | string | Map or map-set code | | endpoints? | Partial<IMultisetSdkEndpoints> | Override default API endpoints | | isRightHanded? | boolean | Handedness sent to the API. Default true | | convertToGeoCoordinates? | boolean | Request geographic coordinates in the response | | hintPosition? | string | Local-space position hint "x,y,z" | | hintRadius? | number \| string | Search radius in metres (1–100). Requires hintPosition or passGeoPose | | hintMapCodes? | string[] | Narrow candidates by map code. Only valid when mapType: 'map-set' | | passGeoPose? | boolean | Send a geoHint from the Geolocation API with each request | | use2DFiltering? | boolean | Skip altitude in geo filtering. Only valid when passGeoPose: true |

Object tracking mode (mapType: 'object-tracking')

| Parameter | Type | Description | |---|---|---| | clientId | string | Your MultiSet client ID | | clientSecret | string | Your MultiSet client secret | | mapType | 'object-tracking' | Enables object tracking mode. No map code required. | | code | string[] | Object codes to detect and track | | isRightHanded? | boolean | Handedness sent to the API. Default true | | endpoints? | Partial<IMultisetSdkEndpoints> | Override default API endpoints |

Methods

| Method | Returns | Description | |---|---|---| | authorize() | Promise<string> | Authenticate and cache an access token. Call before any other method. | | localizeWithFrame(frame, intrinsics) | Promise<ILocalizeAndMapDetails \| null> | Submit a captured frame for VPS localization. | | trackObject(frame, intrinsics) | Promise<IObjectTrackingResponse \| null> | Submit a captured frame for object detection. Uses code from the client config. Returns null when no object is detected. | | downloadObjectMesh(objectCode) | Promise<string \| null> | Fetch a signed download URL for the 3D mesh of an object. | | fetchMapDetails(mapCode) | Promise<IGetMapsDetailsResponse \| null> | Fetch map metadata by code (result is cached). |

XRSessionManager

Owns the WebXR session lifecycle — frame loop, camera capture, localization, object tracking, and tracking-loss recovery. Zero Three.js dependency.

import { XRSessionManager } from '@multisetai/vps/core';

new XRSessionManager(gl: WebGL2RenderingContext, options: IXRSessionOptions)

IXRSessionOptions

| Parameter | Type | Description | |---|---|---| | client | MultisetClient | Required. | | overlayRoot? | HTMLElement | Root element for the WebXR DOM overlay. | | autoLocalize? | boolean | Run one localization automatically when the session starts. | | relocalization? | boolean | Re-localize whenever tracking is lost and then recovered. | | confidenceCheck? | boolean | Only accept results with confidence >= confidenceThreshold. Applies to both VPS and object tracking. | | confidenceThreshold? | number | Minimum confidence (0.2–0.8). Default 0.5. | | poseTimeoutMs? | number | Max ms to wait for a valid viewer pose before failing. Default 10000. | | localizationTrackingTimeoutMs? | number | Deprecated — use poseTimeoutMs. Will be removed in v3. | | backgroundLocalization? | boolean | Periodically send localization/tracking requests in the background while the session is active. | | bgLocalizationInterval? | number | Interval in seconds between background attempts. Clamped to 10–180 s. Default 30 for VPS modes, 10 for object tracking. | | autoTracking? | boolean | Call trackObjects() once automatically when the session starts. Requires mapType: 'object-tracking' on the client. | | restartTracking? | boolean | Re-attempt tracking whenever XR tracking is lost and then recovered. | | trackingCaptureDelayMs? | number | Milliseconds to wait before capturing a frame when trackObjects() is called. Useful for camera stabilisation. Default 0. | | referenceSpaceType? | XRReferenceSpaceType | XR reference space type. Default 'local'. Use 'local-floor' for floor-relative tracking if the device supports it. | | framebufferScaleFactor? | number | XR framebuffer scale relative to native resolution. Values < 1 reduce GPU load; values > 1 supersample. | | onSessionStart? | () => void | Called when the AR session starts. | | onSessionEnd? | () => void | Called when the AR session ends. | | onLocalizationInit? | () => void | Called at the start of a VPS localization run. | | onLocalizationResult? | (result: ILocalizeAndMapDetails) => void | Called when VPS localization succeeds (and passes the confidence check, if enabled). | | onLocalizationSuccess? | (result: ILocalizeAndMapDetails) => void | Deprecated — use onLocalizationResult. If using ThreeAdapter, use its onLocalizationSuccess which also provides worldFromMap. Will be removed in v3. | | onLocalizationFailure? | (reason?: string) => void | Called when VPS localization fails or falls below the confidence threshold. | | onFrameCaptured? | (frame: IFrameCaptureEvent) => void | Called when a camera frame is captured for localization. | | onCameraIntrinsics? | (intrinsics: ICameraIntrinsicsEvent) => void | Called with camera intrinsic parameters for the captured frame. | | onPoseResult? | (pose: IPoseResultEvent) => void | Called with the raw pose result from the VPS backend. | | onObjectTrackingInit? | () => void | Called at the start of an object tracking run. | | onObjectTrackingRequested? | (frame: IFrameCaptureEvent, intrinsics: ICameraIntrinsicsEvent) => void | Called just before the tracking request is sent, with the captured frame and intrinsics. | | onObjectTrackingSuccess? | (result: IObjectTrackingResponse) => void | Called when object tracking succeeds and passes the confidence check (if enabled). | | onObjectTrackingFailure? | (reason?: string) => void | Called when object tracking fails or falls below the confidence threshold. | | onError? | (error: unknown) => void | Called when any error occurs. | | onContextLost? | () => void | Called when the WebGL context is lost. The active session is ended automatically. | | onContextRestored? | () => void | Called when the WebGL context is restored. The user may restart the session. |

Static methods

| Method | Returns | Description | |---|---|---| | XRSessionManager.isSupported() | Promise<boolean> | Returns true if the browser supports immersive-ar WebXR sessions. Use this to conditionally show AR UI before creating any objects. |

Methods

| Method | Returns | Description | |---|---|---| | createButton() | HTMLButtonElement | Create the built-in styled AR button. Shows START AR / STOP AR and toggles the session on click. | | startSession() | Promise<void> | Start an AR session programmatically. Must be called from within a user gesture handler (click/tap). | | stopSession() | void | Stop the active AR session. No-op if no session is running. | | localizeFrame() | Promise<ILocalizeAndMapDetails \| null> | Capture and localize one frame against the configured map. Requires an active session. | | trackObjects() | Promise<IObjectTrackingResponse \| null> | Capture one frame and run object detection. Requires an active session and mapType: 'object-tracking' on the client. | | isActive() | boolean | Whether an XR session is currently running. | | isLocalizing | boolean | Whether a localization or tracking run is currently in progress. | | getClient() | MultisetClient | Access the underlying MultisetClient. | | dispose() | void | End the session, clear background timers, remove context loss listeners, and release all resources. |

Adapter hooks

Used internally by ThreeAdapter. Only call these when building a custom renderer adapter.

| Method | Description | |---|---| | setXRFrameHandler(fn) | Called every XR frame with pose, view, viewport, and framebuffer info. | | setAdapterResultHandler(fn) | Called after a successful VPS localization with the result and tracker-space matrix. | | setAdapterObjectTrackingHandler(fn) | Called after a successful object tracking result with the result and tracker-space matrix. | | setAdapterSessionHandlers(onStart, onEnd) | Called on session start/end, before user callbacks. |

ThreeAdapter

Wires XRSessionManager to a Three.js renderer. Handles XR framebuffer binding, camera matrix sync, preview loop, resize, and optional map mesh / gizmo / object mesh display.

import { ThreeAdapter } from '@multisetai/vps/three';

new ThreeAdapter(options: IThreeAdapterOptions)

IThreeAdapterOptions

| Parameter | Type | Description | |---|---|---| | session | XRSessionManager | Required. | | renderer | THREE.WebGLRenderer | Required. | | scene | THREE.Scene | Required. | | camera | THREE.PerspectiveCamera | Required. | | showMesh? | boolean | Show the VPS map mesh after localization. Default false. | | showGizmo? | boolean | Show a transform gizmo after localization. Default true. | | showObjectMeshes? | boolean | Load and display a 3D outline mesh for each detected object. Default false. | | useDefaultButton? | boolean | Mount the built-in START AR / STOP AR button. Default true. Set to false to drive the session via startSession() / stopSession(). | | buttonContainer? | HTMLElement | Where to append the built-in button. Defaults to overlayRoot or document.body. | | onButtonCreated? | (button: HTMLButtonElement) => void | Called after the built-in button is created. | | onXRFrame? | (event: IXRFrameEvent) => void | Called every XR frame after camera matrices are synced, before the scene is rendered. Use this to update scene objects each frame. | | onLocalizationSuccess? | (result: ILocalizeAndMapDetails, worldFromMap: THREE.Matrix4) => void | Called immediately after a successful VPS localization. worldFromMap transforms map-space coordinates to Three.js world space — use it to place content at known map coordinates. | | onObjectMeshLoaded? | (objectCode: string) => void | Called when a detected object's 3D mesh has been loaded and placed in the scene. Only fires when showObjectMeshes: true. |

Static methods

| Method | Returns | Description | |---|---|---| | ThreeAdapter.isSupported() | Promise<boolean> | Returns true if the browser supports immersive-ar WebXR sessions. |

Methods

| Method | Returns | Description | |---|---|---| | initialize(buttonContainer?) | HTMLButtonElement \| null | Start the preview render loop, attach resize handler, and mount the built-in button. Returns null when useDefaultButton: false. | | isActive() | boolean | Whether an XR session is currently running. | | isLocalizing | boolean | Whether a localization or tracking run is currently in progress. | | startSession() | Promise<void> | Start an AR session. Must be called from within a user gesture handler. | | stopSession() | void | Stop the active AR session. No-op if no session is running. | | localizeFrame() | Promise<ILocalizeAndMapDetails \| null> | Capture and localize one frame. | | trackObjects() | Promise<IObjectTrackingResponse \| null> | Capture one frame and run object detection. Requires mapType: 'object-tracking' on the client. | | clearObjectMeshes() | void | Remove all object meshes from the scene that were placed by showObjectMeshes. | | dispose() | void | Stop loops, remove listeners, dispose Three.js resources, and end the session. |

NeedleAdapter

Needle Engine Behaviour that wires XRSessionManager to Needle's renderer and scene. Handles XR framebuffer binding, camera matrix sync, AR passthrough (transparent background), and optional map mesh / gizmo / object mesh display.

import { NeedleAdapter } from '@multisetai/vps/needle';

new NeedleAdapter(options: INeedleAdapterOptions)

Add to a scene via addNewComponent(gameObject, adapter) — this triggers awake(), which creates the session and mounts the button.

INeedleAdapterOptions

| Parameter | Type | Description | |---|---|---| | client | MultisetClient | Required. | | sessionOptions? | Omit<IXRSessionOptions, 'client'> | All session options and callbacks — forwarded directly to XRSessionManager. See IXRSessionOptions for the full list. | | showMesh? | boolean | Show the VPS map mesh after localization. Default false. | | showGizmo? | boolean | Show a transform gizmo after localization. Default false. | | showObjectMeshes? | boolean | Load and display a 3D outline mesh for each detected object. Default false. | | useDefaultButton? | boolean | Mount the built-in START AR / STOP AR button automatically in awake(). Default true. Set to false to drive the session via startSession() / stopSession(). | | buttonContainer? | HTMLElement | Where to append the built-in button. Defaults to overlayRoot or document.body. | | onButtonCreated? | (button: HTMLButtonElement) => void | Called after the built-in button is created. | | onXRFrame? | (event: IXRFrameEvent) => void | Called every XR frame after camera matrices are synced, before the scene is rendered. | | onLocalizationSuccess? | (result: ILocalizeAndMapDetails, worldFromMap: THREE.Matrix4) => void | Called after a successful VPS localization. worldFromMap transforms map-space coordinates to Three.js world space. | | onObjectMeshLoaded? | (objectCode: string) => void | Called when a detected object's 3D mesh has been loaded and placed in the scene. Only fires when showObjectMeshes: true. |

Static methods

| Method | Returns | Description | |---|---|---| | NeedleAdapter.isSupported() | Promise<boolean> | Returns true if the browser supports immersive-ar WebXR sessions. |

Methods

| Method | Returns | Description | |---|---|---| | isActive() | boolean | Whether an XR session is currently running. | | isLocalizing | boolean | Whether a localization or tracking run is currently in progress. | | startSession() | Promise<void> | Start an AR session. Must be called from within a user gesture handler. | | stopSession() | void | Stop the active AR session. No-op if no session is running. | | localizeFrame() | Promise<ILocalizeAndMapDetails \| null> | Capture and localize one frame. | | trackObjects() | Promise<IObjectTrackingResponse \| null> | Capture one frame and run object detection. Requires mapType: 'object-tracking' on the client. | | clearObjectMeshes() | void | Remove all object meshes placed by showObjectMeshes. | | registerAnchor(anchor) | void | Connect a dynamically created MapAnchor to the adapter. Registers localization listeners and immediately applies the last localization result if the session is already active. | | addLocalizationListener(fn) | void | Subscribe to localization events from any Needle component. fn receives (result, worldFromMap). Prefer this over the constructor onLocalizationSuccess when multiple components need to react. | | removeLocalizationListener(fn) | void | Unsubscribe a previously added localization listener. Call from onDestroy. | | addSessionStartListener(fn) | void | Subscribe to session-start events. Useful for showing in-session UI (buttons, overlays) when AR begins. | | removeSessionStartListener(fn) | void | Unsubscribe a previously added session-start listener. Call from onDestroy. | | addSessionEndListener(fn) | void | Subscribe to session-end events. Useful for hiding anchored content between sessions. | | removeSessionEndListener(fn) | void | Unsubscribe a previously added session-end listener. Call from onDestroy. | | getSession() | XRSessionManager | Access the underlying session manager directly. Use only when you need low-level session control. |

MapAnchor

Needle Engine Behaviour that anchors a GameObject to the VPS map origin after localization. Exported from @multisetai/vps/needle.

In Unity, add the component via the Inspector — MultisetVPS discovers and wires all MapAnchor instances at startup automatically.

For runtime-spawned objects use NeedleAdapter.registerAnchor() (see MapAnchor — zero-code object placement above).

Fields

| Field | Type | Default | Description | |---|---|---|---| | offset | THREE.Vector3 | (0, 0, 0) | Position offset from the map origin in metres. When isRightHanded is false (default), enter Unity Inspector values directly — X is negated automatically. When true, supply Three.js values as-is. | | matchOrientation | boolean | true | Align the object's rotation to the map's orientation. | | rotationOffset | THREE.Euler | (0°, 0°, 0°) | Additional rotation applied on top of the map orientation. Only applied when matchOrientation is true. When isRightHanded is false (default), enter Unity Inspector values — Y and Z are negated automatically. | | hideUntilLocalized | boolean | true | Hide the object until the first successful localization. Re-hides on session end. | | isRightHanded | boolean | false | When false (default), offset and rotationOffset are treated as Unity (left-handed) values and converted automatically (negate X in position; negate Y and Z in rotation). Set to true if you are supplying Three.js (right-handed) values directly. |

Methods

| Method | Description | |---|---| | connectAdapter(adapter) | Called automatically by MultisetVPS. Registers localization and session-end listeners on the adapter. | | applyLocalization(result, worldFromMap) | Apply a localization result directly — positions and shows the object. Called internally by registerAnchor; also useful for testing or custom placement logic. |

MultisetVPS

Inspector-driven Needle Engine component that bootstraps the full VPS stack — credentials, session, UI buttons, and MapAnchor discovery — from Unity Inspector fields alone. Exported from @multisetai/vps/needle.

import { MultisetVPS, MapType } from '@multisetai/vps/needle';

See Unity Inspector workflow for setup. The template files handle TypeStore registration automatically — no manual register() call needed.

Inspector fields

| Field | Type | Default | Description | |---|---|---|---| | clientId | string | "" | Your Multiset client ID. | | clientSecret | string | "" | Your Multiset client secret. | | mapCode | string | "" | Map/map-set code, or comma-separated object codes for object-tracking mode. | | mapType | MapType | SingleMap | SingleMap, MapSet, or ObjectTracking. | | showMesh | boolean | true | Show the VPS map mesh after localization (VPS modes only). | | showGizmo | boolean | true | Show a transform gizmo after localization (VPS modes only). | | showObjectMeshes | boolean | false | Load and display a 3D outline mesh for each detected object (OT mode only). | | autoLocalize | boolean | true | Run one localization automatically when the session starts (VPS modes). | | relocalization | boolean | false | Re-localize whenever tracking is lost and then recovered. | | backgroundLocalization | boolean | false | Periodically localize in the background while the session is active. | | bgLocalizationInterval | number | 0 | Seconds between background localization attempts (10–180). 0 = SDK default. | | confidenceCheck | boolean | false | Only accept results above confidenceThreshold. | | confidenceThreshold | number | 0.5 | Minimum confidence (0.2–0.8). | | poseTimeoutMs | number | 0 | Max ms to wait for a valid viewer pose. 0 = SDK default (10 000 ms). | | convertToGeoCoordinates | boolean | false | Request geographic coordinates in the localization response. | | passGeoPose | boolean | false | Send a geolocation hint with each request. | | use2DFiltering | boolean | false | Skip altitude in geo filtering. Only valid when passGeoPose is enabled. | | hintPosition | string | "" | Local-space position hint "x,y,z". | | hintRadius | number | 0 | Search radius in metres around hintPosition. | | hintMapCodes | string | "" | Comma-separated map codes to restrict search (map-set mode only). | | autoTracking | boolean | false | Detect objects automatically when the session starts (OT mode). | | restartTracking | boolean | false | Re-attempt tracking when XR tracking is lost and recovered (OT mode). | | trackingCaptureDelayMs | number | 0 | Ms to wait before capturing a frame for tracking. |

Properties

| Property | Type | Description | |---|---|---| | adapter | NeedleAdapter \| null | The underlying NeedleAdapter after start() completes. Use this to call registerAnchor(), add listeners, or access the session directly. null until start resolves. |

Placing Content at Map Coordinates

After localization, the onLocalizationSuccess callback on ThreeAdapter provides a worldFromMap matrix that converts any point from VPS map space into Three.js world space. Use this to anchor scene objects to specific physical locations in the scanned map — independently of where the user started the AR session.

const adapter = new ThreeAdapter({
  session,
  renderer, scene, camera,
  onLocalizationSuccess: (result, worldFromMap) => {
    // mapPoint is a position you measured from the scanned map (in metres)
    const mapPoint = new THREE.Vector3(1.5, 0, -2.0);

    const marker = new THREE.Mesh(
      new THREE.SphereGeometry(0.05),
      new THREE.MeshBasicMaterial({ color: 0x00ff88 })
    );
    marker.position.copy(mapPoint.applyMatrix4(worldFromMap));
    scene.add(marker);
  },
});

Note — worldFromMap is recomputed on every successful localization. If you re-localize, update or re-add your objects so they stay in sync with the latest result.

Styling the AR Button

The built-in button ships with minimal inline styles. Use these CSS classes to override appearance from your own stylesheet:

| Class | When present | |---|---| | .multiset-ar-button | Always — use for base styles | | .multiset-ar-inactive | No active session (START AR state) | | .multiset-ar-active | During an active AR session (STOP AR state) |

.multiset-ar-button {
  font-family: 'Your Font', sans-serif;
  border-radius: 8px;
}

.multiset-ar-inactive {
  background: rgba(0, 0, 0, 0.5);
}

.multiset-ar-active {
  background: rgba(200, 0, 0, 0.6);
  border-color: #ff4444;
}

Custom Start / Stop Button

Set useDefaultButton: false to manage the session yourself:

const adapter = new ThreeAdapter({
  session,
  renderer, scene, camera,
  useDefaultButton: false,
});

adapter.initialize();

myButton.addEventListener('click', () => {
  if (adapter.isActive()) {
    adapter.stopSession();
  } else {
    // startSession() must be the first call inside the gesture handler —
    // any await before it consumes the browser's user activation token.
    void adapter.startSession();
  }
});

Type Definitions

import type {
  IMultisetClientConfig,
  IMultisetSdkEndpoints,
  IXRSessionOptions,
  IXRFrameEvent,
  IFrameCaptureEvent,
  ICameraIntrinsicsEvent,
  IPoseResultEvent,
  ILocalizeAndMapDetails,
  IObjectTrackingResponse,
  IGetMapsDetailsResponse,
  MapType,
} from '@multisetai/vps/core';

import type { IThreeAdapterOptions } from '@multisetai/vps/three';

import type { INeedleAdapterOptions } from '@multisetai/vps/needle';
import { NeedleAdapter, MapAnchor, MultisetVPS, MapType } from '@multisetai/vps/needle';

IObjectTrackingResponse

Returned by trackObjects() and MultisetClient.trackObject() when an object is detected.

| Field | Type | Description | |---|---|---| | poseFound | boolean | Whether the API found a valid pose for the object. | | position | IPosition | Position of the detected object in tracker space { x, y, z }. | | rotation | IRotation | Orientation of the detected object as a quaternion { x, y, z, w }. | | confidence | number | Detection confidence score (0–1). | | objectCodes | string[] | The object code(s) that were matched. |

Troubleshooting

AR button doesn't appear

• useDefaultButton: false set? — you must call startSession() from your own button. For ThreeAdapter, ensure you still call adapter.initialize(). For NeedleAdapter, the button mounts automatically in awake() unless useDefaultButton: false.
• authorize() not called? — call client.authorize() before any session or localization attempt.
• WebXR component conflict (Needle)? — remove any Needle WebXR component from the same scene. NeedleAdapter must own the WebXR session directly (it bypasses renderer.xr to avoid a Chrome/WebXR camera-access regression). A Needle WebXR component starts a second session that conflicts with this setup. Delete the WebXR component from the scene hierarchy before adding MultisetVPS.

Camera passthrough not showing (only 3D scene visible)

The XR compositor shows the camera feed wherever the rendered output has alpha = 0. If you see only the 3D scene with an opaque background, ThreeAdapter or NeedleAdapter handle this automatically. If you're using a custom renderer, set scene.background = null and renderer.setClearAlpha(0) at the start of each XR frame.

CORS error on every API request

Your domain is not whitelisted. Go to the MultiSet Dashboard → Credentials → Settings → Domains and add your origin (e.g. https://localhost:5173).

authorize() throws or returns 401

• Verify clientId and clientSecret are correct.
• Tokens expire — call authorize() again if you receive a 401 after a long session.

Localization always fails / low confidence

• Ensure the physical space matches the scanned map (lighting, object placement).
• Enable confidenceCheck: true and lower confidenceThreshold (default 0.5) if you want to accept lower-quality results.
• Try relocalization: true to automatically re-localize when tracking is lost.

WebGL context lost

The browser reclaims the GPU context on mobile when the app is backgrounded. onContextLost fires and the session ends automatically. Re-prompt the user to start a new session when onContextRestored fires.

MapAnchor not appearing after localization (Needle)

• Runtime-spawned anchors — MapAnchor instances created after MultisetVPS.start() (or your equivalent) are not auto-discovered by findObjectsOfType. Call adapter.registerAnchor(anchor) explicitly after adding the component. If localization has already succeeded the anchor is placed immediately via last-localization replay.
• connectAdapter never called — if you are not using MultisetVPS, ensure your setup code calls adapter.registerAnchor(anchor) (or anchor.connectAdapter(adapter)) for every MapAnchor in the scene. Without this the anchor never receives localization events.
• Wrong GameObject — confirm the MapAnchor component is on the exact object (or a parent of the object) you intend to place. The anchor repositions this.gameObject — not the scene root.

Component start() never called (Needle)

A component whose start() never fires is almost always on a GameObject that was marked invisible before start() ran. In Needle Engine, visible = false makes the whole object inactive — every component on it skips start().

The most common cause: the component is on the same GameObject as a MapAnchor that has hideUntilLocalized: true. MapAnchor.awake() sets visible = false immediately, which silently disables all sibling components including your own.

Fix: keep MapAnchor on the object you want placed. Move any other logic components (custom UI, session listeners, test utilities) to a different, always-visible GameObject in the scene.

startSession() throws "must be called from a user gesture"

Browsers only allow WebXR session creation in direct response to a user gesture (click/tap). Do not await anything before calling startSession() inside a click handler — any await consumes the activation token.

// Correct
button.addEventListener('click', () => {
  void adapter.startSession(); // no await before this
});

// Wrong — the await before startSession loses the gesture token
button.addEventListener('click', async () => {
  await someOtherThing();
  void adapter.startSession(); // will throw
});

License