artoolkit5-ts 🎯

TypeScript marker tracking for the browser, built on a WebAssembly build of ARToolkit5 (WebARKitLib).

This is the replacement for artoolkit5-js, which is a fork of andypotato/artoolkit5-js — itself an ES6 module port of artoolkit5.

artoolkit5-ts is a rewrite rather than another fork in that line. It is written in TypeScript against a maintained WebAssembly build, and it drops the monolithic ARController class those ports carried forward in favour of plain data and free functions. Nothing is hidden behind a class, so nothing has to be constructed before it can be used, tested, or tree-shaken.

⚠️ Status: alpha. Pattern markers work end to end, but the API is not stable yet — expect breaking changes before 1.0. See Roadmap.

🧩 Where this fits

artoolkit5-ts is the detection engine layer of the AR.js-next ecosystem:

AR.js-next                ECS core, event bus, frame pump
  arjs-plugin-artoolkit     ECS plugin: Web Worker, ImageBitmap, marker events
    artoolkit5-ts           ← this library
      artoolkit5-wasm       Emscripten / C++ bindings
        artoolkit5-constants  ARToolkit5 constants, extracted from the headers

It is renderer-agnostic and DOM-free. It gives you marker poses as matrices; what you draw with them is your business — Three.js, Babylon.js, raw WebGL, or nothing at all.

📦 Installation

npm install @ar-js-org/artoolkit5-ts

@ar-js-org/artoolkit5-wasm (^0.3.0) provides the WebAssembly engine. It installs automatically as a dependency, and is left external rather than bundled so the .wasm binary is fetched once and cached instead of being copied into every bundle that depends on it.

three is only needed to run the examples, not the library.

🚀 Quick start

import {
  createARToolKitState,
  loadPatternMarker,
  trackMarker,
  processFrame,
} from '@ar-js-org/artoolkit5-ts';

// 1. Initialise once — loads the WASM module and camera calibration
const state = await createARToolKitState(640, 480, './data/camera_para.dat');

// 2. Register the markers you care about.
//    The ID is assigned by the engine — never hardcode it.
const markerId = await loadPatternMarker(state, './data/patt.hiro');
trackMarker(state, markerId, 1.0);

// 3. Per frame: pass RGBA pixels in, get poses out.
//    Draw your video to a canvas and read it back; the library never
//    touches the DOM, so obtaining the pixels is your side of the line.
const pixels = ctx.getImageData(0, 0, 640, 480).data;
const { detected, lost } = processFrame(state, pixels);

for (const marker of detected) {
  // marker.matrixGL is a 4x4 column-major right-handed matrix,
  // ready to hand to WebGL or Three.js.
  // With Three.js, set mesh.matrixAutoUpdate = false once beforehand,
  // or it recomputes the matrix from position/quaternion/scale and
  // discards the pose you just wrote.
  mesh.matrix.fromArray(marker.matrixGL);
}

// `lost` holds markers that were visible last frame and are not now —
// reported once, on the frame they disappear
for (const marker of lost) {
  hideObjectFor(marker.type, marker.id);
}

Two complete working examples — webcam capture, marker tracking and a Three.js cube overlay:

npm run dev

examples/webcam tracks a pattern marker; you will need the Hiro marker printed or on a second screen. examples/barcode tracks a matrix code marker instead — the marker image it needs ships in examples/barcode/data/.

🧠 Why functions instead of a controller class

The ARController that artoolkit5-js inherited from jsartoolkit5 was a God Object: it owned the WASM module, the canvas, the video element, marker state and the render loop. That made it impossible to tree-shake, awkward to run in a Worker, and hard to test without a browser.

Here, ARToolKitState is a plain data container with no methods, and every operation takes it as its first argument:

• Tree-shakeable — you bundle only the functions you import
• Testable — functions take input and return output; no mocking a class hierarchy
• Worker-friendly — no DOM anywhere in src/, so state can live off the main thread
• Framework-agnostic — nothing assumes React, Vue, or any renderer

The trade-off is deliberate: this library will not open your camera, create a canvas, or run a render loop for you. Those belong to your application.

📖 API

createARToolKitState(width, height, cameraUrl, wasmUrl?)

Initialises the WASM module and camera parameters. Returns Promise<ARToolKitState>.

| Parameter | Type | Description | |---|---|---| | width | number | Frame width; must match the frames you pass to processFrame | | height | number | Frame height | | cameraUrl | string | URL of an ARToolKit camera_para.dat calibration file | | wasmUrl | string? | Explicit URL for artoolkit5.wasm. Required when your bundler rewrites asset paths, as Vite does |

loadPatternMarker(state, markerUrl)

Downloads a .patt file, writes it to the WASM virtual filesystem and registers it. Returns Promise<number> — the engine-assigned marker ID.

Loading a marker does not start tracking it; pass the ID to trackMarker.

trackMarker(state, pattId, markerWidth?)

Registers a marker for tracking and allocates its reusable pose buffers.

markerWidth defaults to 1.0. Whatever unit you choose here is the unit all returned translations are expressed in — use millimetres if you want millimetres.

trackBarcodeMarker(state, barcodeId, markerWidth?)

Registers a barcode (matrix code) marker for tracking. Unlike a pattern marker, there is nothing to load first: the ID is encoded directly in the marker's geometry, so barcodeId is a value you choose when generating the marker, not one the engine assigns — pass it straight to this function.

Detecting a barcode marker also requires configureDetector to have set a matrix-capable detectionMode ('matrix', 'color_and_matrix', or 'mono_and_matrix') and a matrixCodeType matching the marker.

Pattern and barcode markers have independent ID spaces, and are kept in separate registries. Pattern IDs are assigned by the engine starting at 0; barcode IDs are encoded in the marker's own geometry and chosen by whoever printed it. So 7 in one family is unrelated to 7 in the other, and both can be tracked at once:

// Pattern IDs come from the engine — never hardcode them
const patternId = await loadPatternMarker(state, './data/patt.hiro');
trackMarker(state, patternId);           // -> state.patternMarkers

// Barcode IDs are yours: encoded in the marker you printed
trackBarcodeMarker(state, 0);            // -> state.barcodeMarkers

If patternId also happens to be 0 — and it usually is, since the engine assigns from zero — both are tracked independently. Detections and losses carry type, so you can always tell which family a result came from.

The engine reports each family through its own field (idPatt / idMatrix), so a detection is only ever matched against the registry it belongs to.

configureDetector(state, opts)

Tunes the underlying detector. Only the keys you pass are changed — call it again later with a single option to adjust just that one, mid-session.

configureDetector(state, {
  detectionMode: 'matrix',       // 'color' | 'mono' | 'matrix' | 'color_and_matrix' | 'mono_and_matrix'
  matrixCodeType: '4x4_BCH_13_9_3',
  thresholdMode: 'auto_otsu',    // 'manual' | 'auto_median' | 'auto_otsu' | 'auto_bracketing'
  threshold: 100,                // 0–255, only meaningful when thresholdMode is 'manual'
  labelingMode: 'black_region',  // 'white_region' | 'black_region' — the engine default
  imageProcMode: 'frame',        // 'frame' | 'field'
  patternRatio: 0.5,             // > 0 and < 1, exclusive
  nearPlane: 1,
  farPlane: 1000,
  minConfidence: { pattern: 0, barcode: 0 },   // 0–1 per family; see below
                                               // before choosing a value
});

An invalid string value or an out-of-range threshold/patternRatio throws ARToolKitError naming the option and, for string options, listing what it does accept — the engine itself would otherwise silently ignore the bad value and keep its previous setting, which is a much harder bug to notice.

'auto_adaptive' threshold mode is not offered: the WebARKitLib build this library ships compiles that mode's implementation out, so passing it would silently degrade to 'manual' while claiming to work.

minConfidence — rejecting weak matches

Every other option here is handed to the engine. minConfidence is the exception: ARToolKit's own confidence cutoff is a compile-time constant with no setter, so this threshold is applied by processFrame instead. It can only ever be stricter than the engine's built-in 0.5.

The two families take separate thresholds because their confidences are not comparable.

There is no safe default value, and this library does not ship one. Measured on a real camera with a Hiro pattern marker and 3x3 matrix markers:

| | genuine match | false match | |---|---|---| | pattern (template matching) | 0.506 – 0.923 | 0.526 – 0.554 (read off a barcode square) | | barcode (matrix code) | 0.500 – 1.000 | 0.633 – 0.867 (read off a pattern square) |

Both ranges overlap, in both directions. A genuine pattern match scored 0.506, below a false one at 0.554. A genuine barcode scored 0.500 at an awkward angle while a phantom barcode — the engine decoding a Hiro marker's interior as a 3x3 grid — reached 0.867. The same barcode marker, in the same detection mode minutes apart, ranged from 0.500 to 0.967 purely on viewing angle and focus.

So confidence is a continuous quality score, not a verdict, for both families. Matrix codes are not digital in this respect: a clean decode does not imply 1.0.

What that means in practice:

• Both thresholds default to 0 — nothing is filtered beyond the engine's own 0.5 cutoff.
• Any threshold you set trades missed real markers against admitted phantoms. There is no value that avoids both.
• Measure your markers, in your lighting, at the angles you expect. Log marker.confidence for a while before choosing a number.
• A threshold is most defensible when you control the conditions — fixed mounting, known print quality, consistent lighting — and least defensible in an uncontrolled environment.

processFrame(state, videoFrame)

Detects registered markers in one frame. Returns a FrameResult:

interface FrameResult {
  detected: MarkerPose[];  // visible in this frame
  lost: LostMarker[];      // { id, type } visible last frame, gone in this one
}

Each lost entry carries type as well as id, because the two families have independent ID spaces — a pattern 7 and a barcode 7 may both be registered, and an ID alone could not say which disappeared.

lost is reported exactly once, on the frame a marker disappears — it does not repeat while the marker stays absent. Tracking already computes this transition internally, so exposing it saves every consumer from diffing successive results to recover it.

videoFrame is a Uint8ClampedArray of RGBA pixels matching the width and height the state was created with — typically ctx.getImageData(...).data.

This runs on every animation frame and allocates no typed arrays: poses are written into buffers owned by the marker's tracking state, and those buffers are reused next frame. Copy the values if you need to retain them.

disposeARToolKitState(state)

Releases the WASM resources the state holds. Call it when tracking stops — otherwise a page that starts and stops AR leaks the C++ instance and its heap allocations every time.

const state = await createARToolKitState(640, 480, cameraUrl);
// … track markers …
disposeARToolKitState(state);

Safe to call more than once. Afterwards every other operation on that state throws ARToolKitError rather than reaching freed memory, so a use-after-dispose gives you a clear message instead of a crash inside the WASM module.

ARToolKitError

Thrown for misuse of this API — currently, using a state after disposing it. Distinct from a plain Error so you can tell an API mistake apart from a failure inside the WASM module or your own code.

getCameraProjectionMatrix(state)

Returns the 4×4 projection matrix ARToolKit computed from your camera_para.dat, as a Float64Array. Use it in place of a generic perspective camera: it carries the measured focal length and principal point of the actual lens, so rendered geometry lines up with the video rather than merely sitting near it. (Radial distortion is not part of this matrix — no projection matrix can express it. ARToolKit corrects for it separately, when un-distorting detected marker corners.)

transMatToGLMat(transMat, out?) / arglCameraViewRHf(glMatrix, out?, scale?)

Matrix helpers, exported because they are occasionally useful directly. processFrame already applies both.

ARToolKit produces a 3×4 row-major pose; WebGL wants a 4×4 column-major matrix in a right-handed system. transMatToGLMat expands the matrix, arglCameraViewRHf negates the Y and Z axes. Without the second step, poses render behind the camera.

Both take an optional output buffer — supply one in hot paths to avoid allocating.

Types

ARToolKitState, MarkerPose, FrameResult, LostMarker, TrackedMarkerState, MarkerType, plus ARToolKitModule, ARToolKitCore and MarkerInfo describing the WASM boundary. DetectorOptions and its option types (DetectionMode, MatrixCodeType, ThresholdMode, LabelingMode, ImageProcMode) describe configureDetector's input.

interface MarkerPose {
  id: number;
  type: 'pattern' | 'barcode';
  confidence: number;      // 0–1, from this marker's own family
  matrix: Float64Array;    // 3x4, row-major, as ARToolKit produces it
  matrixGL: Float32Array;  // 4x4, column-major, right-handed, WebGL-ready
  vertex: [number, number][];  // the square's 4 corners, camera image coords
}

vertex gives the four corners of the detected square in camera image coordinates, origin at top-left — enough to outline a marker, hit-test it or build an occlusion mask without touching the camera projection matrix. Two things differ from the pose matrices:

• It is freshly allocated per frame, not a view onto a reused buffer, so it is safe to retain without copying. matrix and matrixGL are the opposite.
• Corner order follows the marker's rotation. vertex[(4 - dir) % 4] is the marker's own top-left corner, the rest clockwise from there. Outlining the square can ignore this; anything orientation-sensitive cannot.

The webcam example draws exactly this outline, marking corner 0 so the ordering is visible: see createOutlineDrawer in examples/webcam/main.ts.

confidence is read from the field belonging to the marker's family — cfPatt or cfMatrix — so it is comparable within a family but not across them. See minConfidence for measured ranges.

type says which family a detection came from. The engine reports the two through separate fields — idPatt for pattern markers, idMatrix for barcode markers — and each is matched only against its own registry, so type follows from which registry answered rather than from any value the engine supplies. This is also why the families have independent ID spaces: the same integer in each is two unrelated markers.

🖼️ Feeding frames from an ImageBitmap

processFrame takes raw pixels, so src/ never touches a canvas API. If your frames arrive as ImageBitmap — as they do in AR.js-next — convert them yourself, reusing one canvas rather than creating one per frame:

// The same dimensions the state was created with. Reading back any other
// size gives processFrame a buffer it will misinterpret.
const WIDTH = 640;
const HEIGHT = 480;

const canvas = new OffscreenCanvas(WIDTH, HEIGHT);
const ctx = canvas.getContext('2d', { willReadFrequently: true })!;

function toPixels(bitmap: ImageBitmap): Uint8ClampedArray {
  ctx.drawImage(bitmap, 0, 0, WIDTH, HEIGHT);
  return ctx.getImageData(0, 0, WIDTH, HEIGHT).data;
}

A helper that does this is on the roadmap; until then it is a few lines you own.

⚠️ Limitations

• Combined detection requires @ar-js-org/artoolkit5-wasm >= 0.3.0. 'color_and_matrix' and 'mono_and_matrix' rely on the per-mode marker fields (idPatt/idMatrix), which earlier versions of the binding did not expose — against 0.2.0 or older those modes silently detect nothing, or report the wrong marker. The dependency range already requires ^0.3.0; this matters only if you override it.
• Worker support is untested. Nothing in src/ touches the DOM, which is necessary but not proof — WASM instantiation in worker scope has not been verified.
• NFT markers are out of scope for this project — see Roadmap.

🗺️ Roadmap

Detailed design lives in docs/DESIGN-v0.1.md and, for the detector and barcode work, docs/DESIGN-detector-and-barcode.md; work is tracked in issues.

v0.1 (done) — lifecycle, packaging, marker-lost reporting from processFrame, a test suite and CI.

v0.2 (done) — configureDetector, barcode markers, independent ID registries for the two families, combined pattern+barcode detection verified against a real camera, and per-family match confidence.

Next — a verified Worker example, an ImageBitmap conversion helper, and multi-marker sets.

Out of scope — NFT tracking. This project and artoolkit5-wasm cover pattern and barcode markers; NFT belongs to other projects in the ecosystem.

🛠️ Development

npm run dev        # Vite dev server, opens the webcam example
npm run build      # library build (ES + UMD) plus type declarations
npm run preview    # preview the production build
npm test           # run the test suite once
npm run test:watch # re-run tests on change
npm run typecheck  # tsc --noEmit

Tests

Vitest covers the matrix maths, the marker visibility state machine and the dispose lifecycle. The suite runs in well under a second because the WASM boundary is faked: test/mock-core.ts stands in for the Emscripten module and the bound C++ instance, so neither a browser nor a compiled binary is needed.

The suite aims at the code that fails quietly rather than at a line-count target — a transposed matrix still renders, just in the wrong place, and a marker-lost event that fires twice looks fine until something downstream double-handles it.

It is validated by mutation: deliberately breaking the collection order, the Float32Array return type, or the continuous-tracking condition each makes exactly one test fail. If you add tests, check they can actually fail.

Contributing

Branch from dev; main holds release-ready code only. Commits follow Conventional Commits. Fuller guidance is in AGENTS.md.

Releasing

Releases are cut by the Release workflow, run manually from the Actions tab. Its only required input is the version to publish, without a leading v — for example 0.1.0.

Everything after that is automatic: it runs typecheck, tests and build, sets the version, promotes the changelog, derives release notes from the commits, commits, tags vX.Y.Z, creates the GitHub Release and publishes to npm with provenance — so the package carries a verifiable link back to the commit and workflow run that built it.

Run it with dry_run first. That performs every check and prints the notes and the tarball contents without tagging, committing or publishing. It is the only way to rehearse: npm never allows a published version to be replaced.

Before running for real, the workflow refuses to start unless:

• the version is valid semver, not already tagged, and not already on npm
• the branch is main
• the repository is public — npm will not generate provenance from a private repository

Preparing a release means writing the changelog. Add entries to ## [Unreleased] as you go; the workflow renames that heading to the released version and opens a fresh one. Anything between <!-- promote:strip --> markers is dropped during promotion, so notes meant only for editors do not survive into a released section. npm run release-notes prints the Conventional Commits since the last tag if you want to see what has accumulated.

It is a single workflow rather than a "create release" and a "publish" pair because a Release created with the default GITHUB_TOKEN does not trigger other workflows — GitHub blocks that to prevent recursion, so the second one would silently never fire.

📄 Licence

MIT — see LICENSE.

This library wraps a WebAssembly build of ARToolkit5 (WebARKitLib), which is licensed under the LGPL v3.0. The MIT licence covers this TypeScript code, not the engine underneath: redistributing a build that includes the ARToolkit5 (WebARKitLib) WebAssembly binary carries that licence's obligations as well.