ml-web-scanner

iOS-style document scanner for the web. Detects document corners with a DocAligner ONNX model running in ONNX Runtime Web — WebGPU when available, WASM (CPU) fallback — inside a Web Worker so the main thread never blocks.

• 🔵 iOS-look overlay: blue detection quad, dimmed surroundings, animated grid
• 🧵 ≥4 detections/sec in a background worker; spring-smoothed corner animation at render rate
• 📸 Manual capture with page-present check; shutter flash + shrink-to-page-count animation
• 📐 Perspective-corrected (de-skewed) page output via a dependency-free homography warp
• 🧩 Pure TypeScript core API and a ready-made Vue 3 component (@andor83/ml-web-scanner/vue)

Quick start (this repo)

npm install
npm run download-models    # fetches the DocAligner lcnet100 model into models/
npm run build
cd example
npm install
npm run build
node server.mjs            # → https://localhost:8443 (vanilla + Vue demos)

The example server (and npm run dev) defaults to HTTPS with a self-signed certificate — getUserMedia only works in secure contexts, and while http://localhost qualifies, opening the demo from a phone over the LAN does not: mobile Chrome silently blocks the camera without HTTPS. The cert is generated into example/.cert/ on first run (requires openssl, present on macOS/Linux) with your LAN IPs in the SANs and regenerates when they change; accept the browser's one-time warning per device. The server prints the LAN URLs to open from a phone. Set HTTP=1 node server.mjs for plain HTTP. Append ?ep=wasm to a demo URL to force the CPU execution provider.

Installation (your app)

npm install @andor83/ml-web-scanner
# vue is an optional peer dependency — only needed for @andor83/ml-web-scanner/vue

You host two kinds of assets yourself (see Serving assets):

1. a DocAligner heatmap model (.onnx), passed as modelUrl
2. onnxruntime-web's runtime files, passed as ortWasmPaths

Vue 3

<script setup lang="ts">
import { DocumentScannerView } from '@andor83/ml-web-scanner/vue';
import type { CapturedPage } from '@andor83/ml-web-scanner';
import '@andor83/ml-web-scanner/style.css';

function onPage(page: CapturedPage) {
  // page.image is a perspective-corrected ImageBitmap
}
</script>

<template>
  <DocumentScannerView
    model-url="/models/lcnet100_h_e_bifpn_256_fp32.onnx"
    ort-wasm-paths="/ort/"
    @page-captured="onPage"
  />
</template>

The component fills its parent and renders the camera, overlay, shutter button, page-count badge, and Cancel/Done buttons.

Props: model-url, ort-wasm-paths, engine (preloaded ScannerEngine), auto-capture (true or { steadyFrames } — captures once the quad holds still; read at mount), options (remaining ScannerOptions), show-hud.

Events: capture (a page was captured — manual, auto, or programmatic; page-captured is an alias), capture-complete (Done pressed; payload is all captured pages), cancel (Cancel pressed), corners, status-change, error, badge-click.

Slots (all optional, with built-in defaults):

| slot | default content | slot props | | --- | --- | --- | | top-left | Cancel button | pages, status, capture() | | top-right | empty | pages, status, capture() | | bottom-right | Done button (shown once pages exist) | pages, status, capture() | | badge | PageCountBadge | count, thumbnail | | loading | progress ring | progress |

Programmatic capture is also available through the defineExpose'd capture().

Pure TypeScript

import { DocumentScanner, OverlayRenderer, flash, flyToBadge } from '@andor83/ml-web-scanner';
import '@andor83/ml-web-scanner/style.css';

const video = document.querySelector('video')!;   // object-fit: cover
const canvas = document.querySelector('canvas')!; // absolutely positioned over the video

const scanner = new DocumentScanner({
  modelUrl: '/models/lcnet100_h_e_bifpn_256_fp32.onnx',
  ortWasmPaths: '/ort/',
  video,
});
const overlay = new OverlayRenderer(canvas, video);

scanner.on('smoothedCorners', ({ corners, opacity }) => {
  overlay.render(corners, opacity, scanner.status, performance.now());
});
scanner.on('pageCaptured', (page) => {
  flash(container);                       // shutter flash
  flyToBadge(container, page, video, badgeEl); // shrink into the page-count badge
});

await scanner.start();

captureButton.onclick = async () => {
  const page = await scanner.capture(); // null when no page is in view
};

API

new DocumentScanner(options)

| option | default | | | --- | --- | --- | | modelUrl | — (required) | URL of a DocAligner heatmap ONNX model | | video | internal element | existing <video> to attach to | | constraints | rear camera, 1920×1080 ideal | getUserMedia constraints | | detectionIntervalMs | 200 | minimum ms between detections (≥4 Hz) | | executionProviders | ['webgpu', 'wasm'] | preference order | | ortWasmPaths | ort default | base URL of onnxruntime-web runtime files | | workerUrl | bundled worker | escape hatch when your bundler can't handle the worker | | heatmapThreshold | 0.3 | corner heatmap binarization threshold | | cacheModel | true | store the model in IndexedDB; repeat visits skip the download | | smoothing.stiffness | 120 | corner spring stiffness | | autoCapture | disabled | { enabled, steadyFrames } — capture when the quad holds still |

Methods: start(), stop(), dispose(), capture(): Promise<CapturedPage | null>. Properties: status, pages, video, latestDetection.

Events (scanner.on(...)): loadProgress (model download bytes/fraction, then a compile stage during session creation — drive a loading indicator with this), corners (raw, ~detection rate), smoothedCorners (per animation frame), pageCaptured, statusChange (initializing | searching | detected | stable | captured | error), error.

Model loading starts immediately on start() and runs in parallel with the camera permission prompt. The Vue component (and both demos) show a progress ring while the model downloads/compiles and fade the camera view in once detection is live; in the Vue component the indicator is replaceable via the loading slot.

Preloading (start the model before the scanner is shown)

The worker + model live in a ScannerEngine that can be created eagerly at app startup — the model downloads and compiles immediately, and the scanner UI, whenever it appears, picks it up mid-load or fully ready:

// app entry, runs at load
import { ScannerEngine } from '@andor83/ml-web-scanner';
export const engine = new ScannerEngine({
  modelUrl: '/models/lcnet100_h_e_bifpn_256_fp32.onnx',
  ortWasmPaths: '/ort/',
});

// later, when the scanner UI mounts
const scanner = new DocumentScanner({ engine, video });
await scanner.start(); // only waits for the camera (and any remaining load)

or in Vue: <DocumentScannerView :engine="engine" /> (the model-url prop is then unnecessary). A supplied engine is never disposed by the scanner or the component, so it also survives unmount/remount — a route change back to the scanner view restarts instantly with no re-download and no re-compile. Call engine.dispose() yourself when the app truly no longer needs it. engine.ready (a promise), engine.isReady, and its loadProgress event let you build your own splash/loading UX.

Model caching

Downloaded model bytes are persisted in IndexedDB, keyed by modelUrl — serve the model under a versioned URL/filename to invalidate, or set cacheModel: false. (IndexedDB rather than the Cache API because Chrome treats self-signed-cert origins as untrustworthy and blocks the Cache API there — exactly the dev-over-LAN setup the example uses.) The compiled session itself cannot be persisted — neither onnxruntime-web nor TF.js can serialize compiled GPU programs — so the "Preparing model…" stage always runs, but keeping a preloaded ScannerEngine alive makes it a once-per-page-load cost, and Chrome's per-origin shader cache speeds it up on trusted (valid-cert or localhost) origins.

CapturedPage: image (warped ImageBitmap), frame (full camera frame), sourceCorners, timestamp.

Detection pipeline

Frames are downscaled to 256×256 and run through the DocAligner heatmap model (input (1,3,256,256), output (1,4,H,W) — one channel per corner). Each channel is thresholded; the largest connected blob's intensity-weighted centroid becomes that corner. A detection gate (confidence, convexity, area, edge-length sanity + enter/leave hysteresis) decides when a page counts as present, and a critically damped spring glides the drawn quad between detections. Only one frame is ever in flight — detection runs as fast as inference allows, capped by detectionIntervalMs, dropping frames rather than queueing them.

Serving assets

Model — download with npm run download-models (or node scripts/download-models.mjs all for every variant), then host the .onnx file anywhere same-origin or CORS-readable:

| variant | file | notes | | --- | --- | --- | | lcnet100 | lcnet100_h_e_bifpn_256_fp32.onnx | default; smallest and fastest (~4.5 MB) | | fastvit_t8 | fastvit_t8_h_e_bifpn_256_fp32.onnx | middle ground | | fastvit_sa24 | fastvit_sa24_h_e_bifpn_256_fp32.onnx | most accurate, heaviest |

onnxruntime-web runtime — copy from your installed version (filenames change across ort minors, so always copy rather than hardcoding a CDN version):

cp node_modules/onnxruntime-web/dist/ort-wasm-simd-threaded{,.jsep}.{wasm,mjs} public/ort/

and pass ortWasmPaths: '/ort/'.

Cross-origin isolation (recommended) — serve your page with

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

WebGPU doesn't need it, but the WASM fallback only uses multithreading when crossOriginIsolated — without the headers it silently runs single-threaded (slower, may miss 4 Hz on weak CPUs). With COEP require-corp, any cross-origin model/runtime assets must send Cross-Origin-Resource-Policy: cross-origin; easiest is hosting everything same-origin. example/server.mjs shows a dependency-free Node server doing this.

Bundler notes

The detection worker ships as dist/detector.worker.js and is referenced via new Worker(new URL('./detector.worker.js', import.meta.url), { type: 'module' }), which Vite, webpack 5, and Rollup detect and re-bundle automatically (onnxruntime-web included).

• Vite dev: add optimizeDeps: { exclude: ['@andor83/ml-web-scanner'] } — esbuild pre-bundling would break the worker reference.
• Other setups: pass workerUrl pointing at a copy of node_modules/@andor83/ml-web-scanner/dist/detector.worker.js (also exposed as the @andor83/ml-web-scanner/worker export). Note the worker itself imports onnxruntime-web as a bare specifier, so unbundled usage needs an import map.

Browser support

| | detection | notes | | --- | --- | --- | | Chrome / Edge | WebGPU | fastest path | | Firefox | WASM | WebGPU EP falls back automatically | | Safari ≥ 16.4 | WASM (WebGPU where available) | createImageBitmap(video) quirks handled via an ImageData fallback |

Development

npm test            # vitest: geometry, homography, warp, smoothing, gate, heatmap decode
npm run typecheck   # vue-tsc
npm run build       # library → dist/
cd example && npm run dev   # hot-reload demos (after copy-assets via predev)

Manual test checklist: vanilla + Vue demos on Chrome (WebGPU), Chrome with ?ep=wasm, Safari; point the camera at a document → blue quad locks on and follows smoothly; shutter with no page in view shakes instead of capturing; capture plays flash + fly-to-badge and the warped page appears in the gallery.

License

MIT for this package. The DocAligner models are by DocsaidLab, Apache-2.0 — see NOTICE. Model files are not redistributed here; the download script fetches them from DocsaidLab's published locations.