yolo-segdetect-js

Browser-side YOLOv8-seg instance segmentation on WebGPU, running ONNX models exported by browser-onnx-tools.

It is the browser counterpart to yolo-segdetect-worker: same tiling strategy, same merge semantics, same thresholds — minus Temporal, Java and OpenCV.

Install

npm install yolo-segdetect-js onnxruntime-web

onnxruntime-web is a peer dependency, so the host controls its version and its asset pipeline.

Quickstart

import { YoloSegmenter, configureOrt } from 'yolo-segdetect-js';

// One-time: tell ORT where its WASM sidecars are served from. They must be
// same-origin — browsers block the cross-origin dynamic import ORT uses.
configureOrt({ wasmPaths: '/assets/ort/' });

const base = 'https://huggingface.co/Ballon999/yolov8x-seg-opticnerve-onnx/resolve/main';

const seg = await YoloSegmenter.fromPretrained(`${base}/model.fp16w.onnx`, {
  preload: true,
  onProgress: ({ loaded, total }) => console.log(`${loaded}/${total ?? '?'} bytes`),
  onStatus: (s) => console.log(s),
});

const ctx = canvas.getContext('2d')!;
const image = ctx.getImageData(0, 0, canvas.width, canvas.height);

const result = await seg.segment(image, {
  confThreshold: 0.6, // per-tile confidence
  iouThreshold: 0.5, // per-tile NMS
  threshold: 0.3, // cross-tile merge (IoS)
  overlapX: 60,
  overlapY: 60,
  onTileProgress: (done, total) => console.log(`tile ${done}/${total}`),
});

for (const d of result.detections) {
  console.log(d.className, d.score, d.polygons?.[0]?.exterior.length, 'vertices');
}

model.json is fetched automatically from alongside the weights; it carries the class names, class count and input size, so a newly exported model needs no code change.

Which model file to load

Every published repo contains model.onnx, model.fp16w.onnx and model.json.

Use model.fp16w.onnx. It stores weights in half precision and computes in fp32 — half the download, numerically identical results. A true-fp16 export (model.fp16.onnx) accumulates in fp16 on WebGPU, and on the x-scale models that degrades WebGPU/WASM agreement to a cosine of 0.967: a correct 0.859-confidence detection becomes a wrong box. Do not ship it.

Execution providers

Defaults to ['webgpu', 'wasm'], dropping WebGPU when the browser has no navigator.gpu.

Because fp16w computes in fp32, WASM produces the same numbers as WebGPU — just far slower (~5s/tile vs ~110ms/tile on an M1 Max). That makes it a real fallback, and a real reference:

const cpu = await YoloSegmenter.fromPretrained(url, { executionProviders: ['wasm'] });

Use it. WebGPU failures in this stack are silently wrong numbers, never exceptions — a wide Concat returning zeros, or fp16 accumulation drift. A same-machine CPU comparison is the cheapest way to catch one.

What runs where

The main thread does nothing but marshal. Crop, letterbox, inference, decode, NMS, cross-tile merge and contour tracing all happen inside the worker; only finished detections come back. cellpose-js learned this the hard way — with postprocessing on the main thread the UI froze for seconds per run, and mask assembly here is comparable work (32 multiply-adds per mask pixel).

Output

interface Detection {
  box: [number, number, number, number]; // sub-pixel, image coordinates
  score: number;
  classId: number;
  className?: string;
  polygons?: MaskPolygon[]; // { exterior, holes }, image coordinates
  mask: Uint8Array | null; // box-local; omitted unless includeMasks
  maskBox: [number, number, number, number] | null;
}

Masks are box-local — a small buffer covering the detection's own box. The Python allocates a full-image plane per detection, which is fine for a 512px crop and gigabytes at whole-slide scale. They are dropped from the result by default once contours are traced; pass includeMasks: true to keep them.

Polygon rings are closed implicitly (the last point is not a repeat of the first) and are simplified with Douglas-Peucker at a 1px tolerance, which collapses pixel staircases into clean diagonals. Pass traceOptions: { simplifyTolerance: 0 } for exact pixel boundaries.

Behaviour inherited from the Python

Kept deliberately, because divergence here means the browser preview disagrees with the server:

• IoS, not IoU, for cross-tile merging. A detection clipped by a tile edge is contained in the full detection next door: low IoU, IoS ≈ 1. Plain IoU leaves a duplicate at every seam.
• The "intelligent sorter". Confidence is binned (10 bins), then ties break by area, largest first — so within a bin the whole object beats the clipped fragment that scored marginally higher.
• Two-stage suppression. Box overlap only nominates a pair; the decision is made on mask overlap. Adjacent structures can have overlapping boxes and disjoint masks.
• 8-connected foreground when tracing contours, matching OpenCV's findContours, which is what Ultralytics uses.

Authenticated model hosting

fromPretrained accepts pre-fetched bytes or a custom HTTP client, so the model need not come from a public origin:

// Pre-fetched
await YoloSegmenter.fromPretrained(url, { modelBytes, meta });

// Or route every request through the host's stack (auth interceptors, proxies)
await YoloSegmenter.fromPretrained(url, { fetcher: myAuthenticatedFetch });

This is a deliberate departure from cellpose-js, which hardcodes a raw fetch. Behind an OAuth2 proxy a raw fetch bypasses the interceptor, 302s to a login page, and then fails CORS.

Caching

Weights and metadata are cached in IndexedDB, keyed by URL plus an ETag/Last-Modified probed via HEAD, so a re-uploaded model invalidates automatically. clearCachedModel(url) drops an entry.

Development

npm install
npm run build       # tsc — per-module ESM, so bundlers can find the worker
npm test            # vitest
npm run typecheck
npm run lint

The build is plain tsc on purpose. Per-module output is what lets new Worker(new URL('./inference.worker.js', import.meta.url)) resolve to a real file that webpack, Vite and the Angular CLI can each detect and emit as its own chunk. A bundled single-file build breaks that.

License

Copyright (C) 2026 The Jackson Laboratory.

AGPL-3.0-only. See LICENSE.

This is not an incidental choice. The tiling and cross-tile merge logic is a port of patched_yolo_infer (AGPL-3.0), which in turn builds on Ultralytics YOLOv8 (AGPL-3.0), and it deliberately reproduces that project's specific design decisions — the confidence-binned "intelligent sorter", the intersection-over-smaller metric, the two-stage box-then-mask suppression — because agreeing with the server result is the whole point. The published model weights are AGPL-3.0 as well, though they are fetched at runtime and are not redistributed here.

If you need this under a permissive license, the tiling/merge stage is the part that would have to be rewritten from documented behaviour rather than ported; the rest of the library (model cache, ORT session handling, worker harness, contour tracing) has no AGPL lineage.