@jax-data-science/sci-image-visualizer

A framework-style Angular 17 library for interactive scientific image visualization and annotation. It renders large / multi-channel images through pluggable backends — a tiled OpenSeadragon viewer, Plotly 2D/3D plots, and a napari-js WebGPU backend — behind a single IVisualizer contract, and provides a rich set of on-canvas region / annotation tools including AI-assisted segmentation that runs entirely in the browser (WebGPU / WASM via onnxruntime-web).

Extracted from jit-ui (JAX Image Tools) — see jit-ui#80 — so the same viewer can be reused across the @jax-data-science portfolio. Open source; contributions welcome.

Installation

npm install @jax-data-science/sci-image-visualizer

Then install the peer dependencies your app doesn't already have:

npm install @angular/animations @angular/router primeng primeicons primeflex \
  image-js file-saver buffer onnxruntime-web

Peer dependencies

| Package | Range | Notes | | -------------------------------------------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | @angular/common · core · forms · animations · router | ^17.3.0 | Angular 17. The library does not use animations or the router itself; PrimeNG's dialog, table, toast and menus need them | | rxjs | ^7.8.0 | | | primeng | ^17.18.0 | toolbar / dialogs / table / dropdown UI | | primeicons · primeflex | ^7.0.0 · ^3.3.0 or ^4.0.0 | optional peers, but the templates use their icon and utility classes; load their CSS (see Host setup) | | image-js | ^0.35.6 | client-side image processing | | file-saver | ^2.0.5 | GeoJSON / mask export | | buffer | ^5.7.1 | Node Buffer polyfill for the Plotly backend | | onnxruntime-web | ~1.26.0 | browser SAM / cellpose inference (WebGPU/WASM) |

The rendering backends and helpers — openseadragon, plotly.js-dist-min, napari-js, cellpose-js, fast-png, and tslib — are declared as regular dependencies and installed automatically; you don't add them yourself. cellpose-js is lazy-imported at runtime, so apps that never open the Cellpose tool don't pay for it in the bundle — but it does need onnxruntime-web (its own peer, listed above) present.

Host setup

Besides the ports, the library expects these from the host app. The host integration guide has the details and examples/browser-image/main.ts shows all of it working.

1. Providers: provideHttpClient(), provideAnimations(), and PrimeNG's MessageService and ConfirmationService.
2. CSS: primeicons/primeicons.css, primeng/resources/primeng.min.css, primeflex/primeflex.css and a PrimeNG theme.
3. Assets: serve the package's src/lib/assets at assets/plotting/ (icons and colormap previews). In angular.json:

   {
     "glob": "**/*",
     "input": "node_modules/@jax-data-science/sci-image-visualizer/src/lib/assets",
     "output": "assets/plotting"
   }

4. onnxruntime-web sidecars at /assets/ort/ (copy node_modules/onnxruntime-web/dist/*.{wasm,mjs}), or call setOrtWasmBase(url) once at startup.
5. Icon stylesheet (optional): the toolbar and context-menu icon classes (.wand-icon, .brush-icon, … and the active context-menu entry) are global, because those menus are appended to <body>. The visualizer defines them when it first loads; to have them defined up front and independent of stylesheet order (say, for a host menu that reuses the icons), add the shipped stylesheet to angular.json:

   "styles": ["node_modules/@jax-data-science/sci-image-visualizer/src/lib/styles/viz-icons.scss"]

Quick start

import { Component } from '@angular/core';
import {
  VisualizerComponent,
  RegionEditorComponent,
  provideVisualization,
  IMAGE_STATE_PORT,
  TILE_ACCESS_PORT,
  REGION_IO_PORT,
  VIZ_CONFIG,
  setSamModelUrls,
} from '@jax-data-science/sci-image-visualizer';

// Once at startup: where the SAM model files are hosted.
setSamModelUrls(
  'microsam-vit-t-lm',
  'https://huggingface.co/jax-image-tools/microsam-vit-t-lm-onnx/resolve/main/encoder.fp16.onnx',
  'https://huggingface.co/jax-image-tools/microsam-vit-t-lm-onnx/resolve/main/decoder.onnx',
);

@Component({
  selector: 'app-viewer',
  standalone: true,
  imports: [VisualizerComponent, RegionEditorComponent],
  template: `<visualizer></visualizer> <region-editor></region-editor>`,
  providers: [
    // The viewer chain. In `bootstrapApplication` providers it is the one app-wide
    // viewer; in a component's providers it is an isolated viewer for that subtree.
    ...provideVisualization(),
    // Your adapters: the library reaches the host only through these ports.
    { provide: IMAGE_STATE_PORT, useClass: MyImageStateAdapter },
    { provide: TILE_ACCESS_PORT, useClass: MyTileAccessAdapter },
    { provide: REGION_IO_PORT, useClass: MyRegionIoAdapter },
    { provide: VIZ_CONFIG, useValue: { slideCropServer: 'https://tiles.example.org' } },
  ],
})
export class ViewerComponent {}

CELL_SEGMENTER is optional: override it only to replace the default CellposeSegmenterService. SPATIAL_DATA_PORT and PREFERENCES_PORT are optional too.

All components are standalone. VisualizationModule still works for NgModule hosts but is deprecated: import the components and add provideVisualization() instead.

The embeddable components use plain, unprefixed selectors: visualizer, region-editor, channel-histogram, hex-color-picker, spatial-controls and spatial-charts. Their inputs and outputs are listed in the host integration guide.

Live demo & running the example

A serverless, in-browser example lives in examples/browser-image/: a gallery of sample images that render in the viewer with the region + zoom tools, with no backend at all (the "Mode B" path — each image is a self-contained tiled: false source).

Live demo: https://thejacksonlaboratory.github.io/sci-image-visualizer/ — published by CI (.github/workflows/pages.yaml) to GitHub Pages on every push to main.

Run on localhost

npm install            # repo + example toolchain (Vite + Analog)
git lfs pull           # fetch the sample-image bytes (they're stored in Git LFS)
npm run build          # build the library into dist/ (the example consumes it)
npm run start:example  # dev server → http://localhost:5173

npm run build:example produces a static build in examples/browser-image/dist/. See examples/browser-image/README.md for the full walkthrough (adapters, TIFF handling, the Angular-17 toolchain note).

Highlights

• Three rendering backends — a tiled OpenSeadragon viewer, Plotly plots, and a napari-js WebGPU backend — behind one IVisualizer contract; the host picks per plot type via RoutingVisualizerService.
• Plot types — Image (OSD), Plotly (Heatmap, Contour, Scatter 2D, Surface 3D, Scatter 3D, Isosurface), and napari · WebGPU (Image, Scatter 2D, Surface, Scatter 3D, Volume, Isosurface, Spatial omics, Spatial omics 3D).
• Region / annotation tools — selection, rectangle, polyline, freeform, polygon + vertex editing, move, Bézier↔polygon, magic wand, brush, vertex eraser.
• Channels & histogram — brightness/contrast/gamma, per-channel display, colormaps / LUTs.
• Region editor — a Regions panel to list, select, rename, classify, recolor, import/export (GeoJSON), and delete regions.
• Image stacks — a z-stack can be a single server-tiled file (internal z) or a set of self-contained per-slice image URLs (IImageInfo.tiled === false, one URL per slice), e.g. a folder of numbered files assembled by the host. Slices load on demand as you scrub.
• Per-slice ROIs + QuPath planes — GeoJSON import/export carries the QuPath image-plane convention (geometry.plane.z), and the host can supply one ROI set per slice (IImageInfo.roiJsonStrs) that the viewer swaps on scrub.
• Browser-side segmentation — promptable SAM (box + interactive points) and automatic cellpose-SAM, all client-side (see below).

Visualization

Open an image and it renders through whichever backend best fits it; the RoutingVisualizerService switches backends per plot type and keeps the shared state (regions, channels, zoom) consistent across them.

OpenSeadragon — tiled image view

The default Image view is a natively tiled, deeply zoomable raster powered by OpenSeadragon. It streams pyramid tiles (the host supplies them through the TILE_ACCESS_PORT, e.g. a /tiles/info + /tile service), so gigapixel slides pan and zoom smoothly without loading the whole image. It includes:

• a navigator minimap and a physical-units scale bar;
• click-to-zoom toward the clicked point, scroll-zoom, and drag-pan;
• a raw-pixel vs. smoothing toggle — images open at nearest-neighbour so zoomed-in pixels stay crisp for inspection; toggle Smoothen for bilinear;
• WYSIWYG PNG download of the current view and a fit-to-view / autoscale reset.

The viewer switches to a finer pyramid level as soon as the current one would be upscaled at all (minPixelRatio: 1), rather than tolerating up to 2× before fetching — so the image does not go soft and then abruptly sharpen between levels, and the only visible pixellation is past 1:1, where the blocks are genuine source pixels. This assumes the host's pyramid actually has intermediate levels: for a flat image whose descriptor jumps straight from a preview to full resolution there is no finer level to switch to, and the setting only moves the jump to full-resolution tiles to a lower zoom.

Plotly — plots & 3D

Non-image plot types render with Plotly and support "real zooming" — a downscaled overview that re-fetches higher-resolution data as you zoom in:

• Contour and Scatter (regions) 2D plots;
• Surface 3D, Scatter 3D, and Isosurface for z-stacks, with an isosurface band slider and full 3D camera controls (zoom, pan, orbit, turntable, reset);
• scalar/3D types expect a grayscale image (the volume types also need a z-stack).

napari-js — WebGPU

GPU-accelerated renderings via napari-js (WebGPU), selectable from the plot-type menu as the "napari · WebGPU" variants of Image, Scatter 2D, Surface, Scatter 3D, Volume, Isosurface, and the two spatial-omics modes (below). It assembles the volume from the slice endpoints with a runtime decimate factor (resolution slider, default ½), a surface wireframe toggle, a 3D axes / scale gizmo, an in-view Z-height drag handle for the volume, cancellable loading, and multichannel volume compositing (one additive tinted layer per channel). Regions and display options stay in sync with the other backends via the shared stores.

Volume and Isosurface take their voxels from the image stack, and only from it — there is one voxel path, so a dataset reaches those modes by being a stack. A stack that declares its slice spacing (imageMeta[0].mppZ, alongside mppX/mppY) is rendered at its true physical extent, so 40 × 40 × 200 µm voxels read as anatomy rather than as a cube-aspect brick, and the axis labels read in real units on all three axes. Without it the world box is resolution-invariant but shape-only, and Z is labelled in slices.

When a stack is open, a slice slider (Image view, and the 2D Spatial omics view) or single-image/stack toggle (other views) navigates the z-dimension. A stack may be one server-tiled file (internal z) or a set of self-contained per-slice URLs (IImageInfo.tiled === false) — the latter (e.g. a host-assembled folder of numbered files) fetches urls[z] per slice on demand. For a per-slice stack the host can also supply a matching ROI GeoJSON per slice (IImageInfo.roiJsonStrs), which the viewer shows for the displayed slice and swaps as you scrub.

Intensity profiles (work in progress)

A line-ROI tool draws coloured lines and plots intensity along each one in a live floating inset chart that re-samples at the current zoom. It works today but the API/UX are still stabilizing (see In progress / roadmap).

Spatial omics

Two napari-js plot modes for spatial-omics datasets — N observations (Visium spots, segmented cells) positioned in a tissue image's pixel space, each with categorical and continuous annotations plus a lazily-fetched gene matrix:

• Spatial omics — one marker per observation over the tissue image, with 10x Xenium cell outlines, transcripts and a density map read in place;
• Spatial omics 3D — the observations as a point cloud under an orbit camera, inside the dataset's reference volume, with per-cluster density volumes.

The data arrive through an optional SPATIAL_DATA_PORT; SpatialDataHttpService is a reference adapter for the bundled example server. A controls panel (<spatial-controls>) colours by column or gene, selects from ROIs or legend rows, and shows linked distributions. The modes appear only when a dataset is published.

Full guide: docs/guides/spatial-omics.md.

Regions & annotation

Regions are stored in a shared region store and use a GeoJSON-friendly model (rectangles, polygons, polylines, Bézier curves) with an optional zero-based z slice — GeoJSON import/export uses QuPath's geometry.plane.z (written only for non-default slices, so single-plane images round-trip byte-identically). Every tool writes to the same store, so regions persist across backend/plot-type switches and are editable from the Regions panel. The on-canvas tools:

• Selection — neutral mode; deactivates any drawing tool.
• Rectangle, Polyline (open LineString), Freeform (drag-to-draw closed polygon), and Polygon (click vertices, click first to close).
• Vertex editing — add vertex, delete vertex, and move region.
• Bézier ↔ polygon — convert a region to a smooth Bézier curve or back.
• Magic wand — grow a region from similar-valued pixels around the click; Ctrl/Cmd for exact-match flood fill; drag to extend; Shift to erase; a sensitivity slider tunes strictness. Growing into another region merges them.
• Brush — paint/erase a region with a circular brush (QuPath-style); a size slider sets the diameter, Shift erases, and erasing across a region can split it.
• Vertex eraser — remove vertices within a radius from any region; the outline is rejoined where it crosses the eraser's edge (a Bézier region loses the anchors outright).
• Delete the selected region.

Wand- and brush-drawn regions default to the legend class; SAM/cellpose masks inherit the color of the rectangle they came from.

Channels, histogram & colormaps

A floating Channels & Histogram dialog controls how intensities are mapped to display: brightness / contrast / gamma, per-channel display for multi-channel images, and — for grayscale — colormap / LUT selection with a reverse toggle (default: inverted greys). Changes apply live on every backend: OSD, napari-js and the Plotly heatmap. (A tiled multichannel image's heatmap is the server's flat composite preview, so its channels are adjusted in the OSD and napari-js image views.)

Region editor (Regions panel)

The <region-editor> component is the Regions tab: a table of all regions where you can select, rename, assign a classification / class name, set per-class colors, toggle labels, import / export ROIs as GeoJSON (REGION_IO_PORT), and delete. Selecting a row highlights the region on the canvas (and vice-versa), so it pairs with the on-canvas tools above.

Segmentation tools (SAM & cellpose)

All segmentation runs client-side with onnxruntime-web (WebGPU where supported, WASM otherwise); models are fetched once and cached.

• Segment — draw rectangles, then each box becomes a SAM box prompt.
• Point prompts — click an object to segment it as a new region; Shift/Alt-click excludes, Enter commits, Esc undoes.
• Model picker — micro-sam ViT-T (default) / ViT-B for light microscopy, patho-sam ViT-B (fp16 or int8) for H&E.
• Cellpose — automatic cellpose-SAM: one region per detected cell inside each rectangle.

Models, hosting and the host configuration: docs/guides/segmentation.md.

Extending the viewer

Other packages can add to the viewer through Angular DI, with plain objects and no Angular compiler required:

• Plot modes on PLOT_TYPE_CONTRIBUTIONS — a new entry in the plot-type selector that rides on a built-in view and gets the viewport, the visualizer and an optional side panel. See docs/guides/plot-type-contributions.md.
• Toolbar tools on TOOLBAR_TOOLS — a parameter dialog that runs once, or an interactive dialog tool (kind: 'dialog') with its own session. See docs/guides/dialog-tools.md.

In progress / roadmap

• Intensity profile tool (work in progress — not yet stable) — coloured line ROIs with a floating inset chart that plots intensity along each line and updates live as the line is dragged. Usable today but the API/UX and multi-line/stack behaviour are still settling.
• Chart → map brushing (planned) — brushing a selection in a spatial-omics distribution chart back onto the map. See docs/design/spatial-omics-plot-mode-design.md.
• SAM 3 model (planned) — a variant: 'sam3' decoder path + export tooling (SAM 2/3 use a different mask I/O than the current SAM-v1 path). See docs/design/sam-segmentation-design.md.
• int8 patho-sam validation — the patho-sam-vit-b-int8 option is sanity-checked (IoU ~0.99 vs fp16 on a synthetic prompt) but not yet validated on real H&E slides, where int8 ViT attention can degrade on subtle boundaries.

Documentation

• docs/ARCHITECTURE.md — how src/lib is laid out: layers, backends, contracts, stores, regions, tools, workers and testing.

• Guides — host integration · spatial omics · segmentation · plot-type contributions · dialog tools.

• Design records (docs/design/) — each with a Status line: SAM segmentation · spatial-omics plot mode · region holes · region boolean ops · MCP control bridge · omics preprocessing in JIT.

• Architecture diagrams (Mermaid sources, ELK layout) in docs/diagrams/:

  • siv-architecture.mmd — host ⇄ contracts ⇄ router ⇄ OSD / Plotly / napari-js backends, the stores, spatial omics, the workers, and the servers the library talks to.
  • siv-regions.mmd — the region interfaces, the three region overlays, the on-canvas tools with their tool kit and geometry, and the RegionStore.

  The PNG/SVG renders in docs/img/ are of the earlier, pre-napari diagrams that the SOW documents embed.

• docs/JIT_UI_visualization_library_SOW.docx — statement of work for extracting/publishing this library.

• History (docs/history/) — the 2026-06 refactoring plan, the shared-backend refactor, the OpenSeadragon backend investigation, past bug write-ups and review records. Kept for context; not current.

Related (host side, in jit-ui):

• USE-VISUALIZATION-LIB-IN-PIPELINE-DIALOG.md — how the processing-pipeline dialog embeds <visualizer> and drives it via RoutingVisualizerService.
• processing-pipeline/ARCHITECTURE.md — the client/server processing-pipeline engines (OpenCV.js, transformers.js, cellpose, client slide-crop, server) that consume this library's region tools.

Scientific references

Segment Anything (SAM) — the promptable segmentation foundation model.

Kirillov, A. et al. Segment Anything. ICCV 2023. arXiv:2304.02643. Code: facebookresearch/segment-anything.

micro-sam — SAM finetuned for microscopy (the *_lm models; default tool).

Archit, A. et al. Segment Anything for Microscopy. Nature Methods (2025); bioRxiv:2023.08.21.554208. Code: computational-cell-analytics/micro-sam.

patho-sam — SAM finetuned for histopathology (the *_histopathology models).

Segment Anything for Histopathology. arXiv:2502.00408 (computational-cell-analytics). Code: computational-cell-analytics/patho-sam.

Cellpose — generalist cellular segmentation (flow-field algorithm).

Stringer, C. et al. Cellpose: a generalist algorithm for cellular segmentation. Nature Methods 18, 100–106 (2021). doi:10.1038/s41592-020-01018-x. Code: MouseLand/cellpose.

Cellpose-SAM — Cellpose built on a SAM ViT backbone (the automatic tool).

Stringer, C. & Pachitariu, M. Cellpose-SAM: superhuman generalization for cellular segmentation. bioRxiv:2025.04.28.651001. Model: mouseland/cellpose-sam.

MobileSAM — the TinyViT encoder behind micro-sam ViT-T.

Zhang, C. et al. Faster Segment Anything: Towards Lightweight SAM for Mobile Applications. arXiv:2306.14289 (2023). Code: ChaoningZhang/MobileSAM.

Rendering & runtime libraries

• OpenSeadragon — deep-zoom tiled image viewer. openseadragon.github.io · GitHub.
• Plotly.js — interactive 2D/3D plotting. plotly.com/javascript · GitHub.
• napari — multi-dimensional (n-D) scientific image viewer; the WebGPU rendering backend here (napari-js) is modeled on it. napari.org · GitHub · doi:10.5281/zenodo.3555620.
• onnxruntime-web — WebGPU/WASM ONNX inference (the SAM & cellpose runtime). GitHub.

Please cite the relevant model papers when publishing results produced with these tools.

Contributing

Issues and pull requests are welcome. CONTRIBUTING.md covers the dev loop (npm run typecheck, npm run lint, npm test, npm run build), the branch and pull-request rules, and how releases are published.

License

MIT © The Jackson Laboratory.