ChromaPakZ

A lossless RGBD video codec (クロマパックZ): a single ordinary .webm that carries one or more viewable RGB tracks alongside bit-exact 16-bit auxiliary signals — depth, object IDs, packed normals, or any other W×H uint16 plane — all kept in sync. Its design goals:

• a legacy player shows plain RGB — the depth rides in extra tracks an ordinary player ignores;
• it uses only royalty-free codecs (VP9 / libvpx, BSD-licensed) — no GPL encoder, no patent pool;
• it runs in the browser through WebCodecs — no WASM on Chromium, with a small libvpx-WASM fallback for engines whose native path isn't bit-exact;
• each 16-bit signal is packed with a single reversible map, not a stack of per-range slices to manage;
• multiple lossless uint16 signals (depth, object IDs, …) share one container, frame-aligned;
• multiple synchronized RGB streams for stereo / multi-camera rigs — one per camera, same clusters, same timeline;
• each stream at its own resolution — depth at sensor size (256×192 LiDAR) beside full-resolution video, frame-aligned all the same;
• the display track can be 8-bit SDR or 10-bit HDR10/HLG (VP9 profile 2, BT.2020, with the WebM Colour element players actually read).

The same format is implemented three times — browser (WebCodecs), C++ (libvpx), and Python — and a file written by any one of them decodes bit-exactly in the other two. One exception, by design: HDR display tracks are written by the C++/Python encoders only, and a browser plays them in <video> rather than decoding them through the JS library (the lossless signals still decode everywhere).

Quickstart

# Python — self-contained wheels (libvpx is linked in; no build tools needed)
pip install chromapakz
python -c "import chromapakz as cz; print(cz.inverse_depth_spec(0.3, 9.0))"
#   cz.encode({"depth": u16}, specs={"depth": cz.inverse_depth_spec(near, far)}, rgb=rgba)

# Browser — streaming encode/decode, WebCodecs with a per-operation WASM fallback (docs/API.md)
npm install chromapakz
#   import { createEncoder, createDecoder } from 'chromapakz';
#   createEncoder({ signals: [{ id:'depth', near, far }, { id:'objectId' }] })
#   createDecoder(bytes).readFrame() -> { rgb, signals: { depth: { u16 }, objectId: { u16 } } }

# Browser demo — encode→file→decode→view, entirely in-page (no WASM on Chromium)
python3 -m http.server 8000      # from the repo root, then open http://localhost:8000/demo/

Building from a checkout instead (C++ CLI, or a platform with no wheel) needs libvpx dev headers, pkg-config, CMake and a C++17 compiler — brew install libvpx pkg-config cmake ninja, or apt-get install libvpx-dev pkg-config cmake ninja-build g++:

pip install .                    # pip compiles the native core via CMake and bundles it

cmake -S . -B build && cmake --build build -j     # or: native/build.sh
./build/dccli selftest
./build/dccli decodesignal clip.webm depth depth.u16

Encoding an HDR display track additionally needs a libvpx built with --enable-vp9-highbitdepth (VP9 profile 2, 10-bit). Homebrew's and Debian's packages have it; a libvpx without it builds and runs everything else fine, and fails only when an HDR encode opens its encoder — encode failed (2) / "the RGB encoder could not be opened". Check yours with nm -g $(pkg-config --variable=libdir vpx)/libvpx.a | grep -c highbd (0 means no), and see scripts/install-libvpx.sh for the flags the wheels are built with.

How it works

| Layer | Choice | |---|---| | Container | WebM / Matroska, multi-track. The primary RGB stream is track 1, so any player shows it; depth tracks are ignored by players that don't know them. A Duration, a Cues index, and ~1 s RGB keyframes make it seekable in <video> (depth stays single-keyframe — it isn't what <video> plays). | | RGB tracks | Normal, viewable video streams: 8-bit VP9, YUV 4:2:0, BT.709 full-range — or, for HDR, VP9 profile 2, 10-bit, BT.2020 broadcast-range with a WebM Colour element. One per camera for stereo / multi-camera rigs (rgbs); legacy readers see the primary. | | Lossless signals | Each signal: optional quant (e.g. inverse-depth for float depth) → uint16 → triangle-fold 8+8 → two VP9 lossless tracks. Add object IDs, labels, etc. as additional signal pairs. | | Metadata | v4 — rgbs[] (every RGB stream) + signals[] (each signal: id, tracks, scheme, quant); any entry may carry its own width/height (files whose streams all share the file resolution stay v3, byte-identical). |

Inverse-depth quantization spends precision where it matters (near surfaces), matching how stereo/ToF sensors behave. Float can't be stored losslessly in 16 bits, so this quantization is the format's defined precision boundary; everything below it is bit-exact.

Triangle-fold is the key trick. Split a 16-bit value into a high and low byte the naive way (d & 0xFF) and the low byte becomes a sawtooth — a hard 255→0 cliff every 256 levels. Those manufactured edges defeat any spatial predictor, since the codec sees a discontinuity wherever the depth simply crosses a byte boundary. Reflecting every other segment (lo = (high & 1) ? 255 - lo : lo) turns that sawtooth into a continuous triangle wave with no cliffs, so VP9's predictor sees smooth gradients again. One reversible map, nothing to manage.

Full color range is signaled in the bitstream (VP9E_SET_COLOR_RANGE), so a range-honouring decoder returns the packed luma unscaled instead of applying a limited-range conversion that would corrupt depth.

Why these choices (measured, not assumed — Chromium 148)

WebCodecs has no "lossless" switch, so every claim here is a measurement from experiments/webcodecs-lossless:

• VP9 at QP 0 is bit-exact through WebCodecs; AV1 is not (AV1 quantizer:0 drifts by up to ~257). So VP9 carries depth; AV1 is fine only for the lossy RGB track.
• Triangle-fold beats a naive byte-split by ~13%, and inter-coding cuts another ~52% (and stays bit-exact across the GOP) — most of what looks like incompressible LSB noise is actually static fold structure that temporal prediction removes.
• 8+8 beats high-bit-depth. 10-bit VP9 encode is available in browsers, but a 10+6 split is ~4% worse than 8+8 and narrows browser reach, so 8+8 wins on both counts.

docs/EVALUATION.md is the full due-diligence record: every codec/container/packing alternative considered, the constraint that eliminates each, a head-to-head benchmark (ChromaPakZ beats FFV1, PNG-16 and x264 on the same 16-bit depth, beats x265/HEVC at matched 11-bit precision, and lands within 1–2% of LZMA), cited licensing/browser facts, and a sensitivity analysis of when a different choice would win.

What it costs

Lossless 16-bit depth of a real sensor is noise-bound: the low bits are largely sensor noise, and lossless coding must preserve every bit of it. On real Kinect data (TUM RGB-D fr1/desk, 30 frames at 640×480, 78% valid):

| track | bits / pixel | |---|---| | RGB | 0.19 | | depth (hi + lo) | 0.50 + 4.35 | | total | 5.04 |

— depth round-tripped bit-exact. Reproduce with examples/tum_fr1desk.py (see its header for the one-line dataset fetch).

The one knob that moves this is the quantization precision relative to the sensor's noise floor. Spread depth across all 65,535 codes and each step is far finer than the noise, so the codec dutifully archives the randomness bit for bit. Coarsen the grid to match the noise and the cost collapses — with no loss of real signal. The sweep below is measured on the synthetic benchmark clip (make_synthetic_rgbd.py, range ≈0.9–7.8 m) — a separate clip from the TUM numbers above:

| effective bits | depth precision at 7.8 m | depth bpp | |---|---|---| | 16 (default) | 0.9 mm per step | 13.2 | | 12 | 14 mm per step | 9.7 | | 11 | 28 mm per step | 8.1 | | 10 | 56 mm per step | 6.9 |

(Reproduce the bpp column with python python/benchmark_codecs.py.) levels is a first-class, metadata-stored parameter (default 65536 = full 16-bit) shared by all three implementations, so reduced-precision files reconstruct identically everywhere. Set it with chromapakz-ingest --depth-bits N or the levels= argument.

Codec rate-distortion

This is a separate axis from precision: how faithfully the codec carries whatever quantized depth you give it. PSNR here is the encode→decode path measured against the source codes.

The lossless codecs all sit on the ∞-dB band — they reproduce depth exactly and differ only in size, where ChromaPakZ (VP9) is smallest, just under FFV1, with PNG-16 well behind. The blue curve is ChromaPakZ's own near-lossless option (sweeping the VP9 quantizer trades fidelity for size), but the default operating point is QP 0, bit-exact. Regenerate with python python/plot_rd.py.

A note on ffmpeg. Decoding ChromaPakZ files with ffmpeg (or any conformant VP9 decoder) is bit-exact. But encode with ChromaPakZ, not the ffmpeg CLI: ffmpeg -c:v libvpx-vp9 -lossless 1 is lossless yet ~3× larger (≈39 vs ≈13 bpp) — same library, far worse coding decisions, and no flag tested closes the gap. python/plot_rd.py therefore uses the real WebCodecs encoder for the VP9 numbers.

Cross-language implementations

All three read and write the identical .webm, verified bit-exact in every direction (browser ⇄ C++ ⇄ Python), and produce standard files — ffprobe reports matroska,webm with one VP9 stream per RGB camera plus two per lossless signal, and ffmpeg decodes track 0 as plain RGB when present. The one asymmetry is HDR: a 10-bit display track is written natively (C++/Python) and played by the browser rather than decoded by the JS library, which skips it and still decodes every signal.

Format schema: docs/FORMAT.md. API: docs/API.md.

| Surface | Codec | Build | |---|---|---| | Browser | WebCodecs VP9 | none — src/chromapakz.js, src/signals.js, src/webm.js. Multi-signal streaming API. | | C++ | libvpx VP9 | CMake → build/_core + dccli (dc_encode_multi, dc_decode_signal) | | Python | ctypes → C++ | pip install . — encode(), create_encoder(), decode(), parse_metadata() |

./build/dccli encodergbd rgb.rgba depth.u16 W H N fps near far kbps out.webm
./build/dccli decodesignal clip.webm objectId ids.u16
./build/dccli decodergb  clip.webm rgb.rgba

Real-data ingestion

Shipped in the wheel, so these work straight after pip install chromapakz:

• chromapakz.ingest — load depth (.exr / .npy / .npz / 16-bit PNG·TIFF / raw) and optional RGB (image sequence, video via ffmpeg, or array), auto-derive inverse-depth near/far from percentiles, encode, and report real per-track bpp. Invalid pixels (<=0/NaN) map to code 0. Importable (from chromapakz.ingest import encode_clip) or as the chromapakz-ingest command: chromapakz-ingest --depth 'd_*.exr' --rgb 'rgb_*.png' -o clip.webm --report --verify
• chromapakz.webm_inspect — pure-Python EBML parser for the per-track byte breakdown (no native deps).

Repo-only dev scripts under python/:

• make_synthetic_rgbd.py — a realistic RGBD generator (smooth surfaces, depth edges, disparity-domain noise, occlusion shadows, dropout holes) for when you don't have a sensor handy.
• benchmark_codecs.py, plot_rd.py — the benchmark and rate-distortion plots.

How it relates to RealSense / Kinect

Depth-camera ecosystems already split into two camps; ChromaPakZ takes the best of both.

• Intel RealSense colorizes 16-bit depth into an RGB image (Hue, ~10.5 effective bits) and encodes that with a stock H.264/H.265 codec. Great for streaming and reuse of hardware codecs, but lossy — unfit for ground-truth or archival depth.
• Kinect / RGBD datasets store depth raw or as 16-bit PNG. Azure Kinect even records to Matroska with a 16-bit depth track (lossless via per-frame PNG); TUM RGB-D, NYU and ScanNet use 16-bit PNG sequences. Bit-exact, but intra-only and large — no temporal compression.

| | RealSense colorize | Kinect / PNG | ChromaPakZ | |---|---|---|---| | bit-exact 16-bit depth | ✗ (lossy) | ✓ | ✓ | | RGB plays in any legacy player | ✓ | — | ✓ | | inter-frame (temporal) compression | ✓ (lossy) | ✗ | ✓ (lossless) | | royalty-free, browser-native (no WASM on Chromium) | — | — | ✓ |

That Azure Kinect already chose Matroska — WebM's basis — is telling. ChromaPakZ differs by compressing depth losslessly (VP9 + triangle-fold, inter-coded) rather than storing raw or intra PNG, and by running in the browser. Sources: RealSense colorized depth, Azure Kinect record format.

Repository layout

src/          chromapakz.js, signals.js, webm.js, chromapakz-core.js
native/       chromapakz.{h,cpp}, dccli.cpp
python/       chromapakz/ (pip package: __init__.py, ingest.py, webm_inspect.py)
              make_synthetic_rgbd.py, benchmark_codecs.py, plot_rd.py   (repo-only dev scripts)
demo/         index.html                     in-browser encode→decode→view
examples/     tum_fr1desk.py
experiments/  webcodecs-lossless/            run.mjs, smoke-demo.mjs, headless tests
docs/         FORMAT.md, API.md, EVALUATION.md, RELEASING.md
tests/        test_*.py (pytest) + *.test.mjs (node --test), both glob-discovered; fixtures/

CI builds and tests on Linux + macOS and runs the in-browser VP9-lossless probe in headless Chromium; docs/RELEASING.md covers wheels and PyPI publishing. The full design rationale and benchmarks are in docs/EVALUATION.md.

Status & limitations

Working end-to-end and verified across all three implementations. Honest caveats:

• Browser support is engine-specific (measured, EVALUATION.md §11): native WebCodecs lossless encode is Chromium-only today (WebKit lacks WebCodecs' quantizer mode; Firefox's QP 0 isn't lossless); native lossless decode works on Chromium and WebKit/Safari, while Firefox decodes VP9 to color-converted BGRX. Where native can't be trusted, the library transparently falls back to a bundled libvpx-WASM codec, chosen per operation by a cached runtime probe — so a decode-only browser (e.g. Safari) downloads only vp9-decode.wasm and never the larger encoder, and vice-versa. Force it with backend: 'webcodecs' | 'wasm' (default 'auto'); see docs/API.md. These are Playwright engine builds — reconfirm on shipping browsers before hard claims.
• "Royalty-free" reflects the AOMedia/Google position on VP9; Sisvel operates pools that dispute it.
• An auto precision picker (estimate the sensor noise floor to choose --depth-bits) is future work.
• Network byte streaming is supported via onChunk on encode and createDecoder() + push()/finish() on decode; Python has the same streaming encoder as create_encoder(), for live recording. See docs/API.md.