@mdaemon/video-effects, camera background blur and replacement for WebRTC video tracks

[ @mdaemon/video-effects on npm ]

Track in, track out. Give it the local camera track, get back a track whose background is blurred or replaced, and publish that instead. Everything downstream — producer.produce(), simulcast, the local preview element — keeps working on an ordinary MediaStreamTrack.

Processing runs sender-side, once per publisher, on that publisher's own outgoing stream. The cost does not grow with the number of people in the room: a 20-person call costs each participant exactly what a 2-person call costs.

Install

$ npm install @mdaemon/video-effects --save

@mediapipe/tasks-vision is an optional peer dependency, and version 1.x is required. Install it too unless you only ever intend to use the browser's native blur:

$ npm install @mediapipe/tasks-vision --save

Serving the assets

MediaPipe needs its WASM runtime and a .tflite model at runtime. They are not bundled — every consumer serves static files from a different place, so you pass the location in as assetBase.

Copy the WASM runtime out of node_modules/@mediapipe/tasks-vision/wasm/ into whatever directory you serve:

vision_wasm_internal.js
vision_wasm_internal.wasm
vision_wasm_nosimd_internal.js
vision_wasm_nosimd_internal.wasm

Both variants are needed because the pair a browser actually fetches depends on whether it supports WASM SIMD; each browser downloads one of them, not both. (The vision_wasm_module_internal.* files in that directory are for MediaPipe's ES-module loader, which this package does not use — skip them.)

The .tflite model is not in the npm package. Download selfie_segmenter_landscape.tflite separately from Google's MediaPipe model garden and serve it from the same directory, or point modelAssetPath wherever you put it.

Budget roughly 11 MB on disk for the SIMD build and about 3.3 MB over the wire once your server gzips it, plus ~250 KB for the model. Serve the directory with compression enabled — the uncompressed transfer is the single biggest cost of turning an effect on for the first time.

If your app sets a Content Security Policy, it needs 'wasm-unsafe-eval' in script-src and worker-src 'self' blob:.

Usage

Node Modules

import VideoEffects from "@mdaemon/video-effects/dist/videoEffects.mjs";

Node CommonJS

const VideoEffects = require("@mdaemon/video-effects/dist/videoEffects.cjs");

Web

<script type="text/javascript" src="/path_to_modules/dist/videoEffects.umd.js"></script>

API

new VideoEffects(options)

| Option | Type | Default | Purpose | |--------|------|---------|---------| | assetBase | string | required | Directory serving the MediaPipe WASM artifacts. | | modelAssetPath | string | <assetBase>/selfie_segmenter_landscape.tflite | Override the model URL. | | targetFps | number | 30 | Output frame rate for the canvas fallback path. | | preferNativeBlur | boolean | true | Use hardware blur when the platform offers it. | | watchdog | WatchdogOptions \| false | {} | Frame-budget monitor; false disables it. | | documentRef | Document | ambient document | Injection point for the canvases and the fallback <video>; only useful in tests. |

setEffect(effect): Promise<void>

await fx.setEffect({ type: "none" });
await fx.setEffect({ type: "blur", strength: 12 });          // strength in px, 0 < n <= 100
await fx.setEffect({ type: "image", source: "/bg/office.jpg" });
await fx.setEffect({ type: "image", source: someImageBitmap }); // pre-decoded, no fetch

Passing a URL means the first call fetches and decodes before the effect takes hold. Pre-decode to an ImageBitmap to avoid a frame or two of unmasked video.

process(track, options?): Promise<MediaStreamTrack>

Wraps a camera track. The returned track may be the input track itself when the effect is none, when the platform blurred it in hardware, or when no capture path is available — callers do not need to branch on that.

const fx = new VideoEffects({ assetBase: "/wasm" });
await fx.setEffect({ type: "blur" });

const raw = (await navigator.mediaDevices.getUserMedia({ video: true })).getVideoTracks()[0];
const masked = await fx.process(raw);

previewElement.srcObject = new MediaStream([masked]);
await producerTransport.produce({ track: masked });

Pass { visionModule } as the second argument when MediaPipe arrives by <script> tag rather than through a bundler.

Toggling the effect on an already-published track needs no renegotiation:

await fx.setEffect({ type: "none" });
await producer.replaceTrack({ track: await fx.process(raw) });

Events

fx.on("degraded", ({ averageMs, budgetMs, reason }) => { /* effect gave up; raw video continues */ });
fx.on("error", (error) => { /* the effect is off, or a run of frames failed */ });
fx.on("effectchange", (effect) => { /* reflect the new state in the UI */ });

degraded fires when the effect is abandoned and the raw camera track takes over. reason says why:

| reason | Meaning | |----------|---------| | "budget" | The watchdog tripped: segmentation is consistently too slow for this machine. | | "segmentation" | MediaPipe failed on every frame for a full second. Unmasked video was published throughout. |

Either way the raw track keeps flowing — a call without a blurred background beats a call that stutters, and beats one that freezes. Recovery is deliberately not automatic; flapping the effect on and off reads as a bug. Call process() again to retry.

A single failed frame is not fatal: it is composited unmasked and published, so the far end sees live video rather than a frozen picture. One error is emitted per run of failures, not one per frame.

Other members

| Member | Description | |--------|-------------| | VideoEffects.isSupported() | Whether any capture path exists in this browser. | | VideoEffects.capabilities() | Full capability breakdown. | | fx.effect | The current effect. | | fx.degraded | Whether the effect has been given up on. | | fx.usingHardwareBlur | Whether the platform is doing the work. | | fx.stop() | Tear down processing; leaves the source track alone. | | fx.destroy() | stop() plus drop all listeners. |

How it picks a path

1. Native backgroundBlur constraint — Chromium 114+ on supported hardware. Costs nothing: no WASM download, no per-frame work, no extra encode. Blur only; there is no platform equivalent for image replacement.
2. WebCodecs — MediaStreamTrackProcessor / MediaStreamTrackGenerator. Frame-by-frame transform, lowest overhead of the two software paths.
3. Canvas capture — a <video> element driven by requestVideoFrameCallback into a canvas, published via captureStream(). Firefox and Safari.

Output tracks are given contentHint = "motion", because canvas-derived tracks otherwise default to preferring resolution over framerate, which fights a simulcast ladder.

Browser support

| | Native blur | WebCodecs | Canvas capture | |---|---|---|---| | Chrome / Edge 114+ | yes, hardware permitting | yes | yes | | Firefox | no | no | yes | | Safari 16.4+ | no | no | yes |

isSupported() is false only where neither software path exists, in which case process() returns the input track untouched and emits error.

License

Published under the LGPL-2.1 license. See LICENSE.

MediaPipe and the selfie segmentation model are Apache-2.0, copyright Google LLC.