@open-rtn/plugin-virtual-background

中文文档

Virtual background extension for open-rtn-sdk.

This package provides a local video pre-processing extension. It takes a camera video track, renders the selected virtual background in a WebGL pipeline, and returns a processed MediaStreamTrack through the SDK processor pipeline.

Installation

Install this package together with open-rtn-sdk:

pnpm add open-rtn-sdk @open-rtn/plugin-virtual-background

In this monorepo, build open-rtn-sdk first, then build this package:

pnpm --filter open-rtn-sdk run build
pnpm --filter @open-rtn/plugin-virtual-background run build

You can also build all packages from the repository root:

pnpm run build:packages

Browser Support

Check support before creating the processor:

import { VirtualBackgroundExtension } from '@open-rtn/plugin-virtual-background';

if (!VirtualBackgroundExtension.isSupported) {
  throw new Error('Virtual background is not supported in this browser');
}

The current implementation requires:

• HTMLVideoElement.requestVideoFrameCallback
• createImageBitmap
• WebGL2
• HTMLCanvasElement.captureStream

In practice, use Chrome 83 or later for the expected runtime behavior.

Quick Start

Register the extension, create a processor, and pipe it to a local video track:

import { OpenRTC } from 'open-rtn-sdk';
import { VirtualBackgroundExtension } from '@open-rtn/plugin-virtual-background';

const extension = new VirtualBackgroundExtension();

if (!VirtualBackgroundExtension.isSupported) {
  throw new Error('Virtual background is not supported in this browser');
}

OpenRTC.registerExtensions([extension]);

const localVideoTrack = await OpenRTC.createCameraVideoTrack();

const processor = extension.createProcessor({
  type: 'blur',
  blurRadius: 16,
  maxFrameRate: 15,
  processingMode: 'auto',
});

localVideoTrack.pipe(processor).pipe(localVideoTrack.processorDestination);
await processor.enable?.();

localVideoTrack.play('#local-preview');
await client.publish([localVideoTrack]);

OpenRTC.registerExtensions() only registers the extension. The virtual background becomes active when the processor is piped to the local video track.

Background Modes

Use setOptions() to switch modes after the processor has been created.

Blur

processor.setOptions?.({
  type: 'blur',
  blurRadius: 20,
});

Image

processor.setOptions?.({
  type: 'image',
  imagePath: '/assets/backgrounds/office.jpg',
  imageFitMode: 'cover',
});

imageFitMode accepts the package FitMode values. Use cover for the common full-frame background behavior.

Color

processor.setOptions?.({
  type: 'color',
  color: '#1f2937',
});

Video

const backgroundVideo = document.createElement('video');
backgroundVideo.src = '/assets/backgrounds/loop.mp4';
backgroundVideo.muted = true;
backgroundVideo.loop = true;
backgroundVideo.playsInline = true;
await backgroundVideo.play();

processor.setOptions?.({
  type: 'video',
  videoElement: backgroundVideo,
});

Passthrough

Use none to keep the processor connected while rendering the original camera video without a virtual background:

processor.setOptions?.({
  type: 'none',
});

Processing Modes

processingMode controls which pipeline the processor uses:

• auto (default): use the Worker + MediaStreamTrack streams pipeline when the browser supports it, otherwise fall back to the main-thread canvas pipeline.
• worker: require the Worker pipeline. This mode throws when the browser does not support MediaStreamTrackProcessor, MediaStreamTrackGenerator, TransformStream, OffscreenCanvas, and WebGL2.
• main-thread: always use the compatibility pipeline based on HTMLVideoElement, WebGL, canvas.captureStream(0), and requestFrame().

The Worker pipeline supports blur, image, color, and none. Video backgrounds stay on the main-thread pipeline because they depend on a live HTMLVideoElement source.

Hosting Runtime Assets

The processor needs a segmentation model and a small WASM runtime. The package build copies every file under src/assets/ to dist/assets/, including:

• selfie_segmenter.tflite
• vision_wasm_internal.js
• vision_wasm_internal.wasm

For production applications, self-host these files under an application-owned static directory and pass stable URLs to the processor. This avoids CDN availability issues and build-tool-specific bundle URL assumptions.

Recommended public directory layout:

public/open-rtn-virtual-background/runtime/vision_wasm_internal.js
public/open-rtn-virtual-background/runtime/vision_wasm_internal.wasm
public/open-rtn-virtual-background/models/virtual-background-segmentation.tflite

Then pass explicit paths:

const processor = extension.createProcessor({
  type: 'blur',
  blurRadius: 16,
  assetPaths: {
    tasksVisionFileSet: '/open-rtn-virtual-background/runtime',
    modelAssetPath: '/open-rtn-virtual-background/models/virtual-background-segmentation.tflite',
  },
});

You can also provide a localModelPath. The processor will check it with a HEAD request and use it when the response is available:

const processor = extension.createProcessor({
  type: 'blur',
  assetPaths: {
    tasksVisionFileSet: '/open-rtn-virtual-background/runtime',
    localModelPath: '/open-rtn-virtual-background/models/virtual-background-segmentation.tflite',
  },
});

The Worker pipeline is bundled inline by default, so applications do not need to host virtual-background.worker.js. If your Content Security Policy blocks blob: workers, host dist/virtual-background.worker.js yourself and pass workerScriptPath.

Cleanup

Remove the processor from the track when the effect is no longer needed:

localVideoTrack.unpipe(processor);
await processor.disable?.();
await processor.release?.();

Call release() before discarding the processor so its internal video element, canvas output track, WebGL resources, and segmentation runtime can be released.

Troubleshooting

If the effect does not appear, verify these points first:

• VirtualBackgroundExtension.isSupported returns true.
• The processor has been connected with localVideoTrack.pipe(processor).pipe(localVideoTrack.processorDestination).
• The local video track is a video track created by open-rtn-sdk, such as OpenRTC.createCameraVideoTrack().
• Self-hosted runtime and model URLs are reachable by the browser.
• Remote users receive the processed track only after the processor has been piped to the local track that is published.

Each processor instance owns its own video element, canvas, WebGL context, and segmentation runtime. Multiple browser tabs are supported, but they compete for CPU/GPU resources. Use maxFrameRate to cap processing load when more than one virtual background instance may run at the same time.

API Exports

import {
  VirtualBackgroundExtension,
  VirtualBackgroundProcessor,
  type FrameProcessingStats,
  type FitMode,
  type ModelAssetPaths,
  type ProcessingMode,
  type VirtualBackgroundOptions,
  type VirtualBackgroundType,
} from '@open-rtn/plugin-virtual-background';