web-media-effects

Web Media Effects is a suite of media effects developed for web SDKs and WebRTC media applications.

Introduction

There are three effects included in this library:

• Virtual background (e.g., blur, image replacement, video replacement)
• Noise reduction (e.g., background noise removal)
• Gain (for testing)

Common Methods

The effects are built on top of a plugin interface that makes building and extending effects more straight-forward.

The effects plugins have four primary methods to control the plugin:

• load(input) accepts a track or stream and returns a new track or stream with the effect applied
• enable() enables the plugin after it's loaded
• disable() disables the plugin after it's loaded
• dispose() tears down the effect
• preloadAssets() fetches all assets (e.g., WASM files, ONNX models) to optimize the load sequence

Upon enabling or disabling the effect, an event is fired.

effect.on('track-updated', (track: MediaStreamTrack) => {
  // do something with the new track.
});

Additionally, there are a few convenience methods:

• getOutputStream() returns the new outgoing (i.e., "effected") stream
• getOutputTrack() returns the active output track
• setEnabled(boolean) sets the effect state by passing in a boolean (convenient for state managers)

Preloading Assets

In an effort to optimize startup time for applying media effects, there is a preloading mechanism. This mechanism fetches critical assets, such as ONNX models for image segmentation, WASM modules for audio processing, and web workers for background processing, in advance of media availability. This ensures smoother integration of effects once the media stream is ready to improve the overall user experience. Assets can be preloaded using either a provided factory function or directly using preloadAssets() API.

Using Factory Function for Asynchronous Initialization

The library includes factory functions for scenarios that require asynchronous operations. Utilizing the async/await pattern, these functions provide a simple method for creating effects with their assets already preloaded. The factory function's second parameter is a boolean that indicates whether the assets should be preloaded.

const noiseReductionEffect = await createNoiseReductionEffect({
    authToken: 'YOUR_AUTH_TOKEN',
    // ...other options
  },
  true
);

const virtualBackgroundEffect = await createVirtualBackgroundEffect({
    authToken: 'YOUR_AUTH_TOKEN',
    mode: 'BLUR',
    // ...other options
  },
  true
);

By incorporating asset preloading, the preload API aims to minimize delays and performance hitches when activating effects to keep the UI fluid and responsive.

Direct Use of preloadAssets() API

For more fine-grained control over the preloading process, you can also directly call the preloadAssets() method on each effect instance. This approach allows you to manually manage when and how assets are preloaded, providing flexibility to fit various application architectures and workflows:

const virtualBackgroundEffect = new VirtualBackgroundEffect(options);
await virtualBackgroundEffect.preloadAssets();

const noiseReductionEffect = new NoiseReductionEffect(options);
await noiseReductionEffect.preloadAssets();

This direct method is useful in scenarios where you might want to preload assets independently of the effect instantiation or in response to specific application states or events. It gives you the ability to strategically preload assets at the most appropriate moment.

Virtual Background

The virtual background effect is a wrapper around ladon-ts that provides a virtual background for video calling. The virtual background may be an image, an mp4 video, or the user's background with blur applied. The blur option allows for varied levels of strength and quality where higher levels require more compute resources.

The virtual-background-effect takes an optional VirtualBackgroundEffectOptions config object in its constructor. The effect's options can be changed at runtime via an updateOptions() method. When disabled, the effect simply passes through the original video images so that the outgoing stream does not need to be changed.

The effect uses a background thread worker by default to prevent slowdowns on the main UI thread. The main UI thread can be used instead by adding the property generator: 'local' in the VirtualBackgroundEffectOptions object. However, this is not recommended as the worker thread performs much better.

NOTE: For backwards compatibility, the default mode is set to BLUR.

Options

There are a few different options that can be supplied to the constructor or updateOptions() method that affect the behavior

| Name | Description | Values | Required | | ------------ | --------------------------------------------------------------------- | ------------------------------------------------- | ---------------------------------- | | authToken | Used to authenticate the request for the backend models | An encoded string token | Yes | | generator | Determines where the model runs (on main thread or background thread) | local worker | Defaults to worker | | frameRate | Determines how many frames per second are sent to the model | 0-60 | Defaults to 30 | | quality | Determines the accuracy of the model (higher requires more CPU) | LOW MEDIUM HIGH ULTRA | Defaults to LOW | | mirror | Whether the output image should be flipped horizontally | true false | Defaults to false | | mode | Determines what kind of background to render behind the user | BLUR IMAGE VIDEO | Defaults to BLUR | | blurStrength | How strongly the background should be blurred | WEAK MODERATE STRONG STRONGER STRONGEST | Required in BLUR mode | | bgImageUrl | Path to the background image to replace the original background | Fully qualified URL | Required in IMAGE mode | | bgVideoUrl | Path to the background video to replace the original background | Fully qualified URL (mp4 only) | Required in VIDEO mode | | env | Which environment the effect is running in. | EffectEnv.Production EffectEnv.Integration | Defaults to EffectEnv.Production | | avoidSimd | Avoid using the SIMD processor, if SIMD is supported (for testing) | true, false | Defaults to false |

Mode

The virtual background plugin applies a background effect to the original media stream by performing image segmentation on the incoming video frames. The plugin is capable of applying three different kinds of effects called modes: background blur, background image replacement, and background video replacement.

The mode configuration option is determines what background effect to apply. There are three accepted values for the mode: BLUR, IMAGE, and VIDEO. Each mode has at least one required option that needs to be set in the options object, which is outlined below in the Options section.

NOTE: For Typescript users, the mode can be selected by using the exported VirtualBackgroundMode enum, for convenience.

Usage

Supply a video stream to the effect and when loaded, it will return a new stream with the effect applied.

// Create a new video stream by a getting user's video media.
const originalVideoStream = await navigator.mediaDevices.getUserMedia({ video: { width, height } });

// Create the effect.
const effect = new VirtualBackgroundEffect({
  authToken: 'YOUR_AUTH_TOKEN',
  mode: `BLUR`,
  blurStrength: `STRONG`,
  quality: `LOW`,
});

// Load the effect with the input stream.
const newStream = await effect.load(originalVideoStream);

// Attach the new stream to a video element to see the effect in action.
myVideoElement.srcObject = myStream;

Rate Estimator

The Rate Estimator is a utility designed to monitor and adapt to changes in the processing rate of media effects, such as frame rates for video effects. It provides insights into performance, allowing applications to respond dynamically to varying processing capabilities by emitting events based on the processing rate's performance against defined thresholds.

Configuration Options

Configure the Rate Estimator with the following options to tailor its behavior to your application's needs:

| Option | Description | Default Value | |-------------------|------------------------------------------------------------------------------------------------------|--------------------| | hysteresisMargin | Margin of tolerance around the low threshold to prevent rapid toggling between states, expressed as a percentage of the lowThreshold. | 0.05 (5%) | | lowDuration | Duration in seconds that the rate must be below the lowThreshold before considering the rate sustainedly low. | 5 seconds | | lowThreshold | Threshold below which the rate is considered low, expressed as a percentage of the target rate. | 80% of target rate | | minSamples | Minimum number of samples to accumulate before making a rate estimation. | 30 | | maxSamples | Maximum number of samples to consider for rate estimation to prevent using stale data. | 120 |

Events

The Rate Estimator emits events to indicate changes in the processing rate. You can use string values or, if using TypeScript, enums provided by RateEstimatorEvent.

| Event | Description | |---------------------|----------------------------------------------------------------------------| | rate-ok or RateEstimatorEvent.RateOk | Fired when the estimated rate returns to normal, above the lowThreshold. | | rate-low or RateEstimatorEvent.RateLow | Fired when the estimated rate falls below the lowThreshold. | | rate-lagging or RateEstimatorEvent.RateLagging | Fired when the low rate is sustained beyond the duration specified by lowDuration. |

Statuses

The Rate Estimator can be in one of the following statuses, accessible as strings or through the RateEstimatorStatus TypeScript enum:

| Status | Description | |------------|---------------------------------------------------------------| | idle or RateEstimatorStatus.Idle | No data is being processed or the estimator has been reset. | | init or RateEstimatorStatus.Init | Initial data is being collected. | | lagging or RateEstimatorStatus.Lagging | The rate has been low for a sustained period. | | low or RateEstimatorStatus.Low | The current rate is below the threshold. | | ok or RateEstimatorStatus.Ok | The rate is normal, above the threshold. |

Usage

Instantiate the Rate Estimator with your target rate and optional configuration settings. Add timestamps periodically as frames or tasks are processed to enable the estimator to assess the current rate.

import { EffectEvent, RateEstimator, RateEstimatorEvent, RateEstimatorStatus } from '@webex/web-media-effects';

const rateEstimator = new RateEstimator(30, {
  lowThreshold: 24, // Consider rate low if below 24 fps
});

// Assuming 'effect' is an instance of a media effect using the Rate Estimator
effect.setOnFrameProcessedCallback((timestamp) => {
  rateEstimator.addTimestamp(timestamp);
});

// Reset the estimator when the effect is enabled or disabled
effect.on(EffectEvent.Enabled, () => rateEstimator.reset());
effect.on(EffectEvent.Disabled, () => rateEstimator.reset());

// Handle rate change events
rateEstimator.on(RateEstimatorEvent.RateLow, (rate) => {
  console.log(`Rate is low: ${rate}`);
});
rateEstimator.on(RateEstimatorEvent.RateOk, (rate) => {
  console.log(`Rate is ok: ${rate}`);
});
rateEstimator.on(RateEstimatorEvent.RateLagging, (rate) => {
  console.log(`Rate is lagging: ${rate}`);
});

Noise Reduction

The noise reduction effect removes background noise from an audio stream to provide clear audio for calling.

The noise-reduction-effect takes a NoiseReductionEffectOptions config object in its constructor. A developer can optionally pass a workletProcessorUrl parameter (or legacyProcessorUrl) in the config to use a different of test version of the audio processor. An audioContext parameter can be passed into the config as well in order to supply an existing AudioContext; otherwise, a new one will be created.

The effect loads the background thread AudioWorkletProcessor into the main thread AudioWorklet in order to keep the audio computations from impacting UI performance.

Options

There are a few different options that can be supplied to the constructor or updateOptions() method that affect the behavior

| Name | Description | Values | Required | | ------------------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------- | --------------------- | | authToken | Used to authenticate the request for the backend processors | An encoded string token | Yes | | audioContext | An optional AudioContext for custom behavior | AudioContext | No | | mode | Determines whether to run in WORKLET mode or LEGACY mode for older browsers | WORKLET LEGACY | Defaults to WORKLET | | legacyProcessorUrl | A url to fetch the legacy processor that attaches to the deprecated ScriptProcessorNode | A fully qualified URL | No | | workletProcessorUrl | A url to fetch the AudioWorkletProcessor to attach to the AudioWorkletNode | A fully qualified URL | No | | env | Which environment the effect is running in. | EffectEnv.Production EffectEnv.Integration | No |

Supported Bitrates

The noise reduction effect supports the following audio bitrates:

• 16 kHz
• 32 kHz
• 44.1 kHz
• 48 kHz

If an unsupported bitrate is detected, the noise reduction effect will throw the following error: Error: noise reduction: worklet processor error, "Error: Sample rate of X is not supported.

Usage

Supply an audio track or stream to the effect, the effect will handle updating the stream on enable/disable. In the case of a track being passed, listen to the 'track-updated' event to receive the updated track on enable/disable.

// Create a new audio stream by getting a user's audio media.
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });

// Create the effect.
const effect = new NoiseReductionEffect({
  authToken: 'YOUR_AUTH_TOKEN',
  workletProcessorUrl: 'https://my-worklet-processor-url', // For 'WORKLET' mode
  legacyProcessorUrl: 'https://my-legacy-processor-url', // For 'LEGACY' mode
  mode: 'WORKLET', // or 'LEGACY'
});

// Load the effect with the input stream.
await effect.load(stream);

Example

The example app included in this repo is designed to help test functionality and troubleshoot issues. You can run the example app by following the instructions in the README in the example folder. You can also view a live example at https://effects.webex.com.

Development

1. Run yarn to install dependencies.
2. Run yarn prepare to prepare dependencies.
3. Run yarn watch to build and watch for updates.
4. Run yarn test to build, run tests, lint, and run test coverage.

Visual Studio Code

Install the recommended extensions when first opening the workspace (there should be a prompt). These plugins will help maintain high code quality and consistency across the project.

NOTE: VS Code is setup to apply formatting and linting rules on save (Prettier runs first, then ESLint). The rules applied are defined in settings.json.