@open-rtn/plugin-ai-denoiser

Browser-side AI noise suppression extension for open-rtn-sdk.

This package provides a local audio pre-processing extension. It takes a local audio track, runs a DTLN model pipeline through ONNX Runtime Web, WASM, Worker, and AudioWorklet, then returns processed audio through the SDK processor pipeline.

Installation

Install this package together with open-rtn-sdk:

pnpm add open-rtn-sdk @open-rtn/plugin-ai-denoiser

In this monorepo, build open-rtn-sdk first, then build this package:

pnpm --filter open-rtn-sdk run build
pnpm --filter @open-rtn/plugin-ai-denoiser run build

Browser Support

Check support before creating the processor:

import { AIDenoiserExtension } from '@open-rtn/plugin-ai-denoiser';

if (!AIDenoiserExtension.isSupported) {
  throw new Error('AI denoiser is not supported in this browser');
}

The current implementation requires:

• AudioWorklet
• Worker
• WebAssembly
• MediaStreamTrack.applyConstraints
• ONNX Runtime Web WASM backend support

Use desktop Chrome or Edge for the expected runtime behavior. Safari, Firefox, mobile browsers, and low-end devices should be validated separately before being enabled in production.

Quick Start

Register the extension, create a processor, and pipe it to a local audio track:

import { OpenRTC } from 'open-rtn-sdk';
import { AIDenoiserExtension } from '@open-rtn/plugin-ai-denoiser';

const extension = new AIDenoiserExtension({
  assetsPath: '/open-rtn-ai-denoiser',
});

if (!AIDenoiserExtension.isSupported) {
  throw new Error('AI denoiser is not supported in this browser');
}

OpenRTC.registerExtensions([extension]);

const localAudioTrack = await OpenRTC.createMicrophoneAudioTrack({
  AEC: true,
  AGC: true,
  ANS: true,
});

const processor = extension.createProcessor({
  latency: 'BALANCED',
  fallback: 'BYPASS',
});

processor.on('pipeerror', (error) => {
  console.error('AI denoiser pipe error', error);
});

processor.on('overload', () => {
  console.warn('AI denoiser overload', processor.getStats());
});

localAudioTrack.pipe(processor).pipe(localAudioTrack.processorDestination);
await processor.enable();

await client.publish([localAudioTrack]);

OpenRTC.registerExtensions() only registers the extension. AI noise suppression becomes active when the processor is piped to the local audio track and enable() resolves.

Hosting Runtime Assets

assetsPath is the single public path configuration entry. Host these files under the same application-owned static directory:

public/open-rtn-ai-denoiser/models/model_1.onnx
public/open-rtn-ai-denoiser/models/model_2.onnx
public/open-rtn-ai-denoiser/workers/dtln-worker.js
public/open-rtn-ai-denoiser/workers/ort-wasm-simd-threaded.mjs
public/open-rtn-ai-denoiser/workers/ort-wasm-simd-threaded.wasm

Then pass the public directory root:

const extension = new AIDenoiserExtension({
  assetsPath: '/open-rtn-ai-denoiser',
});

The package build copies files from models/ and the required ONNX Runtime Web assets into dist/. Applications can copy dist/models and dist/workers to their own public path or CDN.

Use these response headers for hosted assets:

*.wasm  Content-Type: application/wasm
*.js    Content-Type: text/javascript
*.mjs   Content-Type: text/javascript
*.onnx  Content-Type: application/octet-stream

If the assets are hosted on another origin, configure CORS so the browser can fetch models, worker scripts, and WASM files.

Audio Constraints

By default, the extension requests the SDK to keep browser AEC and AGC enabled while disabling browser native noise suppression:

{
  echoCancellation: true,
  autoGainControl: true,
  noiseSuppression: false,
}

This avoids stacking browser NS with AI denoising. disable() and destroy() request the SDK to roll back the constraint change.

Set disableBrowserNoiseSuppression: false only when the application wants to manage capture constraints itself:

const extension = new AIDenoiserExtension({
  assetsPath: '/open-rtn-ai-denoiser',
  disableBrowserNoiseSuppression: false,
});

Processor Options

const processor = extension.createProcessor({
  latency: 'BALANCED',
  fallback: 'BYPASS',
});

latency controls the amount of buffering used by the realtime path:

• BALANCED is the default and recommended setting for calls.
• LOW uses less buffering and may be more sensitive to runtime scheduling jitter.

fallback controls behavior after runtime failure or overload:

• BYPASS: return to unprocessed audio so the call keeps flowing.
• BROWSER_NS: return to unprocessed audio and request browser native noiseSuppression.
• MUTE_ON_FAILURE: mute output after failure until the processor is reset or re-enabled.

Events And Stats

processor.on('pipeerror', (error) => {});
processor.on('overload', () => {});
processor.on('dump', (blob, name) => {});
processor.on('dumpend', () => {});

Use getStats() to inspect runtime state:

const stats = processor.getStats();
console.log(stats.state, stats.queueDepth, stats.averageInferenceCostMs);

The stats object includes frame counts, underruns, overloads, queue depth, sample rate, and inference cost.

dump() emits a diagnostic blob through the dump event. The current package does not expose a WAV capture API.

Cleanup

Remove the processor from the track when AI denoising is no longer needed:

localAudioTrack.unpipe(processor);
await processor.disable();
await processor.release();

Call release() before discarding the processor so its AudioWorklet, Worker, Web Audio nodes, and ONNX Runtime resources can be released.

Troubleshooting

If AI denoising does not start, verify these points first:

• AIDenoiserExtension.isSupported returns true.
• Runtime assets under assetsPath are reachable by the browser.
• The processor has been connected with localAudioTrack.pipe(processor).pipe(localAudioTrack.processorDestination).
• The local track is an audio track created by open-rtn-sdk.
• processor.getStats().state becomes enabled after enable().
• Worker, model, and WASM requests return 200.

For this repository, implementation plans, research notes, and benchmark evidence live under Design/rtc-plugin-ai-denoiser/.