@shariq102/rtc-ai

Optional AI add-ons for the Synqo SDK. Everything here runs in the participant's browser — no video or audio ever leaves the device, and no API key is required. This package is a separate, opt-in install; the core @shariq102/rtc-sdk ships zero AI code and zero model weights.

npm install @shariq102/rtc-ai        # only if you want in-browser AI effects

| Effect | Factory | Applies to | |---|---|---| | Background blur / virtual background | backgroundBlur() | camera | | Auto-framing ("center stage") | autoFrame() | camera | | AI noise suppression (RNNoise) | noiseSuppression() | microphone |

Camera effects (blur, auto-frame)

import { Room } from "@shariq102/rtc-sdk";
import { backgroundBlur, autoFrame } from "@shariq102/rtc-ai";

const room = new Room();
await room.connect(url, token);
await room.localParticipant.enableCamera();

// Toggle live at any time (no renegotiation). Camera effects are mutually
// exclusive — setting one replaces the other. Pass a FRESH instance to enable:
await room.localParticipant.setCameraEffect(backgroundBlur());  // blur on
await room.localParticipant.setCameraEffect(autoFrame());       // switch to auto-frame
await room.localParticipant.setCameraEffect(undefined);         // off

Microphone effect (noise suppression)

Independent of the camera effect — it can be on at the same time:

import { noiseSuppression } from "@shariq102/rtc-ai";

await room.localParticipant.enableMicrophone();
await room.localParticipant.setMicrophoneEffect(noiseSuppression());  // on
await room.localParticipant.setMicrophoneEffect(undefined);           // off

Apply at capture

const room = new Room({
  videoTransforms: [backgroundBlur({ blurRadius: 14 })],
  audioTransforms: [noiseSuppression()],
});

React

import { useCameraEffect, useMicrophoneEffect } from "@shariq102/rtc-react";

const blur = useMemo(() => (blurOn ? backgroundBlur() : undefined), [blurOn]);
useCameraEffect(room, blur);

const denoise = useMemo(() => (nsOn ? noiseSuppression() : undefined), [nsOn]);
useMicrophoneEffect(room, denoise);

Respecting the room's AI policy

When a room's template enables AI in the Synqo dashboard, the policy rides on the join token. Read it from room.aiPolicy and drive your UI off it:

const p = room.aiPolicy;
if (p?.backgroundBlur?.allowed) {
  showBlurButton();
  if (p.backgroundBlur.defaultOn) {
    await room.localParticipant.setCameraEffect(backgroundBlur());
  }
}
// p?.autoFrame?.allowed and p?.noiseSuppression?.allowed likewise

Options

backgroundBlur(options?)

| Option | Default | Notes | |---|---|---| | blurRadius | 12 | Blur strength in px. | | assetBase | CDN | Self-host the MediaPipe wasm/ folder + selfie_segmenter.tflite. | | fps | 30 | Frame rate for the Firefox/Safari canvas fallback. |

autoFrame(options?)

| Option | Default | Notes | |---|---|---| | padding | 0.6 | Headroom around the face(s), as a fraction of the face box. | | maxZoom | 2 | Maximum zoom-in factor (1 = never zoom). | | smoothing | 0.12 | Glide speed 0..1 (higher = snappier). | | assetBase | CDN | Self-host wasm/ + blaze_face_short_range.tflite. | | fps | 30 | Canvas-fallback frame rate. |

noiseSuppression(options?)

| Option | Default | Notes | |---|---|---| | assetBase | CDN | Self-host the @sapphi-red/web-noise-suppressor dist/ (RNNoise worklet + .wasm). |

How it works

• Camera effects — MediaPipe (segmentation / face detection) runs per frame through MediaStreamTrackProcessor on Chromium/Edge, or a <canvas>.captureStream() fallback on Firefox/Safari. Model weights load only when an effect first starts.
• Noise suppression — RNNoise runs in an AudioWorklet, off the main thread.
• Face-geometry only — auto-framing detects where a face is, never who it is; no biometric identity is computed.
• If an effect can't start, the SDK publishes the raw track instead — a broken effect never breaks the call.

Self-hosting the models (recommended for production)

Every effect defaults assetBase to a public CDN for a zero-config start. To keep every byte on your own infrastructure, copy each effect's assets to your own static host and pass assetBase. Then no third party ever sees a frame or byte.