@bbb-mobile/video-effects

Camera video effects for the BigBlueButton meeting SDK for React Native (@bbb-mobile/bigbluebutton-mobile-sdk): background blur and scenes, colour filters, AR effects (face props and particles), touch-up and low light. The pixels change on the device before the video is encoded, so every other participant (web client included) sees the result and nothing happens server-side. The effects only ever apply to the local camera; screen share is never processed.

Developed and maintained by BiggerBlueButton.

Requires the LiveKit WebRTC fork (@livekit/react-native-webrtc 144.2.0) and the Expo modules runtime (Expo SDK 57), set up as the SDK's page describes; built and tested with React Native 0.86.3 on Android and iOS, as part of @bbb-mobile/bigbluebutton-mobile-sdk (0.2.0 and later; each SDK release pins the module version it was built with, 0.1.2 for SDK 0.3.1).

Install

Nothing to install by hand: @bbb-mobile/bigbluebutton-mobile-sdk depends on this Expo module at an exact version, so npm installs it with the SDK and Expo autolinking links it (run pod install on iOS). Hosts do not call it either. The camera sheet shows the picker under the preview, and the own camera tile's long-press menu opens it ("Video effects") while the camera is on. A host that does not want the feature passes videoEffects={false} to <BigBlueButtonMeeting/>: the picker is hidden and no frame processor is attached, but the module, and ML Kit on Android, stays in the app.

How it works

• Both platforms register a frame processor named bigbluebutton with the LiveKit WebRTC fork's ProcessorProvider (@livekit/react-native-webrtc). The client attaches it to its camera tracks with the fork's track._setVideoEffects(['bigbluebutton']) whenever an effect is selected and detaches it when the selection is empty, so a meeting without effects has no per-frame cost.
• Android (android/): OpenGL ES pipeline on an EGL context shared with the fork, ML Kit selfie segmentation (person mask) and face detection (landmarks), bundled models. Adds com.google.mlkit:segmentation-selfie and com.google.mlkit:face-detection to the app: Google's ML Kit, under Google's ML Kit terms, from Google's Maven repository, about 20 MB of native code per ABI. ML Kit sends device and app information, per-installation identifiers and performance metrics to Google (ML Kit data disclosure); declare them in the app's Google Play Data safety form. build.gradle depends on the Gradle project React Native autolinking creates for the fork (:livekit_react-native-webrtc).
• iOS (ios/): Core Image on Metal, Vision person segmentation and face landmarks. System frameworks only; the artwork ships in the BbbVideoEffects resource bundle.
• index.ts: the effect catalog (ids, English labels, picker thumbnails) and videoEffects, the selection manager (setConfig, attach/detach of camera tracks, subscribe).

Android note

The fork's VideoEffectProcessor releases a frame twice when the processor list is empty; the SDK ships a patch for it (patches/@livekit+react-native-webrtc+144.2.0.patch of the main package) and detaches with null on Android regardless.

Changelog

0.1.2

• Page corrected, no code change: the SDK pin and release it ships with are no longer quoted as 0.1.0 / 0.2.0. Pinned by @bbb-mobile/bigbluebutton-mobile-sdk 0.3.1.

0.1.1

• expo-module.config.json lists ios instead of apple: Expo autolinking no longer offers the module to a macOS (react-native-macos) or tvOS target, whose pod install failed on the iOS-only podspec. No change on Android or iOS.

0.1.0

• First release, with @bbb-mobile/bigbluebutton-mobile-sdk 0.2.0: ML Kit segmentation-selfie 16.0.0-beta6 and face-detection 16.1.7 on Android, Vision and Core Image on iOS (16.4 or later).

License

LGPL-3.0 (the LICENSE file in this package), like BigBlueButton, whose meeting client this module belongs to.