BlendVision Communication SDK

A comprehensive communication SDK, providing audio/video communication, participant management, and remote control capabilities for real-time, multi-participant video conferencing and streaming applications.

Core Features

• Audio/Video Communication: Audio/Video stream publishing and subscription using Real-Time Streaming.
• Participant Management: Real-time tracking of participant join/leave events and status.
• Role-based Permission Control: Admin/user roles for managing remote control permissions.
• Real-time Status Monitoring: Audio/video status and volume level monitoring.
• Canvas Rendering: Participant video rendering.
• Remote Control: Admins can remotely mute/unmute participant audio/video via SEI messages.
• Volume Detection: Real-time microphone volume level detection for active speaker indication.
• Multi-Canvas Support: Dynamic creation and management of participant canvases.

Installation

Using npm/yarn

# Install dependencies
npm install @blendvision/web-bv-communication

# Or using yarn
yarn add @blendvision/web-bv-communication

Import in Your Project

import { CommunicationSdk, SdkConfig, SdkCallbacks } from '@blendvision/web-bv-communication';

Integration Guide

The SDK is designed for seamless integration into various web applications, including vanilla JavaScript, React, and other frameworks. Below are detailed steps and examples for integrating the SDK, focusing on initialization, UI integration, and state management.

1. Basic Integration (Vanilla JavaScript)

Initialize the SDK

Create an instance of CommunicationSdk with a configuration object and optional callbacks to handle events.

// Configure Sdk
const config: SdkConfig = {
  token: 'your-ivs-participant-token', // AWS IVS participant token
  role: 'admin', // or 'user'
  videoInputDeviceId: 'camera-device-id',
  audioInputDeviceId: 'microphone-device-id',
  enableInBandMessaging: true, // enable in-band messaging for remote control (default: true)
  canvasConfig: {
    width: 640,
    height: 480,
    style: { borderRadius: '8px' },
  },
};

// Define event callbacks
const callbacks: SdkCallbacks = {
  onInitialized: () => console.log('SDK initialized'),
  onConnected: () => console.log('Connected to Stage'),
  onParticipantJoined: (participantId, attributes) => console.log(`Participant joined: ${participantId}`, attributes),
  onParticipantLeft: (participantId) => console.log(`Participant left: ${participantId}`),
  onRemoteStreamAdded: (stream, participantId) => console.log(`Remote stream received: ${participantId}`),
  onParticipantVolumeChanged: (participantId, volume) => console.log(`Volume changed: ${participantId}, ${volume}`),
  onParticipantCanvasCreated: (participantId, canvas) => console.log(`Canvas created for: ${participantId}`),
  onError: (error) => console.error('SDK error:', error),
};

// Create and initialize Sdk
const sdk = new CommunicationSdk(config, callbacks);
await sdk.initialize();

Start Publishing and Control Media

// Start publishing audio/video
await sdk.startPublishing();

// Toggle video or audio
sdk.toggleVideo(); // Toggles video on/off
sdk.toggleAudio(); // Toggles audio on/off

// Explicitly set media state
sdk.setLocalAudioEnabled(true);
sdk.setLocalVideoEnabled(false);

Render Participant Canvases

Create canvases for participants to display their video or avatar.

// Create a canvas for a specific participant
const container = document.getElementById('participant-1-container') as HTMLElement;
const canvas = sdk.createParticipantCanvas('participant-1', container);

// Remove canvas when no longer needed
sdk.removeParticipantCanvas('participant-1');

2. React Integration

For React applications, use a context-based approach to manage the Sdk instance and state, ensuring clean integration with React components.

Create a Communication Context

import { FC, ReactNode, createContext, useContext, useReducer, useEffect } from 'react';
import { CommunicationSdk, SdkConfig, SdkCallbacks, ParticipantInfo } from '@blendvision/web-bv-communication';

interface CommunicationState {
  sdk: CommunicationSdk | null;
  isConnected: boolean;
  participants: ParticipantInfo[];
  error: string | null;
}

interface CommunicationAction {
  type: 'SET_SDK' | 'SET_CONNECTED' | 'ADD_PARTICIPANT' | 'REMOVE_PARTICIPANT' | 'SET_ERROR';
  payload: any;
}

const initialState: CommunicationState = {
  sdk: null,
  isConnected: false,
  participants: [],
  error: null,
};

const communicationReducer = (state: CommunicationState, action: CommunicationAction): CommunicationState => {
  switch (action.type) {
    case 'SET_SDK':
      return { ...state, sdk: action.payload };
    case 'SET_CONNECTED':
      return { ...state, isConnected: action.payload };
    case 'ADD_PARTICIPANT':
      return { ...state, participants: [...state.participants, action.payload] };
    case 'REMOVE_PARTICIPANT':
      return { ...state, participants: state.participants.filter((p) => p.id !== action.payload) };
    case 'SET_ERROR':
      return { ...state, error: action.payload };
    default:
      return state;
  }
};

const CommunicationContext = createContext<{
  state: CommunicationState;
  initializeSdk: (config: SdkConfig) => Promise<void>;
} | null>(null);

export const CommunicationProvider: FC<{ children: ReactNode }> = ({ children }) => {
  const [state, dispatch] = useReducer(communicationReducer, initialState);

  const initializeSdk = async (config: SdkConfig) => {
    try {
      const callbacks: SdkCallbacks = {
        onInitialized: () => console.log('Sdk initialized'),
        onConnected: () => dispatch({ type: 'SET_CONNECTED', payload: true }),
        onParticipantJoined: (participantId, attributes) => {
          dispatch({ type: 'ADD_PARTICIPANT', payload: { id: participantId, attributes } });
        },
        onParticipantLeft: (participantId) => {
          dispatch({ type: 'REMOVE_PARTICIPANT', payload: participantId });
        },
        onError: (error) => dispatch({ type: 'SET_ERROR', payload: error.message }),
      };

      const sdk = new CommunicationSdk(config, callbacks);
      await sdk.initialize();
      dispatch({ type: 'SET_SDK', payload: sdk });
    } catch (error) {
      dispatch({ type: 'SET_ERROR', payload: (error as Error).message });
    }
  };

  return <CommunicationContext.Provider value={{ state, initializeSdk }}>{children}</CommunicationContext.Provider>;
};

export const useCommunication = () => {
  const context = useContext(CommunicationContext);
  if (!context) {
    throw new Error('useCommunication must be used within CommunicationProvider');
  }
  return context;
};

Participant Video Component

Render participant videos or avatars using a React component.

import { useEffect, useRef, FC } from 'react';
import { useCommunication } from './CommunicationContext';

interface ParticipantVideoProps {
  participantId: string;
}

export const ParticipantVideo: FC<ParticipantVideoProps> = ({ participantId }) => {
  const { state } = useCommunication();
  const containerRef = useRef<HTMLDivElement>(null);

  useEffect(() => {
    if (containerRef.current && state.sdk) {
      const canvas = state.sdk.createParticipantCanvas(participantId, containerRef.current);
      return () => {
        state.sdk.removeParticipantCanvas(participantId);
      };
    }
  }, [participantId, state.sdk]);

  return (
    <div
      ref={containerRef}
      style={{
        width: '100%',
        height: '200px',
        backgroundColor: '#3c4043',
        borderRadius: '8px',
        overflow: 'hidden',
      }}
    />
  );
};

Usage in a React App

import { FC } from 'react';
import { CommunicationProvider, ParticipantVideo } from './CommunicationContext';

const App: FC = () => {
  const { state, initializeSdk } = useCommunication();

  useEffect(() => {
    initializeSdk({
      token: 'your-ivs-participant-token',
      role: 'admin',
      videoInputDeviceId: 'camera-device-id',
      audioInputDeviceId: 'microphone-device-id',
    });
  }, []);

  return (
    <div>
      {state.participants.map((participant) => (
        <ParticipantVideo key={participant.id} participantId={participant.id} />
      ))}
    </div>
  );
};

export default () => (
  <CommunicationProvider>
    <App />
  </CommunicationProvider>
);

3. Integration with Other Frameworks

The Sdk is framework-agnostic and can be integrated with Vue, Angular, or other JavaScript frameworks by following similar patterns:

• Vue: Use a reactive store (e.g., Pinia) to manage the Sdk instance and participant state, similar to the React Context example.
• Angular: Inject the Sdk as a service, using RxJS to handle participant and stream events.
• Svelte: Store the Sdk instance in a Svelte store, binding participant canvases to DOM elements.

4. Admin Remote Control (Admin Role Only)

Admins can control participant media states using SEI messages.

if (config.role === 'admin') {
  // Mute/unmute participant audio or video
  sdk.muteParticipantAudio('participant-id');
  sdk.unmuteParticipantAudio('participant-id');
  sdk.muteParticipantVideo('participant-id');
  sdk.unmuteParticipantVideo('participant-id');

  // Broadcast custom command
  await sdk.broadcastCommand({
    type: 'MUTE_AUDIO',
    targetParticipantId: 'participant-id',
  });
}

5. Best Practices for Integration

• State Management: Use a centralized state (e.g., React Context, Vue Pinia) to track Sdk instance, participants, and errors.
• Resource Cleanup: Always call sdk.leaveStage() when unmounting components or closing the app to release resources.
• Error Handling: Implement onError to display user-friendly messages for permission or network issues.
• Performance: Limit canvas sizes (e.g., canvasConfig: { width: 640, height: 480 }) to reduce rendering load.

Core Features and Implementation Notes

• Canvas Rendering: Renders at 15 FPS for balanced performance. Each canvas displays participant video or initials (if no video) and a volume bar.
• Volume Detection: Monitors volume every 500ms with exponential smoothing (0.6/0.4 weights) for smooth UI updates.
• SEI Messaging: Commands (e.g., mute/unmute) are sent via video track SEI messages, requiring an active video stream.
• Device Checking: Added pre-initialization device availability checks to prevent publishing failures.
• State Reset: Audio/video enabled states reset to true on stopPublishing() to ensure consistent behavior.

Known Limitations

• SEI Dependency: Remote control commands require an active video stream, limiting functionality in audio-only scenarios.
• Browser Compatibility: Full support for Chrome 88+, Firefox 85+, Edge 88+. Safari 14+ has limited volume detection due to AudioContext restrictions.
• Performance: Optimized for small to medium stages (up to 10 participants). Large stages may require further optimization (e.g., dynamic FPS, subscription batching).
• Mobile Support: Not officially tested on mobile browsers; performance may vary.

API Reference

CommunicationSdk

Constructor

constructor(config: SdkConfig, callbacks?: SdkCallbacks)

Main Methods

| Method | Description | Parameters | Return Value | | --------------------------- | -------------------------------------- | ----------------------------------------------- | ----------------------------------- | | initialize() | Initialize Sdk and join stage | - | Promise<void> | | startPublishing() | Start publishing audio/video | - | Promise<void> | | stopPublishing() | Stop publishing and reset media states | - | void | | toggleVideo() | Toggle video on/off | - | void | | toggleAudio() | Toggle audio on/off | - | void | | getParticipants() | Get participants list | - | IterableIterator<ParticipantInfo> | | getSelfId() | Get self id | - | string \| null | | leaveStage() | Leave stage and clean up | - | void | | muteParticipantAudio() | Mute participant audio (admin) | participantId: string | void | | unmuteParticipantAudio() | Unmute participant audio (admin) | participantId: string | void | | muteParticipantVideo() | Mute participant video (admin) | participantId: string | void | | unmuteParticipantVideo() | Unmute participant video (admin) | participantId: string | void | | getParticipantVolume() | Get participant volume | participantId: string | number | | getParticipantCanvases() | Get all participant canvases | - | Map<string, HTMLCanvasElement> | | getParticipantCanvas() | Get specific participant canvas | participantId: string | HTMLCanvasElement \| null | | createParticipantCanvas() | Create custom canvas | participantId: string, container: HTMLElement | HTMLCanvasElement | | removeParticipantCanvas() | Remove custom canvas | participantId: string | void | | broadcastCommand() | Broadcast command via SEI | command: RemoteCommand, repeatCount?: number | Promise<boolean> |

SdkConfig

interface SdkConfig {
  token: string; // Participant token
  role: 'admin' | 'user';
  videoInputDeviceId: string;
  audioInputDeviceId: string;
  enableInBandMessaging?: boolean; // Enable SEI messaging (default: true)
  canvasConfig?: {
    width?: number;
    height?: number;
    style?: Partial<CSSStyleDeclaration>;
  };
}

SdkCallbacks

interface SdkCallbacks {
  onInitialized?: () => void;
  onConnected?: () => void;
  onDisconnected?: () => void;
  onLocalStreamReady?: (stream: MediaStream) => void;
  onRemoteStreamAdded?: (stream: MediaStream, participantId: string) => void;
  onRemoteStreamRemoved?: (participantId: string) => void;
  onParticipantJoined?: (participantId: string, attributes?: Record<string, unknown>) => void;
  onParticipantLeft?: (participantId: string) => void;
  onVideoToggled?: (enabled: boolean) => void;
  onAudioToggled?: (enabled: boolean) => void;
  onPublishingStarted?: () => void;
  onPublishingStopped?: () => void;
  onError?: (error: Error) => void;
  onStageLeft?: (reason?: string) => void;
  onParticipantMuted?: (participantId: string, type: 'audio' | 'video') => void;
  onParticipantUnmuted?: (participantId: string, type: 'audio' | 'video') => void;
  onCommandReceived?: (command: RemoteCommand) => void;
  onStreamStatusChanged?: (participantId: string, status: StreamStatus) => void;
  onParticipantVolumeChanged?: (participantId: string, volume: number) => void;
  onParticipantCanvasCreated?: (participantId: string, canvas: HTMLCanvasElement) => void;
  onParticipantCanvasRemoved?: (participantId: string) => void;
}

Troubleshooting

Common Issues

Permission Errors

• Verify browser permissions for camera/microphone.
• Handle onError callback to display user prompts.

Connection Failures

• Validate token and stage accessibility.
• Monitor onDisconnected for network issues.

Canvas Rendering Issues

• Ensure container elements have valid dimensions.
• Check browser support for Canvas 2D context.

Volume Detection Issues

• Ensure audio tracks are active and permissions granted.

Debugging Tips

const callbacks: SdkCallbacks = {
  onError: (error) => console.error('Sdk error:', error, { timestamp: new Date().toISOString() }),
  onParticipantVolumeChanged: (id, volume) => console.log(`Volume for ${id}: ${volume}`),
};

// Periodically log status
setInterval(() => {
  if (sdk) {
    console.log('SDK status:', {
      participants: Array.from(sdk.getParticipants()),
      canvases: sdk.getParticipantCanvases().size,
    });
  }
}, 5000);

Browser Support

• Chrome 88+: ✅ Full support
• Firefox 85+: ✅ Full support
• Safari 14+: ✅ Full support
• Edge 88+: ✅ Full support
• Mobile Browsers: Not officially tested; expect performance limitations.

🆘 Support

For questions or suggestions, contact the development team at [email protected].