@convai/experience-embed

A library for embedding Pixel Streaming functionality in web applications, supporting both React and vanilla JavaScript/TypeScript implementations.

Installation

npm install @convai/experience-embed

Step-by-Step Usage Guide

React Implementation

Step 1: Import the Component

First, import the PixelStreamComponent in your React application:

import { PixelStreamComponent } from "@convai/experience-embed";

Step 2: Set Up the Component Reference

Create a ref to access the component's methods:

import { useRef } from "react";
import { PixelStreamComponentHandles } from "@convai/experience-embed";

// Inside your component:
const pixelStreamRef = useRef<PixelStreamComponentHandles>(null);

Step 3: Available Methods

The component provides several methods through the ref:

• enableCamera(): Enables the camera for the experience
• disableCamera(): Disables the camera
• enableCharacterAudio(): Enables character audio
• disableCharacterAudio(): Disables character audio
• initializeExperience(): Initializes the experience

Example usage:

const handleEnableCamera = async () => {
  await pixelStreamRef.current?.enableCamera();
};

Step 4: Create the Component

Add the PixelStreamComponent to your JSX with the required props:

<PixelStreamComponent
  ref={pixelStreamRef}
  expId="your-experiment-id"
  endUserId="user-123" // Optional: for user tracking
  InitialScreen={<div>Loading your experience...</div>}
/>

Step 5: Customize the Initial Screen (Optional)

You can create a custom loading screen:

const CustomLoadingScreen = () => (
  <div
    style={{
      width: "100%",
      height: "100%",
      display: "flex",
      alignItems: "center",
      justifyContent: "center",
      background: "#000",
      color: "#fff",
    }}
  >
    <h2>Loading your experience...</h2>
  </div>
);

// Use it in your component:
<PixelStreamComponent
  ref={pixelStreamRef}
  expId="your-experiment-id"
  InitialScreen={<CustomLoadingScreen />}
/>;

Core (Vanilla TypeScript) Implementation

Step 1: Import the Client

Import the PixelStreamClient in your TypeScript file:

import { PixelStreamClient } from "@convai/experience-embed";

Step 2: Create Container Element

Create a container element in your HTML:

<div id="pixel-stream-container" style="width: 100%; height: 600px;"></div>

Step 3: Initialize the Client

Create a new instance of PixelStreamClient:

const container = document.getElementById("pixel-stream-container");
if (container) {
  const pixelStream = new PixelStreamClient({
    container: container,
    expId: "your-experiment-id",
    endUserId: "user-123", // Optional: for user tracking
    InitialScreen: document.createElement("div"), // Optional custom loading screen
  });
}

Step 4: Available Methods

The client provides the following methods:

// Enable camera
pixelStream.enableCamera().then((success) => {
  console.log("Camera enabled:", success);
});

// Disable camera
pixelStream.disableCamera().then((success) => {
  console.log("Camera disabled:", success);
});

// Enable character audio
pixelStream.enableCharacterAudio();

// Disable character audio
pixelStream.disableCharacterAudio();

// Initialize experience
pixelStream.initializeExperience();

CDN Implementation

You can use the @convai/experience-embed package directly in your HTML without any build tools by loading it from a CDN.

Step 1: Include the UMD Script

Include the library via CDN in your HTML:

<script src="https://unpkg.com/@convai/experience-embed/dist/core/convai-embed.umd.js"></script>

Step 2: Create Container Element

Create a container element in your HTML:

<div id="pixel-stream-container" style="width: 100%; height: 600px;"></div>

Step 3: Initialize the Client

Create a new instance of PixelStreamClient:

<script>
  const container = document.getElementById("pixel-stream-container");

  const customInitialScreen = document.createElement("div");
  customInitialScreen.style.cssText = `
    width: 100%;
    height: 100%;
    position: absolute;
    top: 0;
    left: 0;
    z-index: 10;
    display: flex;
    align-items: center;
    justify-content: center;
    background: rgba(0, 0, 0, 0.8);
    color: white;
    font-family: Arial, sans-serif;
  `;
  customInitialScreen.innerHTML = `
    <div style="text-align: center;">
      <h2>Welcome to the Experience</h2>
      <p>Click anywhere to start</p>
    </div>
  `;

  const pixelStream = new PixelStreamCore.PixelStreamClient({
    expId: "your-experiment-id",
    container: container,
    endUserId: "user-123", // Optional: for user tracking
    InitialScreen: customInitialScreen,
  });

  // Make it available globally for debugging if needed
  window.pixelStream = pixelStream;
</script>

Step 4: Available Methods

Once initialized, the following methods are available on the pixelStream object:

initializeExperience() Starts the pixel streaming session:

pixelStream.initializeExperience();

enableCamera() Requests access to the user’s camera and sends the video stream to the experience:

pixelStream.enableCamera().then((success) => {
  console.log("Camera enabled:", success);
});

disableCamera() Stops the camera stream:

pixelStream.disableCamera().then((success) => {
  console.log("Camera disabled:", success);
});

enableCharacterAudio() Enables audio from the character (from Unreal/Convai):

pixelStream.enableCharacterAudio();

disableCharacterAudio() Enables audio from the character (from Unreal/Convai):

pixelStream.disableCharacterAudio();

Step 5: Add Interactive Buttons (Optional)

You can add buttons in your HTML to trigger these methods:

<div class="controls">
  <button onclick="pixelStream.initializeExperience()">Start Experience</button>
  <button onclick="pixelStream.enableCamera()">Enable Camera</button>
  <button onclick="pixelStream.disableCamera()">Disable Camera</button>
  <button onclick="pixelStream.enableCharacterAudio()">Enable Audio</button>
  <button onclick="pixelStream.disableCharacterAudio()">Disable Audio</button>
</div>

API Reference

PixelStreamComponent Props

| Prop | Type | Required | Description | | --------------------------- | --------------- | -------- | ---------------------------------------------------------- | | expId | string | Yes | Your experiment ID | | endUserId | string | No | User identifier for tracking and long-term memory | | InitialScreen | React.ReactNode | No | Custom loading screen component | | serviceUrls | object | No | Custom service URLs configuration | | serviceUrls.sessionFetch | string | No | Custom URL for fetching the session | | serviceUrls.pixelStreamBase | string | No | Custom base URL for the pixel stream |

PixelStreamClient Options

| Option | Type | Required | Description | | --------------------------- | ----------- | -------- | ---------------------------------------------------------- | | container | HTMLElement | Yes | The container element for the stream | | expId | string | Yes | Your experiment ID | | endUserId | string | No | User identifier for tracking and long-term memory | | InitialScreen | HTMLElement | No | Custom loading screen element | | serviceUrls | object | No | Custom service URLs configuration | | serviceUrls.sessionFetch | string | No | Custom URL for fetching the session | | serviceUrls.pixelStreamBase | string | No | Custom base URL for the pixel stream |