Cephable Web SDK

The Cephable Web SDK is a powerful suite of tools designed for developers looking to integrate advanced, AI-driven capabilities into their web applications. This SDK provides services, each catering to specific functionalities such as camera control, voice commands, device profile management, and more. The SDK is exclusively available to users with a valid Cephable license, ensuring that your applications are powered by Cephable’s cutting-edge technologies.

Services

1. CephableService

The CephableService is the central service that integrates various Cephable services into a web application. It greatly simplifies integrations.

Key Features:

• Initialize and manage all other services including voice, camera, auth and profiles.
• Automatic profile switching based on the active application.
• Support for both regular and guest user authentication.

Example Usage:

const cephableService = new CephableService({
    authenticationConfiguration: {
        clientId: 'your-client-id',
        clientSecret: 'your-client-secret',
        redirectUri: location.href,
        autoRefresh: true
    },
    deviceName: 'your-device-or-app-name',
    deviceTypeId: 'your-device-type-id',
    locale: 'en-US',
    includeDefaultControls: true // enabled default commands for clicking things, scrolling, zooming, typing, etc.
    customControls: [] // add custom controls if you want more than just the defaults
    enableRemoteControls: false // set to true to enable remote commands via device hub (only for authenticated users)
    deviceHubConfiguration: {} // set device hub details if using remote controls
});

// alternatively sign in a real cephable user
// pass null for voice config or camera config if you only want one input. Or none if you only want remote controls
await cephableService.initializeWithGuestUser({
    locale: 'en-US',
    onPartialResult: (result) => {
        // do something to show speech if you want
    },
    onFinalResult: (result) => {
        // do something to show speech if you want
    },

}, {
    isDrawingEnabled: true,
    videoElementId: "video",
    canvasElementId: "canvas",
    faceLinesColor: "red",
    baselineFaceLinesColor: "blue"
});

2. VoiceService

The VoiceService implements voice command capabilities, allowing your application to respond to spoken commands. This service integrates with Cephable’s voice recognition models for accurate and reliable performance.

Key Features:

• Real-time speech recognition using the Vosk engine.
• Event handling for partial and final speech recognition results.
• Control over voice recognition operations, including start, pause, resume, and stop.

Example Usage:

const voiceService = new VoiceService({
    locale: 'en-US',
    modelPath: 'path-to-hosted-model',
    audioWorkletPath: './RecognizerAudioProcessor.js', // Optional: specify custom path to audio worklet file
    onPartialResult: (result) => {
        console.log('Partial Result: ', result);
    },
    onFinalResult: (result) => {
        console.log('Final Result: ', result);
    },
});

voiceService.startVoiceControls([]);

3. CameraService

The CameraService is designed to handle camera operations, including capturing video from a webcam and performing face detection. This service provides methods to start, stop, pause, and resume the camera feed and process the video frames to detect and analyze faces in real-time.

Key Features:

• Initialize Service: Loads and caches models for offline use
• Start/Stop Camera: Begins, pauses, resumes, and stops capturing video from the webcam and processes each frame to detect faces.
• Face Detection: Detects faces in the video feed using the BlazeFace ONNX model.

Example Usage:

const cameraService = new CameraService({
    modelDirectoryPath: 'path-to-remote-models',
    isDrawingEnabled: true,
    videoElementId: 'video',
    canvasElementId: 'canvas',
    faceLinesColor: 'red',
    baselineFaceLinesColor: 'blue',
    onGesturesRecognized: (gestures: string[]) => {
        // do something with the gestures
    },
    onFaceProcessed: (face: FaceExpression) => {
        // do something with the face data
    },
});

cameraService.startCameraControls();

Along with the javascript to initialize it, you'll also need a <video> and <canvas> element with distinct IDs to read the video frames from and to draw the frames over.

<video id="video" style="display: none"></video>
<canvas id="canvas"></canvas>

4. DOMInteractiveService

The DOMInteractiveService comes packed with easy to use functions for DOM interactions on behalf of the user such as scrolling, typing, navigation, focusing, and more.

Key Features:

• Use tab order for navigating forward/backward through the DOM
• Fuzzy search for elements on the page and click, scroll to, or focus them
• Scroll to specific locations on the page
• Configurable filtering for automatic interactions to reduce false positives
• Support for excluding specific element types and hidden elements

Configuration Options:

The DomInteractionService constructor accepts an options object with the following properties:

• searchThreshold: (default: 0.4) Fuzzy search threshold for matching elements. Lower values = stricter matching.
• excludedSelectors: (default: []) Array of CSS selectors to exclude from automatic interactions. Example: ['a', '.footer-link'] to exclude all links and elements with class 'footer-link'.
• excludeHidden: (default: false) When true, filters out elements that are not visible (display: none, visibility: hidden, or outside viewport).
• stricterAutoThreshold: (optional) Stricter threshold used specifically for automatic interactions to reduce false positives.

Example Usage:

// Basic usage
const domService = new DomInteractionService();
// these are just sample values, but you Cephable will search the DOM to find an element that matches the display value
// these methods will return an IResult<HTMLElement> with an "Ok" resultType if the element is found and an "Invalid" result type if not
const scrollResult = domService.scrollToElementByDisplayValue('Button one');
const focusResult = domService.focusElementByDisplayValue('Button two');
const clickResult = domService.clickElementByDisplayValue('Button three');

// Advanced configuration for automatic interactions
const restrictedDomService = new DomInteractionService({
    excludedSelectors: ['a', '.navigation', '.footer-link'], // Exclude all links and navigation elements
    excludeHidden: true, // Only interact with visible elements
    stricterAutoThreshold: 0.2, // Require 80% similarity for automatic interactions
});

// Use specialized methods for automatic interactions (applies stricter filtering)
const autoClickResult = restrictedDomService.autoClickElementByDisplayValue('submit');
const autoFocusResult = restrictedDomService.autoFocusElementByDisplayValue('username');

5. AuthenticationService

The AuthenticationService manages user authentication, including OAuth flows and guest users.

Key Features

• User Authentication: Supports OAuth-based authentication for registered users. Guest Authentication: Allows the creation and management of guest users.
• Token Management: Manages OAuth tokens, including automatic refreshing of access tokens. Secure Storage: Stores authentication state securely using an encrypted storage provider.
• Automatic Token Refreshing: Automatically refreshes tokens before they expire, ensuring continuous authentication without user intervention.

Example Usage:

const authService = new AuthenticationService({
    clientId: 'your-client-id',
    clientSecret: 'your-client-secret', // you should not store this in your code
    redirectUri: location.href, // a redirect uri that can handle the auth code
    autoRefresh: true, // leave on if you want Cephable SDK to keep user signed in
});
authService.startUserAuth(false);
// false value is for existing users, true is for new users.
// Either way a user can create a new account from sign in

//...
// then on a page to handle the callback from oauth:
// if there is a code in the query string, exchange it with the auth service and remove the query string
const urlParams = new URLSearchParams(window.location.search);
const code = urlParams.get('code');
const state = urlParams.get('state');
if (code && state) {
    await authService.authenticateFromCode(code, state);
    window.history.replaceState({}, document.title, window.location.pathname);
}

6. DeviceProfileService

The DeviceProfileService manages user device profiles and handles the current profile to use for controls from the CephableService. Profiles can include settings such as macros, keybindings, and application associations.

Key Features:

• Manage user device profiles.
• Store and retrieve profiles from secure storage.

Example Usage:

const authService = new AuthenticationService({
    clientId: 'your-client-id',
    clientSecret: 'your-client-secret', // you should not store this in your code
    redirectUri: location.href, // a redirect uri that can handle the auth code
});
const profileService = new DeviceProfileService(authService);
await profileService.loadProfiles();

profileService.currentProfile = {
    name: 'test',
    configuration: {
        macros: [], // configure macros if you want
        keybindings: [],
        audioEvents: [],
        dictationCommands: ['type'],
    },
};

Getting Started

Installation

1. Ensure you have a valid Cephable license: Before using the SDK, you must have an active Cephable license

2. Install the SDK via npm: You can easily add the Cephable WebSDK to your project using the NuGet Package Manager Console:

   npm install @cephable/cephable-web

3. Configure Your Application:

const cephableService = new CephableService({
    authenticationConfiguration: {
        clientId: 'your-client-id',
        clientSecret: 'your-client-secret',
        redirectUri: location.href,
        autoRefresh: true,
    },
    deviceName: 'your-device-or-app-name',
    deviceTypeId: 'your-device-type-id',
    locale: 'en-US',
    includeDefaultControls: true, // enables default commands for clicking things, scrolling, zooming, typing, etc.
    enableAutoDomActions: true, // enables cephable to take phrases from a user and automatically search for interactable elements to focus and click on without needing a prefix command to be said like 'click' or 'focus'
    domInteractionOptions: {
        // optional: configure automatic DOM interactions to reduce false positives
        excludedSelectors: ['a', '.footer'], // exclude links and footer elements from automatic interactions
        excludeHidden: true, // only interact with visible elements
        stricterAutoThreshold: 0.2, // require 80% similarity for automatic interactions (vs 60% default)
    },
    customControls: [], // add custom controls if you want more than just the defaults
    enableRemoteControls: false, // set to true to enable remote commands via device hub (only for authenticated users)
    deviceHubConfiguration: {}, // set device hub details if using remote controls
});

// alternatively sign in a real cephable user
// pass null for voice config or camera config if you only want one input. Or none if you only want remote controls
await cephableService.initializeWithGuestUser(
    {
        locale: 'en-US',
        onPartialResult: (result) => {
            // do something to show speech if you want
        },
        onFinalResult: (result) => {
            // do something to show speech if you want
        },
    },
    {
        isDrawingEnabled: true,
        videoElementId: 'video',
        canvasElementId: 'canvas',
        faceLinesColor: 'red',
        baselineFaceLinesColor: 'blue',
    }
);

Using intelligent voice commands and natural language understanding

Cephable is equipped with running natural language processing models locally in your website/app and gives you the ability to customize how it works. To get started with the CephableService you'll need to enable a few options:

• enableIntelligentCommands
• customEntities
• (optionally): use your customEntities in your customControls

new CephableService({
    authenticationConfiguration: {
        clientId: 'your-client-id',
        clientSecret: 'your-client-secret', // you should not store this in your code
        redirectUri: location.href,
        autoRefresh: true,
    },
    deviceName: 'Your App',
    deviceTypeId: 'your-device-id',
    locale: 'en-US',
    includeDefaultControls: true,
    enableAutoDomActions: true, // enable automatic focus or selection of elements from voice commands
    enableIntelligentCommands: true, // enable NLU powered commands for profile and custom action controls
    includeBuiltinEntities: true,
    domInteractionOptions: {
        excludedSelectors: ['a', '.footer'], // Exclude links and navigation
        excludeHidden: true, // Only interact with visible elements
        stricterAutoThreshold: 0.2, // Require 80% similarity for auto interactions
    },
    customControls: [
        {
            id: 'alert',
            defaultCommands: ['alert', 'show alert', 'show alert box'],
            description: 'Show an alert box',
        },
        {
            id: 'navigate',
            defaultCommands: ['navigate to @page', 'go to @page'],
            description: 'Navigate to a specific page',
        },
    ],
    customEntities: {
        page: {
            options: {
                schedule: ['schedule', 'calendar'],
                map: ['map'],
                speakers: ['speakers', 'presenters'],
                about: ['about', 'info'],
                tutorials: ['tutorials', 'tutorial'],
            },
        },
    },
    onCustomControlAction: (control, command, additionalinput, intent, entities) => {
        // do something with the custom control info. Return 'true' if handled
    },
    logLevel: 'info', // set log level to debug for development
});

In your customControls defaultCommands utterances, you can add entities from your list with the @ indicator such as the @page sample above which references the page entity from customEntities.

You can handle intents and entities in the onCustomControlAction: (control, command, additionalinput, intent, entities) => {} function in the configuration for the CephableService where the intent is the name of the intent detected if one was, and entities is a collection of entities and relevant information about those entities detected from the command.

Telling Cephable how to interact with text inputs and voice

Cephable can default allow for the "type" command in any text field the user is focused in to type into that field. You can also add data-cephable- attributes to specific elements to inform how typing should work including:

• data-cephable-no-spell: Tells Cephable not to activate spelling mode in that text field
• data-cephable-sentence-case: Tells Cephable to automatically sentence case the text inputs
• data-cephable-title-case: Tells Cephable to automatically capitalize the first letter of each word. Common for name fields
• data-cephable-optimistic: Tells Cephable to automatically enable optimistic mode when in that text field
• data-cephable-alt: String value telling Cephable other ways to search for this element when interacting
• data-cephable-ignore-auto: true/false value telling Cephable to ignore that element when doing automatic dom interactions. These values will only be selectable/interactable via an explicit user action like "click, focus, tap" commands.

4. Start Using the Services: Once configured, you can start utilizing the various services provided by the SDK.

License

The Cephable WebSDK is only available to licensed users. Ensure that your application complies with Cephable’s licensing terms. For more details on obtaining a license, please contact Cephable Sales or online at contact cephable.com.

Support and Contact

For any inquiries, please contact Cephable or your account representatitve.