appliance-plugin

中文文档

This plugin is based on the plugin mechanism of white-web-sdk, and realizes a set of whiteboard teaching AIDS drawing tools. At the same time, it is also based on @netless/window-manager, which can be used on multiple Windows.

Introduction

appliance-plugin, Depend on white-web-SDK, @netless/window-manager, And based on web API support for offscreenCanvas.

Principle

1. The plugin is mainly based on the 2D functionality of SpriteJS, supports WebGL2 rendering, and is backward compatible with downgrades to WebGL and Canvas2D.
2. The plugin uses the dual WebWorker + OffscreenCanvas mechanism to process the drawing calculation and rendering logic in a separate worker thread, not occupying the CPU tasks of the main thread.
3. For mobile terminals that do not support OffscreenCanvas, it will be processed in the main thread.

Plugin usage

Install

npm install @netless/appliance-plugin

Register Plugin

Plugins can support two scenarios, they have different plugin names:

• Multi-window ApplianceMultiPlugin

import { ApplianceMultiPlugin } from '@netless/appliance-plugin';

• Single whiteboard ApplianceSinglePlugin

import { ApplianceSinglePlugin } from '@netless/appliance-plugin';

worker.js file CDN deployment

We use dual worker concurrency to improve drawing efficiency, which makes it more than 40% more efficient than the main thread. However, the common dependencies on the two worker files are duplicated, so if we build them directly into the package, it will greatly increase the package size. So we allow worker.js files to be deployed via CDN. Just deploy the files under @netless/appliance-plugin/cdn to the CDN, and then configure the CDN addresses of the two worker.js files in the second parameter options.cdn of getInstance in the plugin. This solves the problem of excessive package size.

The total package is about 400kB, and the two worker.js files are 800kB each. If you need to consider the size of the build package, please choose to configure CDN.

Access Mode Reference

fastboard (Direct integration with fastboard)

// The method of importing worker.js is optional. If using CDN, you don't need to import from dist. If importing from dist, you need to configure it into options.cdn in the form of a resource module and blob inline. Such as `?raw`, this requires packer support. Vite supports `?raw` by default, webpack needs to configure raw-loader or asset/source.
import fullWorkerString from '@netless/appliance-plugin/dist/fullWorker.js?raw';
import subWorkerString from '@netless/appliance-plugin/dist/subWorker.js?raw';
const fullWorkerBlob = new Blob([fullWorkerString], {type: 'text/javascript'});
const fullWorkerUrl = URL.createObjectURL(fullWorkerBlob);
const subWorkerBlob = new Blob([subWorkerString], {type: 'text/javascript'});
const subWorkerUrl = URL.createObjectURL(subWorkerBlob);

// Integration with fastboard-react
// Full package mode reference
// import { useFastboard, Fastboard } from "@netless/fastboard-react/full";
// Subpackage reference
import { useFastboard, Fastboard } from "@netless/fastboard-react";

const app = useFastboard(() => ({
    sdkConfig: {
      ...
    },
    joinRoom: {
      ...
    },
    managerConfig: {
      cursor: true,
      enableAppliancePlugin: true,
      ...
    },
    enableAppliancePlugin: {
      cdn: {
          fullWorkerUrl,
          subWorkerUrl,
      }
      ...
    }
  }));

// Integration with fastboard
// Full package mode reference
// import { createFastboard, createUI } from "@netless/fastboard/full";
// Subpackage reference
import { createFastboard, createUI } from "@netless/fastboard";

const fastboard = await createFastboard({
    sdkConfig: {
      ...
    },
    joinRoom: {
      ...
    },
    managerConfig: {
      cursor: true,
      supportAppliancePlugin: true,
      ...
    },
    enableAppliancePlugin: {
      cdn: {
          fullWorkerUrl,
          subWorkerUrl,
      }
      ...
    }
  });

Multi-window (Direct integration with window-manager)

import '@netless/window-manager/dist/style.css';
import '@netless/appliance-plugin/dist/style.css';

import { WhiteWebSdk } from "white-web-sdk";
import { WindowManager } from "@netless/window-manager";
import { ApplianceMultiPlugin } from '@netless/appliance-plugin';
// The method of importing worker.js is optional. If using CDN, you don't need to import from dist. If importing from dist, you need to configure it into options.cdn in the form of a resource module and blob inline. Such as `?raw`, this requires packer support. Vite supports `?raw` by default, webpack needs to configure raw-loader or asset/source.
import fullWorkerString from '@netless/appliance-plugin/dist/fullWorker.js?raw';
import subWorkerString from '@netless/appliance-plugin/dist/subWorker.js?raw';
const fullWorkerBlob = new Blob([fullWorkerString], {type: 'text/javascript'});
const fullWorkerUrl = URL.createObjectURL(fullWorkerBlob);
const subWorkerBlob = new Blob([subWorkerString], {type: 'text/javascript'});
const subWorkerUrl = URL.createObjectURL(subWorkerBlob);

const whiteWebSdk = new WhiteWebSdk(...)
const room = await whiteWebSdk.joinRoom({
    ...
    invisiblePlugins: [WindowManager, ApplianceMultiPlugin],
    useMultiViews: true,
})
const manager = await WindowManager.mount({ room, container: elm, chessboard: true, cursor: true, supportAppliancePlugin: true});
if (manager) {
    await manager.switchMainViewToWriter();
    await ApplianceMultiPlugin.getInstance(manager,
        {
            options: {
                cdn: {
                    fullWorkerUrl,
                    subWorkerUrl,
                },
                ...
            }
        }
    );
}

Note The project needs to import the CSS file import '@netless/appliance-plugin/dist/style.css';

Single whiteboard (Direct integration with white-web-sdk)

import '@netless/appliance-plugin/dist/style.css';

import { WhiteWebSdk } from "white-web-sdk";
import { ApplianceSinglePlugin, ApplianceSigleWrapper } from '@netless/appliance-plugin';
// The method of importing worker.js is optional. If using CDN, you don't need to import from dist. If importing from dist, you need to configure it into options.cdn in the form of a resource module and blob inline. Such as `?raw`, this requires packer support. Vite supports `?raw` by default, webpack needs to configure raw-loader or asset/source.
import fullWorkerString from '@netless/appliance-plugin/dist/fullWorker.js?raw';
import subWorkerString from '@netless/appliance-plugin/dist/subWorker.js?raw';
const fullWorkerBlob = new Blob([fullWorkerString], {type: 'text/javascript'});
const fullWorkerUrl = URL.createObjectURL(fullWorkerBlob);
const subWorkerBlob = new Blob([subWorkerString], {type: 'text/javascript'});
const subWorkerUrl = URL.createObjectURL(subWorkerBlob);

const whiteWebSdk = new WhiteWebSdk(...)
const room = await whiteWebSdk.joinRoom({
    ...
    invisiblePlugins: [ApplianceSinglePlugin],
    wrappedComponents: [ApplianceSigleWrapper]
})
await ApplianceSinglePlugin.getInstance(room, 
    {
        options: {
            cdn: {
                fullWorkerUrl,
                subWorkerUrl,
            }
            ...
        }
    }
);

Note The project needs to import the CSS file import '@netless/appliance-plugin/dist/style.css';

About ?raw webpack configuration

module: {
    rules: [
        // ...
        {
            test: /\.m?js$/,
            resourceQuery: { not: [/raw/] },
            use: [ ... ]
        },
        {
            resourceQuery: /raw/,
            type: 'asset/source',
        }
    ]
},

API Introduction

Optimize Original Interfaces

The plugin re-implements some interfaces with the same name on room or windowmanager, but we have internally re-injected them back into the original object through injectMethodToObject. Therefore, external users do not need to make any changes. As follows:

// Internal hack
injectMethodToObject(windowmanager, 'undo');
injectMethodToObject(windowmanager, 'redo');
injectMethodToObject(windowmanager,'cleanCurrentScene');
injectMethodToObject(windowmanager,'insertImage');
injectMethodToObject(windowmanager,'completeImageUpload');
injectMethodToObject(windowmanager,'lockImage');
injectMethodToObject(room,'getImagesInformation');
injectMethodToObject(room,'callbacks');
injectMethodToObject(room,'screenshotToCanvasAsync');
injectMethodToObject(room,'getBoundingRectAsync');
injectMethodToObject(room,'scenePreviewAsync');
injectMethodToObject(windowmanager.mainView,'setMemberState');
// These we can see the call behavior through the front-end log, for example:
// [ApplianceMultiPlugin] setMemberState
// [ApplianceMultiPlugin] cleanCurrentScene

The following interfaces are involved:

1. Interfaces on room

• setMemberState
• undo
• redo
• callbacks
• insertImage
• lockImage
• completeImageUpload
• getImagesInformation
• cleanCurrentScene

2. WindowManager interfaces

• cleanCurrentScene
• canUndoSteps
• canRedoSteps

3. Interfaces on WindowManager's mainView

• setMemberState
• undo
• redo
• callbacks
• insertImage
• lockImage
• completeImageUpload
• getImagesInformation
• cleanCurrentScene

4. Custom interfaces

• getBoundingRectAsync - Replace interface room.getBoundingRect
• screenshotToCanvasAsync - Replace interface room.screenshotToCanvas
• scenePreviewAsync - Replace interface room.scenePreview
• fillSceneSnapshotAsync - Replace interface room.fillSceneSnapshot
• destroy - Destroy the instance of appliance-plugin
• addListener - Add appliance-plugin internal event listener
• removeListener - Remove appliance-plugin internal event listener
• disableDeviceInputs - Replace interface room.disableDeviceInputs
• disableEraseImage - Replace interface room.disableEraseImage This method only prohibits the eraser that erases the entire image from erasing images, partial eraser is invalid
• disableCameraTransform - Replace interface room.disableCameraTransform (Version >=1.1.17)
• insertText - Insert text at the specified position (Version >=1.1.18)
• updateText - Edit the content of the specified text (Version >=1.1.18)
• blurText - Remove text focus (Version >=1.1.19)
• hasElements - Check if there are elements in the specified scene (Version >=1.1.19)
• getElements - Get all elements in the specified scene (Version >=1.1.19)
• stopDraw - Stop Draw event (Version >=1.1.19)
• setViewLocalScenePathChange - Set the local scene path change for the whiteboard view (Version >=1.1.27)

5. Incompatible interfaces

• exportScene - After appliance-plugin is enabled, notes cannot be exported in room mode
• Server-side screenshot - After appliance-plugin is enabled, notes cannot be obtained by calling server-side screenshot, but need to use screenshotToCanvasAsync to obtain the screenshot

New Features

Laser Pen Tool (Version >=1.1.1)

import { EStrokeType, ApplianceNames } from '@netless/appliance-plugin';
room.setMemberState({currentApplianceName: ApplianceNames.laserPen, strokeType: EStrokeType.Normal});

Extended Tools (Version >=1.1.1)

On the original whiteboard tools type, some extended function attributes have been added, as follows:

export enum EStrokeType {
    /** Solid line */
    Normal = 'Normal',
    /** Line with pen edge */
    Stroke = 'Stroke',
    /** Dotted line */
    Dotted = 'Dotted',
    /** Long dotted line */
    LongDotted = 'LongDotted'
};
export type ExtendMemberState = {
    /** The tool selected by the current user */
    currentApplianceName: ApplianceNames;
    /** Whether to enable pen edge */
    strokeType?: EStrokeType;
    /** Whether to delete the entire line segment */
    isLine?: boolean;
    /** Stroke transparency */
    strokeOpacity?: number;
    /** Whether to enable laser pointer */
    useLaserPen?: boolean;
    /** Laser pointer holding time, second */
    duration?: number;
    /** Fill style */
    fillColor?: Color;
    /** Fill transparency */
    fillOpacity?: number;
    /** The specific type of graph to draw when using ``shape`` tool */
    shapeType?: ShapeType;
    /** Number of polygon vertices */
    vertices?:number;
    /** Inner vertex step length of polygon */
    innerVerticeStep?:number;
    /** Ratio of inner vertex radius to outer vertex radius of polygon */
    innerRatio?: number;
    /** Text transparency */
    textOpacity?: number;
    /** Text background color  */
    textBgColor?: Color;
    /** Text background color transparency */
    textBgOpacity?: number;
    /** Placement */
    placement?: SpeechBalloonPlacement;
};
import { ExtendMemberState, ApplianceNames } from '@netless/appliance-plugin';
/** Set tool state  */
room.setMemberState({ ... } as ExtendMemberState);
manager.mainView.setMemberState({ ... } as ExtendMemberState);
appliance.setMemberState({ ... } as ExtendMemberState);

1. Set stroke type:

// Solid line
setMemberState({strokeType: EStrokeType.Normal });
// Line with pen edge
setMemberState({strokeType: EStrokeType.Stroke });
// Dotted line
setMemberState({strokeType: EStrokeType.Dotted });
// Long dotted line
setMemberState({strokeType: EStrokeType.LongDotted });

2. Set stroke and shape border opacity (marker):

setMemberState({strokeOpacity: 0.5 });

3. Set text color, opacity, background color, and opacity

setMemberState({textOpacity: 0.5, textBgOpacity: 0.5, textBgColor:[0, 0, 0]});

4. Set shape fill color and opacity

setMemberState({fillOpacity: 0.5, fillColor:[0, 0, 0]});

5. Custom regular polygon

// Regular pentagon
setMemberState({currentApplianceName: ApplianceNames.shape, shapeType: ShapeType.Polygon, vertices: 5});

6. Custom star shape

// Fat hexagonal star
setMemberState({currentApplianceName: ApplianceNames.shape, shapeType: ShapeType.Star, vertices: 12, innerVerticeStep: 2, innerRatio: 0.8});

7. Custom speech balloon placement

// Speech balloon in the lower left corner
setMemberState({currentApplianceName: ApplianceNames.shape, shapeType: ShapeType.SpeechBalloon, placement: 'bottomLeft'});

Split screen display notes (little whiteboard feature), need to combine @netless/app-little-white-board (Version >=1.1.3)

Minimap function (Version >=1.1.6)

/** Create a minimap
 * @param viewId ID of the whiteboard under multi-whiteboard, the main whiteboard ID is `mainView`, other whiteboard IDs are the appID returned by addApp()
 * @param div Minimap DOM container
 */
createMiniMap(viewId: string, div: HTMLElement): Promise<void>;
/** Destroy minimap */
destroyMiniMap(viewId: string): Promise<boolean>;

Text editing API (Version >=1.1.18)

/** Insert text at the specified position
 * @param x The x coordinate of the left edge midpoint of the first character in the world coordinate system
 * @param y The y coordinate of the left edge midpoint of the first character in the world coordinate system
 * @param textContent Initial text content, empty if not provided
 * @returns The identifier of the text
 */
insertText(x: number, y: number, textContent?: string): string | undefined;

/** Edit the content of the specified text
 * @param identifier The identifier of the text, returned by insertText()
 * @param textContent The new content of the text
 */
updateText(identifier: string, textContent: string): void;

/** Remove text focus */
blurText(): void;

Element query API (Version >=1.1.19)

/** Check if there are elements in the specified scene
 * @param scenePath Scene path, defaults to the currently focused scene
 * @param filter Filter condition
 * @returns Whether elements exist
 */
hasElements(
  scenePath?: string,
  filter?: (toolsType: EToolsKey) => boolean,
): boolean;

/** Get all elements in the specified scene
 * @param scenePath Scene path, defaults to the currently focused scene
 * @param filter Filter condition
 * @returns All elements
 */
getElements(
  scenePath?: string,
  filter?: (toolsType: EToolsKey) => boolean,
): BaseCollectorReducerAction[];

Filter notes (Version >=1.1.6)

/** Filter notes
 * @param viewId ID of the whiteboard under multi-whiteboard, the main whiteboard ID is `mainView`, other whiteboard IDs are the appID returned by addApp()
 * @param filter Filter condition
 *  render: Whether notes can be rendered, [uid1, uid2, ...] or true. true means all will be rendered; [uid1, uid2, ...] is the specified set of user uids to render
 *  hide: Whether notes are hidden, [uid1, uid2, ...] or true. true means all will be hidden; [uid1, uid2, ...] is the specified set of user uids to hide
 *  clear: Whether notes can be erased, [uid1, uid2, ...] or true. true means all can be erased; [uid1, uid2, ...] is the specified set of user uids that can be erased
 * @param isSync Whether to synchronize to the whiteboard room, default is true, meaning the setting will be synchronized to all users
 */
filterRenderByUid(viewId: string, filter: { render?: _ArrayTrue, hide?: _ArrayTrue, clear?: _ArrayTrue}, isSync?:boolean): void;
/** Cancel filter notes
 * @param viewId ID of the whiteboard under multi-whiteboard, the main whiteboard ID is `mainView`, other whiteboard IDs are the appID returned by addApp()
 * @param isSync Whether to synchronize to the whiteboard room, default is true, meaning it will be synchronized to other users. Please keep it consistent with the filterRenderByUid setting
 */
cancelFilterRender(viewId: string, isSync?:boolean): void;

Set whiteboard local scene path change (Version >=1.1.27)

/** Set whiteboard local scene path change
 * @param viewId ID of the whiteboard under multi-whiteboard, the main whiteboard ID is `mainView`, other whiteboard IDs are the appID returned by addApp()
 * @param scenePath The scene path to set
 */
setViewLocalScenePathChange(viewId: string, scenePath: string): Promise<void>;

ExtrasOption custom tool configuration

1. Custom stroke styles

   • Short dotted line style

   export type DottedOpt = {
       /** Dotted line endpoint style, square: flat, round: round, default is round */
       lineCap: "square" | "round";
       /** Dotted line, single segment length, default is 1, meaning single segment length is 1 */
       segment: number;
       /** Dotted line, single segment gap, default is 2, meaning single segment gap is 2 * thickness */
       gap: number;
   };
   /** Short dotted line style */
   dottedStroke: {
       lineCap: "round",
       segment: 1,
       gap: 2,
   },

   

   • Long dotted line style

   export type LongDottedOpt = {
       /** Long dotted line endpoint style, square: flat, round: round, default is round */
       lineCap: "square" | "round";
       /** Long dotted line, single segment length, default is 1, meaning single segment length is 1 * thickness */
       segment: number;
       /** Long dotted line, single segment gap, default is 2, meaning single segment gap is 2 * thickness */
       gap: number;
   };
   /** Long dotted line style */
   longDottedStroke: {
       lineCap: "round",
       segment: 2,
       gap: 3,
   },

   

   • Normal stroke style

   export type NormalOpt = {
       /** Endpoint style, square: flat, round: round, default is round */
       lineCap: "square" | "round";
   };
   /** Normal stroke style */
   normalStroke: {
       lineCap: "round",
   }

   

2. Text custom styles

export type TextEditorOpt = {
    /** Whether to show float bar */
    showFloatBar?: boolean;
    /** Whether can switch by selector tool */
    canSelectorSwitch?: boolean;
    /** Whether right boundary auto wrap */
    rightBoundBreak?: boolean;
    /** Extended font list */
    extendFontFaces?: { fontFamily: string; src: string }[];
    /** Font loading timeout, unit: milliseconds */
    loadFontFacesTimeout?: number;
};
// For example: set unified font library
textEditor: {
  showFloatBar: false,
  canSelectorSwitch: false,
  rightBoundBreak: true,
  extendFontFaces: [
    {
      fontFamily: "Noto Sans SC",
      src: "https://fonts.gstatic.com/s/opensans/v44/memvYaGs126MiZpBA-UvWbX2vVnXBbObj2OVTS-mu0SC55I.woff2",
    },
  ],
  loadFontFacesTimeout: 20000,
},

Need to combine CSS style implementation

@font-face {
    font-family: "Noto Sans SC";
    src: url("https://fonts.gstatic.com/s/opensans/v44/memvYaGs126MiZpBA-UvWbX2vVnXBbObj2OVTS-mu0SC55I.woff2")
        format("woff2");
    font-display: swap;
}
html {
    font-family: "Noto Sans SC";
}

Handwriting graphics automatic association function:autoDraw, need to combine @netless/appliance-extend-auto-draw-plugin

export interface AutoDrawOptions {
    /** API key for accessing all OpenRouter models */
    apiKey?: string;
    /** Custom model to use */
    customModel?: string;
    /** Container for rendering icons */
    container: HTMLDivElement;
    /** Delay time for rendering icons, default is 2000ms */
    delay?: number;
    /**
     * Upload file to OSS server and return URL address, if returns undefined then this feature will not be used
     * @param file File object
     * @returns Image URL string
     */
    uploadFile?: (file: File) => Promise<string | undefined>;
}
import { ApplianceMultiPlugin } from '@netless/appliance-plugin';
import { AutoDrawPlugin } from '@netless/appliance-extend-auto-draw-plugin';
const plugin = await ApplianceMultiPlugin.getInstance(...);
const autoDrawPlugin = new AutoDrawPlugin({
    container: topBarDiv,
    delay: 2000
});
plugin.usePlugin(autoDrawPlugin);

Configuration Parameters

getInstance(wm: WindowManager | Room | Player, adaptor: ApplianceAdaptor)

• wm: WindowManager | Room | Player. In multi-window mode, pass WindowManager, in single-window mode, pass Room or Player (whiteboard playback mode).
• adaptor: Configuration adapter.
  • options: AppliancePluginOptions - Must be configured, where cdn is required.

    export type AppliancePluginOptions = {
        /** CDN configuration item */
        cdn: CdnOpt;
        /** Additional configuration items */
        extras?: ExtrasOptions;
    };
    export type CdnOpt = {
        /** Full worker URL address, thread for drawing complete data */
        fullWorkerUrl?: string;
        /** Sub worker URL address, thread for drawing one frame of data */
        subWorkerUrl?: string;
    };
    export type ExtrasOptions =  {
        /** Whether to use simple mode, default value is ``false``
         * true: Simple mode:
            1. Drawing will use single worker, bezier smoothing cannot be used during drawing.
            2. Remove some new features: minimap, pointerPen (laser pen), autoDraw plugin.
         */
        useSimple: boolean;
        /** Whether to use worker, default value is ``auto``
        * auto: Automatically select (use webWorker if browser supports offscreenCanvas, otherwise use main thread)
        * mainThread: Use main thread, canvas drawing data.
        */
        useWorker?: UseWorkerType;
        /** Synchronization data configuration item */
        syncOpt?: SyncOpt;
        /** Canvas configuration item */
        canvasOpt?: CanvasOpt;
        /** Pointer configuration item */
        cursor?: CursorOpt;
        /** Canvas cache configuration item */
        bufferSize?: BufferSizeOpt;
        /** Bezier optimization configuration item */
        bezier?: BezierOpt;
        /** Partial eraser configuration item */
        pencilEraser?: PencilEraserOpt;
        /** Stroke width range configuration item */
        strokeWidth?: StrokeWidthOpt,
        /** Text editor configuration item */
        textEditor?: TextEditorOpt;
        /** Undo redo configuration item */
        undoRedo?: {
            /** Whether to enable global undo redo, default value is false (Version >=1.1.27) */
            enableGlobal?: boolean;
            /** Maximum stack length for undo redo, default value is 20 */
            maxStackLength?: number;
        };
    }

  • cursorAdapter?: CursorAdapter - Optional, in single whiteboard mode, configure custom mouse style.
  • logger?: Logger - Optional, configure log printer object. If not provided, defaults to local console output. If logs need to be uploaded to a specified server, manual configuration is required.

    If you need to upload to the whiteboard log server, you can configure room.logger to this item.

Front-end Debugging

During the integration process, if you want to understand and track the internal status of the plugin, you can view internal data through the following console commands.

const appliancePlugin = await ApplianceSinglePlugin.getInstance(...)
appliancePlugin.currentManager  // Can view package version number, internal status, etc.
appliancePlugin.currentManager.consoleWorkerInfo()  // Can view drawing information on worker