KHR_interactivity Typescript Engine

This README provides an overview and instructions for connecting the BasicBehaveEngine to a frontend (e.g. DCC tool, renderer).

Table of Contents

• KHR_interactivity Typescript Engine
  • Table of Contents
  • Introduction
  • NPM package
  • Integration
    • Registering Pointer mappings
    • Loading a graph
    • Resetting a graph
    • Events
    • Animations
    • KHR_selectability and KHR_hoverability
    • KHR_physics_rigid_bodies
    • Debugging
    • Important Notes About Tool Execution
  • Development
    • Prerequisites
    • Installation
    • Building
  • Adding Nodes to BasicBehaveEngine
  • TODO
    • Non normative implementations

Introduction

This engine implements the glTF KHR_interactivity extension. The KHR_interactivity extension allows you to add interactive features and behaviors to your 3D models, enhancing the user experience in 3D applications.

NPM package

1. Add this project as a npm package to your project via

npm i @khronosgroup/gltf-interactivity-engine

Integration

Registering Pointer mappings

The KHR_interactivity spec allows for setting and getting properties of an object model, this logic of mapping an object path to a setter and getter is handled by the respective decorator (so a logging engine can set a path in a different way than a Babylon one to help with implementation specific mapping). To add your own pointer, simply add the logic for your getter and setter to the registerKnownPointers function in the respective Decorators.

Loading a graph

A graph can be loaded into the engine by calling:

loadBehaveGraph(behaveGraph: any, run = true)

behaveGraph is the JSON representation of the graph. run can be set to false, if the graph should not be executed automatically after loading.

The event queue can be pause and played by calling:

pauseEventQueue()
playEventQueue()

Be aware that you need to pause/play e.g. animations and hover in your renderer.

Resetting a graph

Resetting the engine needs to be done manually. An empty graph can be loaded and there are several clear functions for different purposes.

Additionally, you need to manually reset the animation state and the whole glTF state which was modified by pointer/set nodes.

Events

To sent custom events to the engine call:

dispatchCustomEvent(name: string, vals: any)

The name needs to be prefixed with KHR_INTERACTIVITY:. Vals is a JSON object containing all values needed for the event.

For implementation details, check out the KHR_interactivity specification.

KHR_selectability and KHR_hoverability

To implement KHR_selectability and KHR_hoverability the following function need to be added to the BasicBehaveEngine:

this.behaveEngine.getParentNodeIndex = (nodeIndex: number) => number | undefined => {
   // Return index of parent node. If no parent exists return undefined.
}

You can pass a selection event via:

select(selectedNodeIndex: number, controllerIndex: number, selectionPoint: [number, number, number] | undefined, selectionRayOrigin: [number, number, number] | undefined);

selectionPoint and selectionRayOrigin are optional.

You can pass a hover event via:

hoverOn(hoveredNodeIndex: number | undefined, controllerIndex: number);

hoveredNodeIndex can be undefined if nothing is hover over.

KHR_physics_rigid_bodies

To implement KHR_physics_rigid_bodies register the nodes using:

this.registerRigidBodyNodes();

You can pass a trigger enter and exit events via:

rigidBodyTriggerEntered(nodeIndex: number, colliderNodeIndex: number, motionNodeIndex: number | undefined);

rigidBodyTriggerExited(nodeIndex: number, colliderNodeIndex: number, motionNodeIndex: number | undefined);

nodeIndex describes the node index of the trigger, colliderNodeIndex describes the node index of the collider and motionNodeIndex describes the index of the motion which might be attached to the collider or one of it parents. This should be called with undefined, if no motion is found (in case of a static collider).

The following functions need to be implemented and need to overwrite their counterparts in the behave engine:

applyImpulseToRigidBody(nodeIndex: number, linearImpulse: [number, number, number], angularImpulse: [number, number, number])
applyPointImpulseToRigidBody(nodeIndex: number, impulse: [number, number, number], position: [number, number, number])
rayCastRigidBodies(rayStart: [number, number, number], rayEnd: [number, number, number], collisionFilterIndex: number): {hitNodeIndex: number, hitFraction: number | undefined, hitNormal: [number, number, number] | undefined}

this.behaveEngine.applyImpulseToRigidBody = this.applyImpulseToRigidBody;
this.behaveEngine.applyPointImpulseToRigidBody = this.applyPointImpulseToRigidBody;
this.behaveEngine.rayCastRigidBodies = this.rayCastRigidBodies;

If a ray cast result in a miss, {hitNodeIndex: -1} should be returned. The full interactivity node definitions and explanations can be read in the spec.

Debugging

The following functions of the BasicBehaveEngine can be used to perform debugging/logging etc.

this.behaveEngine.processNodeStarted = (behaveEngineNode: BehaveEngineNode) => void {
   
};
this.behaveEngine.processAddingNodeToQueue = (flow: IInteractivityFlow) => void {

};
this.behaveEngine.processExecutingNextNode = (flow: IInteractivityFlow) => void {

};

Important Notes About Tool Execution

1. The tool passes around JSON for the execution graph. Be aware that the engine does not necessarily create deep copies of arrays or objects. Modifying these from outside the engine might create undefined/unexpected behavior. Make sure to not modify passed values or create deep copies.
2. The engine expects matrices in 2D structure e.g. [[1.0, 0.0],[0.0, 1.0]]. This needs to be considered for e.g. registering worldMatrix or matrix

Development

Prerequisites

1. Install Node.js. Please use an LTS version of Node.js.

Installation

1. Install the required dependencies:

   npm install

Building

1. Run

   npm run build

All files are build into the build folder. Source maps are generated but not published via npm.

Adding Nodes to BasicBehaveEngine

The provided Basic BehaveEngine uses a decorator approach, so you behave nodes will be defined only once and the Logging and Babylon decorators will decorate certain touch points in the execution logic with their specific functionality. To create a new node...

1. add a new class in the nodes directory which extends BehaveEngineNode.
2. add your REQUIRED_VALUES and/or REQUIRED_CONFIGURATIONS arrays which will be used to validate the node.
3. In your constructor, run super() then set the instance name and finally run the needed validations (flows, values and configurations) for your node
4. override processNode, if your node has custom named out flows (i.e. in branch we have true and false instead of out) use this.processFlow, if not you can either do this.processFlow on the default out flow or run super.processNode
5. Finally, either in BasicBehaveEngine or in the decorator of your choice, call registerBehaveEngineNode with the namespace of your node and the node class name
6. If you want to do some complex behaviors, like async listeners look at CustomEvent/Receive it has a good example of how async nodes post their subsequent flows to the execution queue instead of invoking synchronously and shows how to set up an event listener so it is not triggered by another node but an event.