PA.EmotionsTracker

The EmotionalAwareness-Tool consists of two parts: a tracker, where webcam images are processed and the estimates regarding valence, fatigue and engagement are calculated, and a client that visualizes the estimates.

This is the tracker project, the client can be found here

This project is published on npm

Documentation

Used tensorflow models

• faceAPI for facial expression and head orientation detection
• Face landmarks detection for explicit facial landmarks detection
• PoseNet for pose detection

Endpoints

Communication with the tracker is done over IPC. The following endpoints exist:

Incoming

• start-tracker to start the tracker
• pause-tracker to pause the tracker
• set-low-performance-mode with payload: boolean to activate/deactivate low performance mode

Outgoing

• face-api-expression with payload: FaceApiExpression
• face-api-head-orientation with payload: FaceApiHeadOrientation
• face-api-aggregated-expression with payload: FaceApiAggregatedExpression
• face-api-smiled-once with payload: FaceApiHasSmiled
• face-api-raw-expressionswith payload: string

• facial-landmarks-data with payload: FacialLandmarksData
• facial-landmarks-blink-count with payload: FacialLandmarksBlinkCount

• pose with payload: PoseData

• z-deviation with payload: ZDeviation
• head-orientation-deviation with payload: HeadOrientationDeviation
• metric-log with payload: MetricLog
• estimated-dimensions with payload: EstimatedDimensions
• update-tracker-status with payload: Status to get the current state of the tracker

Main Components

The following diagram shows the main dataflow of the program. For the sake of clarity, the Renderer is not depicted. The Renderer handles incoming requests from the client, like starting/pausing or switching to low performance mode.

Open issues concerning coupling between tracker & client

• Tracker: pa-shared interface is no longer used

• Client: in package.json, we need to know exact folder structure of tracker

  "html": "./node_modules/emotions-tracker/src/renderer/index.html",
  "js": "./node_modules/emotions-tracker/src/renderer/renderer.ts",
  "name": "tracker_window"

• Client: in webpack.plugins.js, we have to copy the whole assets folder from the tracker to the client, because the tracker_window needs access to models / images

  {
    from: path.resolve(__dirname, "node_modules/emotions-tracker/src/renderer/assets"),
    to: path.resolve(__dirname, ".webpack/renderer/assets"),
  }