Pro-BRAVIA AITRIOS Camera Gaze App

This is a HTML + Typescript application to integrate Gaze Detection Applications, deployed on AITRIOS Cameras, with Professional BRAVIA displays.

Here is a scheme of the working application:

Setup

General

The project works with a T3 camera, a PC, and a BRAVIA TV. Be sure that all devices are connected to the same network. For instance you can use a PoE Switch, receiving internet from LAN, connected to both camera (PoE port), BRAVIA, and PC.

Controller/PC

The software on the machine running the webapp, can be set by using the following commands:

$ sudo apt update
$ sudo apt install -y nodejs npm
$ npm install -g pnpm
$ pnpm install

In order to make the PC able to capture events on the BRAVIA tv, substitute the value of BRAVIA_IP in the AITRIOS Vue file with the current IP address of the TV.

BRAVIA display

Below are the installation steps on the BRAVIA TV.

1. Enable Pro Settings: Check the Start Pro Mode guide on the Pro-BRAVIA website.
2. Enable USB auto-run: Check the USB auto-run guide on the Pro-BRAVIA website.
3. AITRIOS bridge installation: Copy the AITRIOS bridge service file into a USB drive, and plug it into a BRAVIA USB port. Follow the installation process on the display. In order to check if the bridge works properly, follow the Controller/PC and Cameras setup step and then:

   • Open http://<your_bravia_ip>:20347/ in the browser

   • Open the browser console (e.g. Ctrl + Shift + I)

   • Type the following code into the console and press Enter:

     (new EventSource('/meta')).onmessage = (event) => console.log(event)

     After this, if everything works fine, you should see something similar:

4. Enable IP Remote control:
   • Set Pre-shared key on your TV from "Settings > Network & Internet > Local network > IP control":
     • Authentication = Normal and Pre-Shared Key
     • Pre-Shared Key = 1234 (any string is ok)
   • Enable "Remote start" from "Settings > Network & Internet > Remote start":
     • On (Powered on by apps)
   • Install the Sony app IP Remote on your smartphone.
   • Add your BRAVIA on IP Remote:
     • IP Address = Your BRAVIA's IP address
     • PSK = string set before
     • App URL = http://<your_pc_ip>:3000/

   [!IMPORTANT] The smartphone should be connected to the same network of the BRAVIA in order to make IP Remote to work properly.

Cameras

You can use a T3P or T3W camera to work with this application. You have to deploy one of the following combinations on it, in order to run properly:

📋 Remember to change the Model ID field in the command parameter files. 📋 Remember to setup flatbuffer .ts files properly in function of the model you are using. The generated files for Pentas and AWL/WTA are under the pentas and wta folders

These applications run in summary mode: this means that, in order to produce a result, there should be changes in the scene (e.g. the user exits from the camera view for few seconds, comes back, etc.)

[!IMPORTANT] There are some debug configurations that disable the summary feature, like pentas_config_debug.json. Bind these command parameter files to the cameras on AITRIOS if you want to capture every output frame, instead of only the meaningful ones.

Usage

The web application can be executed on the pc by running:

$ pnpm dev

A list of IP address where the app is running can appear; select one and check if the application runs properly on your browser.

[!TIP] After any change to the application, re-build it with the command $ pnpm build before to run again.

Web Application on BRAVIA

There are mainly two ways to control the BRAVIA runtime. Both work with any kind of web application, which exploit the bridge installed on the BRAVIA device to receive data:

• IP Remote: This is a smartphone app to easily control the screen, and show the web app directly on the BRAVIA tv. Once started the target web app on the pc, install the app on iOS/Android, configuring the device on it with the right IP and PSK, as explained in the BRAVIA Setup section. Then, on the phone app, press the button with the image below to open the Web App URL on your BRAVIA:

  

• REST API: Through API calls a deeper control can be established. Follow the BRAVIA REST API guide to have a better understanding of them. Here is an example of a generic API call (made with Postman or other) to get the power status of the TV, with Pre-Shared Key: “1234”:

  POST /sony/system HTTP/1.1
  Host: 192.168.0.1
  Accept: */*
  Cache-Control: no-cache
  Connection: close
  Content-Type: application/json; charset=UTF-8
  Pragma: no-cache
  X-Auth-PSK: 1234
  Content-Length: 70
  
  {"method": "getPowerStatus", "params": [], "id": 50, "version": "1.0"}

  In order to set the running app on the BRAVIA, the API setActiveApp will be used, by making a POST call to the url http://<your bravia ip>/sony/appControl, generating a JSON like this:

  {
      "method": "setActiveApp",
      "id": 601,
      "params": [{
          "uri": "localapp://webappruntime?url=<url_of_running_web_app>"
      }],
      "version": "1.0"
  }

  Example with CURL:

  curl -X POST -H "X-Auth-PSK: 1234" -d '{"id":1,"method":"setActiveApp","version":"1.0","params":[{"uri":"localapp://webappruntime?url=http%3A%2F%2F192.168.101.170%3A3000"}]}' 'http://<your bravia ip>/sony/appControl'