@trillboards/edge-sensing

On-device audience sensing with ONNX models. Face detection, gaze tracking, emotion analysis, age/gender classification, audio event detection, and attention scoring — all running locally on the device.

Install

npm install @trillboards/edge-sensing

What This Does

Runs computer vision and audio ML models directly on DOOH/CTV hardware. No cloud calls. No PII leaves the device. Outputs aggregated, anonymized audience metrics.

• Face Detection — BlazeFace ONNX model detects faces, bounding boxes, head pose (yaw/pitch/roll), eye openness, smile probability
• Age/Gender — classifies viewers into 8 age ranges (0-12 through 65+) with gender confidence scores
• Emotion Detection — 8 emotions (happy, surprised, neutral, sad, angry, fearful, disgusted, contempt) with sentiment scoring
• Gaze Classification — categorizes viewer gaze as Direct, Partial, or Away
• Attention Scoring — combines head pose + eye openness into attention score (0-1)
• Audio Classification — YAMNet model classifies ambient sound events (speech, music, crowd noise, silence)
• Foot Traffic Estimation — estimates venue occupancy from face count signals
• VAS (Viewability Attention Score) — weighted composite of attention, emotion, body, and focus signals

Quick Start

import { AudienceSensingService, FaceDetector, AudioClassifier } from '@trillboards/edge-sensing';

// Full orchestrated service
const sensing = new AudienceSensingService({
  enabled: true,
  fps: 10,
  resolution: { width: 640, height: 480 }
});

sensing.on('metrics', (payload) => {
  console.log(`Faces: ${payload.faceCount}, Attention: ${payload.avgAttention}`);
});

await sensing.start();

Individual Components

// Face detection only
const detector = new FaceDetector({
  modelPath: './models/blazeface_128x128.onnx',
  executionProvider: 'cpu' // or 'openvino', 'directml', 'cuda'
});

const faces = await detector.detect(imageBuffer);
// → [{ bbox, yaw, pitch, roll, eyeOpen, smileProb, age: '25-34', gender: 'male', ... }]

// Audio classification only
const audio = new AudioClassifier({
  modelPath: './models/yamnet_classification.onnx',
  minConfidence: 0.3
});

const events = await audio.classify(audioBuffer);
// → [{ label: 'Speech', confidence: 0.87 }, { label: 'Music', confidence: 0.42 }]

Metrics & Scoring Functions

import {
  calculateAttentionScore,
  classifyGaze,
  calculateEngagement,
  computeVAS,
  calculateDwellTimeMetrics,
  getDaypart
} from '@trillboards/edge-sensing';

const attention = calculateAttentionScore(yaw, pitch, eyeOpenness); // → 0.0 - 1.0
const gaze = classifyGaze(yaw, pitch); // → 'Direct' | 'Partial' | 'Away'
const engagement = calculateEngagement(gazeSignals, poseSignals, emotionSignals);
const vas = computeVAS(attentionData, emotionData, bodyData, focusData);
const daypart = getDaypart(new Date()); // → 'morning' | 'afternoon' | 'evening' | 'night'

Execution Providers

| Provider | Platform | Speed | |----------|----------|-------| | cpu | Any | Baseline | | openvino | Intel iGPU | 2-3x faster | | directml | Windows GPU | 2-4x faster | | cuda | NVIDIA GPU | 5-10x faster |

ONNX Models

Download models before first run:

npx trillboards-edge download-models --dir ./models

Downloads:

• blazeface_128x128.onnx — face detection (~1.2 MB)
• yamnet_classification.onnx — audio classification (~3.5 MB)

License

MIT