sense-on

On-device face tracking and attention monitoring for the web. MediaPipe-powered, 100% client-side, privacy-first.

English | 한국어

Features

• 100% client-side — MediaPipe Face Landmarker runs via WASM in a Web Worker. No server, no data leaves the device.
• Attention detection — Classifies user state as attentive, looking_away, drowsy, or absent.
• Head pose estimation — Extracts yaw, pitch, roll from the facial transformation matrix.
• Eye Aspect Ratio (EAR) — Computes eye openness for drowsiness detection.
• One Euro Filter — Jitter-free smoothing for head pose and landmark values.
• Off-main-thread — Face detection runs entirely in a Web Worker for smooth UI.
• Framework-agnostic core — Use with React, Svelte, vanilla JS via sense-on/core.
• Face mesh constants — Tessellation, eye, iris, lip, and face oval connection arrays for visualization.

Architecture

Camera (getUserMedia)
  → CameraStream (requestVideoFrameCallback loop)
    → ImageBitmap → Web Worker
      → MediaPipe FaceLandmarker (WASM, GPU fallback to CPU)
        → TrackingResult { landmarks, headPose, faceCenter }
          → computeAttention() → AttentionState
            → UI (wireframe + metrics)

Quick Start

Install

npm install sense-on @mediapipe/tasks-vision

Download Model

The MediaPipe Face Landmarker model (~5 MB) is required at runtime. Place it in your public directory:

curl -o public/models/face_landmarker.task \
  https://storage.googleapis.com/mediapipe-models/face_landmarker/face_landmarker/float16/latest/face_landmarker.task

Framework-Agnostic Usage

import {
  createCameraStream,
  createTrackerClient,
  computeAttention,
} from "sense-on/core";

const camera = createCameraStream();
const tracker = createTrackerClient({
  modelPath: "/models/face_landmarker.task",
});

await tracker.init();
await camera.start();

tracker.onResult((data) => {
  const attention = computeAttention(data);
  console.log(attention.state); // 'attentive' | 'looking_away' | 'drowsy' | 'absent'
  console.log(attention.headYaw); // degrees
  console.log(attention.eyeOpenness); // EAR value
});

camera.onFrame((frame) => {
  tracker.detect(frame, performance.now());
});

Vue 3 Usage

<script setup>
import { onMounted, onUnmounted } from "vue";
import {
  createCameraStream,
  createTrackerClient,
  computeAttention,
} from "sense-on";

// ... same as above, with reactive state
</script>

API Reference

Core Functions

| Function | Description | | ------------------------------- | -------------------------------------------------- | | createCameraStream(options?) | Creates a camera stream with frame loop | | createTrackerClient(options?) | Creates a Web Worker-based face tracker | | computeAttention(result) | Computes attention state from tracking result | | computeEAR(landmarks) | Computes Eye Aspect Ratio from 478 landmarks | | matrixToEuler(matrix) | Converts 4x4 transformation matrix to Euler angles | | createOneEuroFilter(options?) | Creates a One Euro Filter for smoothing | | createPointFilter(options?) | Creates a 2D point filter |

Types

| Type | Description | | --------------------- | ---------------------------------------------------------------- | | AttentionState | 'attentive' \| 'looking_away' \| 'drowsy' \| 'absent' | | AttentionResult | { state, headYaw, headPitch, eyeOpenness, facePresent } | | TrackingResult | { faceCenter?, headPose?, landmarks?, inferenceMs, timestamp } | | EulerAngles | { yaw, pitch, roll } in degrees | | Point2D / Point3D | Coordinate types |

Attention States

| State | Condition | Color | | -------------- | ---------------------------------------------- | ------ | | attentive | Face present, forward-facing, eyes open | Green | | looking_away | Face present, |yaw| > 25° or |pitch| > 20° | Yellow | | drowsy | Face present, EAR < 0.2 for 2+ seconds | Orange | | absent | No face detected | Red |

Face Mesh Constants

import {
  TESSELATION,
  LEFT_EYE,
  RIGHT_EYE,
  LEFT_IRIS,
  RIGHT_IRIS,
  LIPS,
  FACE_OVAL,
} from "sense-on/core";

Each constant is a [number, number][] array of landmark index pairs for drawing connections.

Browser Requirements

• Modern browser with WebAssembly support
• getUserMedia API (camera access)
• createImageBitmap API
• Web Worker support

Tested on Chrome 90+, Firefox 90+, Edge 90+, Safari 15.4+.

License

Apache 2.0

한국어

브라우저 기반 얼굴 추적 및 주의력 모니터링 라이브러리. MediaPipe 기반, 100% 클라이언트 사이드, 프라이버시 우선.

주요 기능

• 100% 클라이언트 사이드 — 서버 불필요, 데이터가 기기를 떠나지 않음
• 주의력 감지 — attentive(집중), looking_away(시선 이탈), drowsy(졸림), absent(자리 비움)
• 머리 자세 추정 — yaw/pitch/roll 각도 추출
• 눈 개폐 비율(EAR) — 졸음 감지용
• Web Worker 추론 — 메인 스레드 블로킹 없음

빠른 시작

npm install sense-on @mediapipe/tasks-vision

import {
  createCameraStream,
  createTrackerClient,
  computeAttention,
} from "sense-on/core";

const camera = createCameraStream();
const tracker = createTrackerClient({
  modelPath: "/models/face_landmarker.task",
});

await tracker.init();
await camera.start();

tracker.onResult((data) => {
  const { state, headYaw, eyeOpenness } = computeAttention(data);
  // state: 'attentive' | 'looking_away' | 'drowsy' | 'absent'
});

camera.onFrame((frame) => tracker.detect(frame, performance.now()));

주요 API

| 함수 | 설명 | | ------------------------------- | ------------------------------- | | createCameraStream(options?) | 카메라 스트림 + 프레임 루프 | | createTrackerClient(options?) | Web Worker 기반 얼굴 추적기 | | computeAttention(result) | TrackingResult → AttentionState | | computeEAR(landmarks) | 478 랜드마크 → 눈 개폐 비율 |

라이선스

Apache 2.0