@realeye-io/webcam-eyetracker-light-open

v1.1.0

Published

8 days ago

RealEye Webcam EyeTracker Light Open — lightweight browser-based webcam eye-tracker using MediaPipe and ridge regression for gaze prediction

0High
0Medium
0Low

damianrealeye

eye-tracking webcam gaze ridge-regression mediapipe gaze-detection browser eyetracker computer-vision

RealEye Webcam EyeTracker Light Open

🚀 This technology powers RealEye.io — the commercial webcam eyetracking platform with heatmaps, gaze plots, and areas of interest. Learn more about real-world online webcam eyetracking →

👉 Try the live demo: webcam-eyetracker-light-open demo

RealEye Webcam EyeTracker Light Open is a webcam eye-tracker running in your web browser that provides on-screen gaze coordinates. It works with any regular camera — laptop built-in webcam, USB webcam, or mobile device camera. Everything runs locally on your device. No external servers, no dedicated eye-tracking hardware required. Built in TypeScript with custom ridge regression.

🎯 Commercial Platform

This dual-licensed library is the core technology behind RealEye.io, a commercial research platform that adds:

Heatmaps & gaze plots from real users
Areas of Interest (AOIs) and dwell-time metrics
Cloud storage & analytics
Integration with testing tools

See RealEye.io in action →

✨ Features

Zero setup — no eye-tracking hardware required
Pure browser execution — runs entirely client-side
CDN-loaded models — MediaPipe models and WASM runtime load from CDN by default (no local model files needed)
17-point calibration — proven optimal pattern
Head-pose compensation — learned during calibration
TypeScript-first — full type definitions included
100% self-contained — no server-side dependencies
Interactive demo app — calibration wizard, gaze trail animation, technical info panel, click accuracy overlay, webcam selection, background controls, and runtime diagnostics
≈ 120 CSS px accuracy — average gaze error after 17-point calibration

📷 Camera Support

Works with any camera your browser can access — laptop built-in webcam, USB webcam, or mobile device camera.

🖥️ Desktop recommended — The demo application is designed for desktop/laptop browsers with a fixed webcam.

Eye tracking requires facing the camera at a consistent distance and angle
Calibration involves clicking small targets positioned across the entire screen
Holding a phone at arm's length while trying to look at calibration targets is physically impractical
The demo will show a warning banner on mobile devices; the app still runs but accuracy is significantly reduced

📸 Demo App

The demo app provides a full interactive experience: camera preview, face detection overlay, 17-point calibration wizard, and real-time gaze tracking.

🌐 Live Demo — Try it in your browser: realeye-io.github.io/webcam-eyetracker-light-open

Before calibration — the initial landing screen with a Start Camera button (no webcam permission prompt until clicked):

Demo intro

Ready for calibration — face detected in a preview area, status set to Ready for Calibration, and Start Calibration enabled:

Demo ready

During calibration — the 17-point calibration wizard guides you through targets that appear across the screen. Progress is shown as each point is captured:

Demo calibration

During tracking — calibrated gaze prediction with a gaze cursor, animated trail, and session statistics:

Demo tracking

Settings & Diagnostics — expandable side panels for inference settings (Face Detection Mode, Inference Device, Max Resolution) and detailed diagnostics (video/viewport resolution, device pixel ratio, eye crop):

Demo tracking with panels

🚀 Quick Start

MediaPipe models and WASM runtime are loaded from CDN by default — no local model files needed. Just install and initialize:

npm install @realeye-io/webcam-eyetracker-light-open

import {
  WebcamETLight,
  getRecommendedPattern,
  getCalibrationPointsInPixels,
  type CalibrationSample,
} from '@realeye-io/webcam-eyetracker-light-open';

// Initialize (loads models from CDN automatically)
const tracker = new WebcamETLight({ delegate: 'GPU' });
await tracker.initialize();

// Build calibration samples using the recommended 17-point grid
const pattern = getRecommendedPattern(); // GRID_17
const points = getCalibrationPointsInPixels(
  pattern,
  window.innerWidth,
  window.innerHeight
);

const samples: CalibrationSample[] = [];
for (const point of points) {
  // Capture a webcam frame while the user looks at the target
  const imageData = await captureCalibrationFrame(point);
  samples.push({
    image: imageData,
    gazeX: point.x,
    gazeY: point.y,
  });
}

// Fit the ridge regression model
tracker.calibrate(samples);

// Predict gaze (synchronous)
const gazePoint = tracker.predict(liveFrame);
if (gazePoint) {
  console.log(`Gaze at: (${gazePoint.x}, ${gazePoint.y})`);
}

Tip: See the User Manual for a step-by-step walkthrough of the included demo app.

📖 Table of Contents

User Manual — step-by-step guide for the demo app
API Reference
Calibration Pattern
Technical Architecture
Project Structure
Testing
Contributing
Changelog
Security
License
Third-Party Notices
Links

🔌 API Reference

WebcamETLight

import { WebcamETLight, WebcamETLightConfig } from '@realeye-io/webcam-eyetracker-light-open';

const tracker = new WebcamETLight({
  delegate: 'GPU',              // 'GPU' | 'CPU'
  faceDetectorMode: 'landmarker', // 'landmarker' | 'blazeFace'
  // modelPath and wasmPath default to CDN URLs
  // Override with local paths if needed:
  // modelPath: '/local/models/face_landmarker.task',
  // wasmPath: '/local/wasm',
  runningMode: 'VIDEO',         // 'VIDEO' | 'IMAGE'
  ridgeLambda: 1e-5,
  minDetectionConfidence: 0.5,
  useLandmarks: true,
});

await tracker.initialize();
const gaze = tracker.predict(image); // GazePoint | null

Instance Methods

| Method | Description | |--------|-------------| | initialize() | Load MediaPipe models and face detector | | predict(imageData: ImageData): GazePoint \| null | Return predicted gaze in CSS pixels | | calibrate(samples: CalibrationSample[]) | Fit ridge regression on samples | | getState(): TrackerState | Return current tracker state | | isReady(): boolean | Check if initialized and ready | | isCalibrated(): boolean | Check if calibration has been applied | | detectFace(image): FaceDetectionResult \| null | Detect face with landmarks in an image | | dispose() | Release MediaPipe resources |

Calibration Utilities

Standalone functions exported from the library (not methods on WebcamETLight):

| Function | Description | |----------|-------------| | getRecommendedPattern() | Return the recommended CalibrationPattern (GRID_17) | | getCalibrationPoints(pattern): CalibrationPoint[] | Get pattern points normalized 0–1 | | getCalibrationPointsInPixels(pattern, width, height) | Get pattern in screen-pixel coordinates | | toScreenCoordinates(points, width, height) | Convert normalized points to screen pixels | | getPatternInfo(pattern) | Get metadata for a calibration pattern |

Calibration Data Types

interface CalibrationSample {
  image: ImageData;   // Webcam frame
  gazeX: number;      // Screen X in CSS pixels
  gazeY: number;      // Screen Y in CSS pixels
}

interface GazePoint {
  x: number;          // CSS pixels
  y: number;          // CSS pixels
}

Error Handling

If face detection fails, predict() returns null. The library emits clear error states for missing calibration, initialization failures, and low detection confidence.

🎯 Calibration Pattern

The project uses a single 17-point grid calibration pattern, selected for strong full-screen coverage and stable accuracy across corners, edges, center, and intermediate regions.

17-Point Grid Layout

┌─────────────────────────────────────────────┐
│  (0.05,0.05)    (0.50,0.05)   (0.95,0.05)  │
│                                             │
│  (0.05,0.25)    (0.25,0.25)   (0.50,0.25)   (0.75,0.25)   (0.95,0.25)│
│                                             │
│  (0.05,0.50)    (0.50,0.50)   (0.95,0.50)  │
│                                             │
│  (0.05,0.75)    (0.25,0.75)   (0.50,0.75)   (0.75,0.75)   (0.95,0.75)│
│                                             │
│  (0.05,0.95)    (0.50,0.95)   (0.95,0.95)  │
└─────────────────────────────────────────────┘

import { 
  CalibrationPattern, 
  getRecommendedPattern,
  getCalibrationPoints,
  getCalibrationPointsInPixels,
} from '@realeye-io/webcam-eyetracker-light-open';

const pattern = getRecommendedPattern(); // GRID_17
const normalized = getCalibrationPoints(pattern);
const screen = getCalibrationPointsInPixels(pattern, w, h);
// → [{ x: 85, y: 85 }, { x: 800, y: 85 }, ...]

🏗 Technical Architecture

Feature Extraction

For each webcam frame:

Face detection — MediaPipe Face Landmarker (default), BlazeFace fallback
Landmark coordinates — 15 key points × 2 (x, y) = 30 features
Blendshapes — 22 semantic facial parameters (eye blink, brow raise, look direction, ...)
Eye crops — left + right regions resized to 40×20 pixels each (800 px total)
Z-score + bias — normalize and append 1.0

Feature counts:

| Mode | Features | Breakdown | |------|----------|-----------| | Landmarker | 1,653 | 30 + 22 + 2×(40×20) + 1 | | BlazeFace | 1,622 | 21 + 2×(40×20) + 1 | | Landmarks off | 257 | 16×16 face crop + bias |

Demo Application

The included demo app provides a fully-featured interactive playground:

Privacy-first startup — no webcam permission prompt on load; preview starts only after clicking Start Camera
17-Point Calibration Wizard — interactive point-by-point calibration overlay
Real-time Gaze Tracking — live gaze cursor with animated trail effect
Technical Info Panel — collapsible sidebar showing webcam resolution/FPS, tracker configuration (inference device, feature count, calibration points), and runtime metrics (processing time, sampling rate, prediction count, tracking duration)
Click Accuracy Overlay — click anywhere during tracking to see gaze-vs-click distance lines with average error
Gaze Trail Effect — animated trailing dots that visualize recent gaze movement
Background Color Controls — test tracking against different backgrounds (default, white, gray, black, random)
Webcam Selector — enumerate and switch between connected cameras with resolution presets (VGA through 4K)
Session Stats — gaze prediction count and tracking duration counter
Diagnostics Panel — optional deep-dive with video resolution, screen/viewport info, device pixel ratio, and raw eye crop previews

# Run the demo locally
npm install
npm run dev
# Opens at http://localhost:8089

Ridge Regression

Input: 1,653-element feature vector
Output: Gaze { x, y } in CSS pixels
Training: Solve (XᵀX + λI)β = Xᵀy
Lambda: 1e-5 (insensitive to change)
X & Y trained independently
Matrix operations use ml-matrix; the ridge regression algorithm itself is custom-implemented

Coordinate System

Origin: top-left of viewport
X grows right, Y grows down
Units: CSS pixels

Configuration Constants

| Constant | Value | Description | |----------|-------|-------------| | EYE_INPUT_WIDTH | 40 | Regression input width | | EYE_INPUT_HEIGHT | 20 | Regression input height (2:1) | | EYE_BOX_SCALE | 1.2 | Bounding-box expansion | | EYE_CROP_PADDING | 4 | Augmentation pixel margin | | AUGMENTATION_OFFSETS | ±1 px | 9 shifts per capture | | DEFAULT_FEATURE_COUNT | 1,653 | Total features | | RIDGE_LAMBDA | 1e-5 | Regularization | | MIN_CALIBRATION_SAMPLES | 5 | Minimum samples required | | MIN_DETECTION_CONFIDENCE | 0.5 | Face detection threshold |

📂 Project Structure

webcam-eyetracker-light-open/
├── README.md                          # This file
├── CONTRIBUTING.md                    # How to contribute
├── CHANGELOG.md                       # Version history
├── package.json
├── tsconfig.json
├── tsconfig.lib.json
├── vitest.config.ts
├── playwright.config.ts
├── webpack.config.js
├── eslint.config.js
│
├── public/
│   └── models/                         # MediaPipe models
│       ├── blaze_face_short_range.tflite
│       └── face_landmarker.task
│
├── lib/                                # Core library
│   ├── index.ts                        # All public exports
│   ├── WebcamETLight.ts                # Main tracker class
│   ├── types.ts                        # TypeScript interfaces
│   ├── calibration/
│   │   └── CalibrationPatterns.ts
│   ├── math/
│   │   ├── matrix.ts                   # Ridge regression math
│   │   └── matrix-optimized.ts
│   ├── face/
│   │   ├── FaceDetector.ts
│   │   ├── FaceLandmarkerAdapter.ts
│   │   └── BlazeFaceAdapter.ts
│   └── features/
│       ├── FeatureExtractor.ts
│       ├── EyeBoxScale.ts
│       ├── FeatureConfig.ts
│       └── HeadPoseAugmentation.ts
│
├── apps/
│   └── demo_app/                       # Demo (React 18)
│       ├── App.tsx
│       ├── index.tsx
│       ├── demo_app.md                 # Demo app technical documentation
│       ├── components/
│       └── styles/
│
└── tests/                             # Tests (unit & e2e)
  ├── unit/                           # Vitest unit tests
  └── e2e/                            # Playwright E2E

🧪 Testing

This project uses RealTesting Open — a library making it easier to write acceptance tests of web browser apps using webcam, screen recording, full screen mode, WebRTC and more. E2E tests in this repository leverage RealTesting for virtual camera simulation and browser testing.

npm run test:unit                        # Vitest unit tests
npm run test:e2e                         # Playwright E2E (virtual camera)
npm run test:css                         # CSS rendering tests
npm run test:screenshot                  # Screenshot capture tests
npm test                                 # All default test gates
npm run lint                             # ESLint
npm run lint:fix                         # ESLint with auto-fix
npm run typecheck                        # TypeScript type checking (no emit)
npm run build                            # Build library + demo
npm run build:lib                        # Compile library only (for npm publish)
npm run build:demo                       # Production demo build
npm run test:watch                       # Watch mode for unit tests

🤝 Contributing

See CONTRIBUTING.md for:

Development environment setup
Coding standards (TypeScript strict, ESLint)
Pull-request process
Calibration and tracking quality guidance

When submitting changes that affect prediction accuracy:

Update calibration tests
Include example frames if landmark mapping changes
Re-run npm test for regression checks

📄 License

This project (the eye-tracker library and the demo application) was created by Damian Sromek.

This repository is available under either of the following licenses:

GNU AGPL v3 or later — see LICENSE-AGPL.md
Commercial license — see LICENSE-COMMERCIAL.md

No-fee commercial path

Under the commercial option, you may use the project without a license fee if you qualify as either:

a company whose qualifying valuation, acquisition value, or fallback trailing-twelve-month revenue does not exceed USD $1,000,000, including Affiliates; or
a university, academic, or nonprofit research project whose project or lab grant funding does not exceed USD $1,000,000.

If you exceed those thresholds, or if you need written commercial terms beyond the no-fee allowance, use the paid commercial license path described in LICENSE-COMMERCIAL.md.

Third-party components remain subject to their own licenses. See THIRD_PARTY_NOTICES.md and public/models/README.md.

🔗 Links

npm package
RealEye.io — webcam eyetracking platform
GitHub repository
Issues
User Manual — step-by-step demo guide
Demo App Docs — technical demo documentation
MediaPipe Models — local model hosting details
Third-Party Notices — dependency and model attributions
Changelog — version history
Security Policy — vulnerability reporting
RealTesting Open — browser acceptance testing library (webcam, screen recording, WebRTC, and more)

Made with ❤️ by RealEye