@saccadejs/core

Webcam eye tracking in the browser: MediaPipe face landmarks → a 36×144 eye crop → an ONNX embedding → a ridge map from embeddings to screen position, fit from your own calibration points. Plus a screen-to-webcam timing loopback that measures the one lag JavaScript cannot see.

This is the jsPsych-agnostic core. For jsPsych experiments use @saccadejs/extension and the saccade-* plugins, which wrap everything here.

Install

npm install @saccadejs/core

The package ships models/eye_embedding.onnx (opset 17, input eye_image float32 [1,36,144,1] in 0–255; outputs embedding float32 [1,128] and cal_weight float32 [1,1], that frame's quality score in [0, 1]). The embedding width is not fixed at 128 — the fit takes its width from the model — but the crop and its preprocessing are exact; see The model. onnxruntime-web and @mediapipe/tasks-vision are dependencies of this package; the <script> build loads them from a CDN instead (see Asset hosting).

Usage

1. Bundler (ESM)

import { SaccadeTracker, defaultGrid13, runCalibration, runValidation } from "@saccadejs/core";

const tracker = new SaccadeTracker({ smoothingFrames: 1 });
await tracker.init(); // camera prompt + model load
tracker.start();

// Show each target yourself; the driver owns the timing.
await runCalibration(
  tracker,
  defaultGrid13(),
  { settleMs: 1000, captureMs: 500 },
  {
    showTarget(target, phase) {
      // target is {x, y} in viewport fractions, or null when the run is over.
      // phase is "settle" (eye still moving) or "capture" (samples being collected).
      draw(target, phase);
    },
  },
);
tracker.fitCalibration(); // every frame a row; lambda defaults to lambdaFor(nPoints, fit) = 3

tracker.onFrame((f) => {
  if (f.gaze) console.log(f.gaze.x, f.gaze.y, f.time.capture);
});

Gaze coordinates are viewport fractions, origin top-left; multiply by window.innerWidth / innerHeight for pixels.

Showing load progress

init() is a multi-second wait the first time — the eye model alone is about 20 MB — so pass onProgress to tell the participant what is happening rather than leaving them at a blank screen. It receives a SaccadeProgress ({ stage, loaded?, total? }) as init walks its stages, in this order: "camera" (the permission prompt), "mediapipe", "landmarker", "ort", "model", "session", "ready". Only "model" carries byte counts, and total is present only when the file's size is known: always for a published release, and otherwise only when the server sent it uncompressed with a Content-Length. The .onnx is fetched once and reused for every execution-provider attempt. Passing no callback changes nothing about init(); tracker.onProgress(cb) subscribes after construction.

const tracker = new SaccadeTracker({
  onProgress: ({ stage, loaded, total }) => {
    if (stage === "model" && total) {
      const mb = (n) => (n / 1e6).toFixed(1);
      setStatus(`Downloading eye model ${mb(loaded)} / ${mb(total)} MB`);
    } else {
      setStatus(stage);
    }
  },
});
await tracker.init();

2. Plain <script> tag

The browser build exposes everything as the global Saccade and bundles neither onnxruntime-web nor MediaPipe — it fetches them at init() from jsDelivr.

<script src="https://unpkg.com/@saccadejs/core/dist/index.browser.min.js"></script>
<script>
  const tracker = new Saccade.SaccadeTracker();
  tracker.init().then(({ ep }) => {
    console.log("running on", ep); // "webgpu" or "wasm"
    tracker.start();
  });
</script>

3. jsPsych

Register the extension and use the plugins; you never touch this API directly.

const jsPsych = initJsPsych({ extensions: [{ type: jsPsychExtensionSaccade }] });

See the documentation site for the extension and the saccade-preview / saccade-calibrate / saccade-validate / saccade-time-sync plugins.

API sketch

| | | | --------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | new SaccadeTracker(opts) | assets, video: {width, height}, smoothingFrames (default 1), executionProviders (default ["webgpu", "wasm"]), onFrame, onProgress, calibrationFit ("frames" default, or "points") | | init() | camera + MediaPipe + ONNX, warmed up. Idempotent. Resolves {ep, videoWidth, videoHeight} | | start() / stop() / dispose() / running | the frame loop | | video | the live, unmirrored camera element — display it if you like (CSS-mirror the preview, never the pixels the model sees) | | onFrame(cb) | returns an unsubscribe function | | onProgress(cb) | replays the latest load-progress report, then every new one; returns an unsubscribe function | | nextFrame() / nextEmbedding() / nextSample() | one-shot promises; nextSample() pairs the embedding with the model's weight for it | | addCalibrationPoint(target, embeddings, weights?), clearCalibration(), getCalibrationPoints() | calibration set | | fitCalibration({lambda, center, calHead, fit}) | solves the ridge map, returns {lambda, nPoints, weighting, fit} | | getCalWeighting() | how the last fit weighted its rows: "model", "head" or "uniform" | | getCurrentGaze() | latest {gaze, time} or null | | sampleLuminance() | whole-frame mean luminance, what the loopback correlates | | defaultGrid13(), trainingGrid20(), validationGrid9(), lambdaFor(n, fit) | grids and the ridge penalty ("frames": 3; "points": 3 at ≤ 9 points, else 1) | | runCalibration, runValidation, runLoopback | drivers | | extractEyeCrop, clahe, resizeBilinearCv, rgbaToGray, cropBBox | the preprocessing, exported for testing | | solveRidge, predict, calWeight, fitRidge | the ridge fit | | estimateLagEdges, estimateLag, sparseSchedule, seededRandom, splitHalves, intervalStats | pure loopback analysis |

Keep tracker.video in the document

Chrome only delivers camera frames (requestVideoFrameCallback) for a video element that is actually rendered — in the document and not display: none. A hidden or detached element stops the tracker silently: no error, no frames, and every nextFrame() waiter (a calibration capture, the timing loopback) hangs forever. Move the element wherever you like, but hide it with opacity: 0 and/or a 2×2 px size — never with display: none, visibility: hidden, or by unmounting it.

The tracker defends itself on both sides: if the element is not in the document it re-attaches it to a tiny invisible holder of its own on document.body, and if rVFC stops arriving anyway the frame loop falls back to a ~300 ms timer tick (frames then carry time.source === "callback") until it starts again.

As a last line of defense, runCalibration and runValidation take a timeoutMs in their CollectOptions (default 5000, 0 to wait indefinitely). If one camera frame takes longer than that, the run rejects with no camera frames for 5000 ms rather than hanging, so a caller has something to put on screen. withFrameTimeout(promise, ms) is exported for callers that run their own capture loop.

The preprocessing is bit-exact with the Python pipeline the model was trained on (OpenCV BT.601 gray, INTER_LINEAR resize, CLAHE with clip 2 and 8×8 tiles); test/crop.test.ts checks it byte for byte against fixtures generated by that pipeline. Do not "clean it up".

Asset hosting

By default nothing needs to be served by you: the .onnx resolves next to the package when the build itself is served from a @saccadejs/core/dist/, and otherwise from jsDelivr, as do the onnxruntime-web wasm, the MediaPipe wasm, and Google's face_landmarker.task. To host them yourself (offline labs, or to avoid a third-party request), pass assets:

new SaccadeTracker({
  assets: {
    modelUrl: "/static/eye_embedding.onnx",
    ortWasmUrl: "/static/ort/", // directory with ort's .wasm/.mjs
    mediapipeWasmUrl: "/static/mediapipe/wasm", // directory
    faceLandmarkerUrl: "/static/face_landmarker.task",
  },
});

Copy node_modules/onnxruntime-web/dist/*.{wasm,mjs} and node_modules/@mediapipe/tasks-vision/wasm/* into those directories. ort.env.wasm.wasmPaths is set to ortWasmUrl for you, and numThreads is pinned to 1.

The <script>-tag build additionally needs the two libraries themselves as ES modules; it takes ort.bundle.min.mjs from inside ortWasmUrl and the MediaPipe bundle from jsDelivr. Override those with ortModuleUrl / mediapipeModuleUrl if you are fully offline.

Timing: what FrameTime means

Every frame carries a time: FrameTime, all on the performance.now() clock:

• capture — when the camera delivered the frame, from requestVideoFrameCallback's captureTime (falling back to receiveTime, then to the callback time; source says which). This is the timestamp to record with a gaze sample, not the time the prediction became available.
• meanCapture — because gaze is computed from a ring buffer of the last smoothingFrames embeddings, the smoothed estimate refers to the mean capture time of that buffer, not to the newest frame. At the default of 1 it equals capture; with smoothingFrames: 5 at 30 fps it is ≈ 67 ms before capture.
• emit — when the frame reached your callback. emit - capture is pipeline latency, and only matters for gaze-contingent designs.
• dropped / presentedFrames — camera frames the loop never saw.

What none of these can see is display lag + camera lag: the delay from a requestAnimationFrame paint to photons leaving the panel, plus photons to captureTime. That sum is a per-machine constant, and it is what you must subtract from capture to align gaze with stimulus onsets. runLoopback(tracker) measures it: it flashes a full-viewport panel on a sparse schedule (levels held 0.5–1 s — at most one flash per second, well under the WCAG 2.3.1 limit, and not a flicker), correlates the camera's luminance edges against the flip times, and returns lagMs with an honest uncertainty (plateauWidthMs) and a verdict of "OK", "INCONCLUSIVE" or "UNRELIABLE" with a reason. Only trust lagMs when the verdict is "OK".

License

MIT