hand-tracker-interaction

Plug-and-play hand-gesture interaction for the web. Uses MediaPipe to detect both hands in the user's webcam and turns common gestures into native DOM events you already handle (click, contextmenu, scroll, history navigation, two-hand zoom/rotate, drag & drop). Most apps need a single line of JS plus a couple of data-hand-* attributes on existing markup.

Install

npm install hand-tracker-interaction

The library expects a MediaPipe hand model file at /hand_landmarker.task on your site root. Download it once into your public/ folder:

curl -fsSL -o public/hand_landmarker.task \
  https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task

If you serve it from a different path, pass it as an option:

await startHandInteraction({ modelAssetPath: '/assets/hand_landmarker.task' });

Quick start (vanilla)

<script type="module">
  import { startHandInteraction } from 'hand-tracker-interaction';
  await startHandInteraction();
</script>

<button onclick="alert('hi')">Click me</button>
<div data-hand-draggable>Drag me</div>
<div data-hand-zoomable data-hand-rotatable>Pinch with two hands</div>

Quick start (React)

import { HandInteraction } from 'hand-tracker-interaction/react';

export default function App() {
  return (
    <HandInteraction numHands={2}>
      <button onClick={() => console.log('clicked')}>Click</button>
      <div data-hand-draggable>Drag me</div>
    </HandInteraction>
  );
}

Gestures

| Gesture | What it does | DOM signal | |---|---|---| | Pinch (thumb + index, short) | Click | MouseEvent('click') on element under the pinch | | Pinch + move | Drag & drop | uses [data-hand-draggable] | | Thumb + middle finger | Right click / context menu | MouseEvent('contextmenu') | | Index + middle pegged together + move | Scroll vertical/horizontal | scrollBy on the closest scrollable ancestor | | Open palm + horizontal swipe | Browser back / forward | history.back() / history.forward() | | Two-hand pinch | Zoom + rotate | applies CSS transform to [data-hand-zoomable] / [data-hand-rotatable] |

Hit-test for clicks, drags, and right-click happens along the segment between the involved fingers, not just at the fingertip — anything the glowing line crosses is in scope.

Options

await startHandInteraction({
  numHands: 2,
  modelAssetPath: '/hand_landmarker.task',
  showSkeleton: true,           // debug skeleton overlay
  showCursors: true,
  pinchStartThreshold: 0.25,    // dist(thumb, index) / handSize
  pinchEndThreshold: 0.45,
  scrollSpeed: 2,
  scrollInverted: false,        // true = touchpad style
  historyNavigation: true,      // open-palm swipe = history back/forward
  handMissingGraceMs: 220,      // tolerate brief detection drops mid-drag
});

You can opt out of the default history navigation per gesture by listening to the recognizer events and calling preventDefault:

handle.gesture.addEventListener('swipeleft', (e) => e.preventDefault());

Direct API

If you want full control, the lower-level building blocks are also exported from the package root:

import {
  CameraManager,
  HandTrackerEngine,
  GestureRecognizer,
  CursorLayer,
} from 'hand-tracker-interaction';

See BLUEPRINT.md for the architecture and design notes.

License

MIT