ai-scroll2

Control page scrolling with hand movement captured through your webcam.

ai-scroll2 is a React hook powered by MediaPipe Tasks Vision. Hold your hand in a neutral position to calibrate, then move it above or below that position to scroll.

Features

• Webcam hand tracking in the browser
• Automatic neutral-position calibration
• Configurable detection rate, sensitivity, speed, and smoothing
• Scrolls the window or a specific scrollable element
• Typed React and TypeScript API
• Camera and MediaPipe lifecycle cleanup

Requirements

• React 19 or a compatible React version supported by your application
• A browser with webcam support
• Camera permission
• A secure origin when deployed, such as HTTPS or localhost

Installation

npm install ai-scroll2

The package includes @mediapipe/tasks-vision as a runtime dependency.

Basic Usage

Attach the returned videoRef to a video element and call start from a user action. Browsers generally require camera access to begin from a user gesture.

import { useHandScroll } from "ai-scroll2";

export function HandScrollControls() {
  const {
    videoRef,
    start,
    stop,
    isActive,
    isReady,
    error,
    neutralY,
    resetCalibration,
  } = useHandScroll({
    autoCalibrate: true,
    calibrationFrames: 15,
  });

  return (
    <aside>
      <video ref={videoRef} muted playsInline width={320} />

      <button onClick={() => void start()} disabled={isActive}>
        Start
      </button>
      <button onClick={stop} disabled={!isActive}>
        Stop
      </button>
      <button onClick={resetCalibration} disabled={!isActive}>
        Recalibrate
      </button>

      <p>Camera: {isActive ? "Active" : "Inactive"}</p>
      <p>MediaPipe: {isReady ? "Ready" : "Loading"}</p>
      <p>Neutral Y: {neutralY?.toFixed(3) ?? "--"}</p>
      {error && <p role="alert">{error.message}</p>}
    </aside>
  );
}

After starting, keep your hand still until calibration completes. Moving your palm up scrolls upward; moving it down scrolls downward.

Configuration

All options are optional.

| Option | Default | Description | | --- | ---: | --- | | enabled | false | Reserved automatic-start option. Use start() for explicit camera permission flow. | | detectionFps | 20 | Maximum MediaPipe detection rate. | | deadZone | 0.03 | Normalized movement around neutral Y that is ignored. | | maxDistance | 0.25 | Normalized hand distance at which maximum speed is reached. | | maxSpeed | 25 | Maximum scroll speed per animation frame. | | smoothing | 0.15 | Velocity smoothing from 0 to 1. | | neutralY | 0.5 | Initial normalized neutral Y position when auto-calibration is disabled. | | autoCalibrate | true | Average detected palm positions before scrolling begins. | | calibrationFrames | 15 | Number of palm samples used for calibration. | | handLostFrames | 3 | Missing detections before scrolling stops. | | scrollTarget | null | HTMLElement to scroll; otherwise the window is scrolled. | | videoWidth | 640 | Requested camera width. | | videoHeight | 480 | Requested camera height. | | facingMode | "user" | Camera facing mode: "user" or "environment". | | wasmPath | package default | Optional MediaPipe WASM asset path. | | modelPath | package default | Optional hand-landmarker model path. |

Normalized Y values range from 0 at the top of the camera image to 1 at the bottom.

Scrolling a Specific Element

Pass a scrollable element as scrollTarget when the page itself should not move.

const scrollAreaRef = useRef<HTMLDivElement>(null);

const handScroll = useHandScroll({
  scrollTarget: scrollAreaRef.current,
});

return (
  <div ref={scrollAreaRef} style={{ height: 500, overflowY: "auto" }}>
    <video ref={handScroll.videoRef} muted playsInline />
    {/* Scrollable content */}
  </div>
);

For a ref that is initially null, pass the current element through state or update the option after the element mounts so the hook receives the mounted target.

Return Values

• videoRef: Attach to the <video> element used for camera input.
• start(): Loads MediaPipe, requests the camera, calibrates, and starts detection.
• stop(): Stops detection, scrolling, and camera tracks.
• isActive: Whether the camera and detection loop are active.
• isReady: Whether the MediaPipe recognizer has loaded.
• error: Structured camera or startup error, or null.
• neutralY: Current normalized calibration position, or null while calibrating.
• resetCalibration(): Clears calibration and returns to the initial calibration state.

Local Development

Install dependencies from the repository root:

npm install

Build the library:

npm run build

Create a package archive:

npm run pack

The Vite demo is in demo/ and can be run separately:

cd demo
npm install
npm run dev

Troubleshooting

• Camera permission denied: Allow camera access for the site and try again.
• No camera found: Connect or enable a camera, then restart the demo.
• Camera already in use: Close other applications using the camera.
• No scrolling: Keep the hand visible, complete calibration, and move farther than the configured deadZone.
• Production camera failure: Serve the application over HTTPS; most browsers block webcam access on insecure origins.

License

MIT