soupdraw

Draw on your webcam with hand gestures, on-device. A tiny loader around the full SoupDraw pipeline, the exact engine that powers the SoupDraw browser extension.

Give it a mount point, call start(), and you get back an augmented MediaStream (your camera with hand-drawn annotations) plus a simple control API. The camera and hand tracking never leave the browser.

Live demo: draw.soupup.ai

import { SoupDraw } from "soupdraw";

const sd = new SoupDraw({ mount: "#stage" });
const stream = await sd.start();   // pipe into a call, a recording, or a <video>

sd.setColor("#00F0FF");
sd.on("status", (s) => console.log(s.gesture));

Install

npm install soupdraw

Or from a CDN, no build step:

<div id="stage" style="max-width:720px"></div>
<script type="module">
  import { SoupDraw } from "https://esm.sh/soupdraw";
  await new SoupDraw({ mount: "#stage" }).start();
</script>

Full parity with the extension

The SDK runs the real pipeline, so every gesture and feature is the same:

| Gesture | Does | | :-- | :-- | | 🤏 Pinch (thumb + index) | Draw | | 👍 Thumb-out fist | Erase (eraser rides the thumb tip) | | ✌️ Victory | Move a shape, or drag a box to marquee-select | | ✊ Closed fist | Pan the canvas | | 🖐️🖐️ Two-hand five-finger pinch | Scale, rotate, and pan (the selection, or everything) | | 🖐️ Five-finger pinch | Restore a board from the history strip | | ✊✊ Double fist-clench | Clear |

Plus: a history strip you can restore cleared boards from, a minimap, spotlight (dim + glow on your hands), whiteboard mode, shape-assist (rough strokes snap to clean shapes), and guided per-user pinch calibration.

API

new SoupDraw(options)

| Option | Default | Notes | | :-- | :-- | :-- | | mount | — | selector/element to inject the output <video> into | | video | — | bring your own <video> instead of mount | | color | "#ff2d55" | pen colour | | size | 6 | pen size | | mirror | true | selfie-mirror the camera | | assist | true | snap rough strokes into clean shapes | | history | true | history strip (restore cleared boards) | | minimap | false | show a minimap of off-frame drawings | | spotlight | false | dim the feed and glow around hands | | whiteboard | false | draw on a solid board instead of the camera | | boardColor | "#14151a" | whiteboard colour | | runtimeBase | package CDN | where to load the runtime + recognizer from |

Methods: start() → Promise<MediaStream>, stop(), setColor, setSize, setMirror, setAssist, setHistory, setMinimap, setSpotlight, setWhiteboard, setBoardColor, setBinding(gesture, action), undo, redo, clear, calibrate, on(event, cb).

Events: ready, status ({ gesture, hands, strokes, drawing, erasing, transforming, modelReady, historyMode, selection }), error, calibrated, stop.

How it works

On start(), the SDK injects two things (from the package's CDN by default, or your runtimeBase): the pipeline runtime (engine + gesture catalog + the compositor) and a hidden recognizer iframe that runs MediaPipe. It then drives everything by config messages. The pipeline patches getUserMedia so it can hand back the augmented feed, so start() returns a MediaStream that is a drop-in replacement for a raw camera anywhere a MediaStream works.

The ~20 MB hand model and MediaPipe wasm load once from public CDNs. To run with no third-party fetch, self-host the runtime (dist/) and MediaPipe assets and set runtimeBase.

Status

0.2.x. Full gesture and feature parity with the extension. Issues and use cases welcome.

License

MIT © Abhishek Janjalkar (Soup Up). Runs MediaPipe Tasks Vision (Apache-2.0) in the recognizer iframe.