Mitsuke Live

Mitsuke Live is a browser-based real-time object detection library using TensorFlow.js and YOLO models, with flexible post-detection customization.

Mitsuke Live provides two detection modes:

| Mode | Features | | --------- | ----------------------------------------------------------------------------------------------- | | 2D | Basic object detection with 2D center coordinates | | 3D/AR | Object detection with 3D position and orientation, integrated with Three.js for AR applications |

Installation

npm install mitsuke-live

Vite Configuration (Required)

If you're using Vite, add this configuration to your vite.config.ts to ensure proper Web Worker handling:

import { defineConfig } from 'vite';

export default defineConfig({
  optimizeDeps: {
    exclude: ["mitsuke-live"],
  },
});

Important: Without this configuration, Web Workers may not function correctly and you might encounter HTML response errors.

Example

To start the demo application:

npm run example

Access http://localhost:3000 in your browser to view two demos:

• 2d.html - 2D
• 3d.html - 3D/AR

API Parameters

ObjectDetectorOptions

| Parameter | Type | Default | Description | | ------------------------------------------------------ | ---------- | --------- | -------------------------------------------------------------------- | | detection.inferenceInterval | number | 150 | Inference execution interval (milliseconds) | | detection.scoreThreshold | number | 0.7 | Minimum confidence score for detection (0.0-1.0) | | threeDEstimation.objectSize.width | number | - | Real object width in meters (required for 3D mode) | | threeDEstimation.objectSize.height | number | - | Real object height in meters (required for 3D mode) | | threeDEstimation.cameraFov | number | 50 | Camera field of view in degrees | | threeDEstimation.orientationCoefficients.pitch | number | 1.0 | Multiplier for pitch estimation (up-down tilt) | | threeDEstimation.orientationCoefficients.roll | number | 1.0 | Multiplier for roll estimation (left-right tilt) | | performance.backend | string | "webgl" | TensorFlow.js backend ("webgl", "webgpu", "wasm", "cpu") | | performance.memoryThreshold | number | 50 | Tensor count threshold for memory cleanup | | onDetection | function | - | Callback for detection results (not called when detection is paused) | | onCameraReady | function | - | Callback when camera is ready | | onCameraNotAllowed | function | - | Callback when camera access is denied |

Detection Result Properties

| Property | Type | Description | | ------------- | ---------------------------------- | ------------------------------------------------ | | boundingBox | [number, number, number, number] | Object position and size [centerX, centerY, width, height] | | angle | number | Rotation angle in degrees | | score | number | Confidence score (0.0-1.0) |

ARDetection Additional Properties (3D Mode)

| Property | Type | Description | | ------------------- | -------- | ------------------------------------------- | | depth | number | Distance from camera in meters | | orientation.pitch | number | Up-down tilt angle (-90 to 90 degrees) | | orientation.roll | number | Left-right tilt angle (-180 to 180 degrees) |

DetectionController Methods

pause(options?)

Pause detection processing.

// Pause both camera and detection (default behavior)
detector.pause();
detector.pause({ pauseCamera: true });

// Pause only detection, keep camera running
detector.pause({ pauseCamera: false });

resume()

Resume detection processing.

detector.resume();

Partners

This project is supported by the following partners:

License

Apache License 2.0