vision-camera-plugin-object-detector

Real-time object detection for React Native Vision Camera, powered by MLKit.

Integrates with Vision Camera to provide object detection and classification.

Features

• Fast object detection and tracking Detect objects and get their locations in the image. Track objects across successive image frames.
• Optimized on-device model The object detection and tracking model is optimized for mobile devices and intended for use in real-time applications, even on lower-end devices.
• Prominent object detection Automatically determine the most prominent object in an image.
• Coarse classification Classify objects into broad categories, which you can use to filter out objects you're not interested in. The following categories are supported: home goods, fashion goods, food, plants, and places.

Example Result

[
  {
    "trackingId": 1,
    "bounds": { "x": 42, "y": 128, "width": 230, "height": 180 },
    "labels": [{ "text": "Food", "index": 2, "confidence": 0.92 }]
  }
]

Installation

npm install vision-camera-plugin-object-detector

Then you need to add react-native-worklets-core plugin to babel.config.js. More details here.

Usage

useObjectDetector()

import { View, StyleSheet } from 'react-native';
import { Worklets } from 'react-native-worklets-core';

import {
  Camera,
  useCameraDevice,
  useCameraPermission,
  useFrameProcessor,
} from 'react-native-vision-camera';
import { useEffect } from 'react';
import {
  useObjectDetector,
  type DetectedObject,
} from 'vision-camera-plugin-object-detector';

export default function App() {
  const device = useCameraDevice('front');
  const { hasPermission, requestPermission } = useCameraPermission();

  const { detectObjects } = useObjectDetector({
    enableClassification: true,
    enableMultipleObjects: true,
  });

  useEffect(() => {
    if (!hasPermission) {
      requestPermission();
    }
  }, [hasPermission, requestPermission]);

  const onObjectDetected = Worklets.createRunOnJS(
    (objects: DetectedObject[]) => {
      console.log(objects);
    }
  );

  const frameProcessor = useFrameProcessor(
    (frame) => {
      'worklet';
      const detectedObjects = detectObjects(frame);
      onObjectDetected(detectedObjects);
    },
    [onObjectDetected]
  );

  return (
    <View style={styles.container}>
      {hasPermission && device && (
        <Camera
          device={device}
          isActive={true}
          isMirrored
          style={StyleSheet.absoluteFill}
          frameProcessor={frameProcessor}
        />
      )}
    </View>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    alignItems: 'center',
    justifyContent: 'center',
  },
});

detectObjectsFromImage()

import {
  detectObjectsFromImage,
  type DetectedObject,
} from 'vision-camera-plugin-object-detector';

const handleDetectFromFile = async () => {
  const detectObjects: DetectedObject[] = await detectObjectsFromImage(
    'file://path/to/your/image.jpg',
    {
      enableClassification: true,
      enableMultipleObjects: true,
    }
  );
};

Detection Options

| Option | Type | Default | Description | | ----------------------- | --------- | ------- | ---------------------------------------------------------------------------------- | | enableClassification | boolean | false | Enables coarse object classification (e.g. food, place, plant). | | enableMultipleObjects | boolean | false | Detects multiple objects in the same frame instead of only the most prominent one. |

Interfaces

DetectedObject

interface DetectedObject {
  bounds: DetectedObjectBounds;
  labels: DetectedObjectLabel[];
  trackingId?: number;
}

DetectedObjectLabel

interface DetectedObjectLabel {
  text: string;
  index: number;
  confidence: number;
}

DetectedObjectBounds

interface DetectedObjectBounds {
  width: number;
  height: number;
  x: number;
  y: number;
}

PredefinedCategory

enum PredefinedCategory {
  FASHION_GOOD = 'Fashion good',
  HOME_GOOD = 'Home good',
  FOOD = 'Food',
  PLACE = 'Place',
  PLANT = 'Plant',
  FASHION_GOOD_INDEX = 0,
  HOME_GOOD_INDEX = 1,
  FOOD_INDEX = 2,
  PLACE_INDEX = 3,
  PLANT_INDEX = 4,
}

Contributing

• Development workflow
• Sending a pull request
• Code of conduct

License

MIT

Acknowledgements

Special thanks to Marc Rousavy for creating VisionCamera
and to Luiz Carlos for creating face detector plugin.

Made with create-react-native-library