vision-camera-doc-scanner-plugin

Plugin for VisionCamera that adds a Document Scanner to your app.

Plugin is powered by: OpenCV and Deep Learning for the Document Detection (Image Segmentation).

TODO'S:

NOTE: This plugin is still under development

• [ ] add steps to reproduce iOS
• [ ] some improvements
• [ ] test expo compatability
• [ ] expo config plugin
• [ ] Retrain model with more data

Installation

yarn add vision-camera-doc-scanner-plugin
cd ios && pod install

Add the plugin to your babel.config.js:

module.exports = {
  plugins: [
    [
      'react-native-reanimated/plugin',
      {
        globals: ['__detectDocument'],
      },
    ],

    // ...
  ],
};

Note: You have to restart metro-bundler for changes in the babel.config.js file to take effect.

Add dependencies to package.json:

"react-native-reanimated": "2.12.0",
"react-native-vision-camera": "2.15.2"

Add import on top of index.tsx:

import 'react-native-reanimated';

Android

Register package in MainApplication.java (example/android/app/src/main/java/com/example/visioncameradocscannerplugin/MainApplication.java):

import com.visioncameradocscannerplugin.VisionCameraDocScannerPluginPackage;

// ...
@Override
protected List<ReactPackage> getPackages() {
  @SuppressWarnings("UnnecessaryLocalVariable")
  List<ReactPackage> packages = new PackageList(this).getPackages();
  // Packages that cannot be autolinked yet can be added manually here, for example:
  // packages.add(new MyReactNativePackage());
  packages.add(new VisionCameraDocScannerPluginPackage());
  return packages;
}

Add Camera Permission to /android/app/src/main/AndroidManifest.xml

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
  package="com.example.visioncameradocscannerplugin">

    <uses-permission android:name="android.permission.INTERNET" />
    // add following line
    <uses-permission android:name="android.permission.CAMERA" />

    //...

</manifest>

Apply fix for OpenCV in android/app/build.gradle android section:

// fix for opencv4 duplicated files
packagingOptions {
    pickFirst 'lib/arm64-v8a/libc++_shared.so'
    pickFirst 'lib/x86/libc++_shared.so'
    pickFirst 'lib/x86_64/libc++_shared.so'
    pickFirst 'lib/armeabi-v7a/libc++_shared.so'
}

iOS

Add camera usage description in your projects info.plist

<key>NSCameraUsageDescription</key>
<string>Scan Documents</string>

Usage

See the example for how to use this in your app.

import { cropImage, detectDocument } from "vision-camera-doc-scanner-plugin";

// ...
const frameProcessor = useFrameProcessor((frame) => {
  'worklet';
  const coords = detectDocument(frame);
}, []);

const takeSnapshot = async () => {
    const cachedCoords = {value: coords.value};
    if (
      (cachedCoords.value.x1 !== 0 && cachedCoords.value.y1 !== 0) ||
      (cachedCoords.value.x2 !== 0 && cachedCoords.value.y2 !== 0)
    ) {
      const data = await camera.current
        ?.takePhoto({qualityPrioritization: 'speed'})
        .catch(e => console.log(e));
      const dataPath = data?.path;

      if (dataPath) {
        const base64Images = await cropImage(
          cachedCoords,
          dataPath,
        );
        const imageStr = `data:image/jpeg;base64,${base64Images
          .replace('\n', '')
          .replace('\r', '')
          .replace(' ', '')}`;
      } else {
        console.log('no data path');
      }
    } else {
      console.log('There is nothing to crop !');
    }
  };

Data

detectDocument(frame) returns Coords with the following data shape.

 type Coords = {
   x1: number;
   y1: number;
   x2: number;
   y2: number;
   x3: number;
   y3: number;
   x4: number;
   y4: number;
   width: number;
   height: number;
 }

cropImage takes the Coords to crop and an path to the image. It returns a Base64-String where the cropped image is stored.

Additional:

The project contains a python folder which contains the python script to train the image segmentation model and further information.

Contributing

See the contributing guide to learn how to contribute to the repository and the development workflow.

License

MIT

Made with create-react-native-library