@souscheflabs/ml-vision

ML-powered product detection for React Native. Speed up kitchen inventory onboarding with multi-barcode scanning, receipt OCR, and visual product recognition.

Features

• Multi-Barcode Scanning - Scan multiple barcodes from a single photo
• Receipt OCR - Photograph grocery receipts to auto-extract items
• Visual Product Recognition - Recognize fridge/pantry contents by appearance
• Video Scanning - Real-time product detection from camera feed
• On-Device ML - Fast inference using TFLite with GPU acceleration
• Server Fallback - Optional server-side processing for better accuracy

Installation

npm install @souscheflabs/ml-vision
# or
yarn add @souscheflabs/ml-vision

Peer Dependencies

This package requires the following peer dependencies:

npm install react-native-vision-camera react-native-mmkv

For on-device ML inference (Phase 3+):

npm install react-native-fast-tflite

iOS Setup

cd ios && pod install

Add camera permissions to Info.plist:

<key>NSCameraUsageDescription</key>
<string>Camera is used to scan products and receipts</string>

Android Setup

Add camera permission to AndroidManifest.xml:

<uses-permission android:name="android.permission.CAMERA" />

Quick Start

1. Wrap your app with the provider

import { MLVisionProvider } from '@souscheflabs/ml-vision';
import { MMKV } from 'react-native-mmkv';

// Create MMKV instance for caching
const storage = new MMKV({ id: 'ml-vision-cache' });

function App() {
  return (
    <MLVisionProvider
      config={{
        serverUrl: 'http://192.168.1.100:8000', // Optional: your ML server
        cacheEnabled: true,
      }}
      storage={storage}
    >
      <YourApp />
    </MLVisionProvider>
  );
}

2. Use the hooks in your components

import { useMultiBarcodeScanner } from '@souscheflabs/ml-vision';
import { Camera } from 'react-native-vision-camera';

function BarcodeScanScreen() {
  const {
    results,
    isScanning,
    startScanning,
    stopScanning,
    frameProcessor,
  } = useMultiBarcodeScanner({
    formats: ['ean-13', 'upc-a', 'qr'],
    maxBarcodes: 20,
    onDetected: (barcodes) => {
      console.log('Found barcodes:', barcodes);
    },
  });

  return (
    <Camera
      device={device}
      isActive={isScanning}
      frameProcessor={frameProcessor}
    />
  );
}

API Reference

MLVisionProvider

The context provider that must wrap your app.

<MLVisionProvider
  config={{
    serverUrl?: string;        // ML server URL for fallback
    serverTimeout?: number;    // Request timeout (default: 5000ms)
    cacheEnabled?: boolean;    // Enable result caching (default: true)
    cacheTTL?: number;         // Cache TTL in ms (default: 24 hours)
    enableGPUDelegate?: boolean; // Use GPU for inference (default: true)
  }}
  storage={mmkvInstance}       // MMKV instance for caching
  onInitialized={() => {}}     // Called when ready
  onError={(error) => {}}      // Called on init error
>

useMultiBarcodeScanner

Scan multiple barcodes from camera frames or photos.

const {
  // State
  isReady: boolean;
  isScanning: boolean;
  results: BarcodeDetectionResult[];
  error: Error | null;

  // Actions
  startScanning: () => void;
  stopScanning: () => void;
  scanPhoto: (uri: string) => Promise<BarcodeDetectionResult[]>;
  clearResults: () => void;

  // For VisionCamera
  frameProcessor: FrameProcessor;
} = useMultiBarcodeScanner(options);

useProductDetector

Detect products in images using trained ML models.

const {
  // State
  isModelLoaded: boolean;
  isDetecting: boolean;
  detections: ProductDetectionResult[];
  modelVersion: string;

  // Actions
  detectProducts: (uri: string) => Promise<ProductDetectionResult[]>;
  updateModel: () => Promise<void>;

  // For VisionCamera
  frameProcessor: FrameProcessor;
} = useProductDetector({
  model: 'fast' | 'accurate';
  minConfidence?: number;
  serverFallback?: boolean;
});

useReceiptScanner

Extract items from receipt photos.

const {
  // State
  isProcessing: boolean;
  items: ReceiptItem[];
  confidence: number;

  // Actions
  scanReceipt: (uri: string) => Promise<ReceiptScanResult>;
  confirmItem: (item: ReceiptItem) => void;
  rejectItem: (item: ReceiptItem) => void;
} = useReceiptScanner({
  serverFallback?: boolean;
  minConfidence?: number;
});

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Your React Native App                        │
├─────────────────────────────────────────────────────────────────┤
│  @souscheflabs/ml-vision                                            │
│  ├── Hooks (useMultiBarcodeScanner, useProductDetector, etc.)  │
│  ├── Frame Processors (barcode, OCR, TFLite detection)         │
│  └── CacheManager (MMKV-based caching)                         │
├─────────────────────────────────────────────────────────────────┤
│  On-Device Processing (fast, offline)                           │
│  ├── MLKit (barcodes, text recognition)                        │
│  ├── react-native-fast-tflite (custom models)                  │
│  └── CoreML/NNAPI delegates (GPU acceleration)                 │
├─────────────────────────────────────────────────────────────────┤
│  Server Fallback (accurate, requires network)                   │
│  ├── FastAPI inference server                                   │
│  ├── YOLOv8 full models                                        │
│  └── PaddleOCR for complex receipts                            │
└─────────────────────────────────────────────────────────────────┘

Training Your Own Models

See the Training Guide for instructions on:

• Setting up the Docker training environment
• Preparing your dataset
• Training YOLOv8 models
• Exporting to TFLite for mobile

Project Structure

packages/ml-vision/
├── src/
│   ├── index.ts              # Main exports
│   ├── types/                # TypeScript definitions
│   ├── hooks/                # React hooks
│   ├── core/                 # Provider, cache, server client
│   ├── processors/           # Frame processors
│   └── utils/                # Utilities
├── models/                   # Bundled TFLite models
├── docs/                     # Documentation
└── dist/                     # Compiled output

Development

# Build the package
npm run build

# Watch for changes
npm run watch

# Type check
npm run typecheck

License

ISC

Contributing

1. Fork the repository
2. Create your feature branch
3. Make your changes
4. Run tests and type checks
5. Submit a pull request

Troubleshooting

Model Loading Fails on Android

Ensure the model file is in android/app/src/main/assets/. Metro bundler handles this automatically for .tflite files in the models/ directory.

Camera Permission Denied

Add the camera permission to your app and request it at runtime:

import { Camera } from 'react-native-vision-camera';

const permission = await Camera.requestCameraPermission();
if (permission === 'denied') {
  // Handle permission denied
}

Low Detection Accuracy

1. Ensure good lighting conditions
2. Try lowering minConfidence threshold
3. Enable serverFallback for better accuracy on complex scenes

Build Errors with TFLite

If you encounter build errors with react-native-fast-tflite:

# iOS: Clean and rebuild
cd ios && pod deintegrate && pod install && cd ..

# Android: Clean gradle cache
cd android && ./gradlew clean && cd ..

Support

• GitHub Issues
• Documentation
• Changelog