@sency/react-native-smkit

Integrate SMKit pose detection and exercise analysis into your React Native application with a simple, component-based API.

Features

• ✅ Real-time pose detection and movement tracking
• ✅ Exercise-specific feedback and form analysis
• ✅ Rep counting with completion detection
• ✅ Session-based architecture with automatic lifecycle management
• ✅ TypeScript support
• ✅ iOS and Android camera support

Installation

npm install @sency/react-native-smkit
# or
yarn add @sency/react-native-smkit

Compatibility

• @sency/react-native-smkit 1.2.1
• React Native 0.76.9 in the bundled example app
• iOS 17.0+ with SMKit 1.9.5 and SMBase 1.9.5 via CocoaPods
• Android SDK 26+, JDK 17, Android Gradle Plugin 8.6+, and internal SMKit Android 1.7.1
• react-native-svg 15.8.0 in the example app for React Native 0.76 compatibility

Android Setup

Android uses the internal SMKit Maven artifacts:

• com.sency.smkit:smkit:1.7.1
• com.sency.smbase.nativeclient:smbase-native-client:1.7.1

The package Gradle file includes local sibling repository fallbacks for ../smkit_android/repo and Sency Artifactory:

maven { url = uri("https://artifacts.sency.ai/artifactory/release") }

Apps must request android.permission.CAMERA at runtime before starting a session. The library manifest declares CAMERA and INTERNET, but runtime permission is still the app’s responsibility.

SMKit Android 1.7.1 brings CameraX 1.5.x transitively, so consuming Android apps should use Android Gradle Plugin 8.6 or newer. Older AGP versions can fail checkDebugAarMetadata before compilation.

Android currently supports the embedded CameraX session, detection events, position events, native skeleton overlay, body/phone calibration events, guidance controls, feedback exclusion, and config strings. iOS-only options such as wide-angle camera selection, back camera selection, 3D options, workout paused gesture regions, configFileName, and recordExercise=false are accepted by TypeScript but are no-op or reported as unsupported on Android.

iOS Setup

cd ios && pod install && cd ..

After upgrading to 1.2.1, run CocoaPods again so iOS picks up the react-native-smkit podspec with SMKit 1.9.5 and SMBase 1.9.5.

Quick Start

1. Configure SMKit

Call configure() early in your app lifecycle (e.g., on app startup):

import { Platform } from 'react-native';
import { configure, preloadModelsInBackground } from '@sency/react-native-smkit';

useEffect(() => {
  const configureSMKit = async () => {
    try {
      await configure('YOUR_AUTH_KEY', {
        useBundledModelsOnColdStart: true,
        poseModelChoice: Platform.OS === 'android' ? 'AdaptiveChoice' : undefined,
      });
      preloadModelsInBackground();
    } catch (err) {
      console.error('SMKit configuration failed:', err);
    }
  };

  configureSMKit();
}, []);

⚠️ Required: configure() must be called before using SmkitCameraView.

2. Add the Camera View

Use the SmkitCameraView component to integrate pose detection:

import { SmkitCameraView, type SmkitCameraViewRef } from '@sency/react-native-smkit';
import { useRef } from 'react';

const MyApp = () => {
  const cameraRef = useRef<SmkitCameraViewRef>(null);

  return (
    <SmkitCameraView
      ref={cameraRef}
      authKey="YOUR_AUTH_KEY"
      exercise="SquatRegular"
      phonePosition="Floor"
      userHeight={175}
      onDetectionData={(data) => {
        console.log('Movement feedback:', data.feedback);
        console.log('Technique score:', data.techniqueScore);
      }}
      style={{ flex: 1 }}
    />
  );
};

3. Control Sessions

// Start camera session
cameraRef.current?.startSession();

// Start movement detection
cameraRef.current?.startDetection('SquatRegular');

// Receive exercise summary when detection stops
const handleDetectionStopped = (summary) => {
  console.log('Session ended');
  console.log('Total time:', summary.totalTime, 'seconds');
  console.log('Technique score:', summary.techniqueScore);
};

// Stop detection
cameraRef.current?.stopDetection();

// Stop session
cameraRef.current?.stopSession();

API Reference

SmkitCameraView Component

<SmkitCameraView
  ref={cameraRef}
  authKey={string}
  exercise={string}
  phonePosition={PhonePosition}
  userHeight={number}
  useWideAngleCamera={boolean}
  autoStart={boolean}
  onDetectionData={callback}
  onDetectionStopped={callback}
  onError={callback}
  onPreviewReady={callback}
  onLayout={callback}
  style={StyleProp<ViewStyle>}
/>

Props

| Prop | Type | Required | Description | |------|------|----------|-------------| | ref | React.Ref | Yes | Ref to call imperative methods | | authKey | string | Yes | SMKit authentication key | | exercise | string | No | Exercise type to detect (see Supported Exercises) | | phonePosition | 'Floor' | 'Elevated' | No | Phone position relative to user (default: 'Floor') | | jumpRefPoint | 'HIP' | 'KNEE' | 'ANKLE' | No | Reference point for jump exercises | | jumpHeightThreshold | number | No | Minimum jump height threshold in cm | | userHeight | number | No | User height in cm for form analysis | | useWideAngleCamera | boolean | No | Use wide-angle camera lens (default: false) | | autoStart | boolean | No | Auto-start session on mount (default: false) | | showNativeSkeletonOverlay | boolean | No | Android: draw skeleton directly in the native camera view to avoid JS overlay latency | | positionDataFps | number | No | Throttle pose events sent to JS. Android defaults to 15 fps when onPositionData is provided | | detectionDataFps | number | No | Throttle detection updates sent to JS. Finished-rep events are never dropped | | onDetectionData | (data: MovementFeedbackData) => void | No | Called when movement data is detected | | onPositionData | (data: JointData) => void | No | Called with joint position data for advanced features | | onDetectionStopped | (summary: ExerciseSummary) => void | No | Called when detection stops with session summary | | onError | (error: string) => void | No | Called when errors occur | | onPreviewReady | () => void | No | Called when camera preview is ready | | onLayout | (event: LayoutChangeEvent) => void | No | Standard RN layout event | | style | StyleProp | No | View styling |

Ref Methods (SmkitCameraViewRef)

startSession(): void
// Starts the camera session and initializes detection

stopSession(): void
// Stops the camera session and cleans up resources

startDetection(exercise: string): void
// Begins detecting the specified exercise

stopDetection(): void
// Stops detection and returns exercise summary

Supported Exercises

Map user-friendly names to native SMKit exercise types:

const EXERCISE_TYPE_MAP: Record<string, string> = {
  'Squat': 'SquatRegular',
  'Pushup': 'PushupRegular',
  'JumpingJacks': 'JumpingJacks',
  'Plank': 'PlankHighStatic',
  'HighKnees': 'HighKnees',
};

// Pass the mapped native type to startDetection:
cameraRef.current?.startDetection(EXERCISE_TYPE_MAP['Squat']);

MovementFeedbackData

Real-time feedback for each detected movement frame:

interface MovementFeedbackData {
  didFinishMovement: boolean;     // Rep completed
  isShallowRep: boolean;          // Form issue: shallow rep
  isInPosition: boolean;          // User in correct position
  isPerfectForm: boolean;         // Perfect form detected
  techniqueScore: number;         // Technique score (0-100)
  detectionConfidence: number;    // Detection confidence (0-1)
  feedback: string[];             // User feedback messages
  currentRomValue: number;         // Current range of motion
  specialParams: Record<string, number>; // Exercise-specific params
}

ExerciseSummary

Summary data returned when detection stops:

interface ExerciseSummary {
  sessionId: string;              // Unique session ID
  exerciseName: string;           // Exercise that was detected
  startTime: string;              // ISO 8601 start timestamp
  endTime: string;                // ISO 8601 end timestamp
  totalTime: number;              // Duration in seconds
  techniqueScore: number;         // Average technique score (0-100)
  feedbacks?: Record<string, number>; // Aggregated feedback data
}

JointData

Joint position data for advanced pose tracking:

interface JointPosition {
  x: number;  // Preview-frame x coordinate in pixels
  y: number;  // Preview-frame y coordinate in pixels
  confidence?: number; // Confidence level when available
}

interface JointData {
  [jointName: string]: JointPosition;
}

Available joints include: Nose, Neck, RShoulder, RElbow, RWrist, LShoulder, LElbow, LWrist, RHip, RKnee, RAnkle, LHip, LKnee, LAnkle, REye, LEye, REar, LEar, Hip, Chest, Head, LBigToe, RBigToe, LSmallToe, RSmallToe, LHeel, RHeel.

configure(authKey: string, options?)

Initialize SMKit before using the camera view:

import { configure } from '@sency/react-native-smkit';

try {
  await configure('YOUR_AUTH_KEY');
  // SMKit is now ready
} catch (error) {
  console.error('Configuration failed:', error);
}

useBundledModelsOnColdStart defaults to false. When enabled on iOS, the SDK can use bundled cold-start models, including PSTMO refinement, while server models are still being downloaded.

Android accepts poseModelChoice (AdaptiveChoice, Prime, Pro, Lite, UltraLite, Basic) and voiceFeedbackLanguage. If poseModelChoice is omitted, Android defaults to bundled-safe UltraLite so first camera startup does not depend on remote pose-model downloads. Pass AdaptiveChoice when you want the Android SDK to choose the model profile.

await configure('YOUR_AUTH_KEY', {
  useBundledModelsOnColdStart: true,
  poseModelChoice: 'AdaptiveChoice',
  voiceFeedbackLanguage: 'en',
});

Complete Example

Here's a complete working example with session management and rep counting:

import React, { useRef, useState, useEffect } from 'react';
import { Platform, View, Text, Button, StyleSheet } from 'react-native';
import {
  SmkitCameraView,
  configure,
  type SmkitCameraViewRef,
  type MovementFeedbackData,
  type ExerciseSummary,
} from '@sency/react-native-smkit';

const EXERCISE_TYPE_MAP = {
  'Squat': 'SquatRegular',
  'Pushup': 'PushupRegular',
  'JumpingJacks': 'JumpingJacks',
  'Plank': 'PlankHighStatic',
  'HighKnees': 'HighKnees',
};

export default function ExerciseApp() {
  const cameraRef = useRef<SmkitCameraViewRef>(null);
  const [sessionStarted, setSessionStarted] = useState(false);
  const [isDetecting, setIsDetecting] = useState(false);
  const [repCount, setRepCount] = useState(0);
  const [feedback, setFeedback] = useState('');
  const [score, setScore] = useState(0);

  // Configure on mount
  useEffect(() => {
    configure('YOUR_API_KEY', {
      useBundledModelsOnColdStart: true,
      poseModelChoice: Platform.OS === 'android' ? 'AdaptiveChoice' : undefined,
    }).catch(console.error);
  }, []);

  const handleDetectionData = (data: MovementFeedbackData) => {
    if (data.didFinishMovement) {
      setRepCount(prev => prev + 1);
    }
    setFeedback(data.feedback[0] || '');
    setScore(Math.round(data.techniqueScore));
  };

  const handleDetectionStopped = (summary: ExerciseSummary) => {
    console.log('Session complete!');
    console.log('Total time:', summary.totalTime, 'seconds');
    console.log('Average score:', summary.techniqueScore);
    setIsDetecting(false);
  };

  const startSession = () => {
    setSessionStarted(true);
    setTimeout(() => {
      cameraRef.current?.startSession();
    }, 100);
  };

  const startDetection = () => {
    setIsDetecting(true);
    cameraRef.current?.startDetection(EXERCISE_TYPE_MAP['Squat']);
  };

  const stopDetection = () => {
    cameraRef.current?.stopDetection();
  };

  return (
    <View style={styles.container}>
      {sessionStarted && (
        <SmkitCameraView
          ref={cameraRef}
          authKey="YOUR_API_KEY"
          exercise="SquatRegular"
          phonePosition="Floor"
          userHeight={175}
          onDetectionData={handleDetectionData}
          onDetectionStopped={handleDetectionStopped}
          onError={(error) => console.error(error)}
          onPreviewReady={() => console.log('Camera ready')}
          style={styles.camera}
        />
      )}
      
      <View style={styles.stats}>
        <Text>Reps: {repCount}</Text>
        <Text>Score: {score}%</Text>
        <Text>{feedback}</Text>
      </View>

      <View style={styles.controls}>
        {!sessionStarted && (
          <Button title="Start Session" onPress={startSession} />
        )}
        {sessionStarted && !isDetecting && (
          <Button title="Start Exercise" onPress={startDetection} />
        )}
        {isDetecting && (
          <Button title="Stop Exercise" onPress={stopDetection} />
        )}
      </View>
    </View>
  );
}

const styles = StyleSheet.create({
  container: { flex: 1 },
  camera: { flex: 1 },
  stats: { padding: 20, backgroundColor: 'white' },
  controls: { padding: 20 },
});

For a more advanced example with multiple exercises, see the example app in the repository.

Architecture

SmkitCameraView uses a component-based architecture with imperative ref methods:

1. Component State: React component manages UI state (selected exercise, rep count, etc.)
2. Ref Commands: Use cameraRef.current?.startSession() to control native layer
3. Event Callbacks: Receive continuous onDetectionData updates and onDetectionStopped summary
4. Lifecycle Management: Session and detection automatically clean up on unmount

Troubleshooting

"SMKit not configured" error

• Call configure(key) at app startup before rendering SmkitCameraView

Camera not showing

• Ensure onLayout or onPreviewReady callbacks are firing
• Check that camera permissions are granted in Info.plist on iOS and requested at runtime on Android

No detection data

• Verify correct exercise type is passed to startDetection()
• Check user height and phone position are correct
• Ensure user is in frame with good lighting

Performance issues

• Keep Android showNativeSkeletonOverlay enabled when drawing skeletons
• Avoid subscribing to onPositionData unless the screen needs per-joint data
• Consider reducing the frequency of UI updates from onDetectionData

Platform Support

• iOS: Full support
• Android: CameraX preview, SMKit Android 1.7.1 detection, position events, native skeleton overlay, config overrides, and calibration events

License

See LICENSE file for details.