@sency/react-native-smkit

Integrate SMKit pose detection and exercise analysis into your React Native application with a simple, component-based API.

Features

• ✅ Real-time pose detection and movement tracking
• ✅ Exercise-specific feedback and form analysis
• ✅ Rep counting with completion detection
• ✅ Session-based architecture with automatic lifecycle management
• ✅ TypeScript support
• ✅ iOS support (Android coming soon)

Installation

npm install @sency/react-native-smkit
# or
yarn add @sency/react-native-smkit

iOS Setup

cd ios && pod install && cd ..

Quick Start

1. Configure SMKit

Call configure() early in your app lifecycle (e.g., on app startup):

import { configure } from '@sency/react-native-smkit';

useEffect(() => {
  configure('YOUR_AUTH_KEY').catch(err => {
    console.error('SMKit configuration failed:', err);
  });
}, []);

⚠️ Required: configure() must be called before using SmkitCameraView.

2. Add the Camera View

Use the SmkitCameraView component to integrate pose detection:

import { SmkitCameraView, type SmkitCameraViewRef } from '@sency/react-native-smkit';
import { useRef } from 'react';

const MyApp = () => {
  const cameraRef = useRef<SmkitCameraViewRef>(null);

  return (
    <SmkitCameraView
      ref={cameraRef}
      authKey="YOUR_AUTH_KEY"
      exercise="SquatRegular"
      phonePosition="Floor"
      userHeight={175}
      onDetectionData={(data) => {
        console.log('Movement feedback:', data.feedback);
        console.log('Technique score:', data.techniqueScore);
      }}
      style={{ flex: 1 }}
    />
  );
};

3. Control Sessions

// Start camera session
cameraRef.current?.startSession();

// Start movement detection
cameraRef.current?.startDetection('SquatRegular');

// Receive exercise summary when detection stops
const handleDetectionStopped = (summary) => {
  console.log('Session ended');
  console.log('Total time:', summary.totalTime, 'seconds');
  console.log('Technique score:', summary.techniqueScore);
};

// Stop detection
cameraRef.current?.stopDetection();

// Stop session
cameraRef.current?.stopSession();

API Reference

SmkitCameraView Component

<SmkitCameraView
  ref={cameraRef}
  authKey={string}
  exercise={string}
  phonePosition={PhonePosition}
  userHeight={number}
  autoStart={boolean}
  onDetectionData={callback}
  onDetectionStopped={callback}
  onError={callback}
  onPreviewReady={callback}
  onLayout={callback}
  style={StyleProp<ViewStyle>}
/>

Props

| Prop | Type | Required | Description | |------|------|----------|-------------| | ref | React.Ref | Yes | Ref to call imperative methods | | authKey | string | Yes | SMKit authentication key | | exercise | string | No | Exercise type to detect (see Supported Exercises) | | phonePosition | 'Floor' | 'Elevated' | No | Phone position relative to user (default: 'Floor') | | jumpRefPoint | 'HIP' | 'KNEE' | 'ANKLE' | No | Reference point for jump exercises | | jumpHeightThreshold | number | No | Minimum jump height threshold in cm | | userHeight | number | No | User height in cm for form analysis | | autoStart | boolean | No | Auto-start session on mount (default: false) | | onDetectionData | (data: MovementFeedbackData) => void | No | Called when movement data is detected | | onPositionData | (data: JointData) => void | No | Called with joint position data for advanced features | | onDetectionStopped | (summary: ExerciseSummary) => void | No | Called when detection stops with session summary | | onError | (error: string) => void | No | Called when errors occur | | onPreviewReady | () => void | No | Called when camera preview is ready | | onLayout | (event: LayoutChangeEvent) => void | No | Standard RN layout event | | style | StyleProp | No | View styling |

Ref Methods (SmkitCameraViewRef)

startSession(): void
// Starts the camera session and initializes detection

stopSession(): void
// Stops the camera session and cleans up resources

startDetection(exercise: string): void
// Begins detecting the specified exercise

stopDetection(): void
// Stops detection and returns exercise summary

Supported Exercises

Map user-friendly names to native SMKit exercise types:

const EXERCISE_TYPE_MAP: Record<string, string> = {
  'Squat': 'SquatRegular',
  'Pushup': 'PushupRegular',
  'JumpingJacks': 'JumpingJacks',
  'Plank': 'PlankHighStatic',
  'HighKnees': 'HighKnees',
};

// Pass the mapped native type to startDetection:
cameraRef.current?.startDetection(EXERCISE_TYPE_MAP['Squat']);

MovementFeedbackData

Real-time feedback for each detected movement frame:

interface MovementFeedbackData {
  didFinishMovement: boolean;     // Rep completed
  isShallowRep: boolean;          // Form issue: shallow rep
  isInPosition: boolean;          // User in correct position
  isPerfectForm: boolean;         // Perfect form detected
  techniqueScore: number;         // Technique score (0-100)
  detectionConfidence: number;    // Detection confidence (0-1)
  feedback: string[];             // User feedback messages
  currentRomValue: number;         // Current range of motion
  specialParams: Record<string, number>; // Exercise-specific params
}

ExerciseSummary

Summary data returned when detection stops:

interface ExerciseSummary {
  sessionId: string;              // Unique session ID
  exerciseName: string;           // Exercise that was detected
  startTime: string;              // ISO 8601 start timestamp
  endTime: string;                // ISO 8601 end timestamp
  totalTime: number;              // Duration in seconds
  techniqueScore: number;         // Average technique score (0-100)
  feedbacks?: Record<string, number>; // Aggregated feedback data
}

JointData

Joint position data for advanced pose tracking:

interface JointPosition {
  x: number;  // Normalized x coordinate (0-1)
  y: number;  // Normalized y coordinate (0-1)
}

interface JointData {
  [jointName: string]: JointPosition;
}

Available joints include: Nose, Neck, RShoulder, RElbow, RWrist, LShoulder, LElbow, LWrist, RHip, RKnee, RAnkle, LHip, LKnee, LAnkle, REye, LEye, REar, LEar, Hip, Chest, Head, LBigToe, RBigToe, LSmallToe, RSmallToe, LHeel, RHeel.

configure(authKey: string)

Initialize SMKit before using the camera view:

import { configure } from '@sency/react-native-smkit';

try {
  await configure('YOUR_AUTH_KEY');
  // SMKit is now ready
} catch (error) {
  console.error('Configuration failed:', error);
}

Complete Example

Here's a complete working example with session management and rep counting:

import React, { useRef, useState, useEffect } from 'react';
import { View, Text, Button, StyleSheet } from 'react-native';
import {
  SmkitCameraView,
  configure,
  type SmkitCameraViewRef,
  type MovementFeedbackData,
  type ExerciseSummary,
} from '@sency/react-native-smkit';

const EXERCISE_TYPE_MAP = {
  'Squat': 'SquatRegular',
  'Pushup': 'PushupRegular',
  'JumpingJacks': 'JumpingJacks',
  'Plank': 'PlankHighStatic',
  'HighKnees': 'HighKnees',
};

export default function ExerciseApp() {
  const cameraRef = useRef<SmkitCameraViewRef>(null);
  const [sessionStarted, setSessionStarted] = useState(false);
  const [isDetecting, setIsDetecting] = useState(false);
  const [repCount, setRepCount] = useState(0);
  const [feedback, setFeedback] = useState('');
  const [score, setScore] = useState(0);

  // Configure on mount
  useEffect(() => {
    configure('YOUR_API_KEY').catch(console.error);
  }, []);

  const handleDetectionData = (data: MovementFeedbackData) => {
    if (data.didFinishMovement) {
      setRepCount(prev => prev + 1);
    }
    setFeedback(data.feedback[0] || '');
    setScore(Math.round(data.techniqueScore));
  };

  const handleDetectionStopped = (summary: ExerciseSummary) => {
    console.log('Session complete!');
    console.log('Total time:', summary.totalTime, 'seconds');
    console.log('Average score:', summary.techniqueScore);
    setIsDetecting(false);
  };

  const startSession = () => {
    setSessionStarted(true);
    setTimeout(() => {
      cameraRef.current?.startSession();
    }, 100);
  };

  const startDetection = () => {
    setIsDetecting(true);
    cameraRef.current?.startDetection(EXERCISE_TYPE_MAP['Squat']);
  };

  const stopDetection = () => {
    cameraRef.current?.stopDetection();
  };

  return (
    <View style={styles.container}>
      {sessionStarted && (
        <SmkitCameraView
          ref={cameraRef}
          authKey="YOUR_API_KEY"
          exercise="SquatRegular"
          phonePosition="Floor"
          userHeight={175}
          onDetectionData={handleDetectionData}
          onDetectionStopped={handleDetectionStopped}
          onError={(error) => console.error(error)}
          onPreviewReady={() => console.log('Camera ready')}
          style={styles.camera}
        />
      )}
      
      <View style={styles.stats}>
        <Text>Reps: {repCount}</Text>
        <Text>Score: {score}%</Text>
        <Text>{feedback}</Text>
      </View>

      <View style={styles.controls}>
        {!sessionStarted && (
          <Button title="Start Session" onPress={startSession} />
        )}
        {sessionStarted && !isDetecting && (
          <Button title="Start Exercise" onPress={startDetection} />
        )}
        {isDetecting && (
          <Button title="Stop Exercise" onPress={stopDetection} />
        )}
      </View>
    </View>
  );
}

const styles = StyleSheet.create({
  container: { flex: 1 },
  camera: { flex: 1 },
  stats: { padding: 20, backgroundColor: 'white' },
  controls: { padding: 20 },
});

For a more advanced example with multiple exercises, see the example app in the repository.

Architecture

SmkitCameraView uses a component-based architecture with imperative ref methods:

1. Component State: React component manages UI state (selected exercise, rep count, etc.)
2. Ref Commands: Use cameraRef.current?.startSession() to control native layer
3. Event Callbacks: Receive continuous onDetectionData updates and onDetectionStopped summary
4. Lifecycle Management: Session and detection automatically clean up on unmount

Troubleshooting

"SMKit not configured" error

• Call configure(key) at app startup before rendering SmkitCameraView

Camera not showing

• Ensure onLayout or onPreviewReady callbacks are firing
• Check that camera permissions are granted in Info.plist

No detection data

• Verify correct exercise type is passed to startDetection()
• Check user height and phone position are correct
• Ensure user is in frame with good lighting

Performance issues

• Consider reducing the frequency of UI updates from onDetectionData
• Ensure no heavy computations in callback handlers

Platform Support

• iOS: ✅ Full support
• Android: 🔜 Coming soon

License

See LICENSE file for details.