expo-whisper# expo-whisper

🎤 High-performance speech-to-text transcription for React Native/Expo apps using OpenAI's Whisper model High-performance speech-to-text transcription for React Native/Expo apps using OpenAI's Whisper model

✅ Build Fix (v1.0.10)## Features

This version completely fixes the "Unresolved reference 'whisper'" build errors! The entire package has been rewritten following official Expo Modules API standards.- Real-time transcription with streaming audio support

• Cross-platform - Works on iOS, Android, and Web

🚀 Features- High accuracy using OpenAI's Whisper models

• Native performance with C++ integration

• ✅ Native performance with C++ Whisper integration- Easy integration with React hooks

• ✅ Cross-platform - Works on iOS and Android- Multi-language support (100+ languages)

• ✅ Offline capable - No internet required after model loading- Flexible configuration for different use cases

• ✅ High accuracy using OpenAI's Whisper models- Offline capable - No internet required

• ✅ Easy integration with React hooks

• ✅ Expo SDK 51+ compatible## Getting Started

• ✅ No more build errors - completely fixed!

Prerequisites

📦 Installation

Before you begin, make sure you have:

# Install the package- **Development build** environment set up (expo-whisper cannot run in Expo Go)

npm install expo-whisper- **Android Studio** (for Android builds) or **Xcode** (for iOS builds)



# Install peer dependencies### Step 1: Installation

npx expo install expo-build-properties

``````bash

# Install the package

## ⚙️ Configurationnpm install expo-whisper



Add the plugin to your `app.json`:# Install required peer dependencies

npx expo install expo-build-properties

```json```

{

  "expo": {### Step 2: Configure app.json

    "name": "Your App",

    "plugins": [Add the expo-whisper plugin to your `app.json` configuration:

      [

        "expo-build-properties",```json

        {{

          "android": {  "expo": {

            "minSdkVersion": 24,    "name": "Your App",

            "compileSdkVersion": 36,    "plugins": [

            "targetSdkVersion": 36      [

          },        "expo-build-properties",

          "ios": {        {

            "deploymentTarget": "11.0"          "android": {

          }            "minSdkVersion": 24,

        }            "compileSdkVersion": 34,

      ],            "targetSdkVersion": 34,

      "expo-whisper"            "buildToolsVersion": "34.0.0",

    ]            "enableProguardInReleaseBuilds": true,

  }            "packagingOptions": {

}              "pickFirst": ["**/libc++_shared.so", "**/libjsc.so"]

```            }

          },

## 🎯 Basic Usage          "ios": {

            "deploymentTarget": "11.0"

### Simple API          }

        }

```tsx      ],

import { ExpoWhisper } from 'expo-whisper';      "expo-whisper"

    ],

// Load a model    "android": {

await ExpoWhisper.loadModel('/path/to/ggml-model.bin');      "permissions": [

        "android.permission.RECORD_AUDIO",

// Transcribe audio file        "android.permission.READ_EXTERNAL_STORAGE",

const text = await ExpoWhisper.transcribe('/path/to/audio.wav');        "android.permission.WRITE_EXTERNAL_STORAGE"

console.log('Transcription:', text);      ]

    },

// Release model when done    "ios": {

await ExpoWhisper.releaseModel();      "infoPlist": {

```        "NSMicrophoneUsageDescription": "This app needs access to microphone for speech recognition."

      }

### With React Hook    }

  }

```tsx}

import { useWhisper } from 'expo-whisper';```



export default function App() {### Step 3: Download a Whisper Model

  const {

    loadModel,Choose a model based on your performance and accuracy needs:

    transcribe,

    releaseModel,| Model | Size | Speed | Accuracy | Use Case |

    isModelLoaded,|-------|------|-------|----------|----------|

    isLoading,| `tiny.en` | 39 MB | Fastest | Basic | Real-time, mobile |

    error,| `base.en` | 74 MB | Fast | Good | Mobile apps |

    lastResult,| `small.en` | 244 MB | Medium | Better | General purpose |

  } = useWhisper();| `medium.en` | 769 MB | Slow | High | High accuracy needed |



  const handleLoadModel = async () => {**Download options:**

    try {```bash

      await loadModel('/path/to/your/model.bin');# Option 1: Download to assets folder

      console.log('Model loaded successfully!');mkdir -p assets/models

    } catch (error) {curl -L "https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin" -o assets/models/ggml-base.en.bin

      console.error('Failed to load model:', error);

    }# Option 2: Use react-native-fs to download at runtime

  };npm install react-native-fs

const handleTranscribe = async () => {

if (!isModelLoaded) {### Step 4: Build Your App

  console.warn('Model not loaded');

  return;**Important**: expo-whisper requires a development build:

}

    try {# For Android

      const result = await transcribe('/path/to/audio.wav');npx expo run:android

      console.log('Transcription result:', result);

    } catch (error) {# For iOS  

      console.error('Transcription failed:', error);npx expo run:ios

    }

  };# Or use EAS Build

eas build --platform android --profile development

  return (eas build --platform ios --profile development

    <View style={{ flex: 1, justifyContent: 'center', alignItems: 'center' }}>```

      <Text>Model loaded: {isModelLoaded ? '✅' : '❌'}</Text>

      <Text>Loading: {isLoading ? '⏳' : '✅'}</Text>>  **Cannot use Expo Go** - Native libraries require a development build

      {error && <Text style={{ color: 'red' }}>Error: {error}</Text>}

      {lastResult && <Text>Result: {lastResult}</Text>}### Step 5: Basic Implementation

      

      <Button title="Load Model" onPress={handleLoadModel} />```typescript

      <Button title="Transcribe" onPress={handleTranscribe} />import React, { useEffect, useState } from 'react';

    </View>import { View, Button, Text, Alert } from 'react-native';

  );import { useWhisper } from 'expo-whisper';

}import { Audio } from 'expo-av';

export default function App() {

🔧 API Reference const {

isModelLoaded,

ExpoWhisper Class isTranscribing,

lastResult,

loadModel(modelPath: string): Promise<void> error,

Load a Whisper model from the given path. loadModel,

transcribeFile,

transcribe(audioPath: string): Promise<string> clearError,

Transcribe an audio file and return the text. } = useWhisper();

transcribeFile(filePath: string): Promise<string> const [recording, setRecording] = useState<Audio.Recording>();

Alias for transcribe() method.

useEffect(() => {

transcribePCM(pcmData: number[], sampleRate?: number): Promise<string> // Load model when app starts

Transcribe raw PCM audio data (float array). const initializeWhisper = async () => {

  try {

isModelLoaded(): boolean // Path to your downloaded model

Check if a model is currently loaded. await loadModel('/path/to/assets/models/ggml-base.en.bin');

    Alert.alert('Success', 'Whisper model loaded successfully!');

releaseModel(): Promise<void> } catch (error) {

Release the loaded model and free memory. Alert.alert('Error', Failed to load model: ${error.message});

  }

useWhisper Hook };

Returns an object with: initializeWhisper();

• State: isModelLoaded, isLoading, error, lastResult }, []);

• Actions: loadModel, transcribe, transcribeFile, transcribePCM, releaseModel

  const startRecording = async () => {

📁 Model Setup try {

  const permission = await Audio.requestPermissionsAsync();

1. Download a Whisper model (GGML format): if (permission.status !== 'granted') {

   
   # Example: Base English model (~74MB)        return;
   
   curl -L "https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin" -o ggml-base.en.bin      }
   

   await Audio.setAudioModeAsync({

2. Place in your app's accessible directory: allowsRecordingIOS: true,

   • iOS: Document Directory or Bundle playsInSilentModeIOS: true,

   • Android: Internal storage or assets folder });

3. Use the correct path when calling loadModel() const { recording } = await Audio.Recording.createAsync(

    Audio.RECORDING_OPTIONS_PRESET_HIGH_QUALITY

🏗️ Building );

  setRecording(recording);

# Create development build      console.error('Failed to start recording', err);

npx expo prebuild --clean    }

npx expo run:android  };

npx expo run:ios

```  const stopRecording = async () => {

    if (!recording) return;

**Note**: This package requires a development build. It cannot run in Expo Go.

    setRecording(undefined);

## 🔧 Troubleshooting    await recording.stopAndUnloadAsync();

    

### Build Errors Fixed ✅    const uri = recording.getURI();

    if (uri) {

- ✅ "Unresolved reference 'whisper'" - **FIXED in v1.0.10**      try {

- ✅ MainApplication.kt compilation errors - **FIXED**        const result = await transcribeFile(uri, {

- ✅ Kotlin/Java compatibility issues - **FIXED**          language: 'en',

          temperature: 0.0,

### Common Issues        });

        Alert.alert('Transcription', result.text);

1. **Model not loading**: Ensure the model file exists and is accessible      } catch (error) {

2. **Audio not transcribing**: Check file format (WAV recommended) and permissions        Alert.alert('Error', `Transcription failed: ${error.message}`);

3. **Memory issues**: Release models when not needed with `releaseModel()`      }

    }

## 📋 Requirements  };



- **Expo SDK 51+** or **React Native 0.74+**  return (

- **Android**: Min SDK 24, Target SDK 36    <View style={{ flex: 1, padding: 20, justifyContent: 'center' }}>

- **iOS**: Deployment target 11.0+      <Text style={{ fontSize: 18, marginBottom: 20, textAlign: 'center' }}>

- **Development build** (cannot use Expo Go)        Whisper Speech-to-Text

      </Text>

## 🤝 Contributing

      <Text style={{ marginBottom: 10 }}>

1. Fork the repository        Model Status: {isModelLoaded ? ' Loaded' : ' Loading...'}

2. Create your feature branch (`git checkout -b feature/amazing-feature`)      </Text>

3. Commit your changes (`git commit -m 'Add amazing feature'`)

4. Push to the branch (`git push origin feature/amazing-feature`)      <Button

5. Open a Pull Request        title={recording ? 'Stop Recording' : 'Start Recording'}

        onPress={recording ? stopRecording : startRecording}

## 📄 License        disabled={!isModelLoaded || isTranscribing}

      />

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

      {lastResult && (

## 🙏 Acknowledgments        <View style={{ marginTop: 20, padding: 10, backgroundColor: '#f0f0f0' }}>

          <Text style={{ fontWeight: 'bold' }}>Last Transcription:</Text>

- [OpenAI Whisper](https://github.com/openai/whisper) for the amazing speech recognition model          <Text>{lastResult.text}</Text>

- [whisper.cpp](https://github.com/ggerganov/whisper.cpp) for the C++ implementation        </View>

- Expo team for the excellent modules framework      )}



---      {error && (

        <View style={{ marginTop: 20 }}>

**Need help?** [Open an issue](https://github.com/poovarasan4046/expo-whisper/issues) or check our [troubleshooting guide](https://github.com/poovarasan4046/expo-whisper/blob/main/REAL_WORLD_EXAMPLE.md).          <Text style={{ color: 'red' }}>Error: {error}</Text>
          <Button title="Clear Error" onPress={clearError} />
        </View>
      )}
    </View>
  );
}

Step 6: Test Your Implementation

1. Build and install your development build
2. Grant microphone permissions when prompted
3. Tap "Start Recording" and speak clearly
4. Tap "Stop Recording" to see transcription results

Configuration Options

Advanced app.json Setup

For production apps, you may want additional configuration:

{
  "expo": {
    "plugins": [
      [
        "expo-build-properties",
        {
          "android": {
            "minSdkVersion": 24,
            "compileSdkVersion": 34,
            "targetSdkVersion": 34,
            "proguardMinifyEnabled": true,
            "enableProguardInReleaseBuilds": true,
            "packagingOptions": {
              "pickFirst": [
                "**/libc++_shared.so",
                "**/libjsc.so",
                "**/libfbjni.so"
              ]
            }
          },
          "ios": {
            "deploymentTarget": "11.0",
            "bundler": "metro"
          }
        }
      ],
      [
        "expo-whisper",
        {
          "modelPath": "assets/models/ggml-base.en.bin",
          "enableMicrophone": true,
          "enableAudioSession": true
        }
      ]
    ]
  }
}

EAS Build Configuration

Create eas.json for cloud builds:

{
  "cli": {
    "version": ">= 5.9.0"
  },
  "build": {
    "development": {
      "developmentClient": true,
      "distribution": "internal",
      "android": {
        "buildType": "developmentBuild"
      },
      "ios": {
        "buildConfiguration": "Debug"
      }
    },
    "preview": {
      "distribution": "internal",
      "android": {
        "buildType": "apk"
      }
    },
    "production": {
      "android": {
        "buildType": "app-bundle"
      }
    }
  }
}

Development Build Required

expo-whisper uses native libraries and cannot run in Expo Go. You must use a development build:

Why Development Build is Required:

• Native Libraries: expo-whisper includes compiled C++ libraries (libwhisper.so)
• Custom Native Code: Direct integration with Whisper C++ implementation
• Platform-specific Optimizations: Hardware-accelerated audio processing

Setting Up Development Build:

# Install development build tools
npx expo install expo-dev-client

# Build for development
npx expo run:android  # Local Android build
npx expo run:ios      # Local iOS build

# Or use EAS Build (recommended)
eas build --platform android --profile development
eas build --platform ios --profile development

Next Steps

After completing the setup:

1. Explore the API: Check out the API Reference section
2. Real-time Streaming: Learn about streaming transcription
3. Performance Optimization: Read our Audio Guide
4. Real-world Examples: See REAL_WORLD_EXAMPLE.md

API Reference

Hooks

useWhisper()

The main hook for managing Whisper transcription state and actions.

Returns:

{
  // State
  isModelLoaded: boolean;
  isLoading: boolean;
  isTranscribing: boolean;
  error: string | null;
  lastResult: WhisperResult | null;

  // Actions
  loadModel: (modelPath: string) => Promise<void>;
  transcribe: (audioPath: string, config?: WhisperConfig) => Promise<WhisperResult>;
  transcribeFile: (audioPath: string, config?: WhisperConfig) => Promise<WhisperResult>;
  transcribePCM: (pcmData: Uint8Array, config?: WhisperPCMConfig) => Promise<WhisperResult>;
  releaseModel: () => Promise<void>;
  getModelInfo: () => Promise<string>;
  getSupportedFormats: () => Promise<string>;
  clearError: () => void;
}

Core API

ExpoWhisper.loadModel(modelPath: string)

Load a Whisper model from the given file path.

ExpoWhisper.transcribeFile(audioPath: string, config?: WhisperConfig)

Transcribe an audio file. Supports WAV, MP3, M4A formats.

ExpoWhisper.transcribePCM(pcmData: Uint8Array, config?: WhisperPCMConfig)

Transcribe raw PCM audio data for real-time streaming.

Configuration Options

WhisperConfig

interface WhisperConfig {
  language?: string;           // Language code (e.g., 'en', 'es', 'auto')
  temperature?: number;        // Sampling temperature (0.0 - 1.0)
  maxTokens?: number;         // Maximum tokens to generate
  beamSize?: number;          // Beam search size
  bestOf?: number;            // Number of candidates
  patience?: number;          // Beam search patience
  lengthPenalty?: number;     // Length penalty
  suppressTokens?: number[];  // Tokens to suppress
  initialPrompt?: string;     // Initial prompt text
  wordTimestamps?: boolean;   // Include word-level timestamps
  preprendPunctuations?: string;
  appendPunctuations?: string;
}

WhisperPCMConfig

interface WhisperPCMConfig {
  sampleRate: number;         // Audio sample rate (16000 recommended)
  channels: number;           // Number of channels (1 for mono)
  realtime?: boolean;         // Enable real-time processing
  chunkSize?: number;         // Processing chunk size
}

Audio Format Support

Supported Formats:

• WAV (recommended): 16kHz, 16-bit PCM, mono/stereo
• MP3: Various bitrates and sample rates
• M4A: AAC encoded audio
• Raw PCM: For streaming applications

Optimal Settings for Best Performance:

• Sample Rate: 16kHz
• Bit Depth: 16-bit
• Channels: Mono (1 channel)
• Format: WAV or raw PCM for streaming

Advanced Usage

Real-time Streaming

import { useWhisper } from 'expo-whisper';

function StreamingComponent() {
  const { transcribePCM } = useWhisper();

  const processAudioChunk = async (audioData: Uint8Array) => {
    const result = await transcribePCM(audioData, {
      sampleRate: 16000,
      channels: 1,
      realtime: true,
    });

    console.log('Live transcription:', result.text);
  };
}

Batch Processing

const processMultipleFiles = async (filePaths: string[]) => {
  const results = [];

  for (const path of filePaths) {
    try {
      const result = await transcribeFile(path, {
        language: 'auto',
        temperature: 0.0,
      });
      results.push({ path, text: result.text });
    } catch (error) {
      results.push({ path, error: error.message });
    }
  }

  return results;
};

Custom Configuration Presets

// High accuracy preset
const highAccuracyConfig = {
  temperature: 0.0,
  beamSize: 5,
  bestOf: 5,
  wordTimestamps: true,
};

// Fast processing preset
const fastConfig = {
  temperature: 0.1,
  beamSize: 1,
  bestOf: 1,
  maxTokens: 224,
};

// Real-time streaming preset
const streamingConfig = {
  sampleRate: 16000,
  channels: 1,
  realtime: true,
  chunkSize: 1024,
};

Platform Support

| Platform | Status | Native Library | |----------|--------|----------------| | iOS | | libwhisper.a | | Android | | libwhisper.so | | Web | | WASM (planned) |

Development

Building from Source

# Clone the repository
git clone https://github.com/poovarasan4046/expo-whisper.git
cd expo-whisper

# Install dependencies
npm install

# Build the package
npm run build

# Run tests
npm test

Native Dependencies

The package includes pre-compiled Whisper libraries:

• Android: libwhisper.so (ARM64, ARMv7)
• iOS: libwhisper.a (ARM64, x86_64)

Contributing

We welcome contributions! Please see our Contributing Guide for details.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• OpenAI Whisper - The amazing speech recognition model
• whisper.cpp - C++ implementation
• Expo - Development platform

Troubleshooting

Common Build Issues

"Unresolved reference 'whisper'" in MainApplication.kt

Error: Unresolved reference 'whisper' at line 18 in MainApplication.kt

Cause: This usually happens when:

1. You manually imported expo-whisper modules in MainApplication.kt
2. There's an incorrect import statement
3. The module isn't properly auto-registered

Solution:

1. Remove manual imports: Don't manually import expo-whisper in MainApplication.kt
2. Use Expo's auto-registration: Let Expo handle module registration automatically
3. Check your imports: Remove any lines like:

   import expo.modules.whisper  // Remove this

4. Clean and rebuild:

   cd android && ./gradlew clean
   cd .. && npx expo run:android

minSdkVersion Compatibility

Error: User has minSdkVersion X but library was built for Y

Solution: Ensure your app uses minSdkVersion 24 or higher:

{
  "expo": {
    "plugins": [
      ["expo-build-properties", {
        "android": {
          "minSdkVersion": 24
        }
      }]
    ]
  }
}

Native Library Loading Issues

Error: UnsatisfiedLinkError: couldn't find libwhisper_jni.so

Solution:

1. Ensure you're using a development build (not Expo Go)
2. Clean and rebuild the project
3. Check that native libraries are properly copied

Getting Help

If you encounter other issues:

1. Check our GitHub Issues
2. Search for similar problems in the community
3. Create a new issue with detailed error logs

Support

• Documentation
• Audio Guide
• Issue Tracker
• Discussions

Made with for the React Native community