@movementinfra/expo-audiosync

High-performance audio synchronization for React Native/Expo using Rust-based spectral fingerprinting.

Overview

This Expo module wraps the AudioSync Rust library, providing native iOS functionality to find the time offset between two audio files with high accuracy. It uses a Shazam-style algorithm with spectral fingerprinting for robust matching even with background noise.

Platform Support: iOS only (Android not yet implemented)

Simulator Support: Not included in npm package. See Local Development to enable.

Supported Audio Formats: MP3, M4A/AAC, WAV

Installation

npx expo install @movementinfra/expo-audiosync

This module requires a development build — it won't work with Expo Go.

npx expo prebuild
npx expo run:ios

Quick Start

Full Sync Pipeline (Recommended)

Sync a video with a reference song in one call:

import { syncVideoWithAudio } from '@movementinfra/expo-audiosync';

const result = await syncVideoWithAudio(
  '/path/to/dance-video.mp4',   // Video with misaligned audio
  '/path/to/song.mp3',          // Reference song
  '/path/to/synced-output.mp4'  // Output path
);

console.log(`Synced with ${result.offsetMs}ms offset`);
console.log(`Confidence: ${result.confidence}`);

Audio-Only Offset Detection

Find the time offset between two audio files:

import { findOffset, isAvailable } from '@movementinfra/expo-audiosync';

if (!isAvailable()) {
  console.log('AudioSync is not available on this platform');
  return;
}

const result = await findOffset(
  '/path/to/reference.mp3',  // Full song
  '/path/to/video-audio.mp3' // Audio extracted from video
);

console.log(`Offset: ${result.offsetMs}ms`);
console.log(`Confidence: ${result.confidence}`);

API Reference

Core Functions

syncVideoWithAudio(videoPath, audioPath, outputPath, config?)

Full pipeline: extracts audio from video, detects offset, creates synced video.

const result = await syncVideoWithAudio(videoPath, audioPath, outputPath);
// Returns: { outputPath, offsetMs, confidence, correlation, matchCount }

findOffset(referencePath, queryPath, config?)

Find the time offset between two audio files.

const result = await findOffset(referencePath, queryPath);
// Returns: { offsetMs, offsetSamples, confidence, correlation, matchCount }

extractAudio(videoPath, outputPath, options?)

Extract audio track from a video file.

const result = await extractAudio(videoPath, outputPath, { format: 'm4a' });
// Returns: { outputPath, durationMs, sampleRate, channels }

createSyncedVideo(videoPath, audioPath, outputPath, offsetMs)

Create a synced video with a known offset.

const result = await createSyncedVideo(videoPath, audioPath, outputPath, offsetMs);
// Returns: { outputPath, durationMs, offsetAppliedMs }

isAvailable()

Check if the module is available (iOS only).

getVersion()

Get the native library version string.

Configuration

interface SyncConfig {
  sampleRate?: number;           // Default: 16000
  fftSize?: number;              // Default: 1024
  hopSize?: number;              // Default: 256
  numThreads?: number;           // Default: 2
  refinementEnabled?: boolean;   // Default: true
  refinementWindowSecs?: number; // Default: 1.5
  minConfidence?: number;        // Default: 1.5
}

Error Handling

import { AudioSyncError, AudioSyncErrorCode } from '@movementinfra/expo-audiosync';

try {
  const result = await findOffset(refPath, queryPath);
} catch (error) {
  if (error instanceof AudioSyncError) {
    switch (error.code) {
      case AudioSyncErrorCode.NoMatch:
        console.log('No matching audio found');
        break;
      case AudioSyncErrorCode.AudioTooShort:
        console.log('Audio must be at least 500ms');
        break;
      // ... handle other cases
    }
  }
}

Error Codes: FileReadError, DecodeError, AudioTooShort, NoMatch, LowConfidence, InvalidConfig, InternalError, PlatformNotSupported, NoAudioTrack, NoVideoTrack, ExportFailed

Local Development with Simulator

The npm package only includes device binaries to minimize size. To enable iOS Simulator support for local development:

# Clone the repository
git clone https://github.com/movementinfra/audiosync.git
cd audiosync

# Build with simulator support (requires Rust)
./build-ios.sh --release

# Link to your project
cd your-expo-app
npm install ../audiosync/expo-module
npx expo prebuild --clean
npx expo run:ios

Requirements: Rust 1.70+ with aarch64-apple-ios-sim target, Apple Silicon Mac.

Example App

An example app is included in the repository at expo-module/example/. See its README for setup instructions.

How It Works

The library uses a Shazam-style audio fingerprinting algorithm:

1. Audio Loading — Decode audio files and convert to mono
2. Spectrogram — Compute Short-Time Fourier Transform (STFT)
3. Peak Detection — Find local maxima in time-frequency domain
4. Fingerprinting — Generate 32-bit hashes from peak pairs
5. Matching — Build offset histogram from hash matches
6. Refinement — Cross-correlation for sub-sample precision

Building from Source

Prerequisites

• Rust 1.70+ with iOS targets: rustup target add aarch64-apple-ios aarch64-apple-ios-sim x86_64-apple-ios
• Xcode command line tools
• Node.js 18+

Build Steps

# 1. Build Rust library for iOS
./build-ios.sh --release

# 2. Build TypeScript module
cd expo-module
npm install
npm run build

Troubleshooting

"Module not found" error

Ensure you're using a development build, not Expo Go:

npx expo prebuild --clean
npx expo run:ios

"No match found" errors

• Ensure audio files have at least 500ms overlap
• Check that query audio contains recognizable content from reference
• Try lowering minConfidence in config

Linker errors with libc++

Add to your app's Podfile:

post_install do |installer|
  installer.pods_project.targets.each do |target|
    target.build_configurations.each do |config|
      config.build_settings['OTHER_LDFLAGS'] ||= ['$(inherited)']
      config.build_settings['OTHER_LDFLAGS'] << '-lc++'
    end
  end
end

License

MIT

Credits

Built with Rust, Symphonia, RustFFT, and Expo Modules.