react-native-audio-intelligence

Native audio analysis for React Native via Nitro Modules. Decodes WAV files on-device, extracts DSP features (RMS, transient density, windowing), detects content heuristics, scores prominence, and selects highlight segments — all without leaving the JS thread blocked.

Features

• WAV decoding — iOS via AVFoundation; Android via a built-in 16-bit PCM parser
• Overlapping window analysis — 1.5 s windows with 0.75 s hop
• Transient density — per-window sharp energy-change detection (transients/s)
• Event detection — ONNX Runtime inference (audio_classifier.onnx) with DSP heuristic fallback: speech, music, impact, ambient, silence
• Prominence scoring — weighted score in [0, 1] with quality penalties
• Highlight selection — up to 3 non-overlapping peak windows
• Synchronous version info — getVersion() and getPlatformInfo() run on the native thread

Requirements

• React Native ≥ 0.76 (tested on 0.85)
• react-native-nitro-modules ≥ 0.35.2
• ONNX Runtime (installed automatically via CocoaPods on iOS and Gradle on Android)
  • iOS: onnxruntime-objc ~> 1.22.0
  • Android: onnxruntime-android 1.22.0

Installation

npm install react-native-audio-intelligence react-native-nitro-modules
# or
yarn add react-native-audio-intelligence react-native-nitro-modules

Then rebuild the native app:

npx pod-install   # iOS
npx react-native run-ios
npx react-native run-android

Quick start

import AudioIntelligence from 'react-native-audio-intelligence'

const version = AudioIntelligence.getVersion()
// => "1.1.0"

const result = await AudioIntelligence.analyzeAudioFile('/path/to/audio.wav')
console.log(result.score) // 0.0 – 1.0 prominence
console.log(result.detectedEvents) // [{ label: "speech", confidence: 0.47 }, ...]
console.log(result.windows.length) // overlapping analysis windows

The file path must be an absolute path readable by native code. React Native asset bundles are not supported directly — copy or record to a temp path first.

Platform file paths

| Platform | Example path | Notes | | -------- | -------------------------- | ----------------------------------------------- | | iOS | /tmp/real_test.wav | Simulator/device temp directory | | Android | /data/local/tmp/test.wav | Push with adb push audio.wav /data/local/tmp/ |

# Push a test file to an Android device/emulator
adb push ./fixtures/test.wav /data/local/tmp/test_transient.wav

API

getVersion(): string

Returns the native module version synchronously.

getPlatformInfo(): PlatformInfo

Returns the host OS name and version.

{ platform: 'ios' | 'android', version: string }

analyzeAudioFile(path: string): Promise<AudioAnalysisResult>

Analyzes a WAV file at the given absolute path. Resolves with a full result object; rejects if the file is missing, unreadable, or not a supported format.

Supported formats

• iOS: any format AVFoundation can decode (WAV, AAC, MP3, etc.)
• Android: 16-bit PCM WAV only

Types

AudioAnalysisResult

| Field | Type | Description | | ---------------- | ----------------- | -------------------------------------------------- | | score | number | Overall prominence score, clamped to [0, 1] | | tags | string[] | Top-level tags (e.g. ["voice"], ["silence"]) | | detectedEvents | DetectedEvent[] | Heuristic content detections, sorted by confidence | | highlights | Highlight[] | Up to 3 peak windows suitable for preview clips | | quality | AudioQuality | Clipping, silence, and wind-noise flags | | windows | AudioWindow[] | Overlapping per-window DSP metrics |

AudioWindow

| Field | Type | Description | | ------------------ | -------- | ----------------------------------- | | startMs | number | Window start offset in milliseconds | | durationMs | number | Window length in milliseconds | | rms | number | Root-mean-square energy [0, 1] | | transientDensity | number | Sharp energy rises per second |

DetectedEvent

{
  label: string
  confidence: number
} // confidence in [0, 1]

Labels: speech, music, impact, ambient, silence.

Highlight

{ startMs: number; durationMs: number; score: number; tags: string[] }

AudioQuality

{
  clipping: boolean
  windNoise: boolean
  silence: boolean
}

Example

A minimal screen that runs analysis on launch and renders the results inline. Adjust TEST_WAV_PATH for your platform.

import React, { useCallback, useEffect, useState } from 'react'
import {
  ActivityIndicator,
  Platform,
  ScrollView,
  StyleSheet,
  Text,
  View,
} from 'react-native'
import AudioIntelligence, {
  type AudioAnalysisResult,
} from 'react-native-audio-intelligence'

const TEST_WAV_PATH =
  Platform.OS === 'ios'
    ? '/tmp/real_test.wav'
    : '/data/local/tmp/test_transient.wav'

const LOG = '[AudioIntelligence]'

function logAnalysis(result: AudioAnalysisResult, platform: string) {
  console.log(`${LOG} ── Analysis complete (${platform}) ──`)
  console.log(`${LOG} Prominence score: ${result.score.toFixed(5)}`)
  console.log(`${LOG} Tags: ${result.tags.join(', ') || '—'}`)
  console.log(
    `${LOG} Quality: clipping=${result.quality.clipping} ` +
      `silence=${result.quality.silence} wind=${result.quality.windNoise}`
  )
  console.log(`${LOG} Windows (${result.windows.length}):`)
  result.windows.forEach((w, i) => {
    console.log(
      `${LOG}   [${i}] ${w.startMs.toFixed(0)}ms–` +
        `${(w.startMs + w.durationMs).toFixed(0)}ms  ` +
        `rms=${w.rms.toFixed(4)}  transients=${w.transientDensity.toFixed(2)}/s`
    )
  })
  if (result.highlights.length > 0) {
    console.log(`${LOG} Highlights (${result.highlights.length}):`)
    result.highlights.forEach((h, i) => {
      console.log(
        `${LOG}   [${i}] ${h.startMs.toFixed(0)}ms  score=${h.score.toFixed(3)}`
      )
    })
  }
  console.log(`${LOG} Events (${result.detectedEvents.length}):`)
  result.detectedEvents.forEach((e, i) => {
    console.log(
      `${LOG}   [${i}] ${e.label} (${(e.confidence * 100).toFixed(1)}%)`
    )
  })
}

function formatPercent(value: number) {
  return `${(value * 100).toFixed(1)}%`
}

export default function App() {
  const [status, setStatus] = useState<'idle' | 'loading' | 'done' | 'error'>(
    'idle'
  )
  const [error, setError] = useState<string | null>(null)
  const [result, setResult] = useState<AudioAnalysisResult | null>(null)
  const [version, setVersion] = useState<string | null>(null)

  const runAnalysis = useCallback(async () => {
    setStatus('loading')
    setError(null)
    setResult(null)

    try {
      const v = AudioIntelligence.getVersion()
      setVersion(v)
      console.log(`${LOG} Module v${v} · ${Platform.OS}`)
      console.log(`${LOG} File: ${TEST_WAV_PATH}`)

      const analysis = await AudioIntelligence.analyzeAudioFile(TEST_WAV_PATH)
      logAnalysis(analysis, Platform.OS)
      setResult(analysis)
      setStatus('done')
    } catch (e) {
      const message = e instanceof Error ? e.message : String(e)
      console.error(`${LOG} Analysis failed:`, message)
      setError(message)
      setStatus('error')
    }
  }, [])

  useEffect(() => {
    runAnalysis()
  }, [runAnalysis])

  return (
    <View style={styles.root}>
      <View style={styles.header}>
        <Text style={styles.title}>Audio Intelligence</Text>
        {version && <Text style={styles.subtitle}>v{version}</Text>}
      </View>

      {status === 'loading' && (
        <View style={styles.centered}>
          <ActivityIndicator size="large" color="#6C8EFF" />
          <Text style={styles.muted}>Analyzing audio…</Text>
        </View>
      )}

      {status === 'error' && (
        <View style={styles.card}>
          <Text style={styles.errorLabel}>Analysis failed</Text>
          <Text style={styles.errorText}>{error}</Text>
        </View>
      )}

      {result && (
        <ScrollView
          style={styles.scroll}
          contentContainerStyle={styles.scrollContent}
        >
          <View style={styles.scoreCard}>
            <Text style={styles.scoreLabel}>Prominence</Text>
            <Text style={styles.scoreValue}>{formatPercent(result.score)}</Text>
            <Text style={styles.tags}>{result.tags.join(' · ')}</Text>
          </View>

          <Section title={`Events (${result.detectedEvents.length})`}>
            {result.detectedEvents.map((e, i) => (
              <Row
                key={`${e.label}-${i}`}
                label={e.label}
                value={formatPercent(e.confidence)}
              />
            ))}
          </Section>

          <Section title={`Windows (${result.windows.length})`}>
            {result.windows.map((w, i) => (
              <Row
                key={i}
                label={`${w.startMs.toFixed(0)}ms`}
                value={`rms ${w.rms.toFixed(3)} · ${w.transientDensity.toFixed(1)}/s`}
              />
            ))}
          </Section>

          {result.highlights.length > 0 && (
            <Section title={`Highlights (${result.highlights.length})`}>
              {result.highlights.map((h, i) => (
                <Row
                  key={i}
                  label={`${h.startMs.toFixed(0)}ms`}
                  value={formatPercent(h.score)}
                />
              ))}
            </Section>
          )}

          <Section title="Quality">
            <Row
              label="Clipping"
              value={result.quality.clipping ? 'yes' : 'no'}
            />
            <Row
              label="Silence"
              value={result.quality.silence ? 'yes' : 'no'}
            />
            <Row
              label="Wind noise"
              value={result.quality.windNoise ? 'yes' : 'no'}
            />
          </Section>
        </ScrollView>
      )}
    </View>
  )
}

function Section({
  title,
  children,
}: {
  title: string
  children: React.ReactNode
}) {
  return (
    <View style={styles.section}>
      <Text style={styles.sectionTitle}>{title}</Text>
      <View style={styles.sectionBody}>{children}</View>
    </View>
  )
}

function Row({ label, value }: { label: string; value: string }) {
  return (
    <View style={styles.row}>
      <Text style={styles.rowLabel}>{label}</Text>
      <Text style={styles.rowValue}>{value}</Text>
    </View>
  )
}

const styles = StyleSheet.create({
  root: {
    flex: 1,
    backgroundColor: '#0D0F14',
  },
  header: {
    paddingTop: 60,
    paddingBottom: 20,
    paddingHorizontal: 24,
    borderBottomWidth: StyleSheet.hairlineWidth,
    borderBottomColor: '#1E2230',
  },
  title: {
    fontSize: 28,
    fontWeight: '700',
    color: '#F0F2F8',
    letterSpacing: -0.5,
  },
  subtitle: {
    marginTop: 4,
    fontSize: 14,
    color: '#6C8EFF',
    fontWeight: '500',
  },
  centered: {
    flex: 1,
    justifyContent: 'center',
    alignItems: 'center',
    gap: 12,
  },
  muted: {
    color: '#6B7280',
    fontSize: 15,
  },
  scroll: {
    flex: 1,
  },
  scrollContent: {
    padding: 20,
    gap: 16,
    paddingBottom: 40,
  },
  scoreCard: {
    backgroundColor: '#161A24',
    borderRadius: 16,
    padding: 24,
    alignItems: 'center',
    borderWidth: 1,
    borderColor: '#1E2230',
  },
  scoreLabel: {
    fontSize: 13,
    fontWeight: '600',
    color: '#6B7280',
    textTransform: 'uppercase',
    letterSpacing: 1,
  },
  scoreValue: {
    fontSize: 48,
    fontWeight: '700',
    color: '#6C8EFF',
    marginTop: 4,
  },
  tags: {
    marginTop: 8,
    fontSize: 14,
    color: '#9CA3AF',
  },
  section: {
    backgroundColor: '#161A24',
    borderRadius: 12,
    overflow: 'hidden',
    borderWidth: 1,
    borderColor: '#1E2230',
  },
  sectionTitle: {
    fontSize: 13,
    fontWeight: '600',
    color: '#6B7280',
    textTransform: 'uppercase',
    letterSpacing: 0.8,
    paddingHorizontal: 16,
    paddingTop: 14,
    paddingBottom: 8,
  },
  sectionBody: {
    paddingBottom: 4,
  },
  row: {
    flexDirection: 'row',
    justifyContent: 'space-between',
    alignItems: 'center',
    paddingHorizontal: 16,
    paddingVertical: 11,
    borderTopWidth: StyleSheet.hairlineWidth,
    borderTopColor: '#1E2230',
  },
  rowLabel: {
    fontSize: 15,
    color: '#D1D5DB',
    textTransform: 'capitalize',
  },
  rowValue: {
    fontSize: 15,
    color: '#9CA3AF',
    fontVariant: ['tabular-nums'],
  },
  card: {
    margin: 20,
    backgroundColor: '#1C1014',
    borderRadius: 12,
    padding: 20,
    borderWidth: 1,
    borderColor: '#3B1C24',
  },
  errorLabel: {
    fontSize: 16,
    fontWeight: '600',
    color: '#F87171',
    marginBottom: 8,
  },
  errorText: {
    fontSize: 14,
    color: '#FCA5A5',
    lineHeight: 20,
  },
})

Sample console output

[AudioIntelligence] Module v0.3.0 · android
[AudioIntelligence] File: /data/local/tmp/test_transient.wav
[AudioIntelligence] ── Analysis complete (android) ──
[AudioIntelligence] Prominence score: 0.38922
[AudioIntelligence] Tags: voice
[AudioIntelligence] Quality: clipping=false silence=false wind=false
[AudioIntelligence] Windows (4):
[AudioIntelligence]   [0] 0ms–1500ms  rms=0.0413  transients=0.67/s
[AudioIntelligence]   [1] 750ms–2250ms  rms=0.0598  transients=1.33/s
[AudioIntelligence]   [2] 1500ms–3000ms  rms=0.0890  transients=1.33/s
[AudioIntelligence]   [3] 2250ms–3000ms  rms=0.1099  transients=1.33/s
[AudioIntelligence] Events (1):
[AudioIntelligence]   [0] speech (47.0%)

How scoring works

The prominence score combines five weighted components:

| Component | Weight | Source | | ------------------ | ------ | ---------------------------------------- | | Volume | 0.35 | Whole-file RMS | | Transients | 0.25 | Average transient density across windows | | Frequency richness | 0.20 | RMS variance across windows | | Voice presence | 0.10 | speech event confidence | | Contrast | 0.10 | Peak-to-floor RMS ratio across windows |

Penalties are applied for clipping (−0.15), silence (−0.50), and wind noise (−0.10). The final value is clamped to [0, 1].

Event detection runs ONNX Runtime inference using the bundled audio_classifier.onnx model (mono 16 kHz input). If the model cannot be loaded or inference fails, the module falls back to DSP heuristics so analysis still completes.

Error handling

analyzeAudioFile rejects when:

• The file does not exist at the given path
• The file cannot be decoded (unsupported format, corrupt data)
• On Android: the file is not 16-bit PCM WAV

Always use a try/catch (or .catch()) around the promise and surface the error message to the user.

License

MIT