Smart Turn Detection

A plug-and-play audio pipeline for real-time conversational turn detection. Automatically detects when a user has finished speaking using machine learning.

Features

• Real-time Turn Detection - Detects when someone has finished speaking
• Voice Activity Detection (VAD) - Built-in energy-based speech detection
• Dual Backend Support - WebGPU for speed, WASM for compatibility
• Web Worker Mel Spectrogram - Non-blocking audio processing with PFFFT.wasm
• Zero Configuration - Works out of the box with sensible defaults
• ONNX Runtime - Production-ready ML inference

Installation

npm install smart-turn-detection

Quick Start

import { SmartTurnPipeline } from 'smart-turn-detection';

const pipeline = new SmartTurnPipeline({
  onTurnComplete: (probability) => {
    console.log(`Turn complete! Confidence: ${(probability * 100).toFixed(1)}%`);
    // Trigger your response logic here
  },
  onTurnIncomplete: () => {
    console.log('User is still thinking...');
  },
  onSpeechStart: () => {
    console.log('User started speaking');
  },
  onSpeechEnd: () => {
    console.log('User stopped speaking');
  }
});

// Start listening
await pipeline.start();

// Stop when done
await pipeline.stop();

API Reference

SmartTurnPipeline

Main orchestrator class for turn detection.

Constructor Options

const pipeline = new SmartTurnPipeline({
  // Backend selection
  backend: 'auto', // 'auto' | 'webgpu' | 'wasm'
  
  // Custom model paths (optional)
  wasmModelPath: './custom-model-cpu.onnx',
  webgpuModelPath: './custom-model-gpu.onnx',
  
  // VAD configuration
  vad: {
    threshold: 0.008,              // Energy threshold for speech detection
    minSilenceDuration: 800,       // ms of silence before turn end
    minSpeechDuration: 300         // ms of minimum speech to trigger analysis
  },
  
  // Mel spectrogram configuration
  mel: {
    nMels: 80,                     // Number of mel bins
    nFft: 400,                     // FFT size
    hopLength: 160                 // Hop length in samples
  },
  
  // Callbacks
  onSpeechStart: () => {},
  onSpeechEnd: () => {},
  onTurnComplete: (probability) => {},
  onTurnIncomplete: () => {},
  onError: (error) => {},
  
  // Debug mode
  debug: false
});

Methods

• start() - Begin audio capture and turn detection
• stop() - Stop listening
• toggle() - Toggle listening state
• isActive() - Check if currently listening
• getBackendInfo() - Get current backend information
• dispose() - Cleanup all resources

Backends

WASM Backend (CPU)

import { WASMBackend } from 'smart-turn-detection';

const backend = new WASMBackend('./path/to/model.onnx');
await backend.init();
const probability = await backend.predict(melSpectrogram);

WebGPU Backend (GPU)

import { WebGPUBackend } from 'smart-turn-detection';

const backend = new WebGPUBackend('./path/to/model.onnx');
await backend.init();
const probability = await backend.predict(melSpectrogram);

Backend Selection

The pipeline automatically selects the best available backend:

1. WebGPU (preferred) - GPU acceleration for fastest inference (~30-50ms)
2. WASM (fallback) - CPU-based inference, universally supported (~100-200ms)

To force a specific backend:

const pipeline = new SmartTurnPipeline({
  backend: 'webgpu' // or 'wasm'
});

Architecture

┌─────────────────────────────────────────────────────────────┐
│                   SmartTurnPipeline                          │
├─────────────────────────────────────────────────────────────┤
│  AudioCapture → MelSpectrogram (Web Worker) → TurnDetector  │
├─────────────────────────────────────────────────────────────┤
│  Voice Activity Detection (VAD)                               │
├─────────────────────────────────────────────────────────────┤
│  ONNX Backend (WebGPU or WASM)                               │
└─────────────────────────────────────────────────────────────┘

AudioCapture - Captures microphone audio in real-time using Web Audio API MelSpectrogram - Computes mel spectrograms in a Web Worker using PFFFT.wasm TurnDetector - Runs VAD and coordinates ONNX inference Backends - WebGPU or WASM inference engines

How It Works

1. Audio Capture - Continuously captures 8 seconds of audio in a circular buffer
2. Voice Activity Detection - Detects when speech starts and stops using energy threshold
3. Mel Spectrogram - When silence is detected after speech, computes mel spectrogram in Web Worker
4. ONNX Inference - Runs the turn detection model to predict completion probability
5. Callbacks - Notifies your application of turn status

Browser Requirements

• Modern browsers with Web Audio API support
• WebGPU - Chrome 113+, Edge 113+, Firefox (nightly), Safari (technical preview)
• Microphone permission required

Package Contents

smart-turn-detection/
├── src/
│   ├── core/
│   │   ├── SmartTurnPipeline.js    # Main orchestrator
│   │   ├── AudioCapture.js         # Microphone capture
│   │   ├── MelSpectrogram.js       # Worker wrapper
│   │   └── TurnDetector.js         # VAD + inference coordinator
│   ├── backends/
│   │   ├── WASMBackend.js          # CPU inference
│   │   └── WebGPUBackend.js        # GPU inference
│   └── index.js                   # Package exports
├── dist/
│   ├── mel-worker.js              # Mel computation worker
│   ├── pffft.js                   # PFFFT standard
│   ├── pffft.simd.js              # PFFFT SIMD (recommended)
│   ├── pffft.wasm                 # PFFFT WASM
│   └── pffft.simd.wasm            # PFFFT SIMD WASM
└── models/
    ├── smart-turn-v3.2-cpu.onnx   # 8.3MB WASM model
    └── smart-turn-v3.2-gpu.onnx   # 31MB WebGPU model

Size Information

• Package Size: ~39.5MB (includes both CPU and GPU models)
• CPU-only Install: ~8.5MB (use bundler tree-shaking)
• Runtime Dependencies: onnxruntime-web (~3MB loaded from CDN)

Performance

Typical processing times on modern hardware:

• Mel Spectrogram: ~60-80ms (PFFFT.wasm)
• WASM Inference: ~100-200ms
• WebGPU Inference: ~30-50ms
• Total Latency: ~100-300ms from silence detection to result

Custom Models

You can use your own ONNX models:

const pipeline = new SmartTurnPipeline({
  wasmModelPath: './my-custom-model.onnx',
  backend: 'wasm'
});

Model requirements:

• Input: [1, 80, 800] float32 mel spectrogram
• Output: Single float probability (0-1)

Troubleshooting

WebGPU Not Available

The pipeline automatically falls back to WASM if WebGPU is not available.

Microphone Permission Denied

Ensure your site uses HTTPS (required for getUserMedia). Check browser permission settings.

Worker Loading Issues

If the mel worker fails to load, ensure your server serves .js and .wasm files with correct MIME types:

• .js: application/javascript
• .wasm: application/wasm

Development

Copy Assets from Web Demo

npm run copy-assets

This copies the necessary runtime files from smart-turn-web-demo.

Credits

This package implements the turn detection approach from pipecat-ai/smart-turn.

• Turn Detection Model: Based on pipecat-ai research
• Mel Spectrogram: Whisper-compatible implementation using PFFFT.wasm
• ONNX Runtime: Microsoft

License

MIT

Support

For issues and questions, please visit the GitHub repository.