av-drift-detector (Node.js)

Measure audio/video sync to a fraction of a millisecond, frame by frame.

Node.js bindings to av-drift-detector, the Rust library behind the av-drift command. Generate a test video whose every frame, and every frame's worth of audio, carries its own frame number. Play it through whatever you want to test (an encoder, a streaming pipeline, a TV, a video call), record the result, and measure how far the sound has drifted from the picture at every frame.

import avd from 'av-drift-detector'

await avd.generate({ output: 'sync.mp4', fps: '29.97', duration: 10, audioOffset: 0.04 })
// ... play sync.mp4 through the system under test and record it as captured.mp4 ...
const report = await avd.analyze('captured.mp4')
console.log(report.summary.meanMs) // positive means the audio is late

Install

npm install av-drift-detector

Node.js 20 or later, on macOS (Apple silicon and Intel), Linux (x64 and arm64, glibc 2.17+) or Windows (x64). TypeScript types are included.

Reading and writing files runs the ffmpeg and ffprobe programs, so install FFmpeg 5.1 or later (brew install ffmpeg, sudo apt install ffmpeg, winget install Gyan.FFmpeg) and keep them on your PATH, or set AV_DRIFT_FFMPEG and AV_DRIFT_FFPROBE to their paths. Writing H.264 needs an FFmpeg with libx264, as those packages have. The in-memory functions below don't need FFmpeg.

Files

• generate(options, onProgress?) writes a test video and resolves to a summary. Only output is required; the others default to fps: '30', duration: 10 (seconds), 1280×720, sampleRate: 48000, videoCodec: 'h264', audioCodec: 'aac', audioOffset: 0 (seconds; positive delays the audio) and music: true (a tune under the tones that doesn't affect analysis). onProgress({ framesDone, framesTotal }) is called about once a percent. FFV1 video and PCM audio need .mov or .mkv.
• analyze(path, options?) measures every frame of a capture. Options: offset and duration (a window, in seconds), fps (override the pattern's rate), useQr, and capture: 'auto', 'file' (a digital copy) or 'camera' (a phone filming a screen).

Both run on libuv's thread pool and return promises. Reports have the content of av-drift analyze --json (format) with camelCase names:

const { meanMs, stddevMs, skewPpm } = report.summary
for (const f of report.frames) {
  if (f.status === 'ok') console.log(f.time, f.videoFrame, f.driftMs)
}

Frames and audio from elsewhere

If you already have decoded frames and audio (a capture card, WebRTC, a camera), measure them without files:

const reader = new avd.StampReader()
for (const { time, luma } of myFrames) {
  reader.read(luma, width, height, time)      // 8-bit luma (Y) plane, time in seconds
}
const report = await avd.analyzeFrames({
  frames: reader.finish(),
  audio: { sampleRate: 48000, startTime: t0, samples: monoFloat32Array },
  width,
  height,
})

Frame times and startTime must be on the same clock, in seconds.

• new StampReader(tryQr?) reads the frames of one capture in order. read(luma, width, height, time, stride?, exposure?, readout?) returns the frame ({ time, stamp, qrLocated, exposure, readout }); finish() returns every frame read, with any that could only be identified from later frames filled in. It follows the QR code from frame to frame, which is faster than reading each frame alone, and it reads pictures a camera caught while the screen switched frames, measuring how far through the switch each was (the stamp's blend). Use it for camera footage. For a camera, time is when the picture's top row began its exposure, as phones' camera APIs record it; if the camera reports each frame's exposure time and rolling-shutter readout (seconds), pass them too, and a camera locked to the pattern's rate is timed to a millisecond or two.
• readFrame(luma, width, height, stride?, tryQr?) reads one frame's stamp (barcode, then QR code), or returns null.
• new PatternRenderer({ fps, width, height, sampleRate, duration }).render(frameId) draws a test-pattern frame as a luma Buffer. With duration (seconds), the info line shows the file's length.
• synthesizeAudio({ count, fps, sampleRate, delay, start, music }) makes the test audio as a Float32Array, with the music unless music is false.

Async functions reject on bad arguments; readFrame and the renderer throw.

More

The repository has the guides: measuring with a camera, the test pattern specification, and how the analysis works. Licensed under the BSD-2-Clause Plus Patent License.