life2film-engine

On-device video analysis, compiled to WebAssembly. Finds the beat in music, the cuts in a video, and scores frames for picture quality — in a browser tab or in Node, with no server, no upload and no API key.

This is the analysis engine behind Life2Film and the free tools on that site. Those pages are the fastest way to see what it does: drop a file in and watch it work, then come back here to call it yourself.

Licence: noncommercial. Free for hobby projects, study, research, charities and public institutions. Commercial use needs a separate licence — see Licence below. Ask, and you will probably get one.

npm i life2film-engine

Beat detection

Takes mono f32 PCM, which is exactly what AudioContext.decodeAudioData gives you.

import init, { detect_beats } from 'life2film-engine';

await init();

const ctx = new AudioContext();
const buffer = await ctx.decodeAudioData(await file.arrayBuffer());

// Down-mix to mono at 22050 Hz — the onset envelope lives far below that, and the source
// rate costs several times the memory for an identical answer.
const offline = new OfflineAudioContext(1, Math.ceil(buffer.duration * 22050), 22050);
const source = offline.createBufferSource();
source.buffer = buffer;
source.connect(offline.destination);
source.start();
const mono = (await offline.startRendering()).getChannelData(0);

const { bpm, beats } = JSON.parse(detect_beats(mono, 22050, null));
// bpm: 103.7
// beats: [0.51, 1.09, 1.67, ...] seconds

Tempo is genuinely ambiguous — a track at 140 BPM is also, correctly, 70 in half time — so treat a result that is out by exactly 2× as the other reading rather than an error.

Shot boundaries

Compare the centre of each sampled frame. Sampling rate matters far more than the algorithm: at two samples per second the boundaries land one to three seconds out; at ten per second they land on the frame.

import init, { detect_scenes_slick, score_frame } from 'life2film-engine';

await init();

// For each sampled frame: mean RGB of the centre column, plus its timestamp.
const { scenes } = JSON.parse(detect_scenes_slick(JSON.stringify({
  pixels,        // [{ r, g, b }, ...]
  timestamps,    // [0, 0.1, 0.2, ...] seconds
  duration,
  algo: 'adjacent_diffs',
})));
// scenes: [[0, 3.0], [3.0, 6.0], ...]

Sixteen algorithms are available (scene_algorithms() lists them). adjacent_diffs is the default and the one to start with.

Frame quality

Thirty-one measurements per frame — sharpness, exposure, contrast, colourfulness, entropy and more — combined into one score, plus a flag for frames not worth keeping.

import init, { score_frame, feature_names } from 'life2film-engine';

await init();

// RGB24, no alpha. 64 px wide is enough: these are large-scale properties.
const { score, features, is_garbage } = JSON.parse(
  score_frame(width, height, rgb24, timestamp, null),
);

Loading it

The package is built for the browser: init() fetches the .wasm alongside the module, which bundlers resolve for you. To point at it explicitly:

import init from 'life2film-engine';
await init({ module_or_path: '/path/to/va_wasm_bg.wasm' });

It is 1.2 MB uncompressed, roughly 410 KB over the wire. Load it when the user acts, not on page load.

In Node it works too, but fetch cannot read a file:// URL, so hand it the bytes:

import init, { detect_beats } from 'life2film-engine';
import { readFileSync } from 'node:fs';
import { createRequire } from 'node:module';

const require = createRequire(import.meta.url);
await init({ module_or_path: readFileSync(require.resolve('life2film-engine/va_wasm_bg.wasm')) });

Node 18 or newer.

Everything else

Nineteen exports in total, typed in va_wasm.d.ts:

| Function | Does | |---|---| | detect_beats | tempo and beat positions | | analyze_audio | silence, sections, spectral flatness, content type | | beat_sync_timeline | place cuts on the beat | | detect_scenes_slick | shot boundaries from centre pixels | | detect_scenes_content | shot boundaries from HSV deltas | | detect_scenes_features | shot boundaries from feature vectors | | score_frame | 31 per-frame measurements | | score_segments, select_segments | rank and choose segments | | compose_montage | assemble a timeline | | build_otio, parse_otio | OpenTimelineIO in and out |

What it does not do

It does not decode video or audio — bring your own frames and PCM. In a browser, <video> plus a canvas, or WebCodecs, will do it; the tools pages show both. It does not encode anything either; for that, mediabunny is excellent and MIT licensed.

It reads no speech and tracks no faces. It looks at pictures and sound.

Licence

PolyForm Noncommercial 1.0.0. Permitted: personal projects, study, research, charities, educational institutions, public bodies. Not permitted: anything commercial, including internal use inside a company.

For a commercial licence, write to [email protected]. This is a real offer, not a formality — the licence is noncommercial so that the terms are a conversation rather than an assumption.

The package ships compiled WebAssembly. The Rust it is built from is not public.

Built by SuperDuperAi. See it running at life2film.com/tools.