@audio/scrub

Hearing a moment: what the sound under a held or dragged caret plays. Eight ways, from a tape head to a phase vocoder with noise, every channel its own, a block at a time, in real time. Compared, timed and measured in the lab: Hearing a moment.

import scrub from '@audio/scrub'

let voice = scrub([left, right], 44100, caret)   // vocoder + noise; or { method: 'tape' }, …
let out = [new Float32Array(128), new Float32Array(128)]
voice.render(out, caret)                         // the next block; move the caret between blocks

In an AudioWorklet, the voice runs inside process(): nothing allocates while it renders. A still caret sustains the moment; a moving one plays the sound at the caret's speed.

Methods

| Method | name | What a held caret plays | µs a block, 1 ch / 2 ch | |---|---|---|---| | tape | Tape | Nothing: the read head chases the caret through two 20 ms lags, so pitch follows speed (Audacity's Scrub) | 1 / 1 | | loop | Loop | The 80 ms around the caret again and again, joined by 8 ms equal-power crossfades (Reaper's looped-segment scrub) | 0.5 / 0.6 | | grains | Grains | 80 ms Hann grains, one every 20 ms, from the caret ± 15 ms [1] | 2.4 / 3 | | bank | Noisc bank | A noiscillator [2] per band of a log grid, 40 Hz–16 kHz, at the power the caret's spectrum holds there: a noise vocoder [3], reassigned [4] so a partial lands in one band, at its frequency | 114 / 405 | | lines | Noisc lines | The tonal peaks as lines as wide as each peak is wider than a sine's [5, 6], followed from hop to hop [7], third-octave noise bands for the rest: sines plus noise [8] | 77 / 239 | | random | Random phase | The caret's spectrum at new random phases every hop (Paulstretch [9]) | 17 / 38 | | vocoder | Vocoder | Two frames a hop apart; each peak's phase advances as it did, its region locked to it [10]. Tones hold exactly; noise freezes into a ring | 28 / 64 | | hybrid | Vocoder + noise | The vocoder for the bins a tonal peak's own leakage explains, random phase for the rest. The closest to the sound around the caret (2.8 dB per bin, exact on sines) | 30 / 68 |

A block is 128 samples at 48 kHz (2,667 µs of sound), on an Apple M4 Max in Node 25. Each make(channels, sampleRate, caret, options) starts one; methods[name].name is what the lab calls it.

Options

• frame (2048), overlap (4), spread (0 ms), band (null, or [low, high] Hz): random phase and the vocoders. A spread analyses each frame anywhere within that many ms of the caret. A band keeps only its bins; set voice.band to change it while it plays.
• bands (256), analysis ('reassigned' or 'bins'): the bank.
• line ('rice', sine and noise; 'band'; 'fm', a Lorentzian line; 'ou', a drifting, nearly Gaussian line [11]): the bank and the lines.

Channels

Every channel keeps its own sound: what is in one channel alone plays in it alone. Tape, loop and grains read every channel at the same place, so the time between channels stays.

The bank and the lines hear each channel, and all of them summed. One oscillator per line plays what the channels share: into each channel at the share of the line's power it holds in common with the others (its coherence with the loudest channel there, over the frames around the caret), at its phase against that one. Each channel's own lines play the rest, so its power stays its own exactly. A tone is shared whole or not at all, as the channels are more alike there than not: two oscillators playing parts of one sine would sum as their phases fall. Held, the same sound stays the same, a sound panned stays panned, noise half shared stays 0.65–0.81 coherent (the source 0.71) and unrelated noises 0.02–0.03 (0.04).

The spectral ways hear each channel on its own, each peak turning by its own phase advance, each noise bin by its own random phase. Then the part of a channel that the channel loudest around a bin predicts, over the frames around the caret, turns as that one does: the least-squares transfer between them, shrunk to their coherence beyond what unrelated noises show over as many frames and bins [12]. A sound panned or the same in several channels turns as one and keeps the phase between them; channels holding different sounds (a reverb's two sides, a part doubled 5 Hz apart, two unrelated noises) turn apart. Held, the output's coherence between channels stays within 0.1 of the source's over the second around the caret; with noise (hybrid), its level difference stays within 1 dB of the moment's in every octave band. Where every channel turns alike (the vocoder dragged at the sound's own speed), the source plays back unchanged.

Attacks

An onset is an event, not a moment. The vocoders find onsets as rises of 12 dB in the energy of every channel's first difference [13, 14], play each attack once, as recorded, when the caret passes or lands on it, and hear the frames without it, the gap filled from its edges [15]; a peak stays a steady tone only if the frame without the attacks keeps it [16]. Held on a click, the vocoder plays it once and then the sound around it; a frame holding the attack, played again every hop, would click 94 times a second at 48 kHz.

Tests

node test.js: levels, pitch, overlap, the vocoder's exact playback, every kind of line against its theory (Kubo's line shape), stereo coherence and level, clicks held, crackle on a slow scrub, seeded repeats.

References

1. C. Roads, Microsound, MIT Press (2001).
2. D. Iv, noiscillators: oscillation with uncertain frequency, a line of any width from a sine to noise.
3. R. V. Shannon, F.-G. Zeng, V. Kamath, J. Wygonski & M. Ekelid, "Speech recognition with primarily temporal cues," Science 270 (1995).
4. K. Kodera, R. Gendrin & C. de Villedary, "Analysis of time-varying signals with small BT values," IEEE Trans. ASSP 26 (1978).
5. J. O. Smith & X. Serra, "PARSHL: an analysis/synthesis program for non-harmonic sounds based on a sinusoidal representation," ICMC (1987).
6. F. J. Harris, "On the use of windows for harmonic analysis with the discrete Fourier transform," Proc. IEEE 66 (1978).
7. R. J. McAulay & T. F. Quatieri, "Speech analysis/synthesis based on a sinusoidal representation," IEEE Trans. ASSP 34 (1986).
8. X. Serra & J. O. Smith, "Spectral modeling synthesis: a sound analysis/synthesis system based on a deterministic plus stochastic decomposition," Computer Music Journal 14 (1990).
9. Nasca Octavian Paul, Paulstretch (2006).
10. J. Laroche & M. Dolson, "Improved phase vocoder time-scale modification of audio," IEEE Trans. Speech and Audio Processing 7 (1999).
11. P. W. Anderson, "A mathematical model for the narrowing of spectral lines by exchange or motion," J. Phys. Soc. Japan 9 (1954); R. Kubo, "Note on the stochastic theory of resonance absorption," J. Phys. Soc. Japan 9 (1954).
12. G. C. Carter, C. H. Knapp & A. H. Nuttall, "Estimation of the magnitude-squared coherence function via overlapped fast Fourier transform processing," IEEE Trans. Audio and Electroacoustics 21 (1973).
13. P. Masri, Computer modelling of sound for transformation and synthesis of musical signals, PhD thesis, University of Bristol (1996).
14. J. P. Bello, L. Daudet, S. Abdallah, C. Duxbury, M. Davies & M. B. Sandler, "A tutorial on onset detection in music signals," IEEE Trans. Speech and Audio Processing 13 (2005).
15. F. Nagel & A. Walther, "A novel transient handling scheme for time stretching algorithms," AES 127th Convention (2009).
16. A. Röbel, "A new approach to transient processing in the phase vocoder," DAFx (2003).