@audiolabtools/stresstest

Stress-test your own speech recogniser or voice agent on a fixed set of recordings (noise, reverb, phone line, packet loss), and compare before and after every change.

Status: in development. The package is marked private, and the public set it is built for is not released yet. The runner works on any folder that follows the format below.

What stays on your machine

• The runner talks only to the system you name (--command, or --base-url for an OpenAI-compatible endpoint). It makes no request to AudioLab unless you add --share (see below). A test checks that the only URLs in the code are the OpenAI default and the --share route.
• Your key is read from an environment variable, sent only to that endpoint, and never written to the report or to disk.
• With --command, each recording is written to a temporary file, passed to your command and deleted. The arguments of your command are not recorded in the report (they may hold a key); name the system with --model instead.
• The report is a local JSON file. Nothing is uploaded unless you add --share.

Use

# any command that prints the transcript of the file it is given (a local Whisper, your own agent)
audiolab-stresstest run --set ./set --provider command --command "my-asr {file}" --model my-asr-small

# an OpenAI-compatible transcription endpoint; the key comes from the environment
OPENAI_API_KEY=... audiolab-stresstest run --set ./set --provider openai --model whisper-1 --language en

# after a change, run again and pair the recordings with the earlier report
audiolab-stresstest run --set ./set --provider openai --previous baseline.json --strict
audiolab-stresstest show baseline.json

For people who build a set: audiolab-stresstest check --set ./set checks the format and the hashes, that every condition has the same utterances with the same text, and that every recording carries a licence and an attribution. It exits 1 and lists the problems if not.

--limit 24 runs the first 24 utterances of every condition: quick and coarse, and the report says so. --strict exits with code 3 if a criterion failed or a condition got worse than the previous run, for use in CI.

A recording that gets no answer is left out of the scores, and the report says which and why (failed_recordings, and a note): a time-out, an exit code of your command, an HTTP status. The reason is never the message of the failure, because the message of a failed command is the command line, and that may hold a key. The command provider gives one call 120 seconds and makes no retries (the openai provider retries 429 and 5xx three times); --timeout-s 300 gives a slow recogniser on a busy machine more, and the limit is recorded with the system in the report. A condition with unanswered recordings is compared with clean on the utterances answered in both.

Share the numbers with your account

Optional, off by default. --share keeps the numbers of a run in your AudioLab account, so the dashboard on audiolab.tools can show the history of your runs.

export AUDIOLAB_KEY=...        # or --key <key>
audiolab-stresstest run --set ./set --provider command --command "my-asr {file}" --share

• Sent: the numbers of the report: per condition the word and character error rates with their intervals, the change against clean, the criteria and their outcomes, the ten worst recordings by id and error counts, the set id and version, the runner version, your label and the name you gave the system (--model, --system-version).
• Not sent: no audio, no transcripts, no reference sentences, no per-recording rows, and no settings of your command. The runner removes these before it sends anything, and the server keeps only a list of numbers on top of that.
• The key is checked before the run starts: without one, --share stops with an error and nothing is run. If sharing fails (a wrong key, too many runs, no connection), the local report stays as it is, the failure is printed with its status and code, and the exit code of the run does not change.
• Remove it: delete the run on the dashboard in your account. The local report is yours and is not touched.
• --share-url <url> points it at another server (for tests).

Does a change help? A cleaning step, a model update, new settings

Run the same set twice and let the second run pair with the first:

audiolab-stresstest run --set ./set --provider command --command "./asr-only.sh {file}"       --label "without cleaning" --out without.json
audiolab-stresstest run --set ./set --provider command --command "./clean-then-asr.sh {file}" --label "with cleaning" --out with.json --previous without.json

Both scripts take the path of one recording and print the transcript; the second cleans the audio first. The comparison table puts the WER of both runs side by side for every condition, with the paired change, its interval and a verdict (worse, better, same). Two rows are worth reading first: the clean row says whether the step changes anything for recordings that were fine ("clean in, clean out"), and the noisy rows say where it helps. --strict exits with code 3 if any condition, the clean one included, got clearly worse. The labels come from --label; a previous report without one reads as "previous run".

What the report says

• WER per condition with a 95% interval (bootstrap over utterances; seed and number of resamples are in the report), the character error rate (CER) next to it, and the errors split into substitutions, deletions and insertions as a share of the reference words (sdi_pct; per recording too). Insertions are words that were written and not said, which is how much of a background voice ends up in the text. The total is unique but the split is not (two alignments can tie), so the backtrace rule is fixed and the same as the reference scorer's: from the end of both word lists, take the diagonal step when its cost matches, otherwise a deletion, otherwise an insertion.
• Per condition, the paired change against the clean condition on the same utterances, with its interval and an exact sign test. The verdict is worse when the interval lies above zero and the change is at least degradation.min_difference_points (2 by default), better for the mirror image, otherwise same.
• With --previous: the same paired comparison between two runs, per condition, with the same three verdicts. It refuses to pair reports of another set, version, manifest or criteria file (their hashes are in every report).
• The criteria written in criteria.json before the first run, with passed or failed.
• The ten recordings that got worse the most, with what was said and what was heard.
• 10 ms of digital silence is put in front of every recording, so a provider cannot answer from a cache; the report says so.
• A request that fails after retries is counted and left out of the scores. A reply with no words is scored as every word missing.

The report has the same keys and rounding as the one the set's Python reference scorer (score.py) writes, so one reader opens both: show renders a report from either tool, and either tool takes the other's report as --previous. Checked on the same 192 answers of Whisper small on the English core set (15 October 2026): errors, words, characters and the split of the errors per recording, WER, CER, the shares of the split, change against clean, sign tests, verdicts, criteria and the hashes are identical. The split of the errors is identical on the 79 cases of parity.json and on 3,000 random word pairs that include ties (npm run crosscheck), which also compares the change against clean in 400 random reports with unanswered recordings (only utterances answered in both count; at an exact tie the last digit can differ, because the runner rounds half up and Python half to even). The intervals differ slightly because the two bootstraps use different random generators.

What it is not

A stress test of your system on this set, not a prediction of how it behaves on your real calls. The set is read speech with edited conditions: dialects, real phone calls and real overlapping speakers are not in it. A condition names how the audio was edited; it is not a claim that AudioLab detects it.

The set format

A folder with manifest.json, criteria.json and the recordings (16-bit PCM WAV).

manifest.json: set_id, version, conditions[] (id, description, parameters) and recordings[] (id, utterance_id, reference_text, path inside the folder, condition, sha256, plus source, licence and attribution). Recordings are checked against their hashes before the run (--skip-verify turns that off), and a path that leaves the folder is refused.

criteria.json (its SHA-256 goes into every report): normalization (unicode, lowercase, strip_punctuation, keep_apostrophes, numbers: "as_is", keep_combining_marks and drop_zero_width_joiners (both true) and a version label, see below; a rule this version does not know is an error; with keep_apostrophes the typographic apostrophes U+2018, U+2019 and U+02BC count as the plain '), reference_condition (default clean), degradation.min_difference_points, bootstrap.seed and .resamples, fresh_bytes_silence_ms, coarse_below_n, and criteria[] (id, condition, measure wer or vs_clean, threshold: the condition passes when the value is at most the threshold).

Other languages and scripts

Since version 0.2 of the normalisation (15 October 2026) the runner and the Python reference scorer treat other scripts the same way, always:

• Combining marks (the vowel signs, the virama, the anusvara and the nukta of Devanagari, Tamil, Bengali, Telugu, Malayalam, Gurmukhi and others) belong to the word. Before, they counted as punctuation and a word fell apart: "हिन्दी" (one word) became five single letters, in this runner and in the reference scorer alike.
• The zero-width characters U+200B to U+200D, U+2060 and U+FEFF are deleted without a space. Recognisers disagree about them and they change no word.

criteria.json declares it with keep_combining_marks: true, drop_zero_width_joiners: true and a version label; false is refused, because the reference scorer cannot do it. The report shows a character error rate (CER) next to the word error rate for every condition, with no interval yet, and the character counts per recording: the code points of the normalised words joined without spaces. It is what counts for languages with long words (Tamil, Malayalam) and for scripts without spaces; the verdicts and the criteria still use the word error rate.

What the scorer does not do, and what you decide: it does not transliterate (Hindi in Latin letters against Devanagari counts as wrong), it does not convert numerals (the digit 5 and the word for five differ), and it does not know code-switching. Write the references the way your recogniser writes. npm run crosscheck (needs Python 3) tokenises 4,000 random mixed-script strings with the runner and with the Python reference, and with the reference scorer itself when it is on the machine: the same words and the same characters.

Your own recordings, in any language

The English core set cannot tell you what noise costs a recogniser in Hindi, Tamil or Dutch. build-set makes a set of the same kind from your own clean recordings and what was said in them:

audiolab-stresstest build-set --audio ./clean --texts ./texts.csv --out ./myset --language hi [--ingredients ./pack] [--conditions core|extended|a,b,c] [--n 48]
audiolab-stresstest check --set ./myset
audiolab-stresstest run --set ./myset --provider command --command "my-asr {file}" --model my-asr

• Texts: a CSV or TSV with two columns (file name, text; UTF-8; a header line file,text is optional), or a folder of .txt files with the same base names. A recording without a text is left out, and the builder says so.
• Recordings: WAV, FLAC, OGG or anything ffmpeg reads, at any rate and with any number of channels. The set has them at 16 kHz mono 16-bit, at an active level of -26 dBFS. Sentences of 4 to 10 seconds work best, and at least 48 of them give intervals that are not marked coarse. --n takes a seeded sample.
• Conditions: core is the eight of the English set (clean, telephone band, street noise at 0 dB, restaurant noise at 5 and 0 dB, a competing voice at 5 and 0 dB, packet loss of about 10%). extended adds low levels (-40, -50, -60 dBFS), clipping (+12, +20, +30 dB), telephone codecs (A-law, AMR-NB, AMR-WB, GSM, Opus), room impulse responses in four bins of reverberation time, echo, and a cut-off head or tail. The competing voice is another recording of your own set, so it speaks your language; it is a different recording, not necessarily a different speaker, so use recordings of several speakers if the competing-voice conditions are to mean something.
• Ingredients: street noise, restaurant noise and (for extended) room impulse responses come from an ingredient pack (--ingredients). Without one, those conditions are not built, and the manifest says which were skipped and why. A codec your ffmpeg cannot encode is reported in the same way and never replaced by another. Nothing is downloaded for you: make-pack assembles a pack from corpora you fetch yourself. The noise is DEMAND, whose Zenodo record names two licences (CC BY 4.0 in its Rights field, CC BY-SA 3.0 in its description; read on 15 October 2026), so a pack made from it is marked "shareable": false until that is settled; for noise whose licence you know, pass --noise-license and --noise-attribution. For the rooms you choose: synthetic rooms made with a specified reverberation time (--rooms-json rooms.json --rooms-license --rooms-attribution; the file lists id, spec_t60_by_band_s, spec_drr_db and the sha256 of one WAV per room, which every room is checked against, and a room goes in the bin of its specified time at 1 kHz), a folder of your own impulse responses with the licence you declare (--ir-folder --ir-license --ir-attribution), or RIRS_NOISES with --databases. RIRS_NOISES is declared Apache 2.0 as a whole, but as far as we could read on the same date the databases inside it have terms of their own (RWCP: research and development use only; AIR: no licence stated; REVERB 2014: not checked), so a pack made from them is also marked not shareable. A pack may be shared only when every file in it is CC BY (without NC, ND or SA), CC0 or public domain. pack.json names the licence of every file, and so does the ATTRIBUTION.md of the set. To use noise you recorded yourself, write the pack by hand: a folder with pack.json (groups: street, restaurant, rt_a to rt_d, each a list of paths to 16 kHz WAV files inside the folder; files: the license and attribution of each path) and the files.
• Recordings that are not one clean voice: a stereo recording is mixed down to mono, so cut one speaker out of a call recording first. The clean condition is your recording at 16 kHz, mono and the common level, nothing else.
• Private: the manifest says "private": true and the licence of every recording is user-provided. Nothing is uploaded; the runner sends the recordings only to the system you name.
• Reproducible: the same recordings, texts, pack and seed give the same files, byte for byte, and the hash of the builder is in the manifest. criteria.json carries the normalisation of the English set (version 0.2) and no thresholds: the English ones do not carry over, so write your own at the baseline run.
• Time and room: core takes seconds. extended runs ffmpeg for the telephone band and the codecs: about a minute for 10 recordings of 7 to 8 seconds, and about 8 MB of audio per recording. Use --n for a sample.

What it does not do: validate the conditions for your language. They were chosen on English read speech, after trials with open Whisper models, and what they cost a recogniser in another language is exactly what you are about to measure. The manifest says so in a note, and the report repeats it. Scripts written without spaces (Chinese, Japanese, Thai) are a limit of the report: its verdicts and criteria use the word error rate, which means little there, and the character error rate beside it has no interval and no verdict yet.

Needs Python 3 with numpy, scipy and soundfile (pip install numpy scipy soundfile), and ffmpeg for the telephone band and the codecs. STRESSTEST_PYTHON names another interpreter. audiolab-stresstest build-set --help lists the options.

Tests

npm test            # the runner, and the Python tools when Python 3 with numpy, scipy and soundfile is there
npm run test:tools  # the Python tools alone

No network. The tests use a synthetic set in a temporary folder and a fake system. testing/parity.json (text pairs with the errors a reference scorer counted) is picked up when it exists. npm run sabotage breaks one rule at a time in a copy of the package, the Python tools included, and checks that a test goes red (SABOTAGE_FILTER=<regex on the name> runs only the entries it matches).