A simple benchmark runner with reporting.

• Run your benchmarks
• Build a profile of your code's performance
• Run your benchmarks again as you change your code and have automatic reporting about when your performance shifts

$ npm install @k/bench-runner

CLI Usage

bench run [options]
Runs your benchmark suite

Options:
  --version         Show version number                                                                        [boolean]
  --help            Show help                                                                                  [boolean]
  --config          Your benchmark config file                                                                  [string]
  --name            The name of your benchmark (usually shows up in the test output)                            [string]
  --require         Any modules that should be required prior to running the benchmark                           [array]
  --files           Your benchmark test files                                                                    [array]
  --warn-threshold  The delta threshold below which a benchmark will throw a warning                            [number]
  --fail-threshold  The delta threshold below which a benchmark will fail                                       [number]
  --reporters       The list of reporters to use (these generate your test output)                               [array]
  --profile         The file that contains your benchmark profile you want to test against                      [string]
  --profile-out     If set, will additionally output a new benchmark profile to this file                       [string]

Example

You can see an example of what a benchmark looks like by looking in the example directory. The .bench-config.json file contains the configuration for the benchmark, and looks like this:

{
	"name": "Example Benchmark",
	"files": "example/**/*.js",
	"warnThreshold": 0.8,
	"failThreshold": 0.5,
	"reporters": [
		["cli", {
			"colors": true
		}]
	]
}

Running the benchmark can be done by with bench run --config ./example/.bench-config.json. You should see output something like this:

  Example Benchmark
  =================

  Suite: Array.forEach (2 pass, 0 warn, 0 fail)
   - Small Array (1000) x 1,903,577 ops/sec ±62.35% (88 runs sampled) [+0.00%]
   - Large Array (1000000) x 87.92 ops/sec ±0.38% (75 runs sampled) [+0.00%]

  Suite: Array.map (2 pass, 0 warn, 0 fail)
   - Small Array (1000) x 371,231 ops/sec ±8.71% (97 runs sampled) [+0.00%]
   - Large Array (1000000) x 63.38 ops/sec ±0.14% (65 runs sampled) [+0.00%]

  Benchmark Complete
  4 pass, 0 warn, 0 fail

Because this isn't running against an existing profile, every test passes. You can see the [+0.00%] on the end of each line. That's where the variance against your previous profile would normally show up. To get the real benefit though, we need a profile to compare against.

Building and Using Profiles

Building a profile is as easy as passing in an extra option when running your benchmarks

bench run ... --profile-out your-profile.json

This will create a new file your-profile.json containing your benchmark profile. You can then compare later runs against this profile by passing the file back in with the profile option.

bench run ... --profile your-profile.json

These later runs will compare the new performance against the expectations set in the profile, and if they vary too much, it will throw out a warning or an error. You can control how much variance to allow by setting the threshold options

bench run ... --warn-threshold 0.8 --fail-threshold 0.5

The thresholds are a ratio you expect to meet. Setting --warn-threshold to 0.8 means that if performance degrades down to 80% or lower of what's recorded in the profile, that test will throw a warning.

Don't set your thresholds too strictly, however. Microbenchmarks tend to give inherently variable results based on all kinds of various conditions you may not be aware of. This tool is meant as a sanity test more than anything; To catch unexpected, large shifts in relative performance.