Mutant

Note: this is very much at the proof-of-concept stage

What is it?

mutant is a mutation testing framework designed to help evaluate the quality of test suites for JavaScript applications.

What's the point?

Code coverage tools are useful to give an idea of how much of your code runs when your tests run, giving an overview of areas that may have been overlooked.

However, they do not provide a great deal of insight into how closely the code behaviour matches that which is delineated in the tests. A test simply triggering a code path is not enough to be confident the test adequately defines the required behaviour to an acceptable level of detail.

Mutation testing, on the other hand, is designed to test how far the code can stray from the original while the tests still pass. It does this by making incremental modifications to the code under test, and then repeatedly running the test suite against the new versions produced. If the tests pass for a mutated version, we may gain some insight into where there might be gaps in how the tests are designed.

Install

You can install mutant globally, but the preferred way is to install it as a local dev-dependency:

npm i -D mutant-test

You can then use it via a script in package.json:

{
  "scripts": {
    "mutant": "mutant"
  }
}

Or access the local binary directly using

./node_modules/.bin/mutant /path/to/test/file.js

Usage

You can use mutant alongside any test framework that can output results in TAP format.

Some examples for popular frameworks:

mocha --reporter tap
ava --tap
tap
tape

Currently mutant only supports running against a single test file at a time, so your test framework will also need to support passing the path to run a subset of tests. Watch this space for the ability to run full test suites.

Config

mutant expects a config file to exist at .mutant/config.js (might make this a little more flexible in the future). This is where you tell mutant how to run your tests via the tests.run property. Note, currently mutant only supports running against a single test file at a time, and the tests.run config property must contain a $FILE placeholder where the filename will be passed into your test runner, e.g. for Mocha, your tests.run config value might look like this:

{
  tests:  {
    run: 'mocha --reporter tap $FILE'
  }
}

Then when mutant comes to run your tests it can simply replace the $FILE placeholder with the path to your test file.

Mutation plugins

Mutations are provided to mutant via plugins and can be selectively enabled / disabled via the config file. Babel is used to parse and traverse your code, as well as to generate the mutants based on the mutation plugins you select.