Deviance

:warning: Not ready for production use. Deviance is currently in active development and as such is likely to change. Use at your own peril.

Nightwatch.js reporting tool with extended commands and assertions to perform visual regressions.

Installation

Using npm:

$ npm i --save-dev deviance

Usage

Settings

Nightwatch.js will need to be told to load in the command and assertion provided by Deviance. To do this, add the following lines to your Nightwatch.js configuration file(s):

custom_commands_path: [
    'node_modules/deviance/dist/commands',
],
custom_assertions_path: [
    'node_modules/deviance/dist/assertions',
],

Note that these properties accept an array and can be used with other commands and assertions. They should sit at the same depth as src_folders within the configuration file. You can read more about configuration settings in the Nightwatch.js configuration documentation.

Deviance will default to the following settings:

const defaultSettings = {
    reporting: {
        enabled: false,
        port: 8083,
    },
    regression: {
        expectedPath: 'tests_output/deviance/regression/expected',
        actualPath: 'tests_output/deviance/regression/actual',
        screenshotsPath: '',
        threshold: 0.05,
        screenshotTimeout: 20000,
    },
};

You can modify these settings from within your globals file, as is defined by globals_path within your Nightwatch.js configuration file.

const Deviance = require('deviance');

/*
 * Deviance can be constructed with settings to fit your requirements.
 * You can override just a single setting, some or all of them.
 */
const deviance = new Deviance({
    reporting: {
        enabled: true,
    },
    regression: {
        expectedPath: 'your/regression/expected/path',
        actualPath: 'your/regression/actual/path',
        screenshotsPath: 'your/nightwatch/screenshots/path',
        threshold: 0.15,
    },
});

/*
 * Then provide Nightwatch.js with the Deviance settings as a global property
 * and optionally define the reporter.
 */
module.exports = {
    deviance: deviance.settings,
    reporter: deviance.reporter,
};

In the example above, the Deviance reporter is optional, but for Deviance to function settings are mandatory.

You can optionally override the enabled setting for the report, by setting a node environment variable when running Nightwatch.js:

OPEN_REPORT=false nightwatch

Command

Deviance provides a single command captureElementScreenshot, typically its usage would look like this:

module.exports = {
    'sample capture element image': function (browser) {
        browser
            .url('http://app.host')
            .waitForElementVisible('body')
            .captureElementScreenshot('.selector')
            .end();
    }
};

This will also work with named selectors from page objects e.g. captureElementScreenshot('@selector'). If no parameter is supplied, it will default to the selector body. Any screenshot generated for the body will define it's boundaries from the document scrollWidth and scrollHeight (i.e. the full size of the document).

The command captures the element defined by the selector and saves it to the path defined by the settings. It then checks for an expected image and performs a diff between the two. It performs no assertions or tests, but will raise errors should they occur.

By default the image will be named after the selector passed to it, for example .selector would be named selector.png and img would be called img.png. The path for saving images is derived from the settings defined in the globals file (or the defaults mentioned above) combined with the test group (the directory structure under test suites root) and the test name. This should hopefully prevent collision of filenames for most tests, however if you do find a collision within a test (capturing an element more than once within the same test) then captureElementScreenshot takes an optional second argument:

module.exports = {
    'sample capture element image': function (browser) {
        browser
            .url('http://app.host')
            .waitForElementVisible('body')
            .captureElementScreenshot('.selector', 'selector_1')
            .someOperationOn('.selector')
            .captureElementScreenshot('.selector', 'selector_2')
            .end();
    }
};

The above will now generate images called selector_1.png and selector_2.png.

The command will also take an optional third parameter as a function:

module.exports = {
    'sample capture element image': function (browser) {
        browser
            .url('http://app.host')
            .waitForElementVisible('body')
            .captureElementScreenshot('.selector', '.selector', function(results) {
                // do stuff
            })
            .end();
    }
};

In the above example results is an object with the following structure:

{
    actual: {
        path: 'output/path/of/captured/image',
        width: 0, // image width as Number
        height: 0, // image height as Number
    },
    // the following are only populated if an expected file exists
    expected: {
        path: 'path/of/expected/image',
        width: 0, // image width as Number
        height: 0, // image height as Number
    },
    diff: {
        path: 'path/of/generated/diff/image',
        percent: 0 // difference in image on a scale from 0 to 1
    },
}

Assertion

Deviance provides a single assertion elementRegresses, typically its usage would look like this:

module.exports = {
    'sample assert element regresses': function (browser) {
        browser
            .url('http://app.host')
            .waitForElementVisible('body')
            .verify.elementRegresses('img')
            .assert.elementRegresses('h1');
    },
};

The assertion (like all Nightwatch.js assertions) will work with verify and assert. It will also work with named selectors from page objects e.g. elementRegresses('@selector'). If no parameter is supplied, it will default to the selector body.

Internally the assertion uses the captureElementScreenshot command and performs some analysis on the results provided. Like the command, it can accept an optional second argument to override the filename of the generated image:

module.exports = {
    'sample assert element regresses': function (browser) {
        browser
            .url('http://app.host')
            .waitForElementVisible('body')
            .verify.elementRegresses('div', 'div_1')
            .someActionToElement('div')
            .verify.elementRegresses('div', 'div_2');
    },
};

Please read the command information above for reasoning behind this.

Optionally the assertion accepts a third command in the form of a Number between 0 and 1 which allows you to override the defined threshold in the globals file (or the defaults mentioned above) for this one regression assertion. The threshold is the ratio of difference that is deemed tolerable before the test should fail:

module.exports = {
    'sample assert element regresses': function (browser) {
        browser
            .url('http://app.host')
            .waitForElementVisible('body')
            .verify.elementRegresses('div', 'div', 0.005); // very strict threshold
    },
};

It is advisable to use a very strict threshold for elements where there is a considerably dominant colour. Where possible though we would advise avoiding the use of the regression assertion on elements like this.

Running in Docker

We explored using Deviance within a Docker image and decided to leave some loose guidance on getting set up with Docker.

We configured our base image with Firefox, Google Chrome, Node.js, Yarn and Java. Within the image we created a volume for our application and a basic entrypoint script that would run yarn followed by running Nightwatch.js. As Deviance will generate some files (images for visual regression) we also created a user (node in our case) so any files would not be generated as root.