n-test

Runs smoke tests with Puppeteer (and optionally Browserstack). Define a set of URLs and expected behaviour in JSON, without the toil of writing full blown tests.

![Node.js version support][shield-node]

[shield-github]: (https://img.shields.io/github/tag/Financial-Times/n-test.svg [shield-node]: https://img.shields.io/badge/node.js%20support->=14.0.0-brightgreen.svg

n-test smoke
n-test smoke --config path/to/config.js --host http://local.ft.com:3002 --header "X-Api-Key: 1234"
n-test smoke basic
n-test smoke -i

n-test open
n-test open headers --breakpoint M --config path/to/config.js --host https://local.ft.com:3002

Table of Contents

• Requirements
• Usage
  • Expectations
  • Request types
  • FT User Sessions
  • Using Programatically
  • Cross Browser Testing
• Contributing

Requirements

n-test requires the following to run:

• [Node.js][node] Version defined by engines.node in package.json. Run command nvm use to switch your local Node version to the one specified in .nvmrc.
• [npm][npm] (normally comes with Node.js)

Usage

n-test is easiest to use as a command line tool, installed by npm.

npm install @financial-times/n-test

You must create a config file containing the set of URLs to test. This will be a javascript file, that exports an array of test suites. The default location is test/smoke.js. This can be overriden with a command line parameter.

module.exports = [
	{
		name: 'basic',
		urls: {
			'/': 200,
			'/redirect': '/'
		},
		description: 'Test suite descriptions are optional',
	}
];

Then, you can run (assuming your application is running on port 8080 - the default is 3002):

n-test smoke -H http://localhost:8080

This will run a headless browser, open the URLs and check (in the above case) the response status is 200 for / and '/redirect' redirects to '/'. If both of those things are true, the command will exit with a success status.

You can also run:

n-test open -H http://localhost:8080

This allows you to select a suite of URLs (in this case, "basic"), and open them in Chromium. This is useful for manually testing a set of URLs.

If, when running locally, you are seeing errors about certificates not being valid, set NODE_ENV to be 'development' e.g. NODE_ENV=development;n-test smoke -H http://localhost:8080. This will use some launch options that ignore certificate errors.

Expectations

Checking response statii is great for checking that your application responds with something, but not necessarily the right thing. n-test comes with a bunch of basic things that you check for.

...
urls: {
	'/article/1234': {
		status: 200,
		elements: {
			'.this-should-exist-somewhere': true,
			'.there-should-be-3-of-these': 3,
			'div[exists=false]': false,
			'#should-contain-text': 'text'
		},
		elementShifts: {
			'.this-should-not-move': { maxCount: 0 },
			'.this-can-move-up-to-3-times': { maxCount: 0 },
			'.this-can-only-move-up-to-100-px': { maxPixels: 100 }
		},
		responseHeaders: {
			'My-Header': 'expected-value'
		},
		pageErrors: 0,
		networkRequests: {
			'/some-third-party.js': 1,
			'tracking.pixel': 4, //asserts 4 network requests were made to a URL containing 'tracking.pixel'
			'/will-have-some-of-these.jpg': true,
			'should-not-load-this.js': false
		},
		content: (content) => {
			return content.includes('some-text');
		},
		visibleContent: {
		  contentSelector: '.headline, .image, .standfirst'
		  threshold: 30 // % of viewport that should be visible content
		},
		performance: true, //checks firstPaint/firstContentfulPaint against baseline. default = 2000, or can specify.
		description: 'Each test may have an optional description. It will display when the test result is reported',
	}
}
...

Request types

By default, URLs are assumed to be GET requests, but you can also specify request method/headers/bodies.

...
urls: {
	'/article/1234': {
		headers: {
			'My-Request-Header': 1
		}
	},
	'/post': {
		body: { "some": "data" },
		method: 'POST',
		status: 200,
		https: true //Force this URL to be requested over HTTPS, even if the host is not
	},
	'/wait-for-load': {
		waitUntil: 'load' //default = domcontentloaded
		elements: {
		  '.loaded-by-js': true
		}
	}
}
...

These can all be set at a suite level, as well as a URL level, like so:

...
{
	name: 'authenticated-requests',
	headers: {
		'api-key': process.env.API_KEY
	},
	urls: {
		'/article/1': 200,
		'/article/2': 200,
		'/article/404': 404
	}
}
...

Actions

n-test allows some basic actions (e.g. clicking, interacting with forms). This has been ported from pa11y - see the section on their README for more.

FT User Sessions

To run a test suite for a type of FT subscriber, add a user property to the suite and it will set the session tokens for that type of user before running the tests in that suite.

For the test to get the user session tokens from next-test-sessions-lambda, it needs to rewrite the URL being tested to an ft.com host. The original URL is set in the FT-Test-Host header, which tells next-router to proxy the test URL rather than production.

The test output will display the original URL.

Options: premium, standard, expired.

Running locally:

Ngrok provides a secure public URL for the local test app and will need to be installed and running on the the app's port. Tests will then use the TEST_URL variable to specify the ngrok URL when starting the service. The local next-router needs to be running, as it will be used to proxy the test URL.

Example steps to run next-article user tests locally:

Run next-article and next-router locally:

$ cd ~/next-article
$ make run
$ cd ~/next-router
4 make run

Run ngrok on next-article's local port:

$ ./ngrok http 3002

Run the test against the ngrok address provided (can be either http or https):

make smoke TEST_URL=https://05bd2344ebca.ngrok.io

Remarks

Needs to set TEST_SESSIONS_URL (url to next-test-sessions-lambda) and TEST_SESSIONS_API_KEY environment variables when running the tests.

Example

[
  {
	user: 'premium',
	urls: [
	  '/these-will': 200,
	  '/run-with-a': 200,
	  '/premium-user': 200
	]
  },
  {
	'user': 'standard',
	'urls': [
	  '/this-will-run-with-a-standard-user': 200
	]
  },
  {
	'urls': [
	  '/these-will-run': 403,
	  '/without-session-token': 403
	]
  }
]

Using Programatically

n-test can also be used programatically. This allows you to extend the functionality by adding custom expectations. Below is an example.

const SmokeTest = require('@financial-times/n-test').SmokeTest;
const smoke = new SmokeTests({ headers: { globalHeader: true },  host: 'local.ft.com:3002' });

//Add custom checks like so:
smoke.addCheck('custom', async (testPage) => {
	const metrics = await testPage.page.metrics();

	return {
		expected: `no more than ${testPage.check.custom} DOM nodes`,
		actual: `${metrics.Nodes} nodes`,
		result: testPage.check.custom >= metrics.Nodes
	}
});

smoke.run()
	.then((results) => { //all passed })
	.catch((results) => { //some failed });

smoke.run(['basic']);

Cross Browser Testing [Experimental]

You can also run your test suite against Browserstack .

Browserstack: you must have BROWSERSTACK_USER and BROWSERSTACK_KEY environment variables set, and enable cross browser tests on a suite/url basis.

Note Browserstack supports running off a local host. If your host is local, it will spin up Browserstack Local and proxy through. Caveat sometimes browserstack local might not clean up properly after itself!

{
	name: 'blah'
	urls: {
		'/only-puppeteer': {
			status: 200
		},
		'/no-element-checks': {
			status: 200,
			browsers: true
		},
		'/runs-all-browsers': {
			status: 200,
			elements: {
				'.js-success': true
			},
			browsers: true //runs against all enabled browsers, default ['chrome', 'firefox', 'safari', 'internet explorer', 'MicrosoftEdge', 'android'];
		},
		'/ios-only': {
			status: 200,
			elements: {
				'.app-install-banner': true
			},
			browsers: ['ios']
		},
	}
}

The set of enabled browsers to run against can be changed on the command line:

n-test smoke --browsers "chrome,internet explorer,android"

Cross Browser Screenshotting [Experimental]

There are two ways to get screenshots generated.

1. As part of your smoke test run, you can generate PNG files with every run:

Example:

{
	name: 'blah'
	urls: {
		'/screenshot-me': {
			status: 200,
			screenshot: {
				path: './tmp/screenshots'
			}
		}
	}
}

2. A command that takes screenshots on multiple browsers, and opens them in a headless chrome window.

n-test screenshot --browsers ie9,safari

HALPPPPP

Call upon n-test HALP to get you through the tough times.

If a test is failing because a subscription has expired, e.g. the premium subscription on [email protected] has expired, email customer support to renew it.