puppeteer-browser-ready

Simple utility to go to a URL and wait for the HTTP response

puppeteer-browser-ready is a helper utility to reduce the amount of boilerplate code needed to tell Puppeteer to visit a web page and and retrieve the HTML.  It's primarily intended for use within Mocha test cases.  In addition to the raw HTML, you get a node-html-parsed root so you can immediately run queries on the DOM.

A) Setup

Install packages:

$ npm install --save-dev puppeteer puppeteer-browser-ready

Import packages:

import puppeteer from 'puppeteer';
import { browserReady } from 'puppeteer-browser-ready';

B) Usage

Use the browserReady.goto(url, options) function to tell Puppeteer which page to open. The Promise will resolve with a Web object containing a title field and a html field. Pass the Web object to the browserReady.close(web) function to disconnect the page.

const url = 'https://pretty-print-json.js.org/';
let web;  //fields: browser, page, response, status, location, title, html, root
before(async () => web = await puppeteer.launch().then(browserReady.goto(url));
after(async () =>  await browserReady.close(web));

goto() Options

| Name (key) | Type | Default | Description | | :----------- | :---------- | :------ | :-------------------------------------------------------- | | parseHtml | boolean | true | Return the DOM root as an HTMLElement (node-html-parsed). | | verbose | boolean | false | Output HTTP connection debug messages. |

startWebServer() Options

| Name (key) | Type | Default| Description | | :------------ | :---------- | :----- | :----------------------------------------------- | | autoCleanup | boolean | true | Terminate connection on interruption (SIGINT). | | folder | string | '.' | Document root for the static web server. | | port | number | 0 | Port number for server (0 find open port). | | verbose | boolean | true | Output informational messages. |

C) TypeScript Declarations

See the TypeScript declarations at the top of the puppeteer-browser-ready.ts file.

The browserReady.goto(url, options) function returns a function that takes a Puppeteer Browser object and returns a Promise that resolves with a Web object:

type Web = {
   browser:  Puppeteer.Browser,
   page:     Puppeteer.Page,
   response: HTTPResponse | null,
   location: Location,
   title:    string,
   html:     string,
   root:     HTMLElement | null,  //see node-html-parsed library
   };

The optional browserReady.startWebServer(options) function starts a static web server and returns a Promise for when the server is ready:

export type Http = {
   server:     Server,
   terminator: httpTerminator.HttpTerminator,
   folder:     string,
   url:        string,
   port:       number,
   verbose:    boolean,
   };

D) Examples

Example 1: Node.js program

Code:

import puppeteer from 'puppeteer';
import { browserReady } from 'puppeteer-browser-ready';

const handleResponse = (web) => {
   console.log('Hello, World!');
   console.log('web fields:', Object.keys(web).join(', '));
   console.log(`The HTML from ${web.location.href} is ${web.html.length} characters`,
      `long and contains ${web.root.querySelectorAll('p').length} <p> tags.`);
   return web;
   };
puppeteer.launch()
   .then(browserReady.goto('https://pretty-print-json.js.org/'))
   .then(handleResponse)
   .then(browserReady.close);

Output:

Hello, World!
web fields: browser, page, response, status, location, title, html, root
The HTML from https://pretty-print-json.js.org/ is 8200 characters
long and contains 7 <p> tags.

Example 2: Mocha specification suite

Code:

// Mocha Specification Suite

// Imports
import puppeteer from 'puppeteer';
import { assertDeepStrictEqual } from 'assert-deep-strict-equal';
import { browserReady } from 'puppeteer-browser-ready';

// Setup
const url = 'https://pretty-print-json.js.org/';
let web;  //fields: browser, page, response, status, location, title, html, root
const loadWebPage = async () =>
   web = await puppeteer.launch().then(browserReady.goto(url));
const closeWebPage = async () =>
   await browserReady.close(web);

/////////////////////////////////////////////////////////////////////////////////////
describe('The web page', () => {
   before(loadWebPage);
   after(closeWebPage);

   it('has the correct URL', () => {
      const actual =   { status: web.status, url: web.location.href };
      const expected = { status: 200,        url: url };
      assertDeepStrictEqual(actual, expected);
      });

   it('title starts with "Pretty-Print JSON"', () => {
      const actual =   { title: web.title.substring(0, 17) };
      const expected = { title: 'Pretty-Print JSON' };
      assertDeepStrictEqual(actual, expected);
      });

   it('body has exactly one header, main, and footer -- node-html-parsed', () => {
      const getTags =  (elems) => [...elems].map(elem => elem.tagName.toLowerCase());
      const actual =   getTags(web.root.querySelectorAll('body >*'));
      const expected = ['header', 'main', 'footer'];
      assertDeepStrictEqual(actual, expected);
      });

   it('body has exactly one header, main, and footer -- page.$$eval()', async () => {
      const getTags =  (elems) => elems.map(elem => elem.nodeName.toLowerCase());
      const actual =   await web.page.$$eval('body >*', getTags);
      const expected = ['header', 'main', 'footer'];
      assertDeepStrictEqual(actual, expected);
      });

   });

/////////////////////////////////////////////////////////////////////////////////////
describe('The document content', () => {
   before(loadWebPage);
   after(closeWebPage);

   it('has a 🚀 traveling to 🪐!', () => {
      const actual =   { '🚀': !!web.html.match(/🚀/g), '🪐': !!web.html.match(/🪐/g) };
      const expected = { '🚀': true,                    '🪐': true };
      assertDeepStrictEqual(actual, expected);
      });

   });

Output:

  The web page
    ✓ has the correct URL
    ✓ title starts with "Pretty-Print JSON"
    ✓ body has exactly one header, main, and footer -- node-html-parsed
    ✓ body has exactly one header, main, and footer -- page.$$eval()

  The document content
    ✓ has a 🚀 traveling to 🪐!

Example 3: Start and shutdown a static web server

The startWebServer(options) and shutdownWebServer(http) functions can be used in global fixtures to start and shutdown a static web server.

For example, the spec/fixtures/setup-teardown.js file below starts a web server on port 7123 with the web root pointed to the project's docs folder.

Code:

// Specification Fixtures
import { browserReady } from 'puppeteer-browser-ready';
let http;  //fields: server, terminator, folder, url, port, verbose

// Setup
const mochaGlobalSetup = async () => {
   http = await browserReady.startWebServer({ folder: 'docs', port: 7123 });
   };

// Teardown
const mochaGlobalTeardown = async () => {
   await browserReady.shutdownWebServer(http);
   };

export { mochaGlobalSetup, mochaGlobalTeardown };

Run specification suites with global fixtures: $ npx mocha spec/*.spec.js --require spec/fixtures/setup-teardown.js

Output:

  [2021-07-14T11:38:22.892Z] Web Server - listening: true 7123 http://localhost:7123/
  ...Output of Mocha specification suites here...
  [2021-07-14T11:38:26.704Z] Web Server - shutdown: true

E) Test Timeout Errors

By default Mocha allows a test 2,000 ms to complete before timing out with a failure.  Web page load times can vary significantly, so it's sometimes a good idea to use the timeout option to bump up the allowed test execution time.

Example configuration in package.json to allow 5,000 ms:

   "scripts": {
      "pretest": "run-scripts clean build",
      "test": "mocha spec/*.spec.js --timeout 7000"
   },

CLI Build Tools for package.json

• 🎋 add-dist-header:  Prepend a one-line banner comment (with license notice) to distribution files
• 📄 copy-file-util:  Copy or rename a file with optional package version number
• 📂 copy-folder-util:  Recursively copy files from one folder to another folder
• 🪺 recursive-exec:  Run a command on each file in a folder and its subfolders
• 🔍 replacer-util:  Find and replace strings or template outputs in text files
• 🔢 rev-web-assets:  Revision web asset filenames with cache busting content hash fingerprints
• 🚆 run-scripts-util:  Organize npm package.json scripts into named groups of easy to manage commands
• 🚦 w3c-html-validator:  Check the markup validity of HTML files using the W3C validator

MIT License