html-snapshots

Takes html snapshots of your site's crawlable pages when an element you select is rendered.

Contents

• html-snapshots
  • Contents
  • Overview
  • Getting Started
    • Installation
    • Grunt Task
  • More Information
    • Process Model
  • API
    • Promise run (options[, callback])
      • Callback
  • Example Usage
    • Quick Example
  • Options
    • Input Control Options
      • Sitemap Only Input Options
      • Origin Options
    • Output Control Options
    • Snapshot Control Options
    • Process Control Options
  • Example Rewrite Rule
    • Connect-modrewrite
    • Middleware Example
  • License

Overview

html-snapshots is a flexible html snapshot library that uses a headless browser to take html snapshots of your webpages served from your site. A snapshot is only taken when a specified selector is detected visible in the output html. This tool is useful when your site is largely ajax content, or an SPA, and you want your dynamic content indexed by search engines.

html-snapshots gets urls to process from either a robots.txt, sitemap.xml, or sitemap-index.xml. Alternatively, you can supply an array with completely arbitrary urls, or a line delimited textfile with arbitrary host-relative paths.

Change History

This is an old project that has evolved since 2013. I've endeavored to keep it easy to use and upgrade. However, before upgrading, please review the release notes and the history for breaking changes.

Getting Started

Installation

The simplest way to install html-snapshots is to use npm. npm install html-snapshots will download html-snapshots and all dependencies. To install the browsers required by playwright, you must also run npx playwright install. See Playwright Docs for additional details.

By default, html-snapshots uses Puppeteer under the hood. You can optionally configure it to use Playwright instead by setting the browser option to "playwright".

Grunt Task

If you are interested in the grunt task that uses this library, check out grunt-html-snapshots.

More Information

Here are some background and other notes regarding this project.

• Why Does This Library Exist?
• How To Use Without Knowing About Page Content
• Caveats
• Support History
• Web Automation & Building AI agents

Process Model

html-snapshots takes snapshots in parallel, each page getting its own browser process. Each browser process dies after snapshotting one page. You can limit the number of browser processes that can ever run at once with the processLimit option. This effectively sets up a process pool for browser instances. The default processLimit is 4 browser instances. When a browser process dies, and another snapshot needs to be taken, a new browser process is spawned to take the vacant slot. This continues until a processLimit number of processes are running at once.

API

The api is just one run method that returns a Promise.

Promise run (options[, callback])

A method that takes options and an optional callback. Returns a Promise.
Syntax:

const htmlSnapshots = require('html-snapshots');

htmlSnapshots.run(options[, callback])
.then(completed => {
  // `completed` is an array of paths to the completed snapshots.
})
.catch(errorObject => {
  // `errorObject` is an instance of Error or AggregateError
  // `errorObject.completed` is an array of paths to the snapshots that did successfully complete.
  // `errorObject.notCompleted` is an array of paths to files that DID NOT successfully complete.
});

Callback

The callback is optional because the run method returns a Promise that resolves on completion. If you supply a callback, it will be called, but the Promise will ALSO resolve. Callback usage is deprecated, and is made available for compatibility with older versions.

Signature of the optional callback:

callback (errorOrAggregateErrorObject, arrayOfPathsToCompletedSnapshots)

For the callback, in the error (or AggregateError) case, the errorObject does not have the new extra properties completed and notCompleted. However, arrayOfPathsToCompletedSnapshots is supplied, and contains the paths to the snapshots that successfully completed.

Example Usage

This example reads the pages from a mix of sitemap or sitemap-index files found in the robots.txt and produces snapshots in the ./snapshots directory. In this example, a selector named "#dynamic-content" appears in all pages across the site. Once this selector is visible in a page, the html snapshot is taken and saved to ./snapshots.

Quick Example

const htmlSnapshots = require('html-snapshots');
htmlSnapshots.run({
  source: 'https://host.domain/robots.txt',
  selector: '#dynamic-content',
  outputDir: './snapshots',
  outputDirClean: true
})
.then(completed => {
  // completed is an array of full file paths to the completed snapshots.
})
.catch(error => {
  // error is an Error instance.
  // error.completed is an array of snapshot file paths that did complete.
  // error.notCompleted is an array of file paths that did NOT complete.
});

More examples can be found in this document. Also, A showcase of runnable examples can be found here.

An older (version 0.13.2), more in depth usage example is located in this article that includes explanation and code of a real usage featuring dynamic app routes, ExpressJS, Heroku, and more.

Options

Every option has a default value except outputDir.

Input Control Options

• input {String}

  • default: "robots"

  • Specifies the input generator to be used to produce the urls.

    Possible values:

    • "sitemap" Supply urls from a local or remote sitemap.xml file. Gzipped sitemaps are supported.
    • "sitemap-index" Supply urls from a local or remote sitemap-index.xml file. Gzipped sitemap indexes are supported.
    • "array", supply arbitrary urls from a javascript array.
    • "robots" Supply urls from a local or remote robots.txt file. Robots.txt is first scanned for Sitemap directives. If found, those are used to drive the crawl. Otherwise, Allow directives are used in conjunction with origin options.
    • "textfile" Supply urls from a local line-oriented text file in the style of robots.txt

• source {String|Array}

  • default: "./robots.txt", "./sitemap.xml", "./sitemap-index.xml", "./line.txt", or [], depending on the input generator.
  • Specifies the input source. This must be a valid array or the location (local or remote) of a robots, text, or sitemap file for the corresponding input generator. For the array input generator, it must be an array of urls.

Sitemap Only Input Options

Options that apply to robots.txt with Sitemap directives, sitemaps, and sitemap-index input

• sitemapPolicy {Boolean}

  • default: false
  • For use only with the robots, sitemap, and sitemap-index input generators. When true, lastmod and/or changefreq sitemap url child elements can be used to determine if a snapshot needs to be taken. Here are the possibilities for usage:
    • Both lastmod and changefreq tags are specified alongside loc tags in the sitemap. In this case, both of these tags are used to determine if the url is out-of-date and needs a snapshot.
    • Only a lastmod tag is specified alongside loc tags in the sitemap. In this case, if an output file from a previous run is found for the url loc, then the file modification time is compared against the lastmod value to see if the url is out-of-date and needs a snapshot.
    • Only a changefreq tag is specified alongside loc tags in the sitemap. In this case, if an output file from a previous run is found for the url loc, then the last file modification time is used as a timespan (from now) and compared against the given changefreq to see if the url is out-of-date and needs a snapshot.

  Note that for sitemap-index, only lastmod policy element is available as a policy control.

  Not all url elements in a sitemap have to have lastmod and/or changefreq (those tags are optional, unlike loc), but the urls you want to be able to skip (if they are current) must make use of those tags. You can intermix usage of these tags, as long as the requirements are met for making an age determination. If a determination on age cannot be made for any reason, the url is processed normally. For more info on sitemap tags and acceptable values, read the wikipedia page.

• sitemapOutputDir {String}

  • default: _sitemaps_
  • For use only with the sitemap-index input generator, this option directs the storage of sitemaps locally. It is a string that defines the name of the subdirectory under the outputDir where sitemaps are stored. Locally stored sitemaps are used for age determinations with incoming lastmod tags. If this option is falsy, it will prevent sitemap storage and thereby disable sitemapPolicy for sitemaps referenced in a sitemap-index.

  The examples directory contains sitemap-index and sitemap usage examples.

Origin Options

Origin options are only useful for Robots.txt files that use Allow directives and Textfile input types.

• hostname {String}

  • default: "localhost"
  • Specifies the hostname to use for paths found in a robots.txt or textfile. Applies to all pages. This option is ignored if you are using the sitemap or array input generators.

• port {Number}

  • default: 80
  • Specifies the port to use for all paths found in a robots.txt or textfile. This option is ignored if you are using the sitemap or array input generators.

• auth {String}

  • default: none
  • Specifies the old-school authentication portion of the url. Applies to all path found in a robots.txt or textfile.

• protocol {String}

  • default: "http"
  • Specifies the protocol to use for all paths found in a robots.txt or textfile. This option is ignored if you are using the sitemap or array input generators.

Output Control Options

• outputDir {String}

  • default: none
  • Required (you must specify a value). Specifies the root output directory to put all the snapshot files in. Paths to the snapshot files in the output directory are defined by the paths in the urls themselves. The snapshot files are always named "index.html".

• outputDirClean {Boolean}

  • default: false
  • Specifies if html-snapshots should clean the output directory before it creates the snapshots. If you are using sitemapPolicy and only specifying one of lastmod or changefreq in your sitemap (thereby relying on file modification times on output files from a previous run) this value must be false.

• outputPath {Object|Function}

  • default: none

  • Specifies per url overrides to the generated snapshot output path. The default output path for a snapshot file, while rooted at outputDir, is simply an echo of the input path - plus any arguments. Depending on your urls, your _escaped_fragment_ rewrite rule (see below), or the characters allowed in directory names in your environment, it might be necessary to use this option to change the output paths.

    The value can be one of these javascript types:

    "object" If the value is an object, it must be a key/value pair object where the key must match the url (or path in the case of robots.txt style) found by the input generator.

    "function" If the value is a function, it is called for every page and passed a single argument that is the url (or path in the case of robots.txt style) found in the input. The value returned for a given page must be a string that can be used on the filesystem for a path.

Notes:

• If you already have pretty urls, this option may not be wanted/needed.
• The output file name is always "index.html".

Snapshot Control Options

• selector {String|Object|Function}

  • default: "body"

  • The selector to wait for in the output that triggers a snapshot to be taken.

    The value can be one of these javascript types:

    "string" If the value is a string, it is used for every page.

    "object" If the value is an object, it is interpreted as key/value pairs where the key must match the url (or path in the case of robots.txt style) found by the input generator. This allows you to specify selectors for individual pages. The reserved key "__default" allows you to specify the default selector so you don't have to specify a selector for every individual page.

    "function" If the value is a function, it is called for every page and passed a single argument that is the url (or path in the case of robots.txt style) found in the input. The function must return a value to use for this option for the page it is given. The value returned for a given page must be a string.

  NOTE: By default, selectors must conform to this spec, as they are used by querySelector. If you need selectors not supported by this, you must load jQuery in your page and use a custom snapshot script.

• snapshotScript {String|Object}

  • default: This library's default snapshot script. Which one is used is determined by the browser option.

  • Specifies the browser script to run to actually produce the snapshot. The script supplied in this option is run per url (or path) by html-snapshots in a separate browser process. Applies to all pages.

    The value can be one of these javascript types:

    "string" If the value is a string, it must an absolute path to a custom script you supply.

    • browser: "puppeteer":
      html-snapshots will spawn your script as a separate process and give it the following arguments:

      • process.argv[0] The path to NodeJS.
      • process.argv[1] The path to your snapshot script.
      • process.argv[2] The output file path.
      • process.argv[3] The url to snapshot.
      • process.argv[4] The selector to wait for to signal page completion.
      • process.argv[5] The overall timeout (milliseconds).
      • process.argv[6] The path to a custom NodeJS module that returns a filter function.
      • process.argv[7] A debug flag to kick the browser into headed, devtools mode.
      • process.argv[8] A slowMo time (milliseconds) to slow the browser down.
      • process.argv[9] Stringified Puppeteer launch options.

    • browser: "playwright":
      html-snapshots will spawn your script as a separate process and give it the following arguments:

      • process.argv[0] The path to NodeJS.
      • process.argv[1] The path to your snapshot script.
      • process.argv[2] The output file path.
      • process.argv[3] The url to snapshot.
      • process.argv[4] The selector to wait for to signal page completion.
      • process.argv[5] The overall timeout (milliseconds).
      • process.argv[6] The path to a custom NodeJS module that returns a filter function.
      • process.argv[7] A debug flag to kick the browser into headed, slowMo mode.
      • process.argv[8] A slowMo time (milliseconds) to slow the browser down.
      • process.argv[9] Stringified Playwright launch options. May include a browserType key ("chromium", "firefox", or "webkit"). Defaults to "chromium".

    "object" If an object is supplied, it has the following properties:

    • script This must be one of the following values:
      • "removeScripts" This runs the default snapshot script with an output filter that removes all script tags from the html snapshot before it is saved.
      • "customFilter" This runs the default snapshot script, but allows you to supply any output filter.
    • module This property is required only if you supplied a value of "customFilter" for the script property. This must be an absolute path to a NodeJS module you supply. Your module will be required and called as a function to filter the html snapshot output. Your module's function will receive the entire raw html content as a single input string, and must return the filtered html content.

    customFilter Example:

    // option snippet showing snapshotScript object with "customFilter":
    {
      snapshotScript: {
        script: 'customFilter',
        module: '/path/to/myFilter.js'
      }
    }
    
    // in myFilter.js:
    module.exports = function(content) {
      return content.replace(/someregex/g, 'somereplacement'); // remove or replace anything
    }

    A more complete example using custom options is available here.

• debug {Object}

  • default: { flag: false, slowMo: 500 }
  • Supported with both puppeteer and playwright browser scripts. Setting the debug.flag to true starts the browser in headed mode with devtools open (devtools is only supported for chromium-based browsers). debug.slowMo is a time in milliseconds to reduce browser processing speed (larger numbers slows down the browser more). Recommended use is with a single problem page input using an Array source.

Process Control Options

• browser {String}

  • default: "puppeteer"
  • Specifies which browser process to use in the crawl. Can be one of "puppeteer" or "playwright".

• timeout {Number|Object|Function}

  • default: 10000 (milliseconds)

  • Specifies the time to wait for the selector to become visible.

    The value can be one of these javascript types:

    "number" If the value is a number, it is used for every page in the website.

    "object" If the value is an object, it is interpreted as key/value pairs where the key must match the url (or path in the case of robots.txt style) found by the input generator. This allows you to specify timeouts for individual pages. The reserved key "__default" allows you to specify the default timeout so you don't have to specify a timeout for every individual page.

    "function" If the value is a function, it is called for every page and passed a single argument that is the url (or path in the case of robots.txt style) found in the input. The function must return a value to use for this option for the page it is given. The value returned for a given page must be a number.

• processLimit {Number}

  • default: 4
  • Limits the number of child browser processes that can ever be actively running in parallel. A value of 1 effectively forces the snapshots to be taken in series (only one at a time). Useful if you need to limit the number of processes spawned by this library. Experiment with what works best. One guideline suggests about 4 per CPU.

• pollInterval {Number}

  • default: 500 (milliseconds)
  • Specifies the rate at which html-snapshots checks to see if a browser script has completed. Applies to all pages.

• puppeteerLaunchOptions {Object|Function}

  This option is only used with the puppeteer browser script

  • default: {}

  • Specifies launch options to give to Puppeteer. Can specify per page or for all pages. Puppeteer function options (like targetFilter) are not supported. Launch options will override any puppeteer debug options supplied.

    The value can be one of these javascript types:

    "object" If the value is an object, it can contain any puppeteer launch options, and is applied for all pages. The object will be given directly to puppeteer.launch.

    "function" If the value is a function, it is called for every page and passed a single argument that is the url found in the input. The function must return puppeteer launch options to use for the page it is given. The value returned for a given page must be an object to be given directly to puppeter.launch.

• playwrightLaunchOptions {Object|Function}

  This option is only used with the playwright browser script

  • default: {}

  • Specifies launch options to give to Playwright. Can specify per page or for all pages. The object may include a browserType key ("chromium", "firefox", or "webkit") to select which browser engine to use. Defaults to "chromium". Launch options will override any playwright debug options supplied.

    The value can be one of these javascript types:

    "object" If the value is an object, it can contain any playwright launch options, and is applied for all pages. The object (minus browserType) will be given directly to browserType.launch.

    "function" If the value is a function, it is called for every page and passed a single argument that is the url found in the input. The function must return playwright launch options to use for the page it is given. The value returned for a given page must be an object to be given directly to browserType.launch.

    Example using Firefox:

    {
      browser: "playwright",
      playwrightLaunchOptions: {
        browserType: "firefox"
      }
    }

Example Rewrite Rule

Here is an example apache rewrite rule for rewriting _escaped_fragment_ requests to the snapshots directory on your server.

<ifModule mod_rewrite.c>
  RewriteCond %{QUERY_STRING} ^_escaped_fragment_=(.*)$
  RewriteCond %{REQUEST_URI} !^/snapshots [NC]
  RewriteRule ^(.*)/?$ /snapshots/$1 [L]
</ifModule>

This serves the snapshot to any request for a url (perhaps found by a bot in your robots.txt or sitemap.xml) to the snapshot output directory. In this example, no translation is done, it simply takes the request as is and serves its corresponding snapshot. So a request for http://mysite.com/?_escaped_fragment_= serves the mysite.com homepage snapshot.

Connect-modrewrite

You can also refer _escaped_fragment_ requests to your snapshots in ExpressJS with a similar method using connect-modrewrite middleware. Here is an analogous example of a connect-modrewrite rule:

  '^(.*)\\?_escaped_fragment_=.*$ /snapshots/$1 [NC L]'

Middleware Example

An ExpressJS middleware example using html-snapshots can be found at wpspa/server/middleware/snapshots.js.
Here is the article on how this middleware works with html-snapshots.

License

This software is free to use under the LocalNerve, LLC MIT license. See the LICENSE file for license text and copyright information.

Third-party open source code used are listed in the package.json file.