Snap

Like wget -r <url> but specifically designed to support "pretty" URLs. With wget, a URL pointing to /foo would result in /foo.html, but this means the URL has now changed.

With snap, it will create the directory /foo and save the file to /foo/index.html so that the URL /foo still works.

Installation

Via npm:

npm install @remy/snap

Usage

$ snap <url> # saves to directory matching the domain
$ snap <url> --output <dir>
$ snap <url> -o <dir> # short hand

Example usage

This is how I'm using snap in an Express web site to generate a static copy of the site before deploying to Amazon S3.

The scripts section of my package.json:

{
  "scripts": {
    "start": "node .",
    "prebuild": "NODE_ENV=production PORT=7331 node . & echo $! > .pid; sleep 1",
    "build": "snap http://localhost:7331 -o www; kill $(cat .pid); rm .pid"
  },
  "devDependencies": {
    "@remy/snap": "^1.0.1"
  }
}

Notes and limitations

• If the output directory already exists, it will be removed first by snap
• Crawling is limited to the domain specified
• Speed and interval has been set to run as fast as possible, so this is ideally used with local domains
• There are no other arguments to snap, if further customisation is needed, please send a PR or look at using simplecrawler