icejaw

Icejaw is a general tool like wget -r that can crawl a stream of urls and spit out a 100% static, ready-to-deploy website in the designated folder.

This way you can develop your static site with whatever dynamic technology you're already familiar with instead of credentializing in a more limited static-site-generator that may not be able to do what you want.

Install

npm install --global icejaw

Usage

Icejaw expects a \n-delimited list of routes.

/
/users
/users/1
/users/2
/faq

Pipe them into icejaw's stdin:

cat routes.txt | icejaw

Icejaw will send a GET request to each one and save the HTML response into the ./build directory.

Icejaw only cares about the path. /hello is equivalent to https://example.com/hello. For example, your sitemap.txt file might be sufficient if you have one.

Query-strings are ignored. /hello?foo=1 and /hello?foo=bar are both considered /hello, and only the first instance will trigger a crawl.

$ node server.js
Express is listening on http://localhost:3000

$ cat sitemap.txt | icejaw --port 3000
200 /        -> ./build/index.html
200 /users   -> ./build/users.html
200 /users/1 -> ./build/users/1.html
200 /users/2 -> ./build/users/2.html
200 /users/3 -> ./build/users/3.html

$ tree build
build
├── index.html
├── test.jpg
├── users.html
└── users
    ├── 1.html
    ├── 2.html
    └── 3.html

CLI Options

• --port <Integer>: The localhost post that the server is running on.
  • Default: 3000
• --concurrency <Integer>: The number of in-flight requests to localhost allowed at a time.
  • Default: 8
• --assets <String>: Path to the folder containing static assets to be copied to the build. May be absolute or relative to current directory.
  • Default: ./public
• --out <String>: Path to the generated build folder. May be absolute or relative to current directory.
  • Default: ./build
• --routes <String>: Comma-delimited list of routes. Useful for testing/sanity-checking. If this is set, then stdin will be ignored.
  • Example: --routes "/,/foo,/bar"
• --ignore404: If this flag is set, icejaw won't stop freezing when it encounters a 404 response. Instead, it will print a warning to stdout and move on. No static page will be generated for the route.
  • Default: false
• --redirect [strategy]: is one of follow, ignore, error.
  • Default: follow

Static Assets

By default, icejaw assumes your static assets are contained in a top-level folder named "public", and it will copy all of its contents into the build folder.

Trailing Backslash

Whether a route ends in a backslash is an important distinction.

• /example gets written to /example.html
• /example/path gets written to /example/path.html
• /example/ gets written to /example/index.html
• /example/path/ gets written to /example/path/index.html

This is consistent with, for example, how https://surge.sh implements clean URL redirection (https://surge.sh/help/using-clean-urls-automatically).

Watching Changes

I'm not sure how to implement a --watch argument since I currently have it so that icejaw reads routes from stdin. I'll have to think about this.

Until then, the watch module can be used to run an arbitrary command when files change.

npm install --save-dev watch

In the following example, we have a dev-only GET /routes endpoint on our localhost server that returns a text/plain document of \n-delimited routes.

package.json:

  "scripts": {
    "build": "http localhost:8080/routes --body | icejaw --port 8080",
    "watch": "watch 'npm run build' src"
  },

npm run watch

Now icejaw will build your project on every change.

Route Generation

You generally need to query your database to generate a list of routes.

This part is up to you. If you simply print routes to stdout, then you can pipe the result directly into icejaw.

Print out routes as soon as you have them so icejaw can consume them faster.

// generate.js
;['/', '/users', '/faq', '/about-us'].forEach((route) => console.log(route))

db.getAllUserIds()
  .then((ids) => {
    ids.forEach((id) => console.log(`/users/${id}`))
  })
  .catch((err) => {
    console.error(err)
    process.exit(1)
  })

node generate.js | icejaw

Programmatic Usage (WIP)

I'm not sure how this API should look, yet. I currently only use it for testing. Ideally you would be able to pipe stuff into icejaw and customize icejaw's output with gulp plugins.

Icejaw takes an options object that's similar to the CLI API with the same defaults. It returns a promise.

const icejaw = require('icejaw')

icejaw({
  port: 5000,
  out: './build',
  assets: './assets',
  concurrency: 8,
  routes: ['/', '/users', '/faq']
}).then(console.log, console.error)