harvest-npm

Harvest npm packages for offline use.

You can install harvest-npm using NPM: npm i -g @p26e/harvest-npm, or use the provided docker image (see below).

Usage

Create a pinfile (e.g. pinfile.json) containing the packages you want to download. Each package must have a corresponding semver string, specifying which versions of the package should be downloaded. All versions of a package matching the given semver will be downloaded.

See harvest-npm --help for a complete list of options.

Example

alice@computer$ cd /var/www

alice@computer$ echo '{ "react": "^17 || 16.8.6", "better-sqlite3": ">= 7.4.3"  }' > pinfile.json

alice@computer$ harvest-npm

// pinfile.json
{
	"react": "^17 || 16.8.6",
	"better-sqlite3": ">= 7.4.3"
}

How it works

TL;DR: Every time harvest-npm is executed, it will find all versions satisfying the given semver for each package specified in the pinfile. It will only download package versions that hasn't previously been downloaded (according to the lockfile). The outputed directory can be served as static files to function as a read-only NPM registry.

The key part of harvest-npm is to create a pinfile. The pinfile is a JSON file containing packages and corresponding semver strings (harvest-npm uses this NPM package to resolve matching packages). Let's see an example of a pinfile:

{
	"react": "^17 || 16.8.6",
	"better-sqlite3": ">= 7.4.3"
}

This pinfile contains two packages that harvest-npm should keep track of. The semver string (the value of the key-value style) specifies which versions that should be downloaded. For the first package, react, this resolves to all version with major version 17, and version 16.8.6. Every time harvest-npm is executed, it will find all packages that satisfy this requirement. At the time of writing, this equates to four packages: 17.0.0, 17.0.1, 17.0.2 (latest) and 16.8.6. The second package's semver, >= 7.4.3, resolves to all versions (including major versions) greater than or equal to 7.4.3. At the time of writing 7.4.3 is the latest version, thus only that version would be downloaded.

When package versions are downloaded, all their dependencies (and all their dependencies etc) are also dowloaded. When a package is downloaded its name and version (and other metadata) are written to a lock file (default lockfile.json). Before new packages are downloaded, harvest-npm will consult this file to see if the spesific version has already been downloaded. If it has, the package will be skipped to save bandwith (and time).

The output directory (specified through the --output-dir option) can be served as static files over http and function as a read-only NPM registry. Remember to set the --base-url to the URL you are going to serve the registry under (e.g. https://mydomain.tld/path/to/registry).

Docker

We have created a docker image for ease of use. To use the docker image, mount a folder containing the pinfile at /registry. The files will be written to /registry/packages and the script is automatically executed every time the pinfile changes.

Example:

alice@computer$ mkdir ~/registry

alice@computer$ echo '{ "react": ">= 17" }' > ~/registry/pinfile.json

alice@computer$ docker run -v /home/alice/registry:/registry ghcr.io/p26e/harvest-npm:latest

The following options are available through environment variables: