js-wacz

JavaScript module and CLI tool for working with web archive data using the WACZ format specification, similar to Webrecorder's py-wacz.

It can be used to combine a set of .warc / .warc.gz files into a single .wacz file:

... programmatically (Node.js):

import { WACZ } from '@harvard-lil/js-wacz'

const archive = new WACZ({ 
  input: 'collection/*.warc.gz', 
  output: 'collection.wacz',
})

await archive.process() // "my-collection.wacz" is ready!

... or via the command line:

js-wacz create -f "collection/*.warc.gz" -o "collection.wacz"

js-wacz makes use of workers to process as many WARC files in parallel as the host machine can handle.

Summary

• Install
• CLI: create command
• Programmatic use
• Feature parity with py-wacz
• Development
• Release notes

Install

js-wacz requires Node JS 20+.

npm can be used to install this package and make the js-wacz command accessible system-wide:

npm install -g @harvard-lil/js-wacz

👆 Back to summary

CLI: create command

The create command helps combine one or multiple .warc or .warc.gz files into a single .wacz file.

js-wacz create -f "collection/*.warc.gz" -o "collection.wacz"

js-wacz accepts the following options and arguments for customizing how the WACZ file is assembled.

--file, -f

This is the only required argument, which indicates what file(s) should be processed and added to the resulting WACZ file.

The target can be a single file, or a glob pattern such as folder/*.warc.gz.

# Single file:
js-wacz create --file archive.warc

# Collection:
js-wacz create --file "collection/*.warc"

Note: When using globs, make sure to surround the path with quotation marks.

--output, -o

Specify where the resulting .wacz file should be created, and what its filename should be.

Defaults to archive.wacz in the current directory if not provided.

The completed archive replaces an existing destination only after successful finalization. Handled errors preserve the previous file and remove temporary output.

js-wacz create --file cool-beans.warc --output cool-beans.wacz

--pages, -p

Path to a folder containing pages.jsonl files (pages.jsonl, extraPages.jsonl ...).

If not provided, js-wacz is going to attempt to detect pages in WARC records to build its own pages.jsonl index.

# Assuming the following file exists: /collections/pages/pages.jsonl
js-wacz create -f "collection/*.warc.gz" --pages collection/pages/

--cdxj

Pass a directory of existing CDXJ files, rather than indexing from WARCs. Must be used in combination with --pages.

js-wacz create -f "collection/*.warc.gz" --pages collection/pages.jsonl --cdxj collection/indexes/

--url

If provided, will be used as the mainPageUrl attribute for datapackage.json.

Must be a valid URL.

js-wacz create -f "collection/*.warc.gz" --url "https://lil.law.harvard.edu"

--ts

If provided, will be used as the mainPageDate attribute for datapackage.json.

Can be any value that can be parsed by JavaScript's Date() constructor.

js-wacz create -f "collection/*.warc.gz" --ts "2023-02-22T12:00:00.000Z"

--title

If provided, will be used as the title attribute for datapackage.json.

js-wacz create -f "collection/*.warc.gz" --title "My collection."

--desc

If provided, will be used as the description attribute for datapackage.json.

js-wacz create -f "collection/*.warc.gz" --desc "My cool collection of web archives."

--signing-url

If provided, will be used as an API endpoint for applying a cryptographic signature to the resulting WACZ file.

This endpoint is expected to be authsign-compatible.

An invalid signing URL or failed signature check rejects the export. The returned hash must match the root manifest and its signature must verify with the supplied key or domain certificate. This check does not verify certificate trust, domain identity, or RFC 3161 timestamps. See the release notes for supported formats and historical hash compatibility.

js-wacz create -f "collection/*.warc.gz" --signing-url "https://example.com/sign"

--signing-token

Used conjointly with --signing-url if provided, in case the signing server requires authentication.

js-wacz create -f "collection/*.warc.gz" --signing-url "https://example.com/sign" --signing-token "FOO-BAR"

--log-level

Can be used to determine how verbose js-wacz needs to be.

• Possible values are: silent, trace, debug, info, warn, error
• Default is: info

js-wacz create -f "collection/*.warc.gz" --log-level trace

--log-directory, -l

Path to a folder containing log files (only files with .log or .txt file extensions will be included in the WACZ).

# Assuming the following file exists: collection/logs/crawl.log
js-wacz create -f "collection/*.warc.gz" --log-directory collection/logs/

👆 Back to summary

Programmatic use

js-wacz's CLI and underlying logic are decoupled, and it can therefore be consumed as a JavaScript module (currently only with Node.js).

Example: Creating a signed WACZ programmatically

import { WACZ } from '@harvard-lil/js-wacz'

try {
  const archive = new WACZ({ 
    input: 'collection/*.warc.gz',
    output: 'collection.wacz',
    signingUrl: 'https://example.com/sign',
    signingToken: 'FOO-BAR',
  })

  await archive.process()

  // collection.wacz is ready
} catch(err) {
  // ...
}

Although a process() convenience method is made available, every step of said process can be run individually and the archive's state inspected / edited throughout.

process() and finalize() release output resources on success or failure. Workers are closed after indexing. Incremental callers should use finally to dispose an archive if any step fails or the archive is abandoned:

const archive = new WACZ({ input: 'collection/*.warc.gz', output: 'collection.wacz' })
try {
  await archive.addFileToZip(Buffer.from('Extra data'), 'extras/data.txt')
  await archive.process()
} finally {
  await archive.dispose()
}

dispose() can be called repeatedly and leaves a successfully finalized file in place. A finalized or disposed instance cannot start another export.

Notable affordances

• WACZ.addPage() allows for manually adding an entry to pages.jsonl.
• WACZ.addFileToZip() allows for manually adding any additional data to the final WACZ file.
• The datapackageExtras option allows for adding an arbitrary JSON-serializable object to datapackage.json under extras.

References:

• WACZ Class
• Available options

👆 Back to summary

Feature parity with py-wacz

js-wacz is aiming at partial feature parity with webrecorder's py-wacz, similar to Webrecorder's py-wacz.

This section lists notable differences in implementation that might affect interoperability.

Main differences in currently implemented features:

• CLI: create --detect-pages: --detect-pages is implied in js-wacz unless --pages is provided.
• CLI: create --file: that argument can be implied in py-wacz, it is always explicit in js-wacz.

👆 Back to summary

Development

Standard JS

This codebase uses the Standard JS coding style.

• npm run lint can be used to check formatting.
• npm run lint-autofix can be used to check formatting and automatically edit files accordingly when possible.
• Most IDEs can be configured to automatically check and enforce this coding style.

JSDoc

JSDoc is used for both documentation and loose type checking purposes on this project.

Testing

This project uses Node.js' built-in test runner.

npm run test

The suite starts and stops its own local HTTP signing servers, generates real cryptographic signatures, and checks independent RSA/ECDSA certificate fixtures.

Available CLI

# Runs test suite
npm run test

# Runs linter
npm run lint

# Runs linter and attempts to automatically fix issues
npm run lint-autofix

# Step-by-step NPM publishing helper
npm run publish-util

👆 Back to summary