website-scrap-engine

Configurable website scraper library in TypeScript. Consumers provide a DownloadOptions config (which includes a ProcessingLifeCycle) and instantiate a downloader to recursively scrape websites to local disk.

Features

• Configurable processing pipeline with hook arrays at every stage
• Single-thread and multi-thread (native worker_threads) downloaders
• HTML, CSS, SVG, and sitemap parsing with automatic link discovery
• CSS url() extraction and rewriting
• srcset, Open Graph meta tags, inline styles, and SVG xlink:href support
• Automatic URL-to-relative-path rewriting so saved sites work offline
• Streaming download support for large binary resources
• PQueue-based concurrency with runtime adjustment
• URL deduplication with configurable search-param stripping
• Configurable retry with exponential backoff, jitter, and Retry-After header support
• Local file:// source support for re-processing previously saved sites
• Configurable logging via log4js with dedicated categories (skip, retry, error, notFound, etc.)

Installation

npm install website-scrap-engine

Requires Node.js >= 18.17.0.

Usage

The downloader takes a path (or file:// URL) to a module that default-exports a DownloadOptions object. This pattern allows worker threads to independently load the same configuration.

Step 1: Create an options module (e.g. my-options.js)

import {lifeCycle, options, resource} from 'website-scrap-engine';

const {defaultLifeCycle} = lifeCycle;
const {defaultDownloadOptions} = options;
const {ResourceType} = resource;

const lc = defaultLifeCycle();

// Example: skip binary resources deeper than depth 2
lc.processBeforeDownload.push((res) => {
  if (res.depth > 2 && res.type === ResourceType.Binary) return;
  return res;
});

export default defaultDownloadOptions({
  ...lc,
  localRoot: '/path/to/save',
  maxDepth: 3,
  initialUrl: ['https://example.com'],
});

Step 2: Create and run the downloader

import path from 'path';
import {downloader} from 'website-scrap-engine';

const {SingleThreadDownloader} = downloader;

const d = new SingleThreadDownloader(
  'file://' + path.resolve('my-options.js')
);
d.start();
d.onIdle().then(() => d.dispose());

For CPU-intensive workloads, use MultiThreadDownloader instead (see Multi-Thread Processing).

You can also pass override options as the second argument to the downloader constructor, which are merged into the options module's export:

new SingleThreadDownloader('file://' + path.resolve('my-options.js'), {
  localRoot: '/different/path',
  concurrency: 8,
});

Adapter Helpers

The library provides adapter functions in lifeCycle.adapter for common customization patterns:

| Adapter | Stage | Description | |---|---|---| | skipProcess(fn) | linkRedirect | Skip URLs matching a predicate | | dropResource(fn) | processBeforeDownload | Mark matching resources as discard-only (replace link but don't download) | | preProcess(fn) | processBeforeDownload | Inspect/modify resources before download | | requestRedirect(fn) | processBeforeDownload | Rewrite the download URL | | redirectFilter(fn) | processAfterDownload | Rewrite or discard redirect URLs | | processHtml(fn) | processAfterDownload | Transform the parsed HTML (cheerio $) | | processHtmlAsync(fn) | processAfterDownload | Async version of processHtml |

import {lifeCycle} from 'website-scrap-engine';

const lc = lifeCycle.defaultLifeCycle();

// Skip all URLs containing "/api/"
lc.linkRedirect.push(lifeCycle.adapter.skipProcess(
  (url) => url.includes('/api/')
));

// Drop images from download but still rewrite their links
lc.processBeforeDownload.push(lifeCycle.adapter.dropResource(
  (res) => res.type === ResourceType.Binary && res.url.endsWith('.png')
));

Architecture

Pipeline Life Cycle

Resources are processed through a sequential pipeline of hook arrays. Each stage is an array of functions executed in order. Returning void/undefined from any function discards the resource from that stage onward.

init (once per downloader/worker startup)
 |
 v
URL
 |
 v
1. linkRedirect -----> skip or redirect URLs before processing
 |
 v
2. detectResourceType -> determine type (Html, Css, Binary, Svg, SiteMap, etc.)
 |
 v
3. createResource ----> build a Resource with save paths and relative replacement paths
 |
 v
4. processBeforeDownload -> filter/modify resources; link replacement in parent happens after this
 |
 v
5. download ----------> fetch resource via HTTP (loop ends early once body is set)
 |
 v
6. processAfterDownload -> parse content, discover child resources via submit() callback
 |
 v
7. saveToDisk --------> write to local filesystem
 |
 v
dispose (once per downloader shutdown / worker exit)

Consumers extend the pipeline by prepending or appending functions to any stage array via defaultLifeCycle(). See Usage for examples.

Default Pipeline Handlers

| Stage | Default handlers | |---|---| | linkRedirect | skipLinks - filters out non-HTTP URI schemes (mailto, javascript, data, etc.) | | detectResourceType | detectResourceType - infers type from element/context | | createResource | createResource - builds Resource with URL resolution, save path, and replace path | | download | downloadResource, downloadStreamingResource, readOrCopyLocalResource | | processAfterDownload | processRedirectedUrl, processHtml, processHtmlMetaRefresh, processSvg, processCss, processSiteMap | | saveToDisk | saveHtmlToDisk, saveResourceToDisk |

Resource Types

Defined in ResourceType enum:

| Type | Encoding | Description | |---|---|---| | Binary | null | Not parsed, saved as-is | | Html | utf8 | Parsed with cheerio, links discovered and rewritten | | Css | utf8 | CSS url() references extracted and rewritten | | CssInline | utf8 | Inline <style> blocks and style attributes | | SiteMap | utf8 | URLs discovered but not rewritten | | Svg | utf8 | Parsed with cheerio (same as HTML) | | StreamingBinary | null | Streamed directly to disk, for large files |

HTML Source Definitions

The scraper discovers linked resources from HTML using configurable source definitions. The defaults cover:

• Images: img[src], img[srcset], picture source[srcset]
• Styles: link[rel="stylesheet"], <style> blocks, [style] attributes
• Scripts: script[src]
• Links: a[href], frame[src], iframe[src]
• Media: video[src], video[poster], audio[src], source[src], track[src]
• SVG: *[xlink:href], *[href]
• Meta: meta[property="og:image"], og:audio, og:video and their variants
• Other: embed[src], object[data], input[src], [background], link[rel*="icon"], link[rel*="preload"]

Override via options.sources with an array of {selector, attr, type} definitions.

Key Abstractions

• Resource (src/resource.ts) - Central data object carrying URL, save path, replacement path, body, and metadata. RawResource is the serializable subset used for cross-thread communication.
• PipelineExecutor (interface in src/life-cycle/pipeline-executor.ts, impl in src/downloader/pipeline-executor-impl.ts) - Orchestrates life cycle execution. createAndProcessResource() runs stages 1-4 in one call.
• AbstractDownloader (src/downloader/main.ts) - Base class with PQueue-based concurrency, URL deduplication, and the download loop.
• SingleThreadDownloader (src/downloader/single.ts) - Runs all pipeline stages in the main thread.
• MultiThreadDownloader (src/downloader/multi.ts) - Downloads in main thread, sends to worker pool for post-processing.

Multi-Thread Processing

Use multi-thread processing when post-download work (HTML/CSS parsing, link discovery) is CPU-intensive.

Main thread:

• Runs the download queue with PQueue concurrency control
• Executes stages 1-5 (linkRedirect through download)
• Transfers downloaded resources to worker threads
• Receives discovered child resources back and enqueues non-duplicates

Worker threads:

• Receive downloaded resources from the main thread
• Execute stages 6-7 (processAfterDownload + saveToDisk)
• Parse HTML/CSS/SVG, discover child resources
• Run stages 1-4 on discovered children to prepare them
• Send prepared child resources back to the main thread as RawResource[]

Worker count defaults to Math.min(concurrency, workerCount). The worker pool uses a 2-pass water-fill algorithm to balance tasks across workers by load.

Logging

The library uses log4js with dedicated logger categories:

| Logger | Purpose | |---|---| | skip | Resources filtered/discarded at any pipeline stage | | skipExternal | External resources skipped by scope | | retry | HTTP retry attempts with backoff details | | error | Download and processing errors | | notFound | 404 responses | | request / response | HTTP request/response logging | | complete | Successfully processed resources | | mkdir | Directory creation | | adjustConcurrency | Runtime concurrency changes |

Configure logging via options.configureLogger and options.logSubDir.

Key Dependencies

• cheerio - HTML/SVG parsing and manipulation
• got - HTTP client with retry logic
• p-queue - Download concurrency control
• urijs - URL resolution and path generation
• css-url-parser - CSS url() extraction
• srcset - srcset attribute parsing

License

ISC