Doc Crawler

A Node.js/TypeScript CLI tool for crawling documentation websites and exporting them to Markdown, built with BunJS.

Features

• Parallel web crawling with configurable concurrency
• Domain-specific crawling option
• Automatic conversion from HTML to Markdown
• Generates a well-formatted document with table of contents
• Handles timeouts and crawling limits for stability
• Built with BunJS for optimal performance

Installation

From npm

# Install globally
npm install -g @saintno/doc-export

# Or with yarn
yarn global add @saintno/doc-export

# Or with pnpm
pnpm add -g @saintno/doc-export

From source

Prerequisites

• Bun (v1.0.0 or higher)

Setup

1. Clone this repository
2. Install dependencies:

bun install

3. Build the project:

bun run build

Usage

Basic usage:

# If installed from npm:
doc-export --url https://example.com/docs --output ./output

# If running from source:
bun run start --url https://example.com/docs --output ./output

Command Line Options

| Option | Alias | Description | Default | |--------|-------|-------------|---------| | --url | -u | URL to start crawling from (required) | - | | --output | -o | Output directory for the Markdown (required) | - | | --concurrency | -c | Maximum number of concurrent requests | 5 | | --same-domain | -s | Only crawl pages within the same domain | true | | --max-urls | | Maximum URLs to crawl per domain | 200 | | --request-timeout | | Request timeout in milliseconds | 5000 | | --max-runtime | | Maximum crawler run time in milliseconds | 30000 | | --allowed-prefixes | | Comma-separated list of URL prefixes to crawl | - | | --split-pages | | How to split pages: "none", "subdirectories", or "flat" | none |

Example

doc-export --url https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide --output ./javascript-guide --concurrency 3 --max-urls 300 --request-timeout 10000 --max-runtime 60000

This example will:

• Start crawling from the MDN JavaScript Guide
• Save files in the ./javascript-guide directory
• Use 3 concurrent requests
• Crawl up to 300 URLs (default is 200)
• Set request timeout to 10 seconds (default is 5 seconds)
• Run the crawler for a maximum of 60 seconds (default is 30 seconds)

URL Prefix Filtering Example

To only crawl URLs with specific prefixes:

doc-export --url https://developer.mozilla.org/en-US/docs/Web/JavaScript --output ./javascript-guide --allowed-prefixes https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide,https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference

This example will:

• Start crawling from the MDN JavaScript documentation
• Only process URLs that start with the specified prefixes (Guide and Reference sections)
• Ignore other URLs even if they are within the same domain

How It Works

1. Crawling Phase: The tool starts from the provided URL and crawls all linked pages (respecting domain restrictions and URL prefix filters if specified)
2. Processing Phase: Each HTML page is converted to Markdown using Turndown
3. Aggregation Phase: All Markdown content is combined into a single document with a table of contents

Filtering Options

The crawler supports two types of URL filtering:

1. Domain Filtering (--same-domain): When enabled, only URLs from the same domain as the starting URL will be crawled.
2. Prefix Filtering (--allowed-prefixes): When specified, only URLs that start with one of the provided prefixes will be crawled. This is useful for limiting the crawl to specific sections of a website.

These filters can be combined to precisely target the content you want to extract.

Implementation Details

• Uses bloom filters for efficient link deduplication
• Implements connection reuse with undici fetch
• Handles memory management for large documents
• Processes pages in parallel for maximum efficiency
• Implements timeouts and limits to prevent crawling issues

PDF Support

The current implementation outputs Markdown (.md) files. To convert to PDF, you can use a third-party tool such as:

• Pandoc: pandoc -f markdown -t pdf -o output.pdf document.md
• mdpdf: mdpdf document.md
• Or any online Markdown to PDF converter

License

MIT