Web Downloader CLI

A terminal tool for archiving complete web pages (HTML, CSS, JavaScript, images, fonts) for offline viewing or markdown export.

wget grabs the file. curl fetches the bytes. web-downloader captures the whole page: rewrites links to local paths, follows the asset graph, and ships you a working offline copy or a clean markdown export.

Features

• Downloads HTML, CSS, JavaScript, images, and other assets
• Rewrites links to local references so pages work offline
• Downloads CDN-hosted page assets, responsive srcset images, CSS url(...) dependencies, and runtime assets referenced from JavaScript/JSON
• Removes stale integrity/crossorigin attributes after local asset rewrites
• Downloads assets concurrently with a configurable limit
• Shows an interactive terminal progress bar for asset downloads
• Saves assets under _assets/ and writes _download-manifest.json with success/failure details
• Optional conversion to Markdown
• Configurable depth for following links
• robots.txt modes: obey, warn, or ignore
• Follows redirects automatically
• Respects crawl-delay directives
• Programmatic API for use as a Node.js module

Install

npm install -g @hashmivue/web-downloader-cli

Usage

web-downloader https://example.com

Options

  -h, --help              Show help message
  -o, --output <dir>      Output directory (default: ./downloaded-site)
  --no-assets             Skip CSS, JS, images, and other assets
  -f, --follow-links      Follow and download linked pages
  -d, --depth <n>         Maximum link-follow depth (default: 1)
  -c, --concurrency <n>   Concurrent asset downloads (default: 6)
  --no-progress           Disable the interactive asset progress bar
  -m, --markdown          Convert HTML to Markdown
  -r, --robots <mode>     How to handle robots.txt (default: ignore)
                          ignore - skip robots.txt entirely
                          obey   - respect rules, skip blocked URLs
                          warn   - warn but proceed

Examples

# Single page with all assets
web-downloader https://example.com

# Use more parallel asset downloads
web-downloader -c 10 https://example.com

# Convert to Markdown
web-downloader -m https://example.com/article

# Follow linked pages, depth 2, custom output
web-downloader -f -d 2 -o ./my-site https://example.com

# Multiple pages in one run
web-downloader https://example.com/page1 https://example.com/page2

# HTML only, skip assets
web-downloader --no-assets https://example.com

# Respect robots.txt
web-downloader -r obey -f https://example.com

Programmatic API

const WebDownloader = require('@hashmivue/web-downloader-cli');

const downloader = new WebDownloader({
  outputDir: './my-downloads',
  includeAssets: true,
  downloadExternalAssets: true,
  assetDir: '_assets',
  concurrency: 6,
  progress: true,
  followLinks: true,
  maxDepth: 2,
  markdown: false,
  robotsTxt: 'obey'
});

downloader.download('https://example.com');

When to use what

| You want to... | Reach for | |---|---| | Download a single file | curl or wget | | Mirror a static site | wget --mirror | | Save one page to read offline (with images, working CSS) | web-downloader | | Save a page as Markdown for LLM ingestion | web-downloader -m | | Crawl a JS-heavy site | A real browser tool (Playwright, Puppeteer) |

License

MIT

Contributing

PRs welcome. Open an issue first for anything substantial.