@mdream/crawl

Multi-page website crawler that generates llms.txt files. Follows internal links and converts HTML to Markdown using mdream.

Upgrading from v1? Read Migrating from v1.

Setup

npm install @mdream/crawl

After a global install (npm install -g @mdream/crawl), run the mdream-crawl command. With npx, use the package name: npx @mdream/crawl.

For JavaScript-heavy sites that require browser rendering, install the optional Playwright dependencies:

npm install crawlee playwright
npx playwright install chromium

CLI Usage

Interactive Mode

Run without arguments to start the interactive prompt-based interface:

npx @mdream/crawl

Direct Mode

Pass arguments directly to skip interactive prompts:

npx @mdream/crawl -u https://docs.example.com

CLI Options

Most flags set the option with the same name in camelCase. For example, --max-pages sets maxPages.

| Flag | Alias | Description | Default | |------|-------|-------------|---------| | --url <url> | -u | URL to crawl. Accepts glob patterns. Repeatable. You can also pass URLs without a flag | Required | | --output <dir> | -o | Output directory | output | | --depth <number> | -d | Link depth from 0 to 10. See Depth | 3 | | --single-page | | Same as --depth 0. It takes precedence over --depth | | | --driver <type> | | Crawler driver: http or playwright | http | | --artifacts <list> | | Comma-separated list of llms.txt, llms-full.txt, markdown | all three | | --origin <url> | | Origin for relative links in page Markdown | the origin of each page | | --site-name <name> | | Site name in llms.txt | the home page title | | --description <text> | | Site description in llms.txt | the home page description | | --max-pages <number> | | Maximum number of pages to fetch | no limit | | --crawl-delay <seconds> | | Delay between requests in seconds | the robots.txt Crawl-delay, or none | | --exclude <pattern> | | Skip URLs that match a glob pattern. Repeatable | none | | --skip-sitemap | | Skip sitemap.xml and robots.txt discovery | false | | --sitemap <url> | | Use this sitemap instead of auto-discovery. Repeatable for multi-part sitemaps | auto-discovered | | --allow-subdomains | | Crawl other subdomains of the same root domain | false | | --keep-boilerplate | | Keep repeated navigation and footers in page Markdown. Sets stripBoilerplate: false | false | | --boilerplate-threshold <n> | | Fraction of pages that a block must appear in to count as boilerplate. Greater than 0 and at most 1 | 0.5 | | --verbose | -v | Log details for each failed URL | false | | --quiet | -q, --silent | Drop all logs. Use it to keep stdout clean for JSON-RPC or MCP | false | | --help | -h | Show the help message | | | --version | | Show the version number | |

A flag can also take its value as --flag=value. If a flag is unknown, the CLI exits with code 1. A long flag needs two dashes: -url is an error.

Flags override values from the config file. Config values override the defaults.

CLI Examples

# Basic crawl with specific artifacts
npx @mdream/crawl -u harlanzw.com --artifacts "llms.txt,markdown"

# Shallow crawl (depth 2) with only llms-full.txt output
npx @mdream/crawl --url https://docs.example.com --depth 2 --artifacts "llms-full.txt"

# Exclude admin and API routes
npx @mdream/crawl -u example.com --exclude "*/admin/*" --exclude "*/api/*"

# Single page mode (no link following)
npx @mdream/crawl -u example.com/pricing --single-page

# Use Playwright for JavaScript-heavy sites
npx @mdream/crawl -u example.com --driver playwright

# Skip sitemap discovery with verbose output
npx @mdream/crawl -u example.com --skip-sitemap --verbose

# Crawl across subdomains (docs.example.com, blog.example.com, etc.)
npx @mdream/crawl -u example.com --allow-subdomains

# Override site metadata
npx @mdream/crawl -u example.com --site-name "My Company" --description "Company documentation"

Glob Patterns

Use glob patterns to select URLs. If you provide a pattern, the crawler uses sitemap discovery to find matching URLs.

# Crawl only the /docs/ section
npx @mdream/crawl -u "docs.example.com/docs/**"

# Crawl pages matching a prefix
npx @mdream/crawl -u "example.com/blog/2024*"

Patterns are matched against the URL pathname using picomatch syntax. A trailing single * (e.g. /fieldtypes*) automatically expands to match both the path itself and all subdirectories.

Programmatic API

crawlAndGenerate(options, onProgress?)

The main entry point for programmatic use. Returns a Promise<CrawlResult[]>.

import { crawlAndGenerate } from '@mdream/crawl'

const results = await crawlAndGenerate({
  urls: ['https://docs.example.com'],
  output: './output',
})

CrawlOptions

crawlAndGenerate, the config file, and the CLI use the same options and the same defaults. Only urls is required.

| Option | Type | Default | Description | |--------|------|---------|-------------| | urls | string[] | Required | Starting URLs. Glob patterns limit the crawl | | output | string | 'output' | Output directory, relative to the working directory | | depth | number | 3 | Link depth from 0 to 10. See Depth | | maxPages | number | no limit | Maximum number of pages to fetch | | artifacts | ('llms.txt' \| 'llms-full.txt' \| 'markdown')[] | all three | Files to write. [] writes no files, for example when only hooks consume the pages | | driver | 'http' \| 'playwright' | 'http' | Crawler driver | | sitemap | string \| string[] | auto-discovered | Sitemap URLs to use instead of auto-discovery. All parts are loaded and merged. Ignored when skipSitemap is set | | skipSitemap | boolean | false | Skip sitemap.xml and robots.txt discovery | | siteName | string | the home page title | Site name in llms.txt | | description | string | the home page description | Site description in llms.txt | | origin | string | the origin of each page | Origin for relative links in page Markdown. The page path stays the same | | exclude | string[] | [] | Glob patterns for URLs to skip | | crawlDelay | number | the robots.txt Crawl-delay | Delay between requests in seconds | | allowSubdomains | boolean | false | Crawl other subdomains of the same root domain, for example docs.example.com and blog.example.com. Output paths start with the hostname, so files do not collide | | stripBoilerplate | boolean | true | Remove repeated navigation and footers from page Markdown and llms-full.txt. Needs at least 3 pages. Does not run at depth 0 | | boilerplateThreshold | number | 0.5 | Fraction of pages that a block must appear in to count as boilerplate. Greater than 0 and at most 1 | | useChrome | boolean | false | Use the system Chrome with the Playwright driver. The CLI turns it on when it finds Chrome, unless you set it to false | | verbose | boolean | false | Log details for each failed URL | | silent | boolean | false | Drop all diagnostic and progress logs. A logger takes precedence | | logger | CrawlLogger | @clack/prompts on stdout | Sink for diagnostic and progress logs, for example one that writes to stderr | | hooks | Partial<CrawlHooks> | | Functions for each stage of the crawl. See Hooks | | globPatterns | ParsedUrlPattern[] | parsed from urls | Parsed URL patterns that replace the patterns from urls. Advanced |

crawlAndGenerate checks the options before it sends a request. An unknown key or an invalid value throws a TypeError. A removed v1 key, such as maxDepth, gives an error that names its replacement.

Depth

depth sets how far the crawler follows links from the starting URLs.

• 0: The crawler processes only the given URLs. It does not read robots.txt or the sitemap, and it does not follow links.
• 1 or more: Unless skipSitemap is set, the crawler reads robots.txt and the sitemap first. If the sitemap gives URLs, the crawler fetches those URLs and does not follow links. If not, the crawler follows links up to depth hops from the starting URLs.

The maximum is 10. Use maxPages to limit the total number of pages.

CrawlResult

interface CrawlResult {
  url: string
  title: string
  content: string
  filePath?: string // Set when artifacts include 'markdown'
  timestamp: number // Unix timestamp of processing time
  success: boolean
  error?: string // Set when success is false
  metadata?: PageMetadata
  depth?: number // Link-following depth at which this page was found
}

interface PageMetadata {
  title: string
  description?: string
  keywords?: string
  author?: string
  links: string[] // Internal links discovered on the page
}

PageData

The shape passed to the crawl:page hook:

interface PageData {
  url: string
  html: string // Raw HTML (empty string if content was already markdown)
  title: string
  metadata: PageMetadata
  origin: string
}

Progress Callback

The optional second argument to crawlAndGenerate receives progress updates:

await crawlAndGenerate(options, (progress) => {
  // progress.sitemap.status: 'discovering' | 'processing' | 'completed'
  // progress.sitemap.found: number of sitemap URLs found
  // progress.sitemap.processed: number of URLs after filtering

  // progress.crawling.status: 'starting' | 'processing' | 'completed'
  // progress.crawling.total: total URLs to process
  // progress.crawling.processed: pages completed so far
  // progress.crawling.failed: pages that errored
  // progress.crawling.currentUrl: URL currently being fetched
  // progress.crawling.latency: { total, min, max, count } in ms

  // progress.generation.status: 'idle' | 'generating' | 'completed'
  // progress.generation.current: description of current generation step
})

Examples

Custom page processing with crawl:page

import { crawlAndGenerate } from '@mdream/crawl'

const pages = []

await crawlAndGenerate({
  urls: ['https://docs.example.com'],
  artifacts: [],
  hooks: {
    'crawl:page': (page) => {
      pages.push({
        url: page.url,
        title: page.title,
        description: page.metadata.description,
      })
    },
  },
})

console.log(`Discovered ${pages.length} pages`)

Glob filtering with exclusions

import { crawlAndGenerate } from '@mdream/crawl'

await crawlAndGenerate({
  urls: ['https://example.com/docs/**'],
  output: './docs-output',
  exclude: ['/docs/deprecated/*', '/docs/internal/*'],
  depth: 2,
})

Crawling across subdomains

await crawlAndGenerate({
  urls: ['https://example.com'],
  allowSubdomains: true, // Will also crawl docs.example.com, blog.example.com, etc.
  depth: 2,
})

Single-page mode

Set depth: 0 to process only the given URLs, with no discovery and no link following:

await crawlAndGenerate({
  urls: ['https://example.com/pricing', 'https://example.com/about'],
  depth: 0,
})

Config File

To set options and register hooks, create mdream.config.ts where you run the CLI. The file can also use .js or .mjs. The CLI loads it with c12 in direct and interactive modes.

import { defineConfig } from '@mdream/crawl'

export default defineConfig({
  exclude: ['*/admin/*', '*/internal/*'],
  depth: 2,
  maxPages: 500,
  artifacts: ['llms.txt', 'markdown'],
  hooks: {
    'crawl:page': (page) => {
      // Strip branding from all page titles
      page.title = page.title.replace(/ \| My Brand$/, '')
    },
  },
})

The config file accepts every CrawlOptions key. Each key is optional. If you set urls, you can run the CLI without a URL argument. Interactive mode uses the config values as its initial answers.

The CLI resolves each option in this order:

1. A CLI flag, or an answer in interactive mode
2. The config file
3. The default

exclude is the exception: the CLI adds its patterns to the config patterns. An unknown key in the config file stops the CLI with an error.

Hooks

Five hooks let you change data during the crawl. Each hook receives a mutable object. Change its properties to transform the output.

crawl:url

Called before fetching a URL. Set ctx.skip = true to skip it entirely (saves the network request).

defineConfig({
  hooks: {
    'crawl:url': (ctx) => {
      // Skip large asset pages
      if (ctx.url.includes('/assets/') || ctx.url.includes('/downloads/'))
        ctx.skip = true
    },
  },
})

crawl:html

Runs after fetching, before HTML-to-Markdown conversion. Change ctx.html to remove elements or add content before conversion. The hook does not run if the fetched content is already Markdown.

defineConfig({
  hooks: {
    'crawl:html': (ctx) => {
      // ctx.url, ctx.html, ctx.origin
      ctx.html = ctx.html.replace(/<header[\s\S]*?<\/header>/, '')
    },
  },
})

crawl:page

Called after HTML-to-Markdown conversion, before storage. Mutate page.title or other fields. This hook replaces the v1 onPage option.

defineConfig({
  hooks: {
    'crawl:page': (page) => {
      // page.url, page.html, page.title, page.metadata, page.origin
      page.title = page.title.replace(/ - Docs$/, '')
    },
  },
})

crawl:content

Called before markdown is written to disk. Transform the final output content or change the file path.

defineConfig({
  hooks: {
    'crawl:content': (ctx) => {
      // ctx.url, ctx.title, ctx.content, ctx.filePath
      ctx.content = ctx.content.replace(/CONFIDENTIAL/g, '[REDACTED]')
      ctx.filePath = ctx.filePath.replace('.md', '.mdx')
    },
  },
})

crawl:done

Called after all pages are crawled, before llms.txt generation. Filter or reorder results.

defineConfig({
  hooks: {
    'crawl:done': (ctx) => {
      // Remove short pages from the final output
      const filtered = ctx.results.filter(r => r.content.length > 100)
      ctx.results.length = 0
      ctx.results.push(...filtered)
    },
  },
})

Programmatic Hooks

Hooks can also be passed directly to crawlAndGenerate:

import { crawlAndGenerate } from '@mdream/crawl'

await crawlAndGenerate({
  urls: ['https://example.com'],
  output: './output',
  hooks: {
    'crawl:page': (page) => {
      page.title = page.title.replace(/ \| Brand$/, '')
    },
    'crawl:done': (ctx) => {
      ctx.results.sort((a, b) => a.url.localeCompare(b.url))
    },
  },
})

Crawl Drivers

HTTP Driver (default)

Uses ofetch for page fetching with up to 20 concurrent requests.

• Automatic retry (2 retries with 500ms delay)
• 10 second request timeout
• Respects Retry-After headers on 429 responses (automatically adjusts crawl delay)
• Detects text/markdown content types and skips HTML-to-Markdown conversion

Playwright Driver

For sites that require a browser to render content. Requires crawlee and playwright as peer dependencies (see Setup).

npx @mdream/crawl -u example.com --driver playwright

await crawlAndGenerate({
  urls: ['https://spa-app.example.com'],
  output: './output',
  driver: 'playwright',
})

Waits for networkidle before extracting content. Automatically detects and uses system Chrome when available, falling back to Playwright's bundled browser.

Sitemap and Robots.txt Discovery

By default, the crawler performs sitemap discovery before crawling:

1. Fetches robots.txt to find Sitemap: directives and Crawl-delay values
2. Loads sitemaps referenced in robots.txt (authoritative, tried first)
3. Falls back to /sitemap.xml
4. Tries common alternatives: /sitemap_index.xml, /sitemaps.xml, /sitemap-index.xml
5. Supports sitemap index files (recursively loads child sitemaps, with cycle protection)
6. Filters discovered URLs against glob patterns and exclusion rules

The first candidate that yields matching URLs wins, so a robots.txt sitemap is never overwritten by a well-known fallback. <loc> values are CDATA-unwrapped and XML-entity-decoded (& → &).

The home page is always included for metadata extraction (site name, description).

Disable with --skip-sitemap or skipSitemap: true.

Overriding the sitemap location

If a site uses a custom sitemap URL or several sitemap files, pass their locations explicitly:

# Single non-standard location
npx @mdream/crawl -u example.com --sitemap https://example.com/custom/sitemap.xml

# Multiple parts (repeatable), all loaded and merged
npx @mdream/crawl -u example.com \
  --sitemap https://example.com/sitemap-posts.xml \
  --sitemap https://example.com/sitemap-pages.xml

Or in mdream.config.ts:

import { defineConfig } from '@mdream/crawl'

export default defineConfig({
  sitemap: [
    'https://example.com/sitemap-posts.xml',
    'https://example.com/sitemap-pages.xml',
  ],
})

An explicit sitemap replaces auto-discovery entirely (robots.txt is still read for Crawl-delay).

Artifacts

Choose the files with the artifacts option or the --artifacts flag. By default, the crawler writes all three.

markdown

One .md file per crawled page, written to the output directory preserving the URL path structure. For example, https://example.com/docs/getting-started becomes output/docs/getting-started.md.

llms.txt

A site overview file following the llms.txt specification, listing all crawled pages with titles and links to their markdown files.

# example.com

## Pages

- [Example Domain](index.md): https://example.com/
- [About Us](about.md): https://example.com/about

llms-full.txt

Same structure as llms.txt but includes the full markdown content of every page inline.

Migrating from v1

These docs cover v2. For an earlier release, read the README at its release tag.

In v2, crawlAndGenerate, the config file, and the CLI accept the same options and defaults. The library uses the names of the CLI flags.

Renamed options

| v1 | v2 | |----|----| | outputDir | output. It is optional now, and the default is 'output' | | maxRequestsPerCrawl | maxPages | | maxDepth and followLinks | depth. See Depth | | sitemapUrls | sitemap. It also accepts one string | | generateLlmsTxt, generateLlmsFullTxt, generateIndividualMd | artifacts: ['llms.txt', 'llms-full.txt', 'markdown'] | | siteNameOverride | siteName | | descriptionOverride | description | | onPage: fn | hooks: { 'crawl:page': fn } | | chunkSize | Remove it. It had no effect | | maxDepth in mdream.config | depth | | MdreamCrawlConfig type | Partial<CrawlOptions> | | crawl command after a global install | mdream-crawl |

  await crawlAndGenerate({
    urls: ['https://example.com'],
-   outputDir: './output',
-   maxRequestsPerCrawl: 100,
-   followLinks: true,
-   maxDepth: 2,
-   generateLlmsFullTxt: true,
-   onPage: page => console.log(page.url),
+   output: './output',
+   maxPages: 100,
+   depth: 2,
+   artifacts: ['llms.txt', 'llms-full.txt', 'markdown'],
+   hooks: { 'crawl:page': page => console.log(page.url) },
  })

A v1 key now throws a TypeError that names its replacement. A key with the value undefined is ignored. The library also checks values the same way as the CLI. For example, depth must be an integer from 0 to 10.

New library defaults

The library defaults are now the CLI defaults:

• depth is 3, so the crawler follows links when the sitemap gives no URLs. By default, the v1 library did not follow links.
• artifacts includes llms-full.txt. By default, the v1 library did not write it. To keep the v1 output, set artifacts: ['llms.txt', 'markdown'].
• output is 'output'. In v1, outputDir was required.

v1 with followLinks: false used the sitemap and did not follow links. v2 has no exact match. Use depth: 0 to process only the given URLs, or depth: 1 to follow links one hop when there is no sitemap.

For your first v2 crawl, process only the supplied URLs and write to a separate directory:

import { crawlAndGenerate } from '@mdream/crawl'

await crawlAndGenerate({
  urls: ['https://example.com'],
  output: './output-v2',
  depth: 0,
  maxPages: 100,
  artifacts: ['llms.txt', 'markdown'],
})

Review the generated files before replacing the previous artifacts. To follow links, increase depth. This example skips sitemap discovery.

CLI changes

• Config values for maxPages, artifacts, driver, and depth now apply. In v1, the CLI ignored them.
• Interactive mode loads the config file and its hooks.
• An unknown flag or a single-dash long flag, such as -url, stops the CLI with exit code 1.
• --crawl-delay accepts decimal values, such as 0.5.