@vibeflow-tools/web-crawl

Crawl any page, get clean markdown. Web crawler that extracts main content, strips boilerplate, and returns LLM-ready markdown — with an MCP server and CLI.

npm install -g @vibeflow-tools/web-crawl
web-crawl crawl https://example.com

Why It Matters

AI agents and LLMs work best with clean, focused content — not raw HTML full of nav bars, cookie banners, and footers. web-crawl fetches a page, extracts the main article via Mozilla Readability, converts it to clean markdown, and optionally filters to topic-relevant sections using an LLM.

• Clean markdown output — boilerplate stripped, GFM preserved
• LLM topic filtering — extract only the sections relevant to your query
• Browser rendering — handles JS-heavy pages via Playwright when needed
• Built-in caching — avoids re-fetching the same URL (8h TTL)
• Web search — search via SearXNG (70+ engines) or DuckDuckGo, no API key required

Quick Start

CLI (recommended)

# Install globally
npm install -g @vibeflow-tools/web-crawl

# Crawl a page
web-crawl crawl https://example.com

# Search the web
web-crawl search "how to use playwright"

# Crawl multiple pages
web-crawl crawl https://a.com https://b.com

# Filter to a specific topic
web-crawl crawl https://docs.example.com --filter "authentication setup"

# Output as JSON
web-crawl crawl https://example.com --format json

MCP Server

Note: The MCP server is not well tested. The CLI is the recommended tool.

Add to your editor's MCP config:

{
  "mcpServers": {
    "web-crawl": {
      "command": "web-crawl-mcp",
      "type": "stdio"
    }
  }
}

Commands

| Command | Description | |---------|-------------| | web-crawl crawl <url> | Crawl a URL and output clean markdown | | web-crawl search <query> | Search the web via SearXNG or DuckDuckGo | | web-crawl status | Show LLM config and cache status |

web-crawl crawl <url> [options]

Crawl one or more URLs and output clean markdown.

web-crawl crawl https://example.com
web-crawl crawl https://a.com https://b.com          # multi-URL
web-crawl crawl https://docs.example.com --filter "auth"  # topic filter
web-crawl crawl https://example.com --browser         # force browser rendering
web-crawl crawl https://example.com --no-cache        # skip cache
web-crawl crawl https://example.com --meta            # include OG tags, description
web-crawl crawl https://example.com --format json     # JSON output

| Flag | Default | Description | |------|---------|-------------| | --filter <topic> | (none) | Filter content to topic-relevant sections via LLM | | --no-cache | cache on | Disable result caching | | --browser | auto | Force browser rendering (Playwright) | | --no-browser | auto | Force HTTP-only, no browser | | --cache-key <key> | (auto) | Custom cache key override | | --retry <n> | 0 | Retry count on 429/503 errors | | --timeout <ms> | 20000 | HTTP request timeout in ms | | --meta | false | Include page metadata (OG tags, description, canonical) | | --concurrent <n> | 5 | Max concurrent crawls for multi-URL mode | | --format <fmt> | markdown | Output format: markdown or json |

web-crawl search <query> [options]

Search the web via SearXNG (70+ engines aggregated) or DuckDuckGo. Free, no API key required.

web-crawl search "playwright browser automation"
web-crawl search "node.js best practices" --max-results 5
web-crawl search "typescript generics" --lang en-us
web-crawl search "react hooks" --engine ddg

| Flag | Default | Description | |------|---------|-------------| | --max-results <n> | 10 | Maximum number of results | | --lang <code> | (none) | Language/region for results (e.g., de-at, en-us) | | --engine <name> | searxng | Search engine: searxng or ddg | | --format <fmt> | markdown | Output format: markdown or json |

MCP Server

Note: The MCP server is not well tested. The CLI is the recommended tool.

The MCP server (web-crawl-mcp) exposes two tools over stdio:

crawl

| Parameter | Type | Required | Description | |-----------|------|----------|-------------| | url | string | Yes* | URL to crawl | | urls | string[] | Yes* | Multiple URLs to crawl concurrently | | filter_topic | string | No | Filter to topic-relevant sections via LLM | | use_cache | boolean | No | Use cached result (default: true) | | use_browser | boolean | No | Use browser rendering (default: auto) | | cache_key | string | No | Custom cache key override | | retry | number | No | Max retries on 429/503 errors | | timeout_ms | number | No | HTTP timeout in ms | | include_meta | boolean | No | Include page metadata | | concurrency | number | No | Max concurrent crawls (default: 5) |

*Either url or urls is required.

search

| Parameter | Type | Required | Description | |-----------|------|----------|-------------| | query | string | Yes | Search query | | max_results | number | No | Maximum results (default: 10) | | lang | string | No | Language/region (e.g., en-us) | | engine | "searxng" | "ddg" | No | Search engine (default: searxng) |

MCP Server Configuration

| Flag | Description | |------|-------------| | --llm-enabled | Enable LLM topic filtering | | --llm-url <url> | OpenAI-compatible endpoint | | --llm-token <token> | API token | | --llm-model <model> | Model name (default: gpt-4o) | | --cache-ttl <seconds> | Cache TTL (default: 28800) | | --http-timeout <ms> | HTTP timeout (default: 20000) | | --browser-timeout <ms> | Browser timeout (default: 30000) | | --no-browser-headless | Disable headless browser mode |

Configuration

Environment Variables

| Variable | Default | Description | |----------|---------|-------------| | CRAWL_MCP_LLM_ENABLED | false | Enable LLM topic filtering | | CRAWL_MCP_LLM_URL | (none) | OpenAI-compatible chat completions endpoint | | CRAWL_MCP_LLM_TOKEN | (none) | API token for the LLM endpoint | | CRAWL_MCP_LLM_MODEL | gpt-4o | Model name for LLM filtering | | CRAWL_MCP_CACHE_TTL_SECONDS | 28800 | Cache time-to-live (8 hours) | | CRAWL_MCP_HTTP_TIMEOUT_MS | 20000 | HTTP request timeout | | CRAWL_MCP_BROWSER_TIMEOUT_MS | 30000 | Browser page-load timeout | | CRAWL_MCP_BROWSER_HEADLESS | true | Set to false for visible browser |

A .env file in the current working directory is loaded automatically (does not overwrite existing env vars). CLI flags override environment variables.

Example

# Enable LLM topic filtering
CRAWL_MCP_LLM_ENABLED=true
CRAWL_MCP_LLM_URL=https://api.openai.com/v1/chat/completions
CRAWL_MCP_LLM_TOKEN=sk-...
CRAWL_MCP_LLM_MODEL=gpt-4o

Or via MCP server flags:

web-crawl-mcp \
  --llm-enabled \
  --llm-url https://api.openai.com/v1/chat/completions \
  --llm-token sk-... \
  --llm-model gpt-4o

How It Works

URL → Rate Limiter → HTTP Fetch (or Browser) → Readability → Turndown MD → Clean → [LLM Filter] → Cache → Return

1. Rate limiter — 2s minimum delay, 50-request session cap with 60s cooldown
2. Fetch — HTTP with gzip/brotli, redirect following, ETag/Last-Modified validation; or Playwright browser for JS-heavy pages
3. Extract — Mozilla Readability pulls main article content
4. Convert — Turndown + GFM produces clean markdown
5. Clean — strips nav, breadcrumbs, footers, and other boilerplate
6. Filter — optional LLM topic filtering (two-pass for structured content, single-pass for flat)
7. Cache — in-memory TTL cache with conditional re-fetching

Installation

npm install -g @vibeflow-tools/web-crawl

Or run without installing:

npx @vibeflow-tools/web-crawl crawl https://example.com

Requirements: Node.js >= 22

Contributing

pnpm install
pnpm --filter @vibeflow-tools/web-crawl run build
pnpm --filter @vibeflow-tools/web-crawl run test
pnpm --filter @vibeflow-tools/web-crawl run test:coverage
pnpm --filter @vibeflow-tools/web-crawl run mutation

License

Apache-2.0