ZanScrape

Web scraping MCP server. Extracts clean markdown content from web pages, crawls websites via BFS, discovers URLs from sitemaps, and searches the web -- all through the Model Context Protocol.

Built with TypeScript, strict mode, hexagonal architecture.

Quick Start

Via npx (recommended)

npx zanscrape

Via npm

npm install -g zanscrape
zanscrape

Via pnpm

pnpm add -g zanscrape
zanscrape

Configuration

MCP Client

Add to your MCP client configuration (Claude Desktop, Cursor, etc.):

{
  "mcpServers": {
    "zanscrape": {
      "command": "npx",
      "args": ["zanscrape"]
    }
  }
}

Docker variant:

{
  "mcpServers": {
    "zanscrape": {
      "command": "docker",
      "args": ["run", "--rm", "-i", "ghcr.io/zansoft/zanscrape:latest"]
    }
  }
}

Environment Variables

| Variable | Default | Description | |---|---|---| | LOG_LEVEL | info | Pino log level (fatal, error, warn, info, debug, trace, silent) | | NODE_ENV | — | Set to development to enable pretty-print logging | | LOG_PRETTY | — | Set to true to enable pretty-print logging (alternative to NODE_ENV) | | ZANSCRAPE_API_KEY | — | API key used by the stdio server (required unless ZANSCRAPE_DEV_MODE=true) | | ZANSCRAPE_API_URL | https://zanscrape-api.zansoft.solutions | REST API used to validate API keys | | ZANSCRAPE_DEV_MODE | — | true disables API key authentication (stdio and HTTP). Local development only | | BRAVE_SEARCH_API_KEY | — | Optional. Enables Brave Search as fallback when DuckDuckGo is blocked, unreachable or its queue is full |

HTTP transport authentication

The HTTP server (dist/http-index.js) accepts initialize, ping, tools/list and notifications without credentials, so clients can discover the tools. Every other request (tools/call) and GET /events/:jobId require Authorization: Bearer <api key>:

| Response | When | |---|---| | 401 + WWW-Authenticate: Bearer | Key missing or invalid | | 429 + Retry-After | Key exceeded the requestsPerMin of its plan | | 503 | The REST API could not validate the key | | 400 / 413 | Body is not JSON / larger than 1 MiB |

Valid keys are cached for 5 minutes and invalid keys for 1 minute.

Tools

scrape

Extrai conteúdo limpo de uma URL como markdown (ideal para LLM), com metadados (título, description, OG tags, idioma). Só o conteúdo principal é retornado por padrão, imagens base64 são removidas e a resposta é limitada por maxTokens — truncagem graciosa com marcador em vez de erro.

| Parameter | Type | Default | Description | |---|---|---|---| | url | string (URL) | — | Target URL to scrape | | timeout | number | 30000 | Request timeout in ms (1000–60000) | | formats | array | ['markdown'] | Output formats: markdown, html, text | | onlyMainContent | boolean | true | Extract only main content (strips nav/footer/ads) | | includeTags | string[] | — | CSS selectors to include | | excludeTags | string[] | — | CSS selectors to exclude | | maxTokens | number | 25000 | Estimated token budget; truncates with marker if exceeded | | waitFor | number | 0 | Delay before scraping in ms (0–30000) | | jsRender | boolean \| object | — | Enable JS rendering via headless browser |

Example call from an MCP client:

{
  "url": "https://example.com",
  "timeout": 15000,
  "maxTokens": 10000
}

Returns: Markdown content + metadata (title, description, ogImage, ogSiteName, language, statusCode).

crawl

Crawl a website using BFS to discover URLs within the same domain.

| Parameter | Type | Default | Description | |---|---|---|---| | url | string (URL) | — | Starting URL for the crawl | | maxDepth | number | 2 | Maximum link depth (1–10) | | maxLinks | number | 100 | Maximum URLs to collect (1–10000) | | sameDomain | boolean | true | Restrict crawl to the same domain | | timeout | number | 30000 | Total crawl budget in ms (1000–120000); also caps each request | | delay | number | 1000 | Delay between requests in ms (0–10000) | | webhookUrl | string (URL) | — | Async mode only: receives a POST with the result (public hosts only) | | maxTokens | number | 25000 | Sync mode only: response budget; extra URLs are reported in omittedUrls |

Two modes, chosen by the server:

• Async — server with DATABASE_URL and an authenticated API key (the hosted HTTP server). Returns { id, status: "running" }; the job belongs to the key's user. Poll it with check_crawl_status.
• Sync — no database (e.g. the npm stdio package) or no authenticated user. Runs immediately and returns { status: "completed", total, visited, duration, urls, errors }.

check_crawl_status

Returns the status of an async crawl job started with crawl. Only jobs owned by the calling key are visible.

| Parameter | Type | Default | Description | |---|---|---|---| | id | string (UUID) | — | Job id returned by crawl | | maxTokens | number | 25000 | Response budget; extra URLs are reported in omittedUrls |

map

Discover URLs from a sitemap (supports sitemap.xml and sitemap index files).

| Parameter | Type | Default | Description | |---|---|---|---| | url | string (URL) | — | URL pointing to a sitemap | | maxLinks | number | 1000 | Maximum URLs to extract (1–10000) | | timeout | number | 30000 | Request timeout in ms (1000–60000) | | maxTokens | number | 25000 | Estimated token budget; truncates URL list with marker if exceeded |

Returns: JSON array of discovered URLs + metadata (total).

search

Pesquisa na web e retorna resultados. Breaking change: por padrão retorna apenas snippets (url, title, description) — sem extrair o conteúdo das páginas. Use scrapeContent: true para extrair o markdown de cada página (comportamento antigo). Quem dependia de markdown nos resultados deve passar scrapeContent: true.

| Parameter | Type | Default | Description | |---|---|---|---| | query | string | — | Search query (1–500 characters) | | maxResults | number | 5 | Number of results to fetch (1–20) | | timeout | number | 30000 | Total search budget in ms (1000–60000), retries included | | scrapeContent | boolean | false | Se true, extrai o markdown de cada página além do snippet | | maxTokensPerResult | number | 5000 | Token budget por resultado (apenas quando scrapeContent: true) | | maxTokens | number | 25000 | Token budget total da resposta; trunca com marcador se excedido |

Returns (default): JSON array com url, title, description para cada resultado + metadata (total).

Returns (scrapeContent: true): JSON array com url, title, description, markdown e metadata de página (title, statusCode) por resultado.

extract

Extrai conteúdo estruturado de uma URL como JSON. Aceita prompt e schema opcionais para guiar a extração. Ideal para puxar campos específicos (preços, metadados, listas) em vez de buscar páginas inteiras. O markdown retornado é limitado por maxTokens — truncagem graciosa com marcador.

| Parameter | Type | Default | Description | |---|---|---|---| | url | string (URL) | — | Target URL to extract from | | prompt | string | — | Prompt opcional para guiar a extração (máx 1000 chars) | | schema | string | — | Schema JSON (string) para estruturar o retorno | | onlyMainContent | boolean | true | Extract only main content | | timeout | number | 30000 | Request timeout in ms (1000–60000) | | jsRender | boolean \| object | — | Enable JS rendering via headless browser | | maxTokens | number | 25000 | Estimated token budget; truncates markdown with marker if exceeded |

Returns: JSON com url, statusCode, title, description, language, markdown, prompt (e schema se fornecido).

Fetch behavior

Applies to every tool that fetches a page (scrape, extract, map, crawl, search).

• Redirects are followed, up to 5 hops. Every hop re-runs the SSRF checks (DNS resolution, private ranges, metadata services) before the request is made, so a redirect cannot be used to reach an internal address. Exceeding the hop limit, hitting a redirect cycle, or landing on a blocked target returns an explicit error naming the cause — never empty content.
• A default Accept-Language: en-US,en;q=0.5 is sent. Without it, some documentation sites answer with an intermittent locale redirect instead of the page.
• HTTP status ≥ 400 is an error, not content. scrape and extract return isError citing the status (and Retry-After when the server sends it) instead of converting the error page to markdown. Statuses below 400 are still reported in metadata.statusCode.

Docker

# Build locally
docker build -t zanscrape/zanscrape .

# Via docker-compose
docker compose up --build

# Via GHCR (GitHub Container Registry)
docker run ghcr.io/zansoft/zanscrape:latest

Stack

| Layer | Technology | |---|---| | MCP Protocol | @modelcontextprotocol/sdk 1.29.x | | Runtime | Node.js 24 LTS + TypeScript 6 (strict mode) | | HTML Parsing | linkedom 0.18.x | | HTML→Markdown | node-html-markdown 2.x | | Validation | Zod 4.x | | Logging | Pino 10.x | | Testing | Vitest 4.x | | Linting | ESLint 10 + typescript-eslint 8 (flat config) | | Formatting | Prettier 3 |

Architecture: Hexagonal / Ports & Adapters (tools/ → engine/ → worker/).

Development

# Install dependencies
pnpm install

# Build
pnpm build

# Dev mode with auto-reload
pnpm dev

# Run tests
pnpm test

# Lint and typecheck
pnpm lint
pnpm typecheck

License

MIT