ZanScrape

Web scraping MCP server. Extracts clean markdown content from web pages, crawls websites via BFS, discovers URLs from sitemaps, and searches the web -- all through the Model Context Protocol.

Built with TypeScript, strict mode, hexagonal architecture.

Quick Start

Via npx (recommended)

npx zanscrape

Via npm

npm install -g zanscrape
zanscrape

Via pnpm

pnpm add -g zanscrape
zanscrape

Configuration

MCP Client

Add to your MCP client configuration (Claude Desktop, Cursor, etc.):

{
  "mcpServers": {
    "zanscrape": {
      "command": "npx",
      "args": ["zanscrape"]
    }
  }
}

Docker variant:

{
  "mcpServers": {
    "zanscrape": {
      "command": "docker",
      "args": ["run", "--rm", "-i", "ghcr.io/zansoft/zanscrape:latest"]
    }
  }
}

Environment Variables

| Variable | Default | Description | |---|---|---| | LOG_LEVEL | info | Pino log level (fatal, error, warn, info, debug, trace, silent) | | NODE_ENV | — | Set to development to enable pretty-print logging | | LOG_PRETTY | — | Set to true to enable pretty-print logging (alternative to NODE_ENV) |

Tools

scrape

Extrai conteúdo limpo de uma URL como markdown (ideal para LLM), com metadados (título, description, OG tags, idioma). Só o conteúdo principal é retornado por padrão, imagens base64 são removidas e a resposta é limitada por maxTokens — truncagem graciosa com marcador em vez de erro.

| Parameter | Type | Default | Description | |---|---|---|---| | url | string (URL) | — | Target URL to scrape | | timeout | number | 30000 | Request timeout in ms (1000–60000) | | formats | array | ['markdown'] | Output formats: markdown, html, text | | onlyMainContent | boolean | true | Extract only main content (strips nav/footer/ads) | | includeTags | string[] | — | CSS selectors to include | | excludeTags | string[] | — | CSS selectors to exclude | | maxTokens | number | 25000 | Estimated token budget; truncates with marker if exceeded | | waitFor | number | 0 | Delay before scraping in ms (0–30000) | | jsRender | boolean \| object | — | Enable JS rendering via headless browser |

Example call from an MCP client:

{
  "url": "https://example.com",
  "timeout": 15000,
  "maxTokens": 10000
}

Returns: Markdown content + metadata (title, description, ogImage, ogSiteName, language, statusCode).

crawl

Crawl a website using BFS to discover URLs within the same domain.

| Parameter | Type | Default | Description | |---|---|---|---| | url | string (URL) | — | Starting URL for the crawl | | maxDepth | number | 2 | Maximum link depth (1–10) | | maxLinks | number | 100 | Maximum URLs to collect (1–10000) | | sameDomain | boolean | true | Restrict crawl to the same domain | | timeout | number | 30000 | Per-request timeout in ms (5000–120000) | | delay | number | 1000 | Delay between requests in ms (0–10000) |

Returns: JSON array of discovered URLs + metadata (total, visited, errors, duration).

map

Discover URLs from a sitemap (supports sitemap.xml and sitemap index files).

| Parameter | Type | Default | Description | |---|---|---|---| | url | string (URL) | — | URL pointing to a sitemap | | maxLinks | number | 1000 | Maximum URLs to extract (1–10000) | | timeout | number | 30000 | Request timeout in ms (1000–60000) | | maxTokens | number | 25000 | Estimated token budget; truncates URL list with marker if exceeded |

Returns: JSON array of discovered URLs + metadata (total).

search

Pesquisa na web e retorna resultados. Breaking change: por padrão retorna apenas snippets (url, title, description) — sem extrair o conteúdo das páginas. Use scrapeContent: true para extrair o markdown de cada página (comportamento antigo). Quem dependia de markdown nos resultados deve passar scrapeContent: true.

| Parameter | Type | Default | Description | |---|---|---|---| | query | string | — | Search query (1–500 characters) | | maxResults | number | 5 | Number of results to fetch (1–20) | | timeout | number | 30000 | Per-request timeout in ms (5000–60000) | | scrapeContent | boolean | false | Se true, extrai o markdown de cada página além do snippet | | maxTokensPerResult | number | 5000 | Token budget por resultado (apenas quando scrapeContent: true) | | maxTokens | number | 25000 | Token budget total da resposta; trunca com marcador se excedido |

Returns (default): JSON array com url, title, description para cada resultado + metadata (total).

Returns (scrapeContent: true): JSON array com url, title, description, markdown e metadata de página (title, statusCode) por resultado.

extract

Extrai conteúdo estruturado de uma URL como JSON. Aceita prompt e schema opcionais para guiar a extração. Ideal para puxar campos específicos (preços, metadados, listas) em vez de buscar páginas inteiras. O markdown retornado é limitado por maxTokens — truncagem graciosa com marcador.

| Parameter | Type | Default | Description | |---|---|---|---| | url | string (URL) | — | Target URL to extract from | | prompt | string | — | Prompt opcional para guiar a extração (máx 1000 chars) | | schema | string | — | Schema JSON (string) para estruturar o retorno | | onlyMainContent | boolean | true | Extract only main content | | timeout | number | 30000 | Request timeout in ms (5000–60000) | | jsRender | boolean \| object | — | Enable JS rendering via headless browser | | maxTokens | number | 25000 | Estimated token budget; truncates markdown with marker if exceeded |

Returns: JSON com url, statusCode, title, description, language, markdown, prompt (e schema se fornecido).

Docker

# Build locally
docker build -t zanscrape/zanscrape .

# Via docker-compose
docker compose up --build

# Via GHCR (GitHub Container Registry)
docker run ghcr.io/zansoft/zanscrape:latest

Stack

| Layer | Technology | |---|---| | MCP Protocol | @modelcontextprotocol/sdk 1.29.x | | Runtime | Node.js 24 LTS + TypeScript 6 (strict mode) | | HTML Parsing | linkedom 0.18.x | | HTML→Markdown | node-html-markdown 2.x | | Validation | Zod 4.x | | Logging | Pino 10.x | | Testing | Vitest 4.x | | Linting | ESLint 10 + typescript-eslint 8 (flat config) | | Formatting | Prettier 3 |

Architecture: Hexagonal / Ports & Adapters (tools/ → engine/ → worker/).

Development

# Install dependencies
pnpm install

# Build
pnpm build

# Dev mode with auto-reload
pnpm dev

# Run tests
pnpm test

# Lint and typecheck
pnpm lint
pnpm typecheck

License

MIT