mcp-webscraper

A production-ready Model Context Protocol server that gives any MCP-compatible LLM (Claude Desktop, Cursor, etc.) the ability to scrape the web — returning clean markdown, raw HTML, structured JSON, or plain text.

• Auto-detects JS-rendered pages — uses cheerio + fetch for static HTML, and falls back to headless playwright Chromium when the page needs JavaScript.
• Structured extraction — pass a { field: cssSelector } schema and get back exactly the JSON you asked for.
• Batch + parallel — scrape up to 50 URLs at a time with bounded concurrency.
• Search-and-scrape — query DuckDuckGo (no API key required) and pull the top N results in one call.
• Stdio transport — drops into Claude Desktop / Cursor / any MCP client.

Install

Run on demand with no install:

npx mcp-webscraper

Or install globally:

npm i -g mcp-webscraper

If you plan to scrape JS-heavy sites (or always pass render: "dynamic"), install the headless Chromium binary once:

npx playwright install chromium

Static scraping works without playwright — the browser is only spun up on demand.

Tools

scrape_url

Fetch a single URL.

| Parameter | Type | Default | Description | | --- | --- | --- | --- | | url | string (URL) | — | Page to fetch. | | output_format | markdown | html | json | text | markdown | How to format the result. | | render | auto | static | dynamic | auto | auto upgrades to headless browser if the page looks JS-rendered. | | wait_for | string | — | Optional CSS selector to wait for (dynamic mode only). |

json returns page metadata: title, description, OG tags, headings, top 100 links, top 50 images.

scrape_structured

Extract specific fields with a CSS-selector schema.

{
  "url": "https://news.ycombinator.com",
  "schema": {
    "titles": { "selector": ".titleline > a", "multiple": true },
    "scores": { "selector": ".score", "multiple": true },
    "first_link": { "selector": ".titleline > a", "attr": "href" }
  }
}

A bare string is shorthand for { selector, multiple: false }:

{ "url": "https://example.com", "schema": { "title": "h1" } }

scrape_batch

Scrape up to 50 URLs in parallel (bounded by SCRAPER_MAX_CONCURRENT). Failures are returned inline rather than aborting the batch.

{
  "urls": ["https://example.com", "https://example.org"],
  "output_format": "markdown"
}

search_and_scrape

Web-search a query (via DuckDuckGo's HTML endpoint — no API key required) and scrape the top N results.

{
  "query": "best typescript build tools",
  "num_results": 5,
  "output_format": "markdown"
}

Configuration

All configuration is via environment variables. All are optional.

| Variable | Default | Description | | --- | --- | --- | | SCRAPER_TIMEOUT | 30000 | Per-request timeout in ms. | | SCRAPER_MAX_CONCURRENT | 5 | Max parallel fetches in batch / search-and-scrape. | | SCRAPER_USER_AGENT | Mozilla/5.0 (compatible; mcp-webscraper/0.1) | Sent on every static + dynamic fetch. |

Client setup

Claude Desktop

Edit ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "webscraper": {
      "command": "npx",
      "args": ["-y", "mcp-webscraper"],
      "env": {
        "SCRAPER_TIMEOUT": "30000",
        "SCRAPER_MAX_CONCURRENT": "5"
      }
    }
  }
}

Restart Claude Desktop. The four tools should appear in the slash-tools picker.

Cursor

Edit ~/.cursor/mcp.json (or .cursor/mcp.json in your project root for per-project setup):

{
  "mcpServers": {
    "webscraper": {
      "command": "npx",
      "args": ["-y", "mcp-webscraper"]
    }
  }
}

Reload Cursor and the tools become available to the agent.

Any other MCP client

This is a standard stdio MCP server. Run mcp-webscraper (or npx mcp-webscraper) and connect via stdio.

Example calls

Get a page as markdown:

{ "name": "scrape_url", "arguments": { "url": "https://example.com" } }

Force the headless browser:

{
  "name": "scrape_url",
  "arguments": {
    "url": "https://app.example.com/dashboard",
    "render": "dynamic",
    "wait_for": "[data-loaded='true']"
  }
}

Extract structured data:

{
  "name": "scrape_structured",
  "arguments": {
    "url": "https://news.ycombinator.com",
    "schema": {
      "stories": {
        "selector": ".titleline > a",
        "multiple": true
      }
    }
  }
}

Local development

git clone <this-repo>
cd mcp-webscraper
npm install
npm run build
npm start              # runs dist/index.js as stdio MCP server

To test from the command line without an MCP client:

printf '%s\n' \
  '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"cli","version":"0"}}}' \
  '{"jsonrpc":"2.0","method":"notifications/initialized"}' \
  '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"scrape_url","arguments":{"url":"https://example.com"}}}' \
| node dist/index.js

License

MIT