XCrawl MCP Server

Model Context Protocol (MCP) server for XCrawl. It exposes scraping, search, map, and crawl tools to MCP clients.

Table of Contents

• Prerequisites
• Quick Start (Stdio)
• Claude Desktop Configuration
• Cloudflare Workers Deployment
• Authentication
• Available Tools
• Request Defaults
• Error Format
• Development
• License
• Support

Prerequisites

• Node.js 18+
• XCrawl API key from xcrawl.com
• Cloudflare account (only for Workers deployment)

Quick Start (Stdio)

Run directly with npx:

XCRAWL_API_KEY=your-api-key npx -y xcrawl-mcp

Or install globally:

npm install -g xcrawl-mcp
XCRAWL_API_KEY=your-api-key xcrawl-mcp

Claude Desktop Configuration

Add to Claude Desktop config:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "xcrawl": {
      "command": "npx",
      "args": ["-y", "xcrawl-mcp"],
      "env": {
        "XCRAWL_API_KEY": "your-api-key"
      }
    }
  }
}

Cloudflare Workers Deployment

git clone <your-repo>
cd xcrawl-mcp
npm install
npm run deploy

After deployment:

• https://xcrawl-mcp.<your-account>.workers.dev/mcp
• https://xcrawl-mcp.<your-account>.workers.dev/health

Authentication

Stdio mode

Use environment variable:

• XCRAWL_API_KEY (required)

Workers mode

Pass API key in request headers (priority order):

1. Authorization: Bearer <api-key>
2. x-api-key: <api-key>
3. x-xcrawl-api-key: <api-key>

Available Tools

| Tool | Purpose | |------|---------| | xcrawl_scrape | Scrape a single page | | xcrawl_check_status | Check async scrape status | | xcrawl_search | Google SERP search | | xcrawl_map | Discover URLs from a site | | xcrawl_crawl | Async multi-page crawl | | xcrawl_check_crawl_status | Check async crawl status |

xcrawl_scrape

Basic request:

{
  "url": "https://example.com"
}

With extraction (json.prompt and json.json_schema are optional):

{
  "url": "https://example.com",
  "output": {
    "formats": ["json"],
    "json": {
      "prompt": "Extract product name and price",
      "json_schema": {
        "type": "object",
        "properties": {
          "name": { "type": "string" },
          "price": { "type": "number" }
        }
      }
    }
  }
}

xcrawl_check_status

{
  "scrape_id": "abc-123-def-456"
}

Possible values: pending, crawling, completed, failed.

xcrawl_search

{
  "query": "latest AI news",
  "location": "New York",
  "language": "en",
  "limit": 10
}

xcrawl_map

{
  "url": "https://example.com",
  "filter": "blog/.*",
  "limit": 1000,
  "include_subdomains": true,
  "ignore_query_parameters": true
}

xcrawl_crawl

{
  "url": "https://example.com",
  "crawler": {
    "limit": 100,
    "include": ["products/.*"],
    "exclude": ["admin/.*", "login/.*"],
    "max_depth": 3
  },
  "output": {
    "formats": ["markdown"]
  }
}

xcrawl_check_crawl_status

{
  "crawl_id": "xyz-789-abc-012"
}

Possible values: pending, crawling, completed, failed.

Request Defaults

Common defaults: Defaults may vary by API version.

| Parameter | Default | Notes | |-----------|---------|-------| | mode | "sync" | Scrape only | | proxy.location | "US" | ISO-3166-1 alpha-2 | | request.device | "desktop" | desktop or mobile | | request.only_main_content | true | Main content filtering | | request.block_ads | true | Ad blocking | | request.skip_tls_verification | true | Skip TLS verification | | js_render.enabled | true | JavaScript rendering | | js_render.wait_until | "load" | load, domcontentloaded, networkidle | | output.formats | [] | If omitted or [], returns metadata only | | output.screenshot | "viewport" | viewport or full_page | | output.json.prompt | - | Optional | | output.json.json_schema | - | Optional | | webhook.events | ["started","completed","failed"] | Optional callback events |

Error Format

Errors are returned in MCP format:

{
  "content": [
    {
      "type": "text",
      "text": "Error: XCrawl API error: 401 Unauthorized - Invalid API key"
    }
  ],
  "isError": true
}

Development

npm install
npm run build
XCRAWL_API_KEY=your-key npm run start:stdio:dev

npm run dev
curl http://localhost:8787/health

License

MIT

Support

• XCrawl API: xcrawl.com
• MCP Server: Open an issue in this repository