XCrawl MCP Server

Model Context Protocol (MCP) server for XCrawl. It exposes scraping, search, map, and crawl tools to MCP clients.

Table of Contents

• Prerequisites
• Remote Hosted URL
• Quick Start (Stdio)
• Claude Desktop Configuration
• Authentication
• Available Tools
• Request Defaults
• Error Format
• Development
• License
• Support

Prerequisites

• Node.js 18+
• XCrawl API key from xcrawl.com

Remote Hosted URL

Use the hosted MCP endpoint:

https://mcp.xcrawl.com/{XCRAWL_API_KEY}/mcp

You can pass the API key in the URL path:

https://mcp.xcrawl.com/{XCRAWL_API_KEY}/mcp

Or via headers. Supported methods:

Authorization: Bearer {XCRAWL_API_KEY}
x-api-key: {XCRAWL_API_KEY}
x-xcrawl-api-key: {XCRAWL_API_KEY}

Quick Start (Stdio)

Run directly with npx:

XCRAWL_API_KEY=YOUR_API_KEY npx -y xcrawl-mcp

Or install globally:

npm install -g xcrawl-mcp
XCRAWL_API_KEY=YOUR_API_KEY xcrawl-mcp

Claude Desktop Configuration

Add to Claude Desktop config:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

Use the hosted MCP endpoint:

{
  "mcpServers": {
    "xcrawl": {
      "url": "https://mcp.xcrawl.com/{XCRAWL_API_KEY}/mcp"
    }
  }
}

Or run locally:

{
  "mcpServers": {
    "xcrawl": {
      "command": "npx",
      "args": ["-y", "xcrawl-mcp"],
      "env": {
        "XCRAWL_API_KEY": "YOUR_API_KEY"
      }
    }
  }
}

Authentication

Stdio mode

Use environment variable:

• XCRAWL_API_KEY (required)

Hosted / Workers mode

Pass API key using one of these methods:

1. URL path: https://mcp.xcrawl.com/{XCRAWL_API_KEY}/mcp
2. Authorization: Bearer {XCRAWL_API_KEY}
3. x-api-key: {XCRAWL_API_KEY}
4. x-xcrawl-api-key: {XCRAWL_API_KEY}

Available Tools

| Tool | Purpose | |------|---------| | xcrawl_scrape | Scrape a single page | | xcrawl_check_status | Check async scrape status | | xcrawl_search | Google SERP search | | xcrawl_map | Discover URLs from a site | | xcrawl_crawl | Async multi-page crawl | | xcrawl_check_crawl_status | Check async crawl status | | xcrawl_key_status | Check whether the current API key is valid and has remaining credits |

xcrawl_scrape

Basic request:

{
  "url": "https://example.com"
}

With extraction (json.prompt and json.json_schema are optional):

{
  "url": "https://example.com",
  "output": {
    "formats": ["json"],
    "json": {
      "prompt": "Extract product name and price",
      "json_schema": {
        "type": "object",
        "properties": {
          "name": { "type": "string" },
          "price": { "type": "number" }
        }
      }
    }
  }
}

xcrawl_check_status

{
  "scrape_id": "abc-123-def-456"
}

Possible values: pending, crawling, completed, failed.

xcrawl_search

{
  "query": "latest AI news",
  "location": "New York",
  "language": "en",
  "limit": 10
}

xcrawl_map

{
  "url": "https://example.com",
  "filter": "blog/.*",
  "limit": 1000,
  "include_subdomains": true,
  "ignore_query_parameters": true
}

xcrawl_crawl

{
  "url": "https://example.com",
  "crawler": {
    "limit": 100,
    "include": ["products/.*"],
    "exclude": ["admin/.*", "login/.*"],
    "max_depth": 3
  },
  "output": {
    "formats": ["markdown"]
  }
}

xcrawl_check_crawl_status

{
  "crawl_id": "xyz-789-abc-012"
}

Possible values: pending, crawling, completed, failed.

xcrawl_key_status

{}

Returns key validity and current credit status, including:

• status
• valid
• authorized
• has_credits
• remain_credits
• expire_at
• expired_at (UTC ISO-8601)
• uid
• username
• concurrent
• message

Request Defaults

Common defaults: Defaults may vary by API version.

| Parameter | Default | Notes | |-----------|---------|-------| | mode | "sync" | Scrape only | | proxy.location | "US" | ISO-3166-1 alpha-2 | | location | "US" | Search only | | request.device | "desktop" | desktop or mobile | | request.only_main_content | true | Main content filtering | | request.block_ads | true | Ad blocking | | request.skip_tls_verification | true | Skip TLS verification | | js_render.enabled | true | JavaScript rendering | | js_render.wait_until | "load" | load, domcontentloaded, networkidle | | output.formats | [] | If omitted or [], returns metadata only | | output.screenshot | "viewport" | viewport or full_page | | output.json.prompt | - | Optional | | output.json.json_schema | - | Optional | | webhook.events | ["started","completed","failed"] | Optional callback events |

Preflight Checks

Before executing high-cost tools, the MCP server performs a key and credits preflight check.

Current tools with preflight checks:

• xcrawl_scrape
• xcrawl_search
• xcrawl_map
• xcrawl_crawl

The preflight check verifies:

• the current API key is valid
• the current API key is authorized
• the account still has remaining credits

Error Format

Errors are returned in MCP format:

{
  "content": [
    {
      "type": "text",
      "text": "Error: XCrawl API error: 401 Unauthorized - Invalid API key"
    }
  ],
  "isError": true
}

Development

npm install
npm run build
XCRAWL_API_KEY=YOUR_API_KEY npm run start:stdio:dev

npm run dev
curl http://localhost:8787/health

License

MIT

Support

• XCrawl API: xcrawl.com
• MCP Server: Open an issue in this repository