create-webclaw

v0.1.7

Published

2 months ago

Set up webclaw MCP server for AI agents (Claude, Cursor, Windsurf, OpenCode, Codex, Antigravity)

Quick Start

npx create-webclaw

That's it. Auto-detects your AI tools and writes the npx @webclaw/mcp config into each — nothing to install.

Works with Claude Desktop, Claude Code, Cursor, Windsurf, VS Code, OpenCode, Codex CLI, and Antigravity.

The Problem

Your AI agent calls fetch() and gets a 403. Cloudflare, Akamai, and every major CDN fingerprint the TLS handshake and block non-browser clients before the request hits the server.

When it does work, you get 100KB+ of raw HTML — navigation, ads, cookie banners, scripts. Your agent burns 4,000+ tokens parsing noise.

The Fix

webclaw impersonates the latest Chrome at the TLS protocol level. Perfect JA4 fingerprint. Perfect HTTP/2 Akamai hash. 99% bypass rate on 102 tested sites.

Then it extracts just the content — clean markdown, 67% fewer tokens.

                     Raw HTML                          webclaw
┌──────────────────────────────────┐    ┌──────────────────────────────────┐
│ <div class="ad-wrapper">         │    │ # Breaking: AI Breakthrough      │
│ <nav class="global-nav">         │    │                                  │
│ <script>window.__NEXT_DATA__     │    │ Researchers achieved 94%         │
│ ={...8KB of JSON...}</script>    │    │ accuracy on cross-domain         │
│ <div class="social-share">       │    │ reasoning benchmarks.            │
│ <!-- 142,847 characters -->      │    │                                  │
│                                  │    │ ## Key Findings                  │
│         4,820 tokens             │    │         1,590 tokens             │
└──────────────────────────────────┘    └──────────────────────────────────┘

What It Does

npx create-webclaw

Detects installed AI tools (Claude, Cursor, Windsurf, VS Code, OpenCode, Codex, Antigravity)
Asks for your API key (optional — works locally without one)
Writes the npx @webclaw/mcp config into each detected tool

The server itself runs via @webclaw/mcp — npx fetches it on first launch and caches it. create-webclaw is just the convenience that writes that config for you.

Prefer to configure by hand?

Add this one block to your client's mcpServers config — it's identical to what create-webclaw writes:

{
  "mcpServers": {
    "webclaw": {
      "command": "npx",
      "args": ["-y", "@webclaw/mcp"]
    }
  }
}

MCP Tools

After setup, your AI agent has access to these 14 tools:

| Tool | What it does | API key needed? | |------|-------------|-----------------| | scrape | Extract content from any URL | No | | crawl | Recursively crawl a website | No | | search | Web search + parallel scrape | Yes (Serper) | | map | Discover URLs from sitemaps | No | | batch | Extract multiple URLs in parallel | No | | extract | LLM-powered structured extraction | Yes | | summarize | Content summarization | Yes | | diff | Track content changes | No | | brand | Extract brand identity | No | | research | Deep multi-page research | Yes | | list_extractors | List the built-in vertical extractors | No | | vertical_scrape | Scrape with a built-in vertical extractor | No | | lead | Enrich a company URL into an outreach-ready lead (leadership + socials, tech, pricing, public emails) | Yes (webclaw) | | lead_batch | Enrich up to 25 company URLs at once | Yes (webclaw) |

8 of the 14 tools work fully offline — no API key, no cloud, no tracking.

Supported Tools

| Tool | Config location | |------|----------------| | Claude Desktop | ~/Library/Application Support/Claude/claude_desktop_config.json | | Claude Code | ~/.claude.json | | Cursor | .cursor/mcp.json | | Windsurf | ~/.codeium/windsurf/mcp_config.json | | VS Code (Continue) | ~/.continue/config.json | | OpenCode | ~/.config/opencode/opencode.json (or ./opencode.json) | | Codex CLI | ~/.codex/config.toml | | Antigravity | ~/.config/antigravity/mcp.json (or ~/.antigravity/mcp.json) |

Sites That Work

webclaw gets through where default fetch() gets blocked:

Nike, Cloudflare, Bloomberg, Zillow, Indeed, Viagogo, Fansale, Wikipedia, Stripe, and 93 more. Tested on 102 sites with 99% success rate.

Alternative Install Methods

Homebrew

brew tap 0xMassi/webclaw && brew install webclaw

Docker

docker run --rm ghcr.io/0xmassi/webclaw https://example.com

Cargo

cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli

Prebuilt Binaries

Download from GitHub Releases for macOS (arm64, x86_64) and Linux (x86_64, aarch64).

License

AGPL-3.0