create-webclaw

v0.1.4

Published

8 days ago

Set up webclaw MCP server for AI agents (Claude, Cursor, Windsurf, OpenCode, Codex, Antigravity)

Quick Start

npx create-webclaw

That's it. Auto-detects your AI tools, downloads the MCP server, configures everything.

Works with Claude Desktop, Claude Code, Cursor, Windsurf, VS Code, OpenCode, Codex CLI, and Antigravity.

The Problem

Your AI agent calls fetch() and gets a 403. Cloudflare, Akamai, and every major CDN fingerprint the TLS handshake and block non-browser clients before the request hits the server.

When it does work, you get 100KB+ of raw HTML — navigation, ads, cookie banners, scripts. Your agent burns 4,000+ tokens parsing noise.

The Fix

webclaw impersonates Chrome 146 at the TLS protocol level. Perfect JA4 fingerprint. Perfect HTTP/2 Akamai hash. 99% bypass rate on 102 tested sites.

Then it extracts just the content — clean markdown, 67% fewer tokens.

                     Raw HTML                          webclaw
┌──────────────────────────────────┐    ┌──────────────────────────────────┐
│ <div class="ad-wrapper">         │    │ # Breaking: AI Breakthrough      │
│ <nav class="global-nav">         │    │                                  │
│ <script>window.__NEXT_DATA__     │    │ Researchers achieved 94%         │
│ ={...8KB of JSON...}</script>    │    │ accuracy on cross-domain         │
│ <div class="social-share">       │    │ reasoning benchmarks.            │
│ <!-- 142,847 characters -->      │    │                                  │
│                                  │    │ ## Key Findings                  │
│         4,820 tokens             │    │         1,590 tokens             │
└──────────────────────────────────┘    └──────────────────────────────────┘

What It Does

npx create-webclaw

Detects installed AI tools (Claude, Cursor, Windsurf, VS Code, OpenCode, Codex, Antigravity)
Downloads the webclaw-mcp binary for your platform (macOS arm64/x86, Linux x86/arm64)
Asks for your API key (optional — works locally without one)
Writes the MCP config for each detected tool

10 MCP Tools

After setup, your AI agent has access to:

| Tool | What it does | API key needed? | |------|-------------|-----------------| | scrape | Extract content from any URL | No | | crawl | Recursively crawl a website | No | | search | Web search + parallel scrape | Yes (Serper) | | map | Discover URLs from sitemaps | No | | batch | Extract multiple URLs in parallel | No | | extract | LLM-powered structured extraction | Yes | | summarize | Content summarization | Yes | | diff | Track content changes | No | | brand | Extract brand identity | No | | research | Deep multi-page research | Yes |

8 of 10 tools work fully offline. No API key, no cloud, no tracking.

Supported Tools

| Tool | Config location | |------|----------------| | Claude Desktop | ~/Library/Application Support/Claude/claude_desktop_config.json | | Claude Code | ~/.claude.json | | Cursor | .cursor/mcp.json | | Windsurf | ~/.codeium/windsurf/mcp_config.json | | VS Code (Continue) | ~/.continue/config.json | | OpenCode | ~/.opencode/config.json | | Codex CLI | ~/.codex/config.json | | Antigravity | ~/.antigravity/mcp.json |

Sites That Work

webclaw gets through where default fetch() gets blocked:

Nike, Cloudflare, Bloomberg, Zillow, Indeed, Viagogo, Fansale, Wikipedia, Stripe, and 93 more. Tested on 102 sites with 99% success rate.

Alternative Install Methods

Homebrew

brew tap 0xMassi/webclaw && brew install webclaw

Docker

docker run --rm ghcr.io/0xmassi/webclaw https://example.com

Cargo

cargo install --git https://github.com/0xMassi/webclaw.git webclaw-cli

Prebuilt Binaries

Download from GitHub Releases for macOS (arm64, x86_64) and Linux (x86_64, aarch64).

License

AGPL-3.0