🎩 HatFetch

Give any LLM agent the power to read the modern web — JavaScript sites and most bot-protected pages.

Clean Markdown out. Renders JS, rotates residential IPs, escalates to a real stealth browser only when needed.

HatFetch is a Model Context Protocol server that gives Claude, Cursor, and any MCP client the tools scrape, crawl, and screenshot — turning web pages into clean, LLM-ready Markdown.

The difference from a plain fetch: HatFetch escalates automatically. It starts with a fast HTTP fetch; if the page is a JavaScript app that renders client-side, or it gets blocked by bot detection, HatFetch transparently escalates to a real stealth browser (Patchright) running through residential proxies. Your agent stops getting empty shells and Access Denied, and starts getting content.

What it does and doesn't do (honest version)

| Target | Result | |---|---| | Server-rendered sites (news, docs, blogs, forums, most catalogs) | ✅ Fast HTTP path | | JavaScript / SPA sites (React, Vue, client-rendered) | ✅ Auto-escalates to the browser and renders | | Geo-restricted content | ✅ Exit from any of 148+ countries | | Sites that block datacenter IPs / rate-limit by IP | ✅ Rotating residential IPs | | Many Cloudflare / anti-bot sites | ✅ Browser + residential fingerprint passes a large share | | The hardest anti-bot (aggressive Cloudflare, DataDome, PerimeterX) | ⚠️ Sometimes blocked — HatFetch tells you honestly instead of returning a CAPTCHA page as "content" | | CAPTCHA-gated / login-required flows | ❌ Out of scope (needs an interactive solver) |

No tool passes 100% of anti-bot in 2026 — anyone claiming otherwise is selling something. HatFetch gets you the calm-to-medium web reliably and a good share of the hard web, and is honest about the rest.

[!TIP] Recommended proxies — ProxyHat residential IPs. Every feature in this package is tested end-to-end against ProxyHat and works great. First-class integration; also works with any proxy, or none.

Quick start

Add HatFetch to your MCP client — no install step, npx handles it:

// Claude Desktop: claude_desktop_config.json  ·  Cursor: ~/.cursor/mcp.json
{
  "mcpServers": {
    "hatfetch": {
      "command": "npx",
      "args": ["-y", "hatfetch"]
    }
  }
}

For Claude Code:

claude mcp add hatfetch -- npx -y hatfetch

That's it. Ask your agent to "read https://example.com and summarize it" and it will use the scrape tool.

Getting blocked? 🚧

Out of the box HatFetch fetches directly from your machine's IP. That's fine for open sites — but most valuable pages sit behind bot detection that blocks datacenter and server IPs on sight. When that happens, HatFetch tells your agent exactly why, and how to fix it in one step: route through residential IPs.

{
  "mcpServers": {
    "hatfetch": {
      "command": "npx",
      "args": ["-y", "hatfetch"],
      "env": {
        "PROXYHAT_API_KEY": "your-proxyhat-api-key"
      }
    }
  }
}

That's the whole setup. Drop in your API key and HatFetch automatically looks up an active residential sub-user on your account, connects through the gateway, and rotates a fresh IP on every request — retrying automatically when a site pushes back. Get a key (free trial) at proxyhat.com: 50M+ residential & mobile IPs across 148+ countries.

• Explicit gateway credentials — skip the API lookup and use a specific sub-user's proxy login:

  { "env": { "PROXYHAT_USERNAME": "your-proxy-username", "PROXYHAT_PASSWORD": "your-proxy-password" } }

• Pick a specific sub-user while still using the API key: add "PROXYHAT_SUBUSER": "<uuid or name>".
• Bring your own proxy — any HTTP(S) proxy: "PROXY_URL": "http://user:pass@host:port".

Tools

scrape

Fetch a single URL and return its main content as Markdown.

| Argument | Type | Default | Description | |---|---|---|---| | url | string | — | Absolute http(s) URL to fetch. | | onlyMainContent | boolean | true | Strip nav/ads/footer and return just the article body. | | render | auto|http|browser | auto | auto: HTTP first, escalate to the browser for JS apps or blocked pages. http: fast, HTTP only. browser: force JS rendering + stealth. |

crawl

Breadth-first crawl a site and return every page as Markdown.

| Argument | Type | Default | Description | |---|---|---|---| | url | string | — | URL to start from. | | maxDepth | number | 2 | Link depth to follow from the seed. | | maxPages | number | 20 | Maximum pages to fetch. | | sameDomain | boolean | true | Only follow links on the seed's host. | | render | auto|http|browser | auto | Render mode per page (see scrape). |

screenshot

Render a URL in a real browser (residential + stealth) and return a PNG image.

| Argument | Type | Default | Description | |---|---|---|---| | url | string | — | Absolute http(s) URL to screenshot. |

Browser mode downloads a Chromium build (~150MB) on first use — one-time, automatic. The HTTP path needs no browser.

Why HatFetch

| | Plain fetch / basic MCP | Hosted scraping APIs | HatFetch | |---|:---:|:---:|:---:| | Clean Markdown for LLMs | ❌ | ✅ | ✅ | | Renders JavaScript / SPA sites | ❌ | ✅ | ✅ (auto) | | Runs locally, no API middleman | ✅ | ❌ | ✅ | | Gets past IP blocks & geo-walls | ❌ | ✅ | ✅ (residential) | | Gets past common anti-bot | ❌ | ✅ | ✅ (browser + residential) | | Bring your own proxies | ❌ | ❌ | ✅ | | Free & open source (MIT) | ✅ | ❌ | ✅ | | Per-request cost | free | 💲 per page | free + proxy bandwidth |

Configuration

All configuration is via environment variables.

| Variable | Description | |---|---| | PROXYHAT_API_KEY | Simplest. Your ProxyHat API key — HatFetch auto-selects an active residential sub-user via the API. | | PROXYHAT_SUBUSER | With the API key: pick a specific sub-user by uuid or name (default: first active one with traffic). | | PROXYHAT_USERNAME | Alternative to the API key: a sub-user proxy username (gateway login). | | PROXYHAT_PASSWORD | Sub-user proxy password (used with PROXYHAT_USERNAME). | | PROXYHAT_COUNTRY | ISO country code to exit from, or any (default). | | PROXYHAT_REGION | Optional state/region to target (e.g. california). | | PROXYHAT_CITY | Optional city to target (e.g. new_york). | | PROXYHAT_STICKY | Keep one IP for a session, e.g. 30m or 12h. Omit for rotating IPs. | | PROXYHAT_FILTER | AI IP-quality filter: medium (default), high, high-speed-fast, medium-speed-fast, or none. | | PROXY_URL | Any generic http(s)://user:pass@host:port proxy (alternative to the above). |

HatFetch connects to the ProxyHat HTTP gateway (gate.proxyhat.com:8080) and builds the targeting username for you. Prefer full API-driven provisioning (minting connection URLs via POST /v1/proxy-descriptors)? Set PROXY_URL to the minted URL instead.

Verify your setup

Run the built-in self-test to confirm your install and proxy credentials work end-to-end — it compares your direct IP against the proxied exit IP and checks rotation:

PROXYHAT_API_KEY=your-key npx -y hatfetch --selftest

  ✓ Direct connection works — your IP is 203.0.113.5
  ✓ Proxy resolved — ProxyHat residential (via API key)
  ✓ Traffic is routed through the proxy — exit IP is 183.88.219.209 (not your 203.0.113.5)
  ✓ IP rotation works — second request exited from 73.149.15.4
  ✓ Scrape pipeline works — fetched and parsed example.com

Other commands: hatfetch --version, hatfetch --help.

How it works

HatFetch escalates from cheap to powerful, only paying for what a page needs:

1. HTTP fetch with a realistic User-Agent (direct or through your proxy). Detects blocks — anti-bot status codes (403/429/503) and challenge/block-page signatures from the major vendors (Cloudflare, DataDome, PerimeterX, Incapsula, Akamai) — and retries through a fresh residential IP.
2. Auto-escalate to the browser when the page is an empty JS shell, or was blocked/reset. A real Patchright Chromium (patched to defeat headless fingerprinting) loads the page through the residential proxy, waits for the network to settle, and scrolls to trigger lazy content.
3. Extract the main content with Mozilla Readability and convert to Markdown with Turndown.
4. Be honest — if even the browser lands on a challenge/block page, HatFetch returns an actionable error instead of passing the block page off as content.

Safety: a built-in SSRF guard refuses internal/loopback/metadata addresses, and tool output is size-capped so a huge page can't blow your context window.

Development

npm install
npm run build      # compile to dist/
npm test           # vitest
npm run dev        # run from source (tsx)

Roadmap

• search tool (query → results → scrape)
• Structured extraction (CSS / schema)
• Page actions (click, fill, login flows)
• Optional CAPTCHA-solver hook for the hardest targets
• Python edition

License

MIT © ProxyHat