crawlee-proxyhat

Route Crawlee crawlers through ProxyHat residential proxies — rotating IPs, geo-targeting, and sticky sessions mapped to Crawlee's own session pool.

Why

Crawling at scale from datacenter IPs gets you blocked and rate-limited. This package plugs ProxyHat's residential IPs (50M+ across 148+ countries) into any Crawlee crawler through the first-class ProxyConfiguration API — a fresh IP per request by default, and one pinned IP per Crawlee session when you use the session pool. No boilerplate, works with CheerioCrawler, PlaywrightCrawler, PuppeteerCrawler, and the rest.

Install

npm install crawlee-proxyhat

crawlee is a peer dependency — bring your own version (>=3).

Quick start

import { CheerioCrawler } from "crawlee";
import { createProxyHatProxyConfiguration } from "crawlee-proxyhat";

// An API key auto-selects an active residential sub-user:
const proxyConfiguration = await createProxyHatProxyConfiguration({
  apiKey: process.env.PROXYHAT_API_KEY,
  targeting: { country: "us" },
});

const crawler = new CheerioCrawler({ proxyConfiguration });
await crawler.run(["https://example.com"]);

Get an API key at proxyhat.com.

Credentials

Pass them explicitly or via environment variables — options win over env:

| Option | Env var | Notes | |---|---|---| | apiKey | PROXYHAT_API_KEY | Auto-selects an active sub-user with remaining traffic | | subUser | PROXYHAT_SUBUSER | Pick a specific sub-user by uuid or name (with an API key) | | username | PROXYHAT_USERNAME | Explicit gateway proxy_username (skips the API) | | password | PROXYHAT_PASSWORD | Explicit gateway proxy_password |

Targeting

const proxyConfiguration = await createProxyHatProxyConfiguration({
  apiKey: process.env.PROXYHAT_API_KEY,
  protocol: "http", // or "socks5"
  targeting: {
    country: "us",      // ISO code or "any" (default)
    region: "california",
    city: "new_york",
    filter: "high",     // AI IP-quality tier
  },
  stickyTtl: "30m",     // sticky-session lifetime (default "30m")
});

Sticky IP per Crawlee session

Most Crawlee crawlers enable the session pool by default. This package maps each Crawlee session to one pinned ProxyHat residential IP: every request sharing a session id exits from the same IP, mimicking a real user. When Crawlee retires a session (block detected, too many errors) it stops requesting that URL, so the IP rotates away on its own.

Want a fresh IP on every request instead? Turn stickiness off:

await createProxyHatProxyConfiguration({ apiKey, stickySessions: false });

Per-request targeting

Choose targeting from the request itself:

await createProxyHatProxyConfiguration({
  apiKey: process.env.PROXYHAT_API_KEY,
  perRequest: (request) =>
    request?.url.includes("/de/") ? { country: "de" } : { country: "us" },
});

Advanced: bring your own ProxyConfiguration

Need to combine ProxyHat with Crawlee's tiered proxies or your own logic? Use the lower-level helpers:

import { ProxyConfiguration } from "crawlee";
import { createNewUrlFunction, resolveCredentials } from "crawlee-proxyhat";

const credentials = await resolveCredentials({ apiKey: process.env.PROXYHAT_API_KEY });
const newUrlFunction = createNewUrlFunction(credentials, { targeting: { country: "us" } });

const proxyConfiguration = new ProxyConfiguration({ newUrlFunction });

How it works

createProxyHatProxyConfiguration resolves your gateway credentials once (via the official proxyhat SDK), then returns a standard Crawlee ProxyConfiguration whose newUrlFunction builds a ProxyHat gateway URL per request. With a session id it caches and reuses the URL (sticky IP); without one it reuses a stable username so the gateway hands out a fresh residential IP per connection.

License

MIT © ProxyHat