CrawlProxy

CrawlProxy is a local MITM proxy written in Rust, shipped as a napi-rs native Node addon so a TypeScript project can import it and drive it in-process — no external process to spawn, health-check, or tear down. It sits between a Crawlee/Playwright crawler and one or more upstream proxies, and routes, blocks, and caches HTTP(S) traffic by URL, entirely off the Node event loop.

Contents

• Features
• Packages
• Usage
• Build
• MITM & certificates
• Testing
• Releasing
• Architecture
• Contributing
• License

Features

• Declarative rules, native execution — route, block, and cache rules are plain RegExp patterns matched natively in Rust (via fancy-regex), off the Node event loop. An optional onRequest JS hook is the only thing that ever crosses into JS, and only when supplied.
• Full HTTPS interception — decrypts all HTTPS traffic so rules and the cache can operate at full URL + response-body granularity, not just CONNECT-tunnel host visibility.
• In-process, not a sidecar — compiled to a napi-rs native addon and loaded directly into your Node process; start()/stop() are just method calls.
• Drop-in with Crawlee/Apify — accepts a ProxyConfiguration instance directly as a route's upstream, preserving its rotation/session behavior.
• Predictable precedence — every request is decided by Block → Cache → Route, else forwarded directly; see ADR-0003.
• Indistinguishable from the browser itself — the outbound request is the browser's own: its headers, in its order, over an HTTP/2 connection whose TLS fingerprint matches the browser's version. Verified rather than assumed, by npm run test:fingerprint — see ADR-0004 and ADR-0005.

Packages

This is an npm-workspaces monorepo with two packages:

| Package | What it is | | --- | --- | | packages/proxy | CrawlProxyServer — the Rust MITM proxy (napi addon) + its TypeScript wrapper. Published to npm as crawl-proxy. | | packages/apify-actor-example | A PlaywrightCrawler whose proxyConfiguration points at CrawlProxyServer. Private, not published. |

See CONTEXT.md for the project glossary and docs/adr/ for the architecture decisions behind the napi addon, full MITM, and native rule matching.

Usage

npm install crawl-proxy

import { CrawlProxyServer } from "crawl-proxy";

const proxy = new CrawlProxyServer({
  routes: [
    // pattern -> upstream proxy URL (http(s):// or socks5://, with auth)
    { pattern: /\.example\.com/, upstream: "http://user:pass@upstream:8000" },
    // ...or a Crawlee/Apify `ProxyConfiguration` instance, called per request:
    // { pattern: /.*/, upstream: apifyProxyConfig },
  ],
  // Refuse requests matching any of these
  block: [/\.(png|jpe?g|woff2?)$/i, /ads\./],
  // Cache responses matching these (in-memory LRU, idle TTL resets on each hit)
  cache: [{ pattern: /\/api\//, ttlSeconds: 3_600 }],
  // Log each request's outcome to stdout, e.g.
  // "GET https://x.example.com/y -> upstream http://... (route rule #0)"
  verbose: true,
});

// start() is the only way to reach url/port/caCertPath/caSpkiFingerprint/
// proxyConfiguration — none of them exist on `proxy` itself.
const { url, proxyConfiguration } = await proxy.start(); // url e.g. http://127.0.0.1:53412
// ... point your crawler at `proxyConfiguration` (or `url` directly) ...
await proxy.stop();

Decision precedence per request: Block → Cache → Route, else forward directly. Regexes are tested against the full absolute URL. Matching runs natively in Rust (see ADR-0003).

Apify Actor (Playwright + TypeScript)

Drop-in for an Actor scaffolded from Apify's playwright-ts template — pass the Actor's own ProxyConfiguration straight through as a route's upstream, and point PlaywrightCrawler at the proxy CrawlProxy hands back:

// src/main.ts
import { Actor } from "apify";
import { PlaywrightCrawler } from "crawlee";
import { CrawlProxyServer } from "crawl-proxy";

await Actor.init();

// Real Apify Proxy, routed through CrawlProxy. Falls back to `undefined`
// without credentials (apify login, or APIFY_PROXY_PASSWORD/APIFY_TOKEN env
// vars) — routes then go direct and block/cache still work.
const apifyProxy = await Actor.createProxyConfiguration({ groups: ["RESIDENTIAL"] });

const crawlProxyServer = new CrawlProxyServer({
  // Pass the `ProxyConfiguration` instance itself (not a resolved URL) — CrawlProxy
  // calls its async newUrl() per request, so rotation/session behavior is preserved.
  routes: [{ pattern: /.*/, upstream: apifyProxy }],
  block: [/\.(png|jpe?g|gif|webp|svg|woff2?)$/i],
  cache: [{ pattern: /\/api\//, ttlSeconds: 3_600 }],
});
const crawlProxy = await crawlProxyServer.start();

const crawler = new PlaywrightCrawler({
  proxyConfiguration: crawlProxy.proxyConfiguration,
  launchContext: {
    launchOptions: {
      // Trust only this run's generated MITM CA — every other site's real
      // certificate is still verified normally (see "MITM & certificates" below).
      args: [`--ignore-certificate-errors-spki-list=${crawlProxy.caSpkiFingerprint}`],
    },
  },
  async requestHandler({ request, enqueueLinks, log }) {
    log.info(request.url);
    await enqueueLinks();
  },
});

try {
  await crawler.run(["https://apify.com"]);
} finally {
  await crawlProxyServer.stop();
  await Actor.exit();
}

A fuller, runnable version of this — with multiple ProxyConfiguration instances routed by pattern, cache/block rules, and a real request handler — lives in packages/apify-actor-example; run it with npm run example from the repo root.

Build

Requires Node ≥ 18, a Rust toolchain, a C toolchain, Go, and Ninja. The C toolchain (cc, cmake, nasm) is for napi build and the aws-lc-rs/ring native crypto (inbound TLS); Go and Ninja are for btls-sys (BoringSSL), which builds the outbound TLS-impersonation client — see ADR-0004. On Fedora: sudo dnf install gcc gcc-c++ cmake nasm golang ninja-build and install Rust via rustup.

npm install                 # install workspace deps
npm run build               # napi build (proxy) -> *.node + dist/, current host only
npm run example             # run the Crawlee example

Cross-compiling for macOS/Windows

npm run build only produces a binary for the machine you run it on. To build for other platforms from one host, npm run build-all cross-compiles for linux-x64, macOS x64/arm64, and Windows x64 in one pass (via napi-rs cross-build: zig for macOS, cargo-xwin for Windows). One-time setup: install zig and clang and put them on PATH — the cargo-zigbuild/cargo-xwin cargo subcommands install themselves on first use.

Only the target matching your current OS can actually be loaded/tested locally; the others are cross-compiled but unverified until run on their real platform. macOS is the riskiest leg: zig can only cross-link pure-Rust dependencies for Darwin targets, and this project depends on aws-lc-rs (a C library, pulled in via hudsucker's rustls) for crypto — if that needs real Apple frameworks to link, the macOS builds will fail here and need to run natively on a Mac (or a macOS CI runner) instead. build-all reports a per-target pass/fail summary rather than stopping at the first failure, so a macOS failure doesn't block the Linux/Windows artifacts.

Since ADR-0004, the outbound client also depends on btls-sys (BoringSSL), a much larger native codebase with its own CMake/Go build — a second, bigger candidate for the cross-compile risk above. All four napi.targets were re-verified by actually cross-compiling them from Linux after adding it: all four produced valid binaries for their target OS (Mach-O for both macOS targets, PE32+ for Windows). The one new requirement surfaced by that: Windows needs Ninja on PATH (cargo-xwin's CMake toolchain file requires the Ninja generator specifically). "Cross-compiles cleanly here" is still not the same as "verified working on a real Mac/Windows machine," which remains untested.

The example uses Actor.createProxyConfiguration() for the Apify Proxy upstream; credentials come from apify login, or the APIFY_PROXY_PASSWORD/APIFY_TOKEN env vars. Without credentials it falls back to undefined and routes go direct, so the example still runs — the CrawlProxyServer features (block/cache) work either way.

MITM & certificates

The proxy decrypts all HTTPS (full MITM) to see URL paths and response bodies. It generates a CA (cert + key) synchronously within start() — cached under the OS temp dir (<tmp>/crawl-proxy) and reused across runs — and mints per-host leaf certs, which hudsucker caches. Every leaf cert embeds the CA's own public key, so the caSpkiFingerprint resolved by start() (a base64 SHA-256 SPKI hash) is the same for every host.

The example launches Chromium with --ignore-certificate-errors-spki-list=${caSpkiFingerprint}, which trusts only certificates with that fingerprint — every other site's real certificate is still verified normally. This is safer than the blanket --ignore-certificate-errors flag, which disables certificate validation altogether.

Note (ADR-0002): the target site sees the Rust client's TLS fingerprint, not Chromium's — since ADR-0004, that outbound fingerprint impersonates a real Chrome (JA3/JA4 + HTTP/2), rather than a generic Rust/rustls one, and per ADR-0005 it impersonates the same Chrome version the browser on the inbound leg reports, with that browser's own headers forwarded verbatim on top of it.

Testing

cd packages/proxy
npm test          # vitest run — smoke, hook, HTTPS, wrapper, cache-rule tests
npm run test:watch
cargo test        # Rust unit tests: URL normalization, profile selection, redaction

CI (.github/workflows/ci.yml) runs the same suite, plus spelling/markdown lint, on every push and pull request against main.

Fingerprint parity check

cd packages/proxy
npm run test:fingerprint

Loads a fingerprint-echo endpoint in Chromium twice — once directly, once through CrawlProxyServer — and diffs what the origin saw each time: HTTP version, JA4, peetprint, the HTTP/2 Akamai fingerprint, :authority, header order, and accept-encoding. Any difference is something an anti-bot layer could key on, so it exits non-zero. Needs real network egress and a Playwright Chromium (npx playwright install chromium); FINGERPRINT_URL overrides the default endpoint (https://tls.peet.ws/api/all), which must answer in that service's JSON shape.

Expect the two TLS signals — and only those — to differ when the local Chromium is newer than the newest emulation profile wreq-util ships; the script says so when that's the case. Nothing else should ever differ.

Releasing

This README is what npm shows on the package page: packages/proxy's prepack script (scripts/sync-readme.mjs) copies it next to the published package.json, flattening links to paths inside this repo into plain text — they'd resolve against packages/proxy/ on npm, and this repo isn't public. The copy is generated and git-ignored; edit this file.

packages/proxy publishes to npm as a single package, crawl-proxy, bundling one compiled .node binary per target in napi.targets directly inside it (via package.json's files: ["*.node", ...]) — index.js picks the right one for the current platform/arch at require() time. Cut a release by tagging a version:

npm version <major|minor|patch> --workspace crawl-proxy
git push --follow-tags

Then publish it: open this repo's Actions tab → Publish → Run workflow, and pick the tag you just pushed from the ref selector. It's a manual (workflow_dispatch) trigger, not automatic on tag push — the tag must already exist and match packages/proxy/package.json's version (checked as the workflow's first step). It then re-runs the test suite, cross-compiles every target, and publishes to npm using npm trusted publishing (GitHub Actions OIDC) — no long-lived NPM_TOKEN secret involved.

One-time setup, before the first release: register this repository and the publish.yml workflow as a trusted publisher for crawl-proxy on npmjs.com (Package Settings → Trusted Publisher). Until that's done, npm publish in CI will fail authentication.

Architecture

• CONTEXT.md — glossary of project terms (CrawlProxyServer, Route, Upstream, Block/Cache rule, onRequest hook, native matching, leaf cert cache).
• ADR-0001 — why a napi-rs native addon instead of a subprocess.
• ADR-0002 — why full MITM instead of selective/tunnelled HTTPS.
• ADR-0003 — why rule matching runs natively in Rust instead of crossing into JS per request.
• ADR-0004 — why the outbound client impersonates a real browser's TLS/HTTP2 fingerprint instead of presenting Rust's own.

Contributing

Issues and pull requests are welcome. Before opening a PR:

npm run lint:text   # spelling (cspell) + markdown lint
npm test            # from packages/proxy, or npm run build && npm test at root

License

MIT