crawlee-ghost-fetch

A Crawlee BaseHttpClient that delegates every HTTP request to the ghost-fetch unblocker. Drop it into a CheerioCrawler (or any HttpCrawler) and your crawl gets ghost-fetch's full unblocking cascade — Tier 1 impit, Tier 2 stealth Chromium, country-aware locale, sticky sessions — without any of the plumbing in your actor.

When to use

You're building (or maintaining) an Apify standby actor and:

• The target site has bot protection — Akamai BMP, Cloudflare, DataDome, or similar.
• You want a single sticky session per actor process so the first call pays the browser-tier warmup and every subsequent call rides the same warmed cookie jar through ghost-fetch's fast Tier 1 (impit) path.
• You're happy to keep the rest of your actor (request builders, handlers, extractors, route mapping) the way you have it today — this library only swaps out the HTTP fetcher.

If you're scraping an unprotected site, you don't need this — vanilla CheerioCrawler is fine.

Upgrading to 0.9.0 (breaking). The client now speaks ghost-fetch's POST /v1/fetch (trace-v4) instead of /fetch_url. Three caller-visible changes: tier/browser config moved under a single render: {...} block (hintFromUrl→render.mode, browser/fox→render.browser/render.firefox); a treatAsSuccess predicate now reads p.response.body (was p.content) and overrides ghost-fetch's own verdict; and a larger maxPoolSize now buys real identity diversity (one {exit IP + cookie jar + device fingerprint} per Crawlee Session — not TLS). See CHANGELOG.md and docs/adr/0001.

Install

npm install crawlee-ghost-fetch

crawlee, @crawlee/core, and apify are peer dependencies — your actor already has them.

You also need a ghost-fetch endpoint to talk to. Either deploy your own ghost-fetch standby on Apify (see the ghost-fetch repo) or run it locally for development. Pass the URL via ghostFetchUrl on the client config or the GHOST_FETCH_URL env var. There is no default — the lib throws on construction if neither is set.

Quick-start (single-country)

import { CheerioCrawler } from 'crawlee';
import {
    GhostFetchHttpClient,
    ghostFetchCrawlerOptions,
} from 'crawlee-ghost-fetch';

const httpClient = new GhostFetchHttpClient({
    name: 'overstock',
    defaultCountry: 'US',
});

const crawler = new CheerioCrawler({
    keepAlive: true,
    httpClient,
    requestHandlerTimeoutSecs: 180,
    navigationTimeoutSecs: 120,
    maxConcurrency: 20,
    maxRequestRetries: 3,
    // Identity binding/rotation is automatic — the client reads the active
    // Crawlee session from `request.sessionToken.id`. No pre-nav hook needed.
    ...ghostFetchCrawlerOptions(),
    requestHandler: async ({ $, request }) => {
        // your handler
    },
});

await crawler.run(['https://www.overstock.com/some/page']);

Multi-country (e.g. kaufland)

When the country is encoded in the URL — TLD, query string, subdomain — pass a countryFromUrl resolver. The library invokes it per request and forwards the result to ghost-fetch's country argument:

const KAUFLAND_TLDS = new Set(['DE', 'CZ', 'PL', 'SK', 'AT', 'IT', 'FR']);

const httpClient = new GhostFetchHttpClient({
    name: 'kaufland',
    defaultCountry: 'DE',
    countryFromUrl: (url) => {
        try {
            const tld = new URL(url).hostname.split('.').pop()?.toUpperCase();
            return tld && KAUFLAND_TLDS.has(tld) ? tld : undefined;
        } catch {
            return undefined;
        }
    },
});

Returning undefined falls back to defaultCountry.

Per-URL warmup and render control

Two more optional resolvers, same shape as countryFromUrl:

• warmupFromUrl(url) → string | undefined — a URL ghost-fetch navigates to before fetching the target, for endpoints that need cookies from a real HTML page nav first. undefined to skip.

• renderFromUrl(url) → Partial<RenderOptions> | undefined — overrides the static render config per URL (shallow-merged; returned keys win).

The render block mirrors ghost-fetch's /v1/fetch render schema:

| field | values | meaning | |---|---|---| | mode | auto (default) · http · browser | cascade control. http = Tier 1 only (fail-fast); browser = skip Tier 1, go straight to the browser tier. | | browser | chrome (default) · firefox | engine family (cloakbrowser / camoufox). | | screenshot | none · viewport · full_page | capture a screenshot (GET only; base64 PNG on response.screenshot). | | firefox | FirefoxOptions | camoufox device-fingerprint knobs (fox_os, fox_locale, …). Firefox-only; dropped on the wire for chrome. |

// Tier-1-only on a cheap API, force the browser tier on a heavy page:
new GhostFetchHttpClient({
    name: 'mysite',
    defaultCountry: 'US',
    renderFromUrl: (url) =>
        url.includes('/api/') ? { mode: 'http' }
            : url.includes('/heavy/') ? { mode: 'browser' }
                : undefined,
});

Firefox (camoufox) device fingerprints

Pick the Firefox engine when a target blocks Chromium fingerprints — it ships a different (authentic, shared) network fingerprint and a separate device-fingerprint stack:

new GhostFetchHttpClient({
    name: 'datadome-target',
    defaultCountry: 'DE',
    render: {
        browser: 'firefox',
        firefox: { fox_os: 'windows', fox_humanize: true, fox_block_images: true },
    },
});

Per-URL:

new GhostFetchHttpClient({
    name: 'multi-target',
    defaultCountry: 'US',
    renderFromUrl: (url) =>
        url.includes('hard-target.com')
            ? { browser: 'firefox', firefox: { fox_os: 'macos', fox_locale: 'en-US' } }
            : undefined,
});

render.firefox is dropped on the wire when the resolved browser is not 'firefox' — safe to set defaults even when most URLs go through Chromium.

Identity diversity (maxPoolSize)

Each Crawlee Session maps to one ghost-fetch Session — one coherent {exit IP + cookie jar + device fingerprint} bundle (the network/TLS fingerprint is authentic and shared per browser, not varied per session). The default maxPoolSize: 1 is one sticky identity per process (maximises Tier-1 warm-jar reuse). For block-heavy or high-concurrency targets, raise it to run several distinct identities concurrently:

...ghostFetchCrawlerOptions({
    sessionPoolOptions: { maxPoolSize: 5 },          // N concurrent identities
    // Retire a burned identity on ghost-fetch's own signal (a fresh Session
    // fixes a fingerprint-rejected block; rate-limit / IP-burn are self-healed
    // server-side — don't retire on those):
    shouldRetire: (_ctx, r) => r.diagnostics?.suspected_fingerprint_rejected === true,
});

See docs/adr/0001 for the trade-off.

See FoxOptions in src/types.ts for the full param list (OS, locale, WebGL config, fonts, addons, blocking toggles, raw fox_config).

Crawler config recipe

ghostFetchCrawlerOptions(opts?) returns the Crawlee opinions that pair with the client. Identity binding is automatic — the client reads the active Crawlee session from request.sessionToken.id (no hook required since 0.9.1). Since 0.7.0, any pre/post navigation hooks you do add compose through the helper arg, not by spread-after:

import {
    GhostFetchHttpClient,
    ghostFetchCrawlerOptions,
} from 'crawlee-ghost-fetch';

const crawler = new CheerioCrawler({
    httpClient: new GhostFetchHttpClient({ name: 'site', defaultCountry: 'US' }),
    ...ghostFetchCrawlerOptions({
        // Optional retire predicate (see below). Receives status +
        // headers only — body content is not yet parsed at this point.
        // shouldRetire: (_, r) => r.statusCode === 403,
    }),
    requestHandler,
});

Do not spread caller hooks AFTER ghostFetchCrawlerOptions(). That overwrites the helper's hooks (including the generated retire hook when shouldRetire is set). Always pass them through the helper arg.

The helper expands to:

{
    additionalMimeTypes: ['application/octet-stream'],
    useSessionPool: true,
    persistCookiesPerSession: true,
    sessionPoolOptions: { /* sticky-by-default — see below */ },
    preNavigationHooks: [...callerHooks],
    postNavigationHooks: [...generatedRetireHook?, ...callerHooks],
}

Sticky-by-default session pool

sessionPoolOptions defaults to ONE session per actor process, never auto-retired on usage / age / 4xx:

| Knob | Default | Why | |---|---|---| | maxPoolSize | 1 | All requests share one session → one IP, one cookie jar, one JA4. Hot-path Tier 1 amortizes fully. | | sessionOptions.maxUsageCount | Number.MAX_SAFE_INTEGER | Don't auto-retire on usage count. | | sessionOptions.maxErrorScore | Number.MAX_SAFE_INTEGER | Don't auto-retire on error score. | | sessionOptions.maxAgeSecs | 31_536_000 (1 year) | Finite (Crawlee builds a Date from this; MAX_SAFE_INTEGER overflows). | | blockedStatusCodes | [] | 4xx is often cookie-rotation noise on WAFs that bind cookies per-response, not a real block. Use shouldRetire for status + header decisions Crawlee's enum can't express. |

Operator overrides via opts.sessionPoolOptions, deep-merged with these defaults (opts.sessionPoolOptions.sessionOptions merges nested-key-wise, not replaces).

Do not enable retryOnBlocked: true on the crawler. With blockedStatusCodes: [], HttpCrawler.isRequestBlocked() falls back to Crawlee's built-in default [401, 403, 429] when the pool list is empty — reintroducing automatic retire outside your shouldRetire predicate.

shouldRetire — retire-and-retry on status + headers

Pass a predicate to retire the active session and re-queue the request when ghost-fetch's response looks bad on signals Crawlee's blockedStatusCodes enum can't express:

ghostFetchCrawlerOptions({
    // Predicate sees status code + headers only; the body has not been
    // parsed at this point. For body-shape decisions, do them inside
    // the request handler and call `ctx.session.retire()` + throw yourself.
    shouldRetire: (_ctx, response) => response.statusCode === 403,
})

The predicate's response arg is a RetirableResponse:

interface RetirableResponse {
    statusCode?: number;
    headers?: Record<string, string | string[] | undefined>;
}

Body parsing happens AFTER postNavigationHooks (where the generated retire hook lives), so any body-shape check inside shouldRetire would see undefined / unparsed data. If you need to retire based on parsed body content, do it inside requestHandler:

requestHandler: async (ctx) => {
    if (looksLikeChallenge(ctx.$('body').text())) {
        ctx.session?.retire();
        throw new Error('challenge body — retiring session');
    }
    // ...normal extraction
}

Returning true from shouldRetire → the active session is marked bad, retired, and the request is thrown so Crawlee re-queues it with a fresh session (new session.id → new ghost-fetch session arg → fresh exit IP + empty cookie jar).

The generated hook runs before any caller-supplied postNavigationHooks, so a known-blocked response can't trigger caller side-effects before the retry throw.

Each shouldRetire: true consumes a maxRequestRetries slot. For block-heavy sites, bump maxRequestRetries from the default 3 to ~8 so a temporary streak of retires doesn't kill the request.

shouldRetire v1 only supports retire-and-retry semantics. There is no "retire but let this response flow to the handler" mode: Crawlee's session.markGood() call after a successful handler subtracts 0.5 from errorScore, undoing Session.retire()'s max-errorScore set. The retire decision would silently evaporate.

Identity binding is automatic (since 0.9.1)

GhostFetchHttpClient reads the active Crawlee session directly from request.sessionToken.id — the Session object Crawlee binds to every client request — and forwards its id as ghost-fetch's session arg. That binds the Apify residential exit IP, cookie jar, and (via ghost-fetch 0.4.0's session-engine recording) JA4 to the Crawlee session lifecycle. When shouldRetire retires a session, the next request carries a new session.id → new ghost-fetch session → fresh exit IP + empty cookie jar. No hook, no wiring.

If no Crawlee session is bound (SessionPool off, or a non-Crawlee caller), the client falls back to a static per-process token (one IP for the actor's life).

ghostFetchPreNavigationHook is deprecated since 0.9.1. It only copied ctx.session.id into ctx.request.userData.sid, which the client still honors as a fallback — so existing wiring keeps working unchanged — but the hook is now redundant. Drop it from preNavigationHooks. The export remains for back-compat and will be removed in a future major.

Recommended caller-side knobs:

| Option | Recommendation | Why | |---|---|---| | maxConcurrency | 20 | Higher and ghost-fetch's session lock will serialize anyway | | maxRequestRetries | 3 (or 8 with shouldRetire) | One cold-start retry + headroom; bump for retire-heavy sites | | navigationTimeoutSecs | 120 | Cold-start browser warmup can take ~30-60s | | requestHandlerTimeoutSecs | 180 | Outer envelope above | | keepAlive | true | Standby actors are long-lived | | retryOnBlocked | false (don't set true) | Bypasses our blockedStatusCodes: [] and reintroduces Crawlee's built-in [401, 403, 429] retire |

False-block recovery (treatAsSuccess)

Some WAFs return an error status (typically 403) alongside the real page body as a deception layer. The wire-level status says "blocked", but the body contains exactly the data the actor wants — Product JSON-LD, full listing tiles, etc.

Without intervention this round-trips badly through Crawlee:

1. The HTTP client returns a response with statusCode: 403 (the truth from the wire).
2. Crawlee's CheerioCrawler checks the status against blockedStatusCodes (default [401, 403, 429]).
3. On match, it calls session.retire() and throws before the request handler runs.
4. The request retries with a fresh session — fresh exit IP, fresh empty cookie jar, fresh cold-warmup cost. The session that just succeeded (the body was real) and the warm cookie jar tied to it are both discarded.

GhostFetchHttpClient exposes an opt-in predicate to recover from this:

import { GhostFetchHttpClient } from 'crawlee-ghost-fetch';

new GhostFetchHttpClient({
    name: 'site',
    defaultCountry: 'US',
    treatAsSuccess: (p) => {
        const c = p.response.body ?? '';
        // Length floor rejects hard-block challenge bodies (~1 KB).
        // Endpoint-specific marker rejects WAF blank-shell error pages
        // (right outer shape, every field empty).
        return c.length > 50_000
            && /"@type":\s*"Product"[^]*?"name":\s*"[^"]+"/.test(c);
    },
});

The predicate receives the raw trace-v4 UnblockResult — read p.response.body, p.verdict, p.usable_content, etc.

Default behaviour (no predicate): the client trusts ghost-fetch's own verdict — a >=400 response the server judged usable (usable_content: true) is presented to Crawlee as 200. treatAsSuccess overrides that judgement both ways:

• returns true for a >=400 response → statusCode rewritten to 200 (SessionPool keeps the session, handler runs), statusMessage becomes OK (upstream <orig>), and the original status is preserved at response.headers['x-ghost-fetch-upstream-status'].
• returns false → the upstream status flows straight through, even if ghost-fetch judged the body usable (force a real block).

How to design the predicate

Two components, both required:

1. Body-length floor. Hard-block challenge bodies are tiny — 500 B to 5 KB depending on the WAF. Real pages are 50+ KB. A floor of ~30–50 KB rules out the obvious hard-block class without thinking.

2. Endpoint-specific success marker. This is the load-bearing check. Length alone false-passes WAF blank-shell error pages — pages that look like the right shape but render every field empty. Pick a marker that's only present when the data is real:

   • Product detail page: Product JSON-LD with a non-empty name. The blank-shell page has "name": "", so "name":\s*"[^"]+" is enough. Don't just match "@type":\s*"Product" — the shell has that too.
   • Listing / search results: at least one product card / tile element from your Phase 2 selector. Challenge and shell pages don't render the card grid.
   • Detail page with API-loaded data: a bound DOM attribute that only the real-data path produces (data-product-id="…" matching a non-empty value, etc.).
   • JSON API: if the API returns application/json, parse it and check for the expected envelope key. Don't string-match — valid API errors often have the same outer shape.

When an actor serves multiple endpoints

Dispatch off the URL inside the predicate. The predicate receives the full payload but not the request URL; pull it from your config or keep markers permissive (any marker satisfies):

treatAsSuccess: (p) => {
    const c = p.response.body ?? '';
    if (c.length < 30_000) return false;
    if (/"@type":\s*"Product"[^]*?"name":\s*"[^"]+"/.test(c)) return true;
    if (/data-tile-id="[^"]+"/.test(c)) return true;
    return false;
},

If the actor talks to a mix of WAF'd and unprotected hosts (e.g. main storefront + a CDN-hosted reviews API), the predicate is only consulted when status ≥ 400, so unprotected hosts (which return 200) bypass it naturally.

What this is NOT

• Not a substitute for retry policy. Real blocks still need rotation. The predicate is a discriminator between "the body has what we asked for despite the status code" and "the body is a challenge / error page". Get the predicate wrong on the conservative side and a real block flows through to the handler → handler crashes / returns nulls → actor loses a retry budget. Get it wrong on the permissive side and SessionPool hangs on to a dead session → every subsequent request through that session fails.

• Not a replacement for the warmup hook. A WAF that demands cookies before serving content still needs warmupFromUrl. The predicate only handles cases where the warmup worked but the WAF is cosmetically masking success.

• Not free. The predicate runs on every response. Keep it regex-based and small — don't parse the full DOM.

Environment variables

| Var | Required | Default | Purpose | |---|---|---|---| | GHOST_FETCH_URL | yes (or pass ghostFetchUrl on the client) | — | Points at your ghost-fetch deployment, e.g. https://<user>--ghost-fetch.apify.actor. The lib throws on construction if neither this nor the constructor arg is set. | | APIFY_TOKEN | yes when ghost-fetch is the Apify standby (Apify-hosted actors auto-inject) | — | Apify auth for the standby HTTP frontend in front of ghost-fetch. Set explicitly only for local dev or non-Apify hosting. | | GHOST_FETCH_SESSION | no | ${name}.{uuid} per process | Pin a session token across process restarts. Rarely useful in production — SessionPool drives rotation. |

Cold-start behavior

The first request through the client triggers ghost-fetch's browser tier (a *_browser strategy — chromium_browser / firefox_browser) to mint a validated session cookie set — typically 20-40s wall clock. Every subsequent request in the same process reuses that session token, so ghost-fetch routes them through its Tier 1 impit path (impit_chrome / impit_firefox) with the warmed cookies — typically 1-3s. The winning tier is reported per attempt in the response's decision_trace (strategy_id + transport).

For an actor's first user-facing call to succeed in spite of the cold start, the standard Crawlee retry config (maxRequestRetries: 3) bridges the gap: if the cold call returns a browser-tier 403 (Akamai sometimes rejects the very first request even with valid sensor data), the retry sees freshly-banked cookies and lands on Tier 1.

Apify deployment notes

The Apify-hosted actor build runs npm install against the public npm registry — no extra setup. APIFY_TOKEN is auto-injected at runtime for any Apify-hosted actor. Set GHOST_FETCH_URL on the actor (Settings → Environment variables) pointing at your ghost-fetch standby.

Troubleshooting

| Symptom | Likely cause | Fix | |---|---|---| | crawlee-ghost-fetch: ghostFetchUrl is required thrown on first request | Neither ghostFetchUrl config nor GHOST_FETCH_URL env is set | Pass one. There is no default. | | First call times out (>120s) | Cold-start browser warmup ran into an interactive challenge (Turnstile / hCaptcha) | Try a different country — Apify's residential pools vary; CZ/DE pools often clean for EU sites | | Every call resolves via a *_browser strategy, never drops to impit_* (Tier 1) | Tier 1 fingerprint mismatch with the residential exit's geo | Likely a ghost-fetch issue — open a ticket. Confirmed working: locale-aware Accept-Language is set per-country | | First call returns 500 from your actor with Request failed: ... | Crawlee aborted on application/octet-stream content-type before handler ran | Confirm you spread ...ghostFetchCrawlerOptions() into your crawler config | | crawlee-ghost-fetch: APIFY_TOKEN env var (or apifyToken config) is required | No token visible to the actor at runtime | Ensure APIFY_TOKEN is in the actor's runtime env (Apify auto-injects it for Apify-hosted actors, set it manually for local dev) | | Multi-country actor: every request goes to defaultCountry | countryFromUrl returning undefined for valid URLs | Log inside the resolver — common bug is checking tld === 'de' (lowercase) when the ToUpperCase comparison expects 'DE' | | SessionPool keeps retiring sessions even when handlers extract data fine | WAF returns 4xx alongside the real page body; default blockedStatusCodes retires the session before your handler can confirm success | Set treatAsSuccess on the client with a length floor + endpoint-specific marker | | Cold call rate is fine but the actor's IP-quality lottery never settles — every request pays the cold warmup | Same as above: the warm session that minted real cookies was retired the moment it returned 4xx, so nothing in the pool ever ages | Same fix; verify by inspecting response.headers['x-ghost-fetch-upstream-status'] in the handler — if the rewrite is firing, the same session id should appear across consecutive successful calls |

Migration from an inline ghost-fetch client

If your actor already has a hand-rolled ghost-fetch-client.ts, the diff is roughly:

- import { GhostFetchHttpClient } from './utils/ghost-fetch-client.js';
+ import {
+     GhostFetchHttpClient,
+     ghostFetchCrawlerOptions,
+ } from 'crawlee-ghost-fetch';

  const crawler = new CheerioCrawler({
-     httpClient: new GhostFetchHttpClient(),
+     httpClient: new GhostFetchHttpClient({ name: 'overstock', defaultCountry: 'US' }),
-     additionalMimeTypes: ['application/octet-stream'],
-     useSessionPool: false,
+     ...ghostFetchCrawlerOptions(),
      ...
  });

Then delete src/crawler/utils/ghost-fetch-client.ts from your actor.

For multi-country actors that previously used a preNavigationHook to stamp x-ghost-country onto request headers, drop the hook entirely — countryFromUrl runs inside the client's sendRequest, no header round-trip needed.

Versioning

This package follows semver. The 0.x line tracks breaking changes freely; we plan to cut 1.0.0 once the API has stabilized across at least three production actors.

See also

• ghost-fetch — the unblocker this library talks to.
• Crawlee — the framework this library plugs into.
• Apify standby actors — the deployment shape both reference actors use.