@watchgold/bot-verify

Verify Googlebot/Bingbot the way the engines document it: published IP-range verification with spoof-resistant X-Forwarded-For parsing. Edge-runtime safe.

Google and Microsoft both publish the exact IP ranges their crawlers operate from, and both document the same authentication method: when a request claims a crawler user-agent, check its source IP against those published ranges. This package implements that method as a small, dependency-free library:

• a user-agent pattern covering exactly the crawlers whose IPs the engines publish for verification (Googlebot variants, Google-InspectionTool, GoogleOther; bingbot, adidxbot, BingPreview),
• BigInt-based IPv4/IPv6 parsing and CIDR containment (no Node APIs),
• a hardened X-Forwarded-For walk that resists client-forged header entries,
• a vendored snapshot of the official ranges as the floor, with an optional throttled background refresh of the live lists.

Why user-agent sniffing alone is not enough

The User-Agent header is free text chosen by the client. Anyone can send Googlebot/2.1 from a laptop, so a UA match alone tells you nothing about who is actually crawling. The engines' answer is IP verification: Google publishes googlebot.json and Microsoft publishes bingbot.json, and a request is only genuinely from that crawler when its IP falls inside those ranges. This package pairs the UA check (cheap pre-filter) with the IP check (the actual proof).

The X-Forwarded-For hardening story

Knowing the "client IP" behind a proxy chain is itself an untrusted-input problem. Clients can send their own X-Forwarded-For header; trustworthy infrastructure appends the address it saw connecting. That means:

• Only the right edge of the header is trustworthy. Anything the client sent itself sits on the left and must never be used — a spoofer would simply put 66.249.66.1 there.
• The walk starts from the right and skips known infrastructure hops: private and loopback addresses, plus a configurable set of trusted load-balancer ranges (trustedInfraRanges). The default is GCP_FRONTEND_RANGES — the Google Front End / external HTTP(S) Load Balancer ranges that legitimately sit at the right edge in front of Cloud Run. Deploying behind a different edge (another cloud, a CDN)? Pass your own platform's published proxy ranges.
• The walk inspects at most maxHops entries (default 3), so a long forged chain cannot walk the parser back into client-controlled territory.

Legitimate uses

Verification answers one question — "is this request really the crawler it claims to be?" — which is useful wherever crawler identity matters:

• Analytics and logs: label genuine crawler traffic (and spot spoofers) instead of trusting user-agent strings.
• Rate limiting / bot protection: exclude verified crawlers from limits that target abusive automation, without opening a UA-string loophole.
• Dynamic rendering / crawler-specific responses: make the render-for-crawlers decision on proof rather than on a spoofable header.

Install

npm install @watchgold/bot-verify

Quick start (Next.js middleware)

// middleware.ts
import { NextResponse, type NextRequest } from 'next/server';
import { createCrawlerVerifier, isSearchBotUserAgent } from '@watchgold/bot-verify';

// One instance per server process: it owns the current range set and the
// refresh throttle.
const verifier = createCrawlerVerifier();

export function middleware(request: NextRequest) {
  if (!isSearchBotUserAgent(request.headers.get('user-agent'))) {
    return NextResponse.next();
  }

  // Fire-and-forget: keeps ranges fresh in the background, never blocks.
  verifier.scheduleRefresh();

  // 'google' | 'bing' | 'dev' (local development) | null (claims a bot UA
  // but the IP says otherwise).
  const verdict = verifier.verifyRequest(request.headers.get('x-forwarded-for'));

  const response = NextResponse.next();
  response.headers.set('x-crawler', verdict ?? 'spoofed');
  return response;
}

Quick start (generic)

import { createCrawlerVerifier, isSearchBotUserAgent } from '@watchgold/bot-verify';

const verifier = createCrawlerVerifier({
  // Not on Google Cloud? Declare your own edge's ranges.
  trustedInfraRanges: ['203.0.113.0/24'],
  // Explicit is fine too; the default is NODE_ENV !== 'production'.
  allowDirectAsDev: false,
});

function classifyRequest(userAgent: string | null, xff: string | null) {
  if (!isSearchBotUserAgent(userAgent)) return 'human';
  return verifier.verifyRequest(xff) ?? 'spoofed-bot';
}

// Or verify a bare IP you already trust:
verifier.verifyIp('66.249.66.1'); // → 'google'
verifier.verifyIp('8.8.8.8');     // → null (Google DNS is not Googlebot)

API

createCrawlerVerifier(options?): CrawlerVerifier

Creates an independent verifier instance (all state — current ranges, refresh throttle — lives on the instance).

| Option | Default | Meaning | | --- | --- | --- | | ranges | vendored Google + Bing snapshot | Record<engine, readonly cidr[]> to verify against. | | trustedInfraRanges | GCP_FRONTEND_RANGES | Proxy/LB CIDRs skipped in the right-edge XFF walk. | | allowDirectAsDev | NODE_ENV !== 'production' | Whether direct/private-network requests yield 'dev' instead of null. Guarded with typeof process; runtimes without process fail closed. | | refresh.sources | official Google/Bing URLs | Record<engine, url> of published range JSON endpoints. | | refresh.minIntervalMs | 24h | Throttle between refresh attempts. | | refresh.timeoutMs | 5000 | Per-request fetch timeout. | | refresh.minPrefixCounts | { google: 50, bing: 5 } | Sanity floors — an undersized payload is refused so a truncated response never shrinks verification below the vendored floor. | | refresh.fetch, refresh.now | globals | Injection points for tests. |

The returned instance:

• verifyIp(rawIp) — which engine (if any) publishes this IP; null otherwise.
• verifyRequest(xff) — verdict for a request whose UA already matched: the engine name, 'dev' for a direct/private origin outside production (typical managed platforms always set X-Forwarded-For in production, so its absence there means "not a real bot"), or null for a claimed bot whose IP says otherwise.
• scheduleRefresh() — fire-and-forget, throttled background refresh; never awaited on the request path.
• currentRanges() — the range set currently in use.

Standalone pieces

• verifyIpAgainstRanges(ranges, rawIp) — the pure range check.
• clientIpFromForwardedFor(xff, { trustedInfraRanges?, maxHops? }) — the hardened right-edge XFF walk.
• parseIp(raw), ipInCidr(ip, cidr), isPrivateIp(raw) — BigInt IP utilities (IPv4, IPv6 with :: shorthand, IPv4-mapped ::ffff: form, zone indexes stripped, junk → null).
• SEARCH_BOT_UA_PATTERN, isSearchBotUserAgent(ua) — the UA pre-filter. AI/training bots (GPTBot, Google-Extended, …) are deliberately excluded: they are governed by robots.txt, not IP verification.
• GCP_FRONTEND_RANGES, vendoredRanges — the shipped data.

Vendored floor + background refresh

The package ships vendoredRanges — a snapshot of the engines' official lists — so verification works offline and from the first request. At runtime, scheduleRefresh() re-fetches the live lists at most once per day per instance and swaps them in only when they look sane (per-engine minimum prefix counts); a truncated or errored payload can never shrink verification below the vendored floor, and a network failure keeps the previous set.

To update the committed snapshot itself:

npm run refresh-ranges   # regenerates src/vendored-ranges.ts from the official sources

Edge-runtime safety

Everything under src/ uses Web-platform APIs only — no node: imports, no Node globals required (process is feature-detected). IP math is plain BigInt arithmetic. This makes the package safe for Next.js middleware, Cloudflare Workers, and other edge runtimes, as well as ordinary Node servers.

License

MIT.

Extracted from the production codebase of WatchGold, a precious-metals market-data platform.