@projecthq/crawlers

Count the search engines and AI crawlers reading your site — the ones your analytics script can never see, because they take the HTML and leave.

Most crawlers that matter today (GPTBot, ClaudeBot, CCBot, PerplexityBot) do not run JavaScript, so no browser tracker will ever record them. They are visible only from your own server, which is where this package lives: a few lines in your middleware, and every declared crawler hit shows up in your ProjectHQ Crawlers report.

Only known crawlers ever leave your server. Every request is matched against an embedded token list before anything else happens. No match — which is to say, a person — and the function returns having made no network call at all. Your visitors' IP addresses and reading habits stay on your infrastructure. There is no sampling of human traffic, no "we filter it on our side": it never goes out.

• Zero dependencies, 2.7 kB gzipped, ESM + CJS
• Node 18+, Bun, Deno, Cloudflare Workers/Pages, Vercel Edge
• Never blocks a response, never throws, never retries into your error budget

Install

npm install @projecthq/crawlers

You need your site id — the same public project UUID your tracker script already uses (tracker.js?site=…), shown in Analytics → Settings. It is not a secret: commit it, ship it in an edge bundle, put it in a public repo.

Next.js

On Next 16 the file is proxy.ts in your project root. On Next 15 and earlier it is middleware.ts and the function is called middleware — everything else is identical.

Most apps already have one, so compose rather than replace:

// proxy.ts  (middleware.ts before Next 16)
import { recordCrawler } from "@projecthq/crawlers";
import { NextResponse } from "next/server";
import type { NextFetchEvent, NextRequest } from "next/server";

export function proxy(request: NextRequest, event: NextFetchEvent) {
  // Not awaited — it returns immediately. `event` lets the report finish after
  // the response has been sent, which is what serverless needs.
  recordCrawler(request, event, { siteId: "your-site-id" });

  // ...whatever your proxy already did
  if (!request.cookies.get("session")) {
    return NextResponse.redirect(new URL("/login", request.url));
  }

  return NextResponse.next();
}

export const config = {
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};

Starting from scratch? The same file with just the recordCrawler line and return NextResponse.next() is a complete integration.

The matcher matters. Without it the proxy runs on every static asset and image request, which costs you invocations and tells you nothing new — a crawler that fetched a page also fetched its CSS.

Cloudflare Workers

import { withCrawlerRecording } from "@projecthq/crawlers";

export default {
  fetch: withCrawlerRecording(async (request, env, ctx) => {
    return new Response("hello");
  }, { siteId: "your-site-id" }),
};

The wrapper reports the status the crawler was served as well, and registers the report on the execution context so it outlives the response.

Cloudflare Pages

// functions/_middleware.ts
import { recordCrawler } from "@projecthq/crawlers";

export const onRequest: PagesFunction = (context) => {
  recordCrawler(context.request, context, { siteId: "your-site-id" });
  return context.next();
};

Express

import express from "express";
import { crawlerRecorder } from "@projecthq/crawlers/express";

const app = express();
app.use(crawlerRecorder({ siteId: "your-site-id" }));

Mount it first, before your routes, so it sees every request. The report goes out on the response's finish event, so the status code — the 404s and 500s you are serving to Googlebot — is recorded too. A request from a person does not even get a listener attached.

Hono

import { Hono } from "hono";
import { honoCrawlerRecorder } from "@projecthq/crawlers/hono";

const app = new Hono();
app.use(honoCrawlerRecorder({ siteId: "your-site-id" }));

Uses c.executionCtx where the runtime has one (workerd, Deno Deploy) and falls back to fire-and-forget where it does not (Node, Bun).

Anything else with a web-standard Request

recordCrawler takes a plain Request, so Deno, Bun, SvelteKit hooks, Nitro, Remix and friends all work directly:

import { recordCrawler } from "@projecthq/crawlers";

Deno.serve((request) => {
  recordCrawler(request, { siteId: "your-site-id" });
  return new Response("hello");
});

API

recordCrawler(request, ctx?, options)

Reports request if it came from a known crawler. Returns void, synchronously — do not await it, there is nothing to await. Never throws.

Pass the runtime context as the middle argument wherever you have one (NextFetchEvent, workerd ExecutionContext, Cloudflare Pages context). It is detected by duck-typing waitUntil, so any runtime offering that method works. Without it the report is fire-and-forget, which is fine on a long-running server and lossy on serverless.

Where there is no context, pass null (or leave the argument out entirely — both forms are typed):

recordCrawler(request, null, { siteId: "your-site-id" });
recordCrawler(request, { siteId: "your-site-id" });

recordCrawlerResponse(request, response, ctx?, options)

Same, plus the response's status code. Use it where you have the response in hand; a wave of 404s served to a crawler is the interesting half of the report.

withCrawlerRecording(handler, options)

Wraps a Cloudflare Workers fetch handler (or a Pages onRequest). Returns a handler with the same signature that reports every crawler hit with its status.

crawlerRecorder(options) — from @projecthq/crawlers/express

Express middleware. Captures the status on finish.

honoCrawlerRecorder(options) — from @projecthq/crawlers/hono

Hono middleware. Captures the status after next().

Options

| Option | Type | Default | Meaning | |---|---|---|---| | siteId | string | required | Your public project UUID. Not a secret. | | endpoint | string | https://api.projecthq.pro/v1/server-hits | Where hits are posted. | | apiKey | string | — | Optional. Sent as X-API-Key when set; nothing is sent when it is not. Only needed if you would rather authenticate with your secret tracking key than with the public site id. | | ignoreCategories | CrawlerCategory[] | [] | Categories to drop locally. Nothing is sent for a filtered hit. | | timeoutMs | number | 3000 | Abort the report after this long. |

Categories: "ai_answers" (assistants fetching a page to answer someone now), "ai_training" (collection for model training), "search_index" (classic search indexing), "seo_tools", "other" (link previews, uptime monitors).

recordCrawler(request, event, {
  siteId: "your-site-id",
  ignoreCategories: ["seo_tools", "other"],
});

What is sent

One POST per crawler hit, keepalive: true, aborted after timeoutMs:

{
  "site_id": "your-site-id",
  "hits": [
    {
      "user_agent": "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)",
      "ip": "203.0.113.7",
      "path": "/blog/post",
      "hostname": "example.com",
      "referrer": "",
      "status": 200,
      "occurred_at": "2026-08-24T10:00:00.000Z"
    }
  ]
}

ip is the first hop of x-forwarded-for, else x-real-ip, else cf-connecting-ip, else an empty string — no address is ever guessed. It is what lets us verify a bot against the ranges its operator publishes, so a spoofed GPTBot user agent can be shown as unverified rather than counted as real. status is present only on the response-side paths (Express, Hono, withCrawlerRecording, recordCrawlerResponse).

hostname comes from x-forwarded-host, then Host, and only then the request URL. That order matters when you self-host behind a reverse proxy: the URL your runtime hands middleware names the container it listens on, not the site the crawler asked for. It has to be your project's domain or a subdomain of it — the collector rejects hits for any other host, which is what stops someone else's site id from polluting your report.

If you are seeing nothing arrive, check what your proxy sets: a proxy that strips Host and sets no x-forwarded-host leaves nothing to identify the site by. Reporting is silent by design, so a rejected hit looks exactly like no traffic.

Hits are sent one at a time rather than batched. After the local prefilter the volume is a handful of requests a minute even on a heavily crawled site, and serverless instances have no state to batch in.

The token list is embedded at build time from the same database that backs the public GET https://api.projecthq.pro/v1/crawler-tokens. Nothing is fetched at runtime: until a crawler actually shows up, this package makes no network calls whatsoever.

Failure behaviour

Every error is swallowed: a network failure, a timeout, a 429 telling you to slow down, a 500 from the collector, a malformed request object. The response is never even read. A lost hit is a lost hit — it is never your incident. Nothing is retried, nothing is queued, and no unhandled rejection reaches your process.

License

MIT