@rankveo/crawler-log

Record which AI crawlers actually read your pages. Part of rankveo — SEO and generative engine optimization for teams who want to be cited by ChatGPT, Claude and Perplexity, not only ranked.

GPTBot, ClaudeBot, PerplexityBot and the rest identify themselves in the User-Agent header, and that header only exists in the log of whatever served the request. No provider reports it and it cannot be inferred from visibility data — so this forwards those requests to rankveo, and nothing else.

npm install @rankveo/crawler-log

Create a token in Dashboard → AI Crawlers, then put it in your environment:

RANKVEO_CRAWLER_TOKEN=rk_ingest_...

Keeping it in the environment rather than in code is the point. A rotated token is a redeploy, not an edit, and npm update picks up new crawlers as they appear — a snippet pasted into a Worker stays frozen at whatever the list looked like the day it was copied.

The token is write-only. It can append crawler events to your workspace and do nothing else, so a leak costs you noise in your own chart rather than access to your data. It still belongs in a server-only variable — never NEXT_PUBLIC_.

Express

import { crawlerLog } from "@rankveo/crawler-log/express";

app.use(crawlerLog());

Mount it before your routes. It records on finish, so the status is the one actually sent.

Cloudflare Workers

export { default } from "@rankveo/crawler-log/cloudflare";

Deploy that on a route and it proxies every request through untouched, recording the crawler ones. Set the token as a secret:

wrangler secret put RANKVEO_CRAWLER_TOKEN

Already have a Worker? Record from inside it instead:

import { logCrawlerHit } from "@rankveo/crawler-log/cloudflare";

export default {
  async fetch(request, env, ctx) {
    const response = await fetch(request);
    logCrawlerHit(request, response, env, ctx);
    return response;
  },
};

Next.js

// middleware.ts
import { withCrawlerLog } from "@rankveo/crawler-log/next";

const record = withCrawlerLog();

export function middleware(request) {
  record(request);
}

Middleware runs before the route, so the real status is not known yet and the hit is recorded as served. If you need true statuses, use the Cloudflare or Express adapter instead.

Anything else

The core is framework-agnostic:

import { createCrawlerLogger, isCrawler } from "@rankveo/crawler-log";

const logger = createCrawlerLogger();

if (isCrawler(userAgent)) {
  logger.record({ userAgent, path, status });
}

await logger.flush(); // on shutdown, if your runtime is short-lived

Is it really that crawler?

A user-agent is a string anyone can send, so every adapter forwards the client address too. rankveo checks it against the addresses each operator publishes — OpenAI and Perplexity do; Anthropic and Google do not, and those hits stay honestly unlabelled rather than being assumed genuine.

The address is checked on arrival and discarded. Only the verdict is stored, so nothing here becomes a log of who visited your site.

A Cloudflare Worker is the most trustworthy place to record from: CF-Connecting-IP is set by Cloudflare and a client cannot forge it. Behind your own proxy, set trust proxy in Express so X-Forwarded-For means something. Without a proxy you control the header is forgeable, and a hit with no address is recorded as unverified rather than not recorded at all.

What leaves your server

This is the whole list. It is short on purpose, and the source is here so you can check rather than take our word for it.

| Field | Example | Why | | --- | --- | --- | | userAgent | compatible; GPTBot/1.2 | Which crawler it was | | path | /pricing | Which page it read. No query string | | status | 200 | Whether you served it or refused it | | at | 2026-09-03T00:00:00Z | When | | ip | 23.102.140.115 | Checked against the crawler's published addresses, then discarded |

Nothing is sent for a request that is not a crawler. record() returns false before anything is queued unless the user-agent matches the crawler list, so your visitors' requests never leave your infrastructure — they are not filtered later on our side, they are never collected in the first place.

No cookies, no request or response bodies, no Referer, no headers beyond the user-agent, no session or account identifiers. Not "we don't look at them" — the code never reads them. Query strings are stripped from the path before it is sent, so tracking parameters and anything else you put in a URL stay with you.

The address is the only field that could identify anybody, and it is a crawler's address by construction, since a hit is only queued once the user-agent matched. rankveo checks it against the operator's published ranges on arrival and stores only the verdict — no column in the database holds an address.

What it will not do

• Delay a response. Hits are queued after the response is on its way and flushed on a timer or when the batch fills.
• Throw into your application. Every failure is swallowed. Pass { debug: true } to hear about it once.
• Forward visitor traffic. Only user-agents on the crawler list are queued. The filtering happens on your server, so ordinary requests never leave it.

Options

| Option | Default | | | --- | --- | --- | | token | RANKVEO_CRAWLER_TOKEN | Write-only ingest token | | apiUrl | RANKVEO_API_URL, else https://api.rankveo.com/api | | | batchSize | 25 | Hits queued before sending | | flushMs | 10000 | How long a partial batch waits | | debug | false | Warn once when something is wrong |

What rankveo does with this

The dashboard shows which crawlers reached which pages, whether you served or refused them, and whether each one was really the crawler it claimed to be. It sits beside the rest of rankveo: rank tracking, site audits, backlinks, and AI answer visibility across ChatGPT, Claude, Perplexity and Google AI Overviews.

• rankveo.com — what it is
• app.rankveo.com — create an account
• help.rankveo.com — documentation

Other rankveo packages

Run your rankveo blog on your own domain: @rankveo/client · @rankveo/next · @rankveo/astro · @rankveo/nuxt · @rankveo/sveltekit · @rankveo/eleventy

MIT © rankveo