@traceten/ai-crawl

Server-side AI crawler tracking for Traceten. See the GPTBot, ClaudeBot and PerplexityBot crawls that browser JavaScript can never see.

AI crawlers do not execute JavaScript. They issue one HTTP GET for the HTML and leave, so the Traceten snippet is structurally blind to them. The only possible observation point is your server, edge function, or worker. This package is that observation point.

What it does

1. Pre-filters locally: only GET/HEAD, skips assets and API routes, skips browser subresource fetches. Non-crawler traffic costs one string scan and no network call.
2. Matches the user agent against a local list of known AI crawler tokens, anchored at token boundaries (a UA containing not-really-GPTBot does not match).
3. Reports plausible AI crawls to POST https://ingest.traceten.com/v1/ai-crawls, authenticated with your crawl token. Delivery is scheduled after the response (via waitUntil where the runtime has one) and never delays or breaks a request.

Provider, category (answer fetch, search index, training, other), verification and confidence are decided by Traceten's servers against the full crawler registry, so new crawlers are covered without upgrading this package.

Contract

• Never blocks the response. Never throws per-request. Failure is silent.
• Zero runtime dependencies.
• 5000 ms delivery timeout, keepalive: true. Delivery is scheduled off your response path, so this bounds a background task and never delays a response.
• Requires Node 18+ or an edge runtime with fetch.
• Each adapter bundle is under 8 KB gzipped (CI-enforced).

Install

npm install @traceten/ai-crawl

Configuration

import { defineAiCrawlConfig } from "@traceten/ai-crawl";

const config = defineAiCrawlConfig({
  siteId: "ttid_7Rb4TrC1dTbnD8w3s1TS12", // your site key (the ttid_… data-site value on the install page; case-sensitive)
  authToken: process.env.TRACETEN_CRAWL_TOKEN!, // tt_bot_..., created in the dashboard. REQUIRED.
});

defineAiCrawlConfig throws on invalid input. Call it at module scope or server boot so a missing token fails your deploy, not your visitors' requests. The endpoint rejects unauthenticated reports, so a missing token would otherwise mean every report silently returns 401.

The authToken is a server-side secret. Do not put it in browser code, and do not reuse the public snippet key in its place.

Options

| Option | Default | Meaning | | ------------------------- | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | endpoint | https://ingest.traceten.com/v1/ai-crawls | Ingestion endpoint | | allowedMethods | ["GET", "HEAD"] | Methods considered at all | | extraDenyPathPrefixes | [] | Added to the built-in deny-list (/api, /_next, /static, ...). Extends, never replaces. Prefixes match whole path segments: /api denies /api and /api/users but not /apidocs | | extraDenyExtensions | [] | Added to the built-in static-extension deny-list. Extends, never replaces | | disableAnswerFetch | false | Skip user-triggered fetchers (ChatGPT-User, Claude-User, ...) | | disableSearchCrawlers | false | Skip AI search index crawlers (OAI-SearchBot, PerplexityBot, ...) | | disableTrainingCrawlers | false | Skip training crawlers (GPTBot, ClaudeBot, CCBot, ...) | | disableOtherCrawlers | false | Skip uncategorised AI bots and unrecognised agents from known vendors | | trustProxy | false | Trust x-forwarded-for for IP resolution. Off by default because the header is spoofable | | proxyDepth | 1 | Number of trusted proxies appending to x-forwarded-for (used only with trustProxy) | | trustCfConnectingIp | false | Trust the cf-connecting-ip header. The Cloudflare Workers and Pages adapters enable this automatically; set it manually only when your origin sits behind Cloudflare. Off by default because any client can forge the header on a non-Cloudflare origin | | publicOrigin | none | Rebuild reported URLs on this origin, for containers behind reverse proxies that see internal hosts. The fetch-based adapters (Next.js, Hono, Cloudflare) already fall back to the incoming Host header when the runtime resolves the request against its own bind address (0.0.0.0, [::]); set this when your Host header itself carries an internal hostname | | onError | none | Called when a report does not land: a non-2xx, or a network/timeout failure. Off by default; delivery stays silent either way |

Knowing when reports are not landing

Delivery is silent by design: a broken analytics call must never surface on your site, so sendReport never throws and never retries. The cost is that a rejected report looks exactly like a delivered one. A wrong authToken, or egress your platform blocks, both read as "no AI crawlers visited" indefinitely.

onError closes that gap without changing the safety posture:

export const aiCrawl = defineAiCrawlConfig({
  siteId: "ttid_7Rb4TrC1dTbnD8w3s1TS12",
  authToken: process.env.TRACETEN_BOT_TOKEN!,
  onError: (e) => {
    // e.kind is "http" (with e.status) or "network" (with e.cause)
    console.warn("[traceten] crawl report failed", e.kind, e.status ?? e.cause);
  },
});

Your callback runs inside a try/catch, so throwing from it cannot break the host response. A 401 or 403 means the token or the origin; repeated network failures mean egress. Worth wiring in staging at minimum: it is the difference between "no crawlers came" and "nothing we sent was accepted".

Crawler-facing paths (/robots.txt, /llms.txt, /llms-full.txt, *sitemap*.xml) are always reported even though .txt and .xml are otherwise denied. A GPTBot hit on /llms.txt is one of the highest-signal events this product can capture.

Adapters

Next.js (proxy / middleware)

// proxy.ts (Next 15.5+; middleware.ts on earlier versions)
import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";
import { defineAiCrawlConfig } from "@traceten/ai-crawl";
import { trackAICrawlerRequest } from "@traceten/ai-crawl/next";

const config = defineAiCrawlConfig({
  siteId: process.env.TRACETEN_SITE_ID!,
  authToken: process.env.TRACETEN_CRAWL_TOKEN!,
});

export function proxy(request: NextRequest, event: NextFetchEvent) {
  trackAICrawlerRequest(request, event, config);
  return NextResponse.next();
}

Cloudflare Workers

For a site already behind Cloudflare this needs no application deploy at all.

import { defineAiCrawlConfig } from "@traceten/ai-crawl";
import { withAICrawlerTracking } from "@traceten/ai-crawl/cloudflare-workers";

const config = defineAiCrawlConfig({
  siteId: "ttid_7Rb4TrC1dTbnD8w3s1TS12",
  authToken: "tt_bot_...", // prefer an env binding in production
});

export default {
  fetch: withAICrawlerTracking(async (request, env, ctx) => {
    return fetch(request); // your existing origin logic
  }, config),
};

The wrapper captures the real response status and reports through ctx.waitUntil. Errors from your handler propagate untouched.

Cloudflare Pages Functions

// functions/_middleware.ts
import { createAICrawlerPagesMiddleware } from "@traceten/ai-crawl/cloudflare-pages";

export const onRequest = createAICrawlerPagesMiddleware({
  siteId: "ttid_7Rb4TrC1dTbnD8w3s1TS12",
  authToken: "tt_bot_...",
});

Express

import express from "express";
import { createTracetenAICrawlerMiddleware } from "@traceten/ai-crawl/express";

const app = express();
app.use(
  createTracetenAICrawlerMiddleware({
    siteId: process.env.TRACETEN_SITE_ID!,
    authToken: process.env.TRACETEN_CRAWL_TOKEN!,
  }),
);

Calls next() immediately and reports on the response finish event, so the real status code is captured and no request is ever delayed.

Set publicOrigin here. Without it the reported URL is built from the request's Host header (which a client can set to anything) with an assumed https scheme. With publicOrigin the reported URL is always on your real origin:

createTracetenAICrawlerMiddleware({
  siteId: process.env.TRACETEN_SITE_ID!,
  authToken: process.env.TRACETEN_CRAWL_TOKEN!,
  publicOrigin: "https://www.example.com",
});

Hono / generic Request+Response

import { Hono } from "hono";
import { aiCrawlerTracking } from "@traceten/ai-crawl/hono";

const app = new Hono();
app.use("*", aiCrawlerTracking({ siteId: "...", authToken: "tt_bot_..." }));

Any other framework that exposes fetch-API objects can use trackAICrawlerFetch(request, response, config, waitUntil?) from the same entry point.

What is sent, including the crawler IP

For each matched crawl the package sends exactly:

{
  "site_id": "ttid_7Rb4TrC1dTbnD8w3s1TS12",
  "url": "https://example.com/docs/pricing",
  "method": "GET",
  "status": 200,
  "user_agent": "Mozilla/5.0 ... compatible; GPTBot/1.1; +https://openai.com/gptbot",
  "ip": "20.15.240.64",
  "ts": 1765432100000
}

The ip field is the crawler's IP address, resolved from cf-connecting-ip (only where trusted, see trustCfConnectingIp), then x-forwarded-for (only when trustProxy is set), then the socket address. The field is omitted when no trustworthy value can be derived.

Sending it is deliberate and required. The crawler's TCP connection terminates at your server, so its IP is observable only there. Traceten uses it to verify the crawl against network evidence (for example OpenAI publishes GPTBot's IP ranges), which is what separates a real GPTBot crawl from a spoofed user agent. Verification runs at Traceten's edge while the value is in scope, and the reported IP is always treated as a claim to be verified, never as proof by itself.

Retention depends on the verdict. Traceten keeps the raw address in exactly two cases: it fell inside a range the vendor publishes for that crawler, or its reverse DNS forward-confirmed to the vendor's own domain. In every other case, including an address that only matches the provider's wider network, the address is discarded at the edge and only an HMAC-SHA-256 hash under a per-site derived key is stored. Those rows are hash-only: Traceten does not resolve or store the network operator (ASN) or the country for a crawler IP, and writes neither field. Verification that needs a reverse-DNS lookup sends the address to Cloudflare's public DNS resolver before anything about it is known. This package never sends visitor analytics, cookies, or request bodies.

No batching in v1: each matched crawl is one report. Crawl volume on typical sites is orders of magnitude below pageview volume because of the pre-filter.

Privacy and GDPR

• Traceten processes crawl reports on your behalf, as your processor under the Traceten Data Processing Agreement.
• Your privacy notice should reflect this transfer: crawler request metadata (URL, user agent, IP address) is shared with Traceten for bot and crawler detection.
• The typical legal basis is legitimate interest in detecting and attributing automated crawler traffic. Confirm this with your own counsel for your jurisdiction.
• Hashed unverified IPs are pseudonymized personal data. Crawl records are records of automated software requests rather than of a visitor, and they carry no visitor or session identifier, so Traceten's per-visitor access and deletion API cannot reach them — it has no key to look one up by. That is a limit of the API, not a claim that the records can never be matched: the hash is deterministic per site, so where a data subject supplies the address itself, Traceten can find and erase their rows. Those requests are handled manually — email [email protected] with the address, the site and a date range. (This is the GDPR Article 11(2) situation: no obligation to acquire extra data to identify someone, full rights where the data subject supplies what makes identification possible.) What otherwise bounds these records: the 90-day limit below, hash-only storage for anything unverified, and whole-account erasure, which removes every crawl record and rollup for every site on the account. See the crawl privacy notes.
• Individual crawl records are deleted after 90 days on every plan. The daily rollups derived from them are counts grouped by crawler and by page path, with no address, no hash and no personal identifier. Page paths therefore persist in aggregate after the records containing them expire.

Development

npm install
npm test        # vitest
npm run typecheck
npm run build    # tsc + size-gate bundles + 8KB check

The local token list lives in src/crawlers.ts. It's a deliberately-loose cost filter behind Traceten's server-side crawler registry, which is the actual source of truth for provider, category, verification and confidence — this list only decides "worth reporting locally." Google-Extended and Applebot-Extended are excluded on purpose: they are robots.txt control tokens and never appear as live user agents.

Versioning

This package follows Semantic Versioning. Before 1.0.0, minor versions may include breaking changes — pin an exact version in production until then. See CHANGELOG.md for release history.

Contributing

Issues and pull requests are welcome. For anything beyond a small fix, please open an issue first to discuss the change. Run the checks below before submitting a PR — CI enforces the same steps on every pull request:

npm install
npm run typecheck
npm run build
npm test

License

MIT © Traceten — see LICENSE.

Links

• Documentation
• Traceten — AI traffic attribution for the AI search era
• Issues
• Changelog
• Node SDK · Python SDK · Go SDK