@datafast/ai-crawl

Server-side AI crawler tracking for DataFast.

Use this package to see when known AI assistants, AI search systems, or training crawlers request pages on your website. It is not a replacement for DataFast web analytics and does not track human visitors, sessions, conversions, attribution, or browser pageviews.

Full setup guide: DataFast bot traffic tracking docs

Installation

npm install @datafast/ai-crawl

How It Works

The package runs on your server, middleware, edge function, or worker. For each incoming request, it checks the user agent and path locally. If the request is not a likely AI/search crawler document request, nothing is sent to DataFast.

When a likely crawler requests a document page, the package sends a small event to DataFast's AI crawl endpoint. In runtimes that support background tasks, such as Vercel and Cloudflare, it uses waitUntil so your page response does not wait for DataFast.

The package is only a cost and safety pre-filter. DataFast's servers are the source of truth for provider classification, category, IP verification, confidence, and crawler list updates. This means DataFast can update crawler user agents and IP ranges without requiring you to upgrade this package for every crawler change.

Status code is optional. The most important signal is the page the AI tried to crawl. Status code is captured only when the backend exposes it without adding latency.

Important: when you pass a Vercel, Cloudflare, or Worker context to trackAICrawlerRequest, do not await it. The package schedules the tracking request with waitUntil and returns immediately.

Tutorials

Next.js / Vercel Middleware

Create or update middleware.ts:

// middleware.ts
import {
  NextResponse,
  type NextFetchEvent,
  type NextRequest,
} from "next/server";
import { trackAICrawlerRequest } from "@datafast/ai-crawl";

export function middleware(request: NextRequest, event: NextFetchEvent) {
  trackAICrawlerRequest(request, event, {
    websiteId: "dfid_your_tracking_id",
  });

  return NextResponse.next();
}

export const config = {
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};

This is non-blocking. The package schedules the DataFast request with event.waitUntil, then your app continues returning the page.

Middleware runs before the final page response, so status code is usually unknown and will be stored as 0.

Cloudflare Pages

Install the package, then create a Pages Functions middleware file:

// functions/_middleware.ts
import { trackAICrawlerRequest } from "@datafast/ai-crawl";

export async function onRequest(context) {
  trackAICrawlerRequest(context.request, context, {
    websiteId: "dfid_your_tracking_id",
  });

  return context.next();
}

Cloudflare Pages provides context.waitUntil, so the DataFast request runs in the background while context.next() continues to your page or static asset.

Do not await trackAICrawlerRequest(...) here. Call it, then immediately return context.next().

For better Cloudflare billing/performance hygiene, also exclude obvious static assets from Function invocation when your framework does not generate this automatically. Add _routes.json to your build output:

{
  "version": 1,
  "include": ["/*"],
  "exclude": [
    "/assets/*",
    "/static/*",
    "/_next/*",
    "/favicon.ico",
    "/robots.txt",
    "/sitemap.xml",
    "/*.css",
    "/*.js",
    "/*.png",
    "/*.jpg",
    "/*.svg",
    "/*.ico",
    "/*.woff2"
  ]
}

The package still filters static assets locally, but _routes.json prevents Cloudflare from invoking your Function for the most obvious asset requests.

Cloudflare docs: Pages middleware, Pages Functions API, routing and _routes.json.

Cloudflare Workers

Wrap your Worker fetch handler:

import { withAICrawlerTracking } from "@datafast/ai-crawl";

export default {
  fetch: withAICrawlerTracking(
    async (request: Request, env: Env, ctx: ExecutionContext) => {
      return fetch(request);
    },
    {
      websiteId: "dfid_your_tracking_id",
    },
  ),
};

This captures status code because the wrapper sees the Response your handler already created. It still uses ctx.waitUntil when available, so the tracking request is best-effort and non-blocking.

Express

import express from "express";
import { createExpressAICrawlerMiddleware } from "@datafast/ai-crawl";

const app = express();

app.use(
  createExpressAICrawlerMiddleware({
    websiteId: "dfid_your_tracking_id",
  }),
);

The Express middleware attaches a finish listener and sends status code after the response has already been sent to the crawler. It calls next() immediately and does not wait for DataFast before your app continues.

Hono

import { Hono } from "hono";
import { trackAICrawlerResponse } from "@datafast/ai-crawl";

const app = new Hono();

app.use("*", async (c, next) => {
  await next();

  trackAICrawlerResponse(c.req.raw, c.res, c.executionCtx, {
    websiteId: "dfid_your_tracking_id",
  });
});

This is useful on Cloudflare Workers or other Hono runtimes where you can access the final response and execution context.

Generic Request / Response Handler

If your backend gives you a standard Request and Response, use:

import { trackAICrawlerResponse } from "@datafast/ai-crawl";

export async function handleRequest(request, context) {
  const response = await yourAppHandler(request);

  trackAICrawlerResponse(request, response, context, {
    websiteId: "dfid_your_tracking_id",
  });

  return response;
}

If your backend only gives you the request before the app response exists, use request-only tracking:

import { trackAICrawlerRequest } from "@datafast/ai-crawl";

export function middleware(request, context) {
  trackAICrawlerRequest(request, context, {
    websiteId: "dfid_your_tracking_id",
  });

  return next();
}

Request-only tracking is enough to know which page the AI tried to crawl. Response-aware tracking only adds status code when it is easy to get.

Filtering

By default, the package tracks every relevant crawler category it recognizes, including AI answer fetchers, search/indexing crawlers, training crawlers, and other AI crawler traffic.

You can opt out of specific categories:

trackAICrawlerRequest(request, event, {
  websiteId: "dfid_your_tracking_id",

  // Optional category opt-outs. All default to false.
  disableAnswerFetch: true,
  disableSearchCrawlers: true,
  disableTrainingCrawlers: true,
  disableOtherCrawlers: true,
});

For backward compatibility, includeSearchCrawlers: false also disables search/indexing crawlers. Prefer disableSearchCrawlers: true for new installs.

What Gets Tracked

The tracker only sends likely AI/search crawler document requests to DataFast. Normal human traffic, API routes, framework internals, and static assets such as CSS, JavaScript, images, fonts, JSON, XML, maps, PDFs, and media files are ignored before any network request is made.

It stores provider, agent, category, page, hostname, status code when available, source, and confidence.

OpenAI / ChatGPT-User  -> answer_fetch
OpenAI / OAI-SearchBot -> search_index
OpenAI / GPTBot        -> training
Googlebot              -> search_index
Google-NotebookLM      -> answer_fetch
PerplexityBot          -> search_index
ClaudeBot              -> training
Bingbot                -> search_index
Applebot               -> search_index
Bytespider             -> training
CCBot                  -> training
xAI-SearchBot          -> answer_fetch
Grok-DeepSearch        -> answer_fetch
GrokBot / xAI-Bot      -> ai_crawler

License

MIT