@spreelytics/ai-crawl

Track AI crawler traffic — GPTBot, ChatGPT-User, Claude-User, Googlebot, PerplexityBot, Bytespider, and dozens more — hitting your site, and see it in your Spreelytics dashboard's Bot traffic card.

AI crawlers fetch your raw HTML and never run JavaScript, so a browser analytics snippet can't see them. This is a tiny (zero-dependency) server-side middleware: it inspects each incoming request's User-Agent, and when it's a known AI crawler it fire-and-forgets a report to Spreelytics. It never blocks or slows the response, and it ignores normal human traffic, static assets, and API routes.

Install

npm i @spreelytics/ai-crawl

You need your Spreelytics client key (the same client_key you pass to Spreelytics.init(...) in the browser SDK).

Quick start

Express

const express = require('express');
const { createExpressAICrawlerMiddleware } = require('@spreelytics/ai-crawl');

const app = express();
app.use(createExpressAICrawlerMiddleware({ clientKey: process.env.SPREELYTICS_CLIENT_KEY }));

Mount it early. It runs on GET/HEAD only, never touches the response, and reports after the request finishes.

Next.js (middleware — Edge or Node runtime)

// middleware.ts
import { NextResponse } from 'next/server';
import { createAICrawlerMiddleware } from '@spreelytics/ai-crawl';

const trackCrawler = createAICrawlerMiddleware({ clientKey: process.env.SPREELYTICS_CLIENT_KEY! });

export function middleware(request: Request, context: { waitUntil?: (p: Promise<unknown>) => void }) {
  trackCrawler(request, context); // fire-and-forget; uses waitUntil so it never delays the response
  return NextResponse.next();
}

export const config = { matcher: '/((?!_next/|api/).*)' };

Any Fetch-style handler (Cloudflare Workers, Hono, Deno, Bun…)

import { withAICrawlerTracking } from '@spreelytics/ai-crawl';

const handler = async (request: Request) => new Response('…');

export default { fetch: withAICrawlerTracking(handler, { clientKey: SPREELYTICS_CLIENT_KEY }) };

Or call it manually where you have the request:

import { trackAICrawlerRequest } from '@spreelytics/ai-crawl';
await trackAICrawlerRequest(request, { clientKey: SPREELYTICS_CLIENT_KEY });

What gets tracked

Each hit is classified into one of four categories (these are the tabs on the dashboard card):

| Category | Meaning | Examples | |---|---|---| | AI answers (answer_fetch) | An AI assistant fetching your page to answer a user's live query | ChatGPT-User, Claude-User, Perplexity-User | | Indexing (search_index) | Search/answer-engine indexing crawlers | Googlebot, Bingbot, OAI-SearchBot, PerplexityBot | | Training (training) | Model-training data collection | GPTBot, ClaudeBot, Bytespider, CCBot, Applebot | | Other (ai_crawler) | Uncategorized AI crawlers | — |

Requests to static assets, /api, framework internals, and common admin/webhook paths are skipped before any network call; robots.txt, llms.txt, and sitemaps are always allowed.

Configuration

Only clientKey is required.

| Option | Default | Description | |---|---|---| | clientKey | — | Required. Your Spreelytics client key. | | domain | request hostname | Your primary domain. | | publicOrigin | — | Public origin (e.g. https://example.com) to rebuild URLs behind a reverse proxy that exposes an internal host. | | enabled | true | Turn tracking off without removing the middleware. | | disableAnswerFetch | false | Don't report AI-answer crawlers (ChatGPT-User, Claude-User…). | | disableSearchCrawlers | false | Don't report indexing crawlers (Googlebot, Bingbot…). | | disableTrainingCrawlers | false | Don't report training crawlers (GPTBot, ClaudeBot…). | | disableOtherCrawlers | false | Don't report uncategorized AI crawlers. | | methods | ['GET','HEAD'] | HTTP methods to track. | | additionalIgnoredPathPrefixes | [] | Extra path prefixes to skip (added to the defaults). | | additionalIgnoredExtensions | [] | Extra file extensions to skip (added to the defaults). | | shouldTrackPath | — | (url, crawler) => boolean — final say; return false to skip a request. | | getIp | edge headers | Override how the client IP is read. | | timeoutMs | 1500 | Timeout for the background report. | | apiUrl | https://sdk.spreeflo.com/1.0/ai-crawls | Override the Spreelytics ingest endpoint (e.g. a first-party proxy). | | debug | false | Log report failures. |

How it works

1. Reads the request User-Agent and matches it against a maintained AI-crawler directory (classifyAICrawlerUserAgent).
2. Skips anything that isn't a tracked crawler, wrong method, a static asset, or an ignored path.
3. Extracts the client IP from the usual edge headers (cf-connecting-ip, x-real-ip, x-forwarded-for, …).
4. Fire-and-forgets a small JSON report to Spreelytics (keepalive, short timeout) — it never delays or alters your response.

Requirements

Node 18+ (uses the global fetch, Request, Headers). Works in Node and Edge/Worker runtimes.

License

MIT.