@seosorted/ai-crawl

One-line, server-side AI-crawler & bot traffic tracking. See exactly what ChatGPT, Claude, Perplexity, Google and others fetch on your site — including the pages they request that 404 (a signal you should build them).

It runs server-side because AI crawlers request raw HTML and skip frontend JavaScript, so a normal analytics tag never sees them. The tracker ignores static assets, framework internals and human browser traffic, and sends a tiny event only for detected bots. It's best-effort and won't slow your site down.

Install

npm install @seosorted/ai-crawl

Grab your websiteId from Dashboard → Bot Traffic → Enable tracking.

Integrate (pick your stack)

Next.js / Vercel — proxy.ts

import { NextResponse, type NextRequest, type NextFetchEvent } from "next/server";
import { trackAICrawlerRequest } from "@seosorted/ai-crawl";

export function proxy(request: NextRequest, event: NextFetchEvent) {
  trackAICrawlerRequest(request, event, { websiteId: "sst_xxxxxxxx" });
  return NextResponse.next();
}

export const config = {
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};

Cloudflare Workers

import { withAICrawlerTracking } from "@seosorted/ai-crawl";

export default {
  fetch: withAICrawlerTracking(
    async (request, env, ctx) => fetch(request),
    { websiteId: "sst_xxxxxxxx" },
  ),
};

Cloudflare Pages — functions/_middleware.ts

import { trackAICrawlerRequest } from "@seosorted/ai-crawl";

export async function onRequest(context) {
  trackAICrawlerRequest(context.request, context, { websiteId: "sst_xxxxxxxx" });
  return context.next();
}

Express

import express from "express";
import { createExpressAICrawlerMiddleware } from "@seosorted/ai-crawl";

const app = express();
app.use(createExpressAICrawlerMiddleware({ websiteId: "sst_xxxxxxxx" }));

Hono

import { Hono } from "hono";
import { trackAICrawlerResponse } from "@seosorted/ai-crawl";

const app = new Hono();
app.use("*", async (c, next) => {
  await next();
  trackAICrawlerResponse(c.req.raw, c.res, c.executionCtx, { websiteId: "sst_xxxxxxxx" });
});

Deploy, and bot traffic starts appearing in your dashboard.

No-code / hosted platforms

WordPress (self-hosted)

WordPress runs PHP on every request, so a tiny plugin is the equivalent of the middleware above — and it captures 404s natively. Save this as wp-content/plugins/seosorted-ai-crawl/seosorted-ai-crawl.php, set your websiteId, and activate it under Plugins.

<?php
/* Plugin Name: SEOSorted AI Crawl */
if (!defined('ABSPATH')) exit;

define('SEOSORTED_WEBSITE_ID', 'sst_xxxxxxxx'); // ← your website id

add_action('shutdown', function () {
  $ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
  if ($ua === '' || !preg_match(
    '/bot|crawl|spider|slurp|gptbot|chatgpt|oai-|claude|anthropic|perplexity|applebot|amazonbot|bytespider|ccbot|baidu|grok|xai|duckassist|google-extended|meta-external|facebookexternal/i',
    $ua
  )) return;

  wp_remote_post('https://app.seosorted.ai/api/v1/track/' . SEOSORTED_WEBSITE_ID, [
    'blocking' => false, // fire-and-forget — never slows the page
    'timeout'  => 1,
    'headers'  => ['Content-Type' => 'application/json'],
    'body'     => wp_json_encode(['hits' => [[
      'path'       => $_SERVER['REQUEST_URI'] ?? '/',
      'method'     => $_SERVER['REQUEST_METHOD'] ?? 'GET',
      'statusCode' => http_response_code() ?: 200, // native 404 capture
      'userAgent'  => $ua,
      'ip'         => $_SERVER['HTTP_CF_CONNECTING_IP']
                      ?? explode(',', $_SERVER['HTTP_X_FORWARDED_FOR'] ?? '')[0]
                      ?: ($_SERVER['REMOTE_ADDR'] ?? null),
      'referer'    => $_SERVER['HTTP_REFERER'] ?? null,
    ]]]),
  ]);
});

Behind Cloudflare (any origin, no npm)

If your domain is proxied through Cloudflare, add a Snippet (Rules → Snippets) or a Worker. Captures 100% of traffic — including 404s — whatever runs at the origin.

export default {
  async fetch(request, env, ctx) {
    const res = await fetch(request);
    const ua = request.headers.get("user-agent") || "";
    const path = new URL(request.url).pathname;
    const isBot = /bot|crawl|spider|gptbot|chatgpt|oai-|claude|anthropic|perplexity|applebot|amazonbot|bytespider|ccbot|baidu|grok|duckassist|google-extended|meta-external/i.test(ua);
    const isAsset = /\.(css|js|png|jpe?g|gif|svg|webp|ico|woff2?|map)$/i.test(path);
    if (isBot && !isAsset) {
      ctx.waitUntil(fetch("https://app.seosorted.ai/api/v1/track/sst_xxxxxxxx", {
        method: "POST",
        headers: { "content-type": "application/json" },
        body: JSON.stringify({ hits: [{
          path, method: request.method, statusCode: res.status, userAgent: ua,
          ip: request.headers.get("cf-connecting-ip"),
          referer: request.headers.get("referer"), occurredAt: Date.now(),
        }] }),
      }));
    }
    return res;
  },
};

Shopify, Wix, Squarespace

These run your site entirely on their own servers and don't allow third-party server-side code. Because AI crawlers skip JavaScript, there is no reliable client-side workaround — bot tracking isn't supported on them today. If you can front the domain with Cloudflare (most custom domains, but not Shopify's storefront), use the Snippet above instead.

Direct API

Integrate from any language by POSTing hits yourself:

POST https://app.seosorted.ai/api/v1/track/<websiteId>
Content-Type: application/json

{ "hits": [{
  "path": "/pricing",          // required
  "method": "GET",             // optional (default "GET")
  "statusCode": 404,           // optional; omit at request time
  "userAgent": "GPTBot/1.1",   // optional but recommended
  "ip": "203.0.113.9",         // optional; needed to verify authenticity
  "referer": "https://…",      // optional
  "occurredAt": 1719000000000  // optional (ms epoch), default = ingest time
}] }

Batch 1–500 hits per request; returns 202 { accepted, jobId }. Send the visitor's IP (not your server's) — verification runs on it.

Platform support

| Platform | Method | Captures 404s? | |---|---|---| | Next.js / Vercel | trackAICrawlerRequest | ⚠️ request-time only | | Cloudflare Workers | withAICrawlerTracking / Snippet | ✅ | | Cloudflare Pages | trackAICrawlerRequest | ⚠️ request-time only | | Express / Node | createExpressAICrawlerMiddleware | ✅ | | Hono | trackAICrawlerResponse | ✅ | | WordPress (self-hosted) | plugin snippet | ✅ | | Behind Cloudflare (any origin) | Snippet / Worker | ✅ | | Shopify / Wix / Squarespace | not supported (platform limitation) | — |

Supported crawlers

Detected out of the box and grouped in your dashboard by what they're doing — each shown with its provider's logo. New crawlers are added regularly.

🟣 AI assistants — fetching to answer a user right now

• OpenAI — ChatGPT-User
• Anthropic — Claude-User
• Google — Google-NotebookLM, Google-Read-Aloud, Google-Agent
• Microsoft — Copilot
• Perplexity — Perplexity-User
• xAI — Grok-DeepSearch
• Meta — meta-externalfetcher
• Amazon — Amzn-User
• DuckDuckGo — DuckAssistBot

🔵 Search indexing — discovering & refreshing content

• OpenAI — OAI-SearchBot
• Anthropic — Claude-SearchBot
• Google — Googlebot
• Microsoft — bingbot, BingPreview
• Perplexity — PerplexityBot
• Apple — Applebot
• xAI — xAI-SearchBot
• Meta — meta-webindexer
• ByteDance — TikTokSpider
• Baidu — Baiduspider
• Moonshot — Kimi-SearchBot
• You.com — YouBot

🟢 AI training — crawling for model data

• OpenAI — GPTBot
• Anthropic — ClaudeBot, anthropic-ai
• Google — Google-Extended, Google-CloudVertexBot
• Amazon — Amazonbot
• Apple — Applebot-Extended
• Common Crawl — CCBot
• ByteDance — Bytespider
• Meta — meta-externalagent, FacebookBot
• xAI — xAI-Web-Crawler
• DeepSeek — DeepSeekBot
• Alibaba — QwenBot
• Baidu — ERNIEBot
• Cohere — cohere-ai
• Allen AI — AI2Bot

🟠 SEO tools

• AhrefsBot, SemrushBot, DataForSeoBot, DotBot, MJ12bot

⚪ Other

• OpenAI OAI-AdsBot, Google GoogleOther, xAI GrokBot, Meta facebookexternalhit, Alibaba TongyiBot, Baidu YiyanBot, plus a generic catch-all for any User-Agent containing bot / crawler / spider.

Notes

• 404 tracking: response-aware integrations (Workers, Express, Hono, WordPress) know the real status code, so bot 404s show up. Request-time hooks (Next.js / Pages edge middleware) run before the response, so they report requests without a status.
• Options: { websiteId, endpoint? } — set endpoint to point at a self-hosted / local ingest host (defaults to production).
• Verification: User-Agents are spoofable, so each hit is confirmed against the source IP (reverse-DNS forward-confirm, or the vendor's published IP ranges). Hits that fake a known bot UA are flagged as spoofed.
• Privacy: IPs are used only to verify authenticity, then truncated before storage — full IPs are never persisted.