@didyouseo/bot-traffic

See which AI assistants and crawlers — ChatGPT, Claude, Perplexity, Googlebot and ~30 more — visit your site and which pages they read.

AI crawlers don't run JavaScript, so client-side analytics (GA4, Plausible, etc.) never see them. This package runs on your server: it spots bot-looking requests and forwards them to DidYouSEO, where the bot is classified (AI answers, AI search, AI training, search indexing, SEO tools, link previews) and its IP is verified via reverse DNS to catch impostors. Classification happens server-side, so crawler lists stay current without upgrading this package. Your report lives in the DidYouSEO dashboard under Track → Bot Traffic.

• Zero dependencies; works on Node 18+, Vercel Edge, Cloudflare Workers/Pages
• Never blocks or slows a page: reports are non-blocking with a 2s timeout, and all errors are swallowed
• Skips static assets and API routes locally — but keeps crawler-facing files (robots.txt, llms.txt, llms-full.txt, sitemap XMLs, .md content) trackable, since AI crawlers request those first
• Human traffic is never stored — non-bot requests forwarded by the liberal pre-filter are discarded server-side
• Optional status-code capture: crawlers repeatedly requesting a 404 page is a content signal — a page users and agents expect to exist

Setup

1. Sign in at didyouseo.com and open Dashboard → Track → Bot Traffic to get your site's tracking token.
2. npm install @didyouseo/bot-traffic
3. Add one tracking call in your backend (examples below), deploy, and watch the dashboard.

Next.js (Vercel or self-hosted)

Add one line to your middleware.ts (or proxy.ts in newer Next.js):

// middleware.ts
import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";
import { trackBotVisit } from "@didyouseo/bot-traffic";

export function middleware(request: NextRequest, event: NextFetchEvent) {
  trackBotVisit(request, event, { token: process.env.DIDYOUSEO_TOKEN! });

  return NextResponse.next(); // or the rest of your existing middleware
}

export const config = {
  // Keep robots.txt, llms.txt, and sitemap files reachable by this middleware —
  // AI crawlers request those before crawling the rest of your site.
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

Pass event so the report runs in the background via waitUntil — call it, then return your response; no await needed. If you have no middleware yet and want zero wiring, withBotTraffic from @didyouseo/bot-traffic/next creates one for you.

Cloudflare Pages

// functions/_middleware.ts
import { trackBotVisit } from "@didyouseo/bot-traffic";

export async function onRequest(context) {
  trackBotVisit(context.request, context, { token: "YOUR_TRACKING_TOKEN" });
  return context.next();
}

Cloudflare Workers

Workers see the final response, so use the response-aware variant to capture status codes:

import { trackBotResponse } from "@didyouseo/bot-traffic";

export default {
  async fetch(request: Request, env: Env, ctx: ExecutionContext) {
    const response = await handleRequest(request);
    trackBotResponse(request, response, ctx, { token: env.DIDYOUSEO_TOKEN });
    return response;
  },
};

Express / Connect

import { botTraffic } from "@didyouseo/bot-traffic/express";

app.use(botTraffic({ token: process.env.DIDYOUSEO_TOKEN! }));

Calls next() immediately and sends the report from the response's finish listener — status code included, zero added latency.

Hono

import { trackBotResponse } from "@didyouseo/bot-traffic";

app.use("*", async (c, next) => {
  await next();
  trackBotResponse(c.req.raw, c.res, c.executionCtx, { token: "YOUR_TRACKING_TOKEN" });
});

Any other server

If you have a Fetch-API Request, use trackBotVisit(request, contextOrNull, options) — await the result if your runtime has no waitUntil. Otherwise build the payload yourself:

import { reportBotVisit } from "@didyouseo/bot-traffic";

await reportBotVisit(
  { path: "/pricing", userAgent: req.headers["user-agent"], ip: clientIp, status: 200 },
  { token: process.env.DIDYOUSEO_TOKEN! }
);

Or skip the package and POST the JSON directly:

POST https://didyouseo.com/api/bot-traffic
Content-Type: application/json

{ "token": "...", "path": "/pricing", "userAgent": "GPTBot/1.2", "ip": "203.0.113.7", "status": 200 }

Options

| Option | Default | What it does | |---|---|---| | token | — (required) | Your site's tracking token from the dashboard | | endpoint | https://didyouseo.com/api/bot-traffic | Ingest URL override | | timeoutMs | 2000 | Max wait before the report is abandoned | | filter | isLikelyBot | Which user-agents to forward; return true to forward everything | | pathFilter | isTrackablePath | Which paths to forward; default skips assets/API routes but keeps crawler-facing files |

License

MIT