@shipwake/next-ai-crawl

Server-side AI crawler tracking for Next.js applications deployed on Vercel or another Next.js-compatible runtime.

The package observes requests from known AI and search crawlers in Next.js proxy.ts and sends a small, authenticated event to Shipwake after the response path has started. It does not use browser JavaScript, cookies, visitor IDs, or IP addresses.

Install

npm install @shipwake/next-ai-crawl

The package supports Next.js 15.1 through 16.x and requires Node.js 18.18 or newer.

Configure

Set these environment variables in Vercel:

SHIPWAKE_SITE_KEY=your_site_public_key
SHIPWAKE_SERVER_TOKEN=your_private_server_token
SHIPWAKE_AI_CRAWL_ENDPOINT=https://shipwake.dev/api/ai-crawls
SHIPWAKE_AI_CRAWL_TIMEOUT=1000

SHIPWAKE_AI_CRAWL_ENDPOINT is optional when using Shipwake production. The server token must never be exposed to client-side code.

Next.js integration

Create src/proxy.ts at the same level as your app directory:

export { proxy } from "@shipwake/next-ai-crawl/proxy";

This package intentionally owns the crawler catalog, filtering, and delivery path. The application only provides its environment configuration. The proxy is safe to run for every request because non-crawler requests return immediately and static assets are filtered before delivery.

Composing an existing proxy

If the application already has a proxy, compose it with the package:

import { withAiCrawlerProxy } from "@shipwake/next-ai-crawl";
import type { ProxyHandler } from "@shipwake/next-ai-crawl";

const applicationProxy: ProxyHandler = (request, event) => {
  return undefined;
};

export const proxy = withAiCrawlerProxy(applicationProxy);

When composing, keep the package's matcher out of the application and let the package filter requests internally. Next.js only accepts one exported config object per project.

Categories

The package and Shipwake keep these signals separate:

• ai_answers: user-initiated assistant fetches and AI search crawlers.
• indexing: conventional search and AI index crawlers.
• training: crawlers identified as collecting training data.

Crawler identity is based on the User-Agent and can be spoofed. This is crawler observability, not proof that a user saw a citation or converted from an AI answer.

Development

npm install
npm test
npm run lint
npm run build