@crawlguard/nextjs

Next.js integration for CrawlGuard bot detection. One line in your proxy.ts — blocking, challenges, and rate limiting are managed from the dashboard.

Install

npm install @crawlguard/nextjs

Setup

Add your site key to .env:

CRAWLGUARD_SITE_KEY=cg_your_site_key

Create proxy.ts in your project root:

export { crawlGuardProxy as proxy } from '@crawlguard/nextjs'

Done. All traffic decisions (block, challenge, throttle, allow) are controlled from your CrawlGuard Dashboard.

Custom Config

import { createCrawlGuard } from '@crawlguard/nextjs'

export const proxy = createCrawlGuard({
  siteKey: 'cg_your_site_key',
  exclude: ['/_next', '/favicon.ico', '/api/health'],
  challengePath: '/challenge',
})

AI Content Delivery (Markdown for Agents)

Serve clean Markdown to identified AI agents instead of raw HTML — 80% token reduction, fewer arms-race blocks, better representation in AI answers. The transform runs inside your Next.js app so the response keeps coming from your own domain (no CrawlGuard sub-host involved).

// middleware.ts
import { NextResponse, type NextRequest } from "next/server";
import { createAIDelivery } from "@crawlguard/nextjs/ai-delivery";
import { crawlGuardProxy } from "@crawlguard/nextjs";

const ai = createAIDelivery({ siteKey: process.env.CRAWLGUARD_SITE_KEY! });

export async function middleware(req: NextRequest) {
  const aiResponse = await ai.middleware(req);
  if (aiResponse) return aiResponse; // rewrite to /api/__cg/md/... or 403
  return crawlGuardProxy(req);
}

export const config = { matcher: "/((?!_next|api/__cg).*)" };

// app/api/__cg/md/[...path]/route.ts
import { createAIDeliveryRoute } from "@crawlguard/nextjs/ai-delivery/route";

export const GET = createAIDeliveryRoute({
  siteKey: process.env.CRAWLGUARD_SITE_KEY!,
});
export const runtime = "nodejs"; // Turndown needs Node, not Edge.

Configure per-agent policy (Markdown / HTML / Block) on the AI Delivery dashboard page. Policy is fetched once and cached in-process for 60s. Markdown bodies are cached per page for the TTL configured in the dashboard (default 5 min).

Triggers detected by the middleware:

• Accept: text/markdown content negotiation header
• Signature-Agent header (ChatGPT Agent pattern)
• Known AI bot User-Agent (Claude / GPTBot / Bytespider / Perplexity / CCBot / Cursor / OpenCode / …)
• .md URL suffix

Identity verification (anti-spoofing)

Each bot family has four policy modes:

| Mode | Behaviour | |------------------|-----------| | MD (verified) | Serve Markdown only after rDNS + RFC 9421 identity check confirms the request really came from the claimed AI lab. Spoofed claimants fall through to the Default policy. | | MD | Serve Markdown based on UA pattern alone. Faster, but a scraper can impersonate Claude / GPTBot etc. | | HTML | Pass-through unchanged. | | Block | Hard 403. |

Defaults ship with MD (verified) for Claude / OpenAI / Perplexity (the AI labs where spoofing is most attractive) and plain MD for the developer-tool family (Cursor / OpenCode). Verification adds one DNS lookup per IP+UA pair the first time it's seen; results are cached for 5 minutes in-process.

Response Headers

Every request gets observability headers:

• x-crawlguard-score — bot score (0–1)
• x-crawlguard-classification — human, bot, suspicious
• x-crawlguard-action — allow, block, challenge, throttle

License

MIT