@withstoryline/crawler-edge

Drop-in Edge SDK for the Storyline AI Crawler Detector. Capture AI-crawler hits (GPTBot, ClaudeBot, PerplexityBot, Amazonbot, Applebot, GoogleOther, and more) at your site's edge and fire-and-forget them to Storyline. Human traffic is untouched, and ingest never blocks a response.

Install

npm i @withstoryline/crawler-edge

Next.js / Vercel

Re-export the ready-made middleware from your root middleware.ts:

export { middleware, config } from '@withstoryline/crawler-edge/next';

Set one secret in your hosting provider:

STORYLINE_CRAWLER_INGEST_TOKEN=<your tracked site's ingest token>

That's it. The ingest URL is baked in (https://withstoryline.com/api/crawler/ingest); override it only for self-host/testing via STORYLINE_CRAWLER_INGEST_URL.

Custom matcher or merging with existing middleware

Use the factory when you already have middleware logic:

import { createMiddleware } from '@withstoryline/crawler-edge/next';

const detect = createMiddleware({
  ingestToken: process.env.STORYLINE_CRAWLER_INGEST_TOKEN!,
});

export function middleware(request, event) {
  detect(request, event); // schedules ingest via event.waitUntil
  return myExistingResponse(request);
}

Behaviour guarantees

• UA match only — non-bot traffic is ignored; no cookies, no client JS.
• Fire-and-forget — ingest failures are swallowed; page loads never block.
• Per-site token — each tracked site has its own token; rotate from /crawler.
• No heavy deps — the bundle ships matcher-only signatures and no zod.