@devtune/ai-traffic

Measure which AI crawlers visit your site with a dependency-free sensor that filters at the edge and forwards only matched machine traffic to DevTune.

Install

pnpm add @devtune/ai-traffic

Create a project-scoped server-side ingest key and expose it only to your server runtime:

DEVTUNE_AI_TRAFFIC_INGEST_KEY=dt_ingest_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Quickstarts

Next.js

For Next.js 16, add proxy.ts. For older Next.js projects, use the same body in middleware.ts and export middleware() instead.

// proxy.ts
import { createDevTuneAiTrafficMiddleware } from "@devtune/ai-traffic";
import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";

const trackAiTraffic = createDevTuneAiTrafficMiddleware({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
});

export function proxy(request: NextRequest, event: NextFetchEvent) {
  trackAiTraffic(request, event);

  return NextResponse.next();
}

Express

import express from "express";
import { createDevTuneAiTrafficExpressMiddleware } from "@devtune/ai-traffic/express";

const app = express();

app.use(
  createDevTuneAiTrafficExpressMiddleware({
    ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  }),
);

The middleware calls next() immediately and records the actual response status after Express emits finish.

Node HTTP

import { createServer } from "node:http";

import { createDevTuneAiTrafficNodeHandler } from "@devtune/ai-traffic/node";

const trackAiTraffic = createDevTuneAiTrafficNodeHandler({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
});

createServer((request, response) => {
  trackAiTraffic(request, response);

  response.statusCode = 200;
  response.end("ok");
}).listen(3000);

The hook records the final response.statusCode without blocking or changing the response.

Privacy by Design

The sensor filters requests where your application runs. It uses a cheap user-agent hint before consulting the AI bot registry, and only matched machine hits are eligible to be forwarded. It uses no cookies, no fingerprinting, and no full request logs.

For a matched crawler, DevTune receives only:

• path
• origin plus path, with query parameters removed
• user agent
• response status when the adapter can know it
• timestamp

Request bodies, cookies, IP addresses, unrelated headers, and query-derived identifiers are not sent.

Detect Without Sending

The standalone classifiers need no ingest key and never make a network request or push data to DevTune:

import { detectAiCrawler, detectAiReferrer } from "@devtune/ai-traffic";

const crawler = detectAiCrawler(request);
const referrer = detectAiReferrer(request);

Each function also accepts the relevant string directly: a user-agent for detectAiCrawler() or a referrer URL for detectAiReferrer().

detectAiCrawler() returns the matched bot registry entry or null:

type AiCrawlerDetection = {
  uaPattern: string;
  llmPlatform: string;
  botClass: "training_crawler" | "index_bot" | "answer_fetcher" | "acting_agent";
  label: string;
  status?: "active" | "retired";
  asnHints?: number[] | null;
  notes?: string | null;
};

detectAiReferrer() returns the matched referrer registry entry or null:

type AiReferrerDetection = {
  hostname: string;
  llmPlatform: string;
  label: string;
};

Both use bundled registry snapshots by default, so they work offline and in CI. Pass registryEntries as the second argument when you need to classify against a pinned or private registry:

const match = detectAiCrawler(userAgent, {
  registryEntries: myRegistryEntries,
});

Machine Traffic and Human Referrals

The sensor deliberately measures machine traffic only. DevTune gets human visits from AI products through its GA4 integration, where sessions, engagement, and conversions provide a richer picture than middleware referrer matching. The referrer detector is available for local classification, but the sensor adapters never forward human referral visits.

Production Notes

Batching and Runtime Lifetime

The default client configuration sends up to 10 matched events per unchanged ingest payload, with a 250 ms flush window for low-volume traffic. Set batchSize: 1 and flushIntervalMs: 0 to opt out and send each matched event immediately.

Express and Node servers are long-lived enough to benefit directly from the default. Next.js proxy and middleware pass the delayed send to event.waitUntil() when available, which prevents the response from waiting but may keep the middleware invocation alive for the short flush window. Use the explicit opt-out in short-lived runtimes that cannot reliably preserve delayed work, or when minimizing middleware duration matters more than request coalescing:

const trackAiTraffic = createDevTuneAiTrafficMiddleware({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  batchSize: 1,
  flushIntervalMs: 0,
});

The middleware helper leaves registry refresh work outside waitUntil by default. Ordinary browser user agents are rejected before refresh or ingest work is scheduled.

Rate-Limit Retries

A 429 from the ingest endpoint is the one failure the server tells us is temporary, so the batch is redelivered rather than dropped. The client waits for the response's Retry-After — delay-seconds or an HTTP date — and falls back to exponential backoff from 250 ms when the header is absent or unusable. The batch being retried is held intact, so events that arrive during a backoff are sent separately rather than folded into it.

Retries are bounded so a short-lived runtime cannot be held open by a throttled endpoint. The budget belongs to one flush, not to each batch: maxRetryAttempts defaults to 3 and maxRetryDelayMs caps each wait at 5000 ms, including a Retry-After longer than the cap, so a flush adds at most 15 s no matter how deep the queue is. When the budget is spent the current batch is dropped with the usual sampled warning and the drain stops; whatever is behind it stays queued for the next flush, which gets a fresh budget. maxRetryAttempts: 0 disables waiting entirely: a 429 drops the batch on its first rejection, as it did before this behaviour existed. The drain still stops there rather than dropping every batch behind it, so the rest stays queued. Tune both:

const trackAiTraffic = createDevTuneAiTrafficMiddleware({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  maxRetryAttempts: 2,
  maxRetryDelayMs: 2_000,
});

Because backing off lets the queue drain slower than traffic arrives, the queue itself is bounded. maxQueuedEvents defaults to 1000; past it the oldest events are shed with a sampled warning, so a sustained throttle cannot grow memory without limit.

Non-429 responses and network errors are unchanged: the batch is dropped after a single attempt, so a hard outage never keeps the queue alive.

Status Semantics

Express and Node adapters observe the completed response and report its actual status. Next.js proxy and middleware cannot observe the final route status after pass-through, so the convenience middleware omits status instead of guessing a 200.

When a Next.js proxy returns a response directly, use the client and pass the known status:

import { createDevTuneAiTraffic } from "@devtune/ai-traffic";
import type { NextFetchEvent, NextRequest } from "next/server";

const aiTraffic = createDevTuneAiTraffic({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  defaultStatus: null,
});

export function proxy(request: NextRequest, event: NextFetchEvent) {
  const response = new Response("Forbidden", { status: 403 });

  aiTraffic.trackRequest(request, event, response.status);

  return response;
}

Use withDevTuneAiTrafficRoute() where you own a Fetch-compatible route handler and want its exact response status captured automatically. notFoundPathPatterns and statusResolver remain available for applications that can provide additional status knowledge.

Registry Refresh and Fallback

Clients start with the bundled AI bot snapshot, refresh from https://devtune.ai/api/v1/llm-traffic/registry, cache active entries for about an hour, and use ETag revalidation. Failed refreshes are guarded and sampled; they never fail the application response, and the bundled snapshot remains usable.

User-agent matching is a conservative signal. Some agents spoof ordinary browsers or require network-level signals, so reported counts are a floor rather than exact bot truth. Cloudflare-proxied sites should prefer DevTune's Cloudflare pull integration when available because it can combine request, bot-score, and network signals without running middleware.

Forwarded Origins

Express and Node adapters build event URLs from the request protocol and host, preferring the first value in standard X-Forwarded-Proto and X-Forwarded-Host chains. Only accept those headers from a trusted proxy. If they are unavailable or not trustworthy in your deployment, pass a fixed origin:

const trackAiTraffic = createDevTuneAiTrafficNodeHandler({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  origin: "https://www.example.com",
});

Failure Behavior

Registry refresh and ingest sends are fire-and-forget and guarded. Network failures may drop telemetry, but they do not block, throw into, or alter the customer response. The ingest endpoint and batched wire format are unchanged.