@devtune/ai-traffic

Measure which AI crawlers visit your site with a dependency-free sensor that filters at the edge and forwards only matched machine traffic to DevTune.

Install

pnpm add @devtune/ai-traffic

Create a project-scoped server-side ingest key and expose it only to your server runtime:

DEVTUNE_AI_TRAFFIC_INGEST_KEY=dt_ingest_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Quickstarts

Next.js

For Next.js 16, add proxy.ts. For older Next.js projects, use the same body in middleware.ts and export middleware() instead.

// proxy.ts
import { createDevTuneAiTrafficMiddleware } from "@devtune/ai-traffic";
import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";

const trackAiTraffic = createDevTuneAiTrafficMiddleware({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
});

export function proxy(request: NextRequest, event: NextFetchEvent) {
  trackAiTraffic(request, event);

  return NextResponse.next();
}

Express

import express from "express";
import { createDevTuneAiTrafficExpressMiddleware } from "@devtune/ai-traffic/express";

const app = express();

app.use(
  createDevTuneAiTrafficExpressMiddleware({
    ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  }),
);

The middleware calls next() immediately and records the actual response status after Express emits finish.

Node HTTP

import { createServer } from "node:http";

import { createDevTuneAiTrafficNodeHandler } from "@devtune/ai-traffic/node";

const trackAiTraffic = createDevTuneAiTrafficNodeHandler({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
});

createServer((request, response) => {
  trackAiTraffic(request, response);

  response.statusCode = 200;
  response.end("ok");
}).listen(3000);

The hook records the final response.statusCode without blocking or changing the response.

Privacy by Design

The sensor filters requests where your application runs. It uses a cheap user-agent hint before consulting the AI bot registry, and only matched machine hits are eligible to be forwarded. It uses no cookies, no fingerprinting, and no full request logs.

Requests that a page makes for itself are never forwarded. A crawler that renders pages runs their JavaScript, and its browser then fires prefetches, client navigations and other fetch() calls from inside each page. Counting them would report crawls of pages it only fetched in the background; on DevTune's own site, one rendering crawler made three such requests to / for every page it loaded. The sensor recognises them from the fetch metadata the browser attaches, so no configuration is needed:

• Sec-Fetch-Dest is present and is not document, iframe, frame or fencedframe. empty marks a fetch() or XHR from a page, and values such as script or image mark subresources.
• Sec-Purpose or Purpose includes prefetch. This catches speculation-rules prefetches, which browsers send as document fetches, and marked prefetches from clients that send no Sec-Fetch-* headers.
• Next-Router-Prefetch is present. Next.js removes this header before middleware runs, so it only applies where the request reaches the sensor intact, such as the Express and Node adapters or a route handler. The fetch metadata above works in middleware too.

Crawlers that do not run a browser usually send no fetch metadata, and their requests are still recorded. The tradeoff: once an AI browser has loaded a page, its client navigations within your site are page-initiated fetches and are not counted. Its first page load is.

Browsers send fetch metadata only to HTTPS origins. On a site served over plain HTTP, page-initiated requests carry no Sec-Fetch-* headers, so the sensor cannot tell them apart from page loads and records them.

The exception is framework data endpoints, which the sensor skips by URL shape unless the browser marks the request as a page load (Sec-Fetch-Dest: document, iframe, frame or fencedframe). Several frameworks load a route's data from a separate URL during client navigation and link prefetching, and their servers never answer those URLs with a page:

• Next.js Pages Router: a path under /_next/data/ ending in .json, or any request carrying x-nextjs-data. Next.js middleware sees these requests at the page's own path, so there the header is what identifies them.
• Remix: any request with a _data query parameter. Without this rule, /pricing?_data=routes/pricing would be recorded as a crawl of /pricing once the query is removed.
• Remix and React Router single fetch: a path ending in .data, such as /pricing.data or /_root.data.
• React Router lazy route discovery: a path ending in /__manifest with a paths query parameter (React Router 7.10+) or p (earlier), such as /__manifest?paths=/pricing. A /__manifest page without it is still recorded.
• SvelteKit: a path ending in /__data.json or .html__data.json.
• Nuxt: a path ending in /_payload.json, or /_payload.js when JSON payloads are turned off.

The match is exact, so pages such as /data or /blog/__data-tips are still recorded. Other framework requests made from inside a page, such as Next.js App Router fetches, still depend on fetch metadata.

For a matched crawler, DevTune receives only:

• path
• origin plus path, with query parameters removed
• user agent
• response status when the adapter can know it
• timestamp
• the address the request arrived from, used to verify the crawler and then discarded

Request bodies, cookies, unrelated headers, and query-derived identifiers are not sent. An address is sent only for a request whose user agent matches the AI bot registry, so ordinary browser traffic never reaches DevTune at all. Note the limit of that statement: the match is on the user agent, and anyone can send a crawler's user agent, so a person whose client does that would have their address sent. That is the same fact the verification exists to handle, and it is why the address is checked and discarded rather than stored. DevTune checks it against the claimed operator's published networks on receipt and keeps the verdict, not the address. Set trustedProxy: "none" to send no address at all; DevTune then records that traffic as unverified.

Bot Verification

A user agent is a claim, not an identity. Anything on the internet can send GPTBot/1.3, so a sensor that reports user agents alone reports what strangers say about themselves. The address the request arrived from is what makes the claim checkable.

The package only reads that address from a source your deployment declares as trustworthy, because a forgeable one verifies nothing:

| trustedProxy | Source | When to use it | | -------------- | ------------------------------------------------------------------------- | --------------------------------------- | | "vercel" | x-vercel-forwarded-for, which Vercel overwrites | Default when VERCEL=1 | | "connection" | Socket peer supplied by the Express and Node adapters | Default everywhere else | | "cloudflare" | cf-connecting-ip, or cf-connecting-ipv6 in Pseudo IPv4 overwrite mode | Cloudflare fronts the app | | "cloudfront" | Rightmost address in x-forwarded-for, appended by CloudFront | CloudFront connects directly to the app | | A number | x-forwarded-for, counting that many trusted hops from the right | Another reverse proxy fronts the app | | "none" | Nothing | Opt out of verification |

x-forwarded-for is only read when you select "cloudfront" or pass a numeric offset, because any client can append to it. When it is read, entries are counted from the right, so a padded chain cannot push the trusted hops out of range. Values that do not parse as an IPv4 or IPv6 literal are dropped.

No address is sent when the declared proxy's or hop count's header is absent, or when an edge runtime has neither a socket nor a declared proxy. DevTune records that traffic as unverified rather than guessing.

const trackAiTraffic = createDevTuneAiTrafficMiddleware({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  trustedProxy: "cloudflare",
});

Detect Without Sending

The standalone classifiers need no ingest key and never make a network request or push data to DevTune:

import { detectAiCrawler, detectAiReferrer } from "@devtune/ai-traffic";

const crawler = detectAiCrawler(request);
const referrer = detectAiReferrer(request);

Each function also accepts the relevant string directly: a user-agent for detectAiCrawler() or a referrer URL for detectAiReferrer().

detectAiCrawler() returns the matched bot registry entry or null:

type AiCrawlerDetection = {
  uaPattern: string;
  llmPlatform: string;
  botClass: "training_crawler" | "index_bot" | "answer_fetcher" | "acting_agent";
  label: string;
  status?: "active" | "retired";
  asnHints?: number[] | null;
  notes?: string | null;
};

detectAiReferrer() returns the matched referrer registry entry or null:

type AiReferrerDetection = {
  hostname: string;
  llmPlatform: string;
  label: string;
};

Both use bundled registry snapshots by default, so they work offline and in CI. Pass registryEntries as the second argument when you need to classify against a pinned or private registry:

const match = detectAiCrawler(userAgent, {
  registryEntries: myRegistryEntries,
});

Version Reporting

Ingest and registry requests include an x-devtune-sdk-version header with the package version, so DevTune can show an upgrade notice when a newer release is available. It carries no visitor data.

Machine Traffic and Human Referrals

The sensor deliberately measures machine traffic only. DevTune gets human visits from AI products through its GA4 integration, where sessions, engagement, and conversions provide a richer picture than middleware referrer matching. The referrer detector is available for local classification, but the sensor adapters never forward human referral visits.

Production Notes

Batching and Runtime Lifetime

The default client configuration sends up to 10 matched events per unchanged ingest payload, with a 250 ms flush window for low-volume traffic. Set batchSize: 1 and flushIntervalMs: 0 to opt out and send each matched event immediately.

Express and Node servers are long-lived enough to benefit directly from the default. Next.js proxy and middleware pass the delayed send to event.waitUntil() when available, which prevents the response from waiting but may keep the middleware invocation alive for the short flush window. Use the explicit opt-out in short-lived runtimes that cannot reliably preserve delayed work, or when minimizing middleware duration matters more than request coalescing:

const trackAiTraffic = createDevTuneAiTrafficMiddleware({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  batchSize: 1,
  flushIntervalMs: 0,
});

Registry refreshes use waitUntil when provided, with a five-second fetch timeout. They run in the background and do not delay the response. Set waitUntilRegistryRefresh: false to opt out of keeping refresh work alive; short-lived runtimes may then stop it before it completes.

Rate-Limit Retries

A 429 from the ingest endpoint is the one failure the server tells us is temporary, so the batch is redelivered rather than dropped. The client waits for the response's Retry-After — delay-seconds or an HTTP date — and falls back to exponential backoff from 250 ms when the header is absent or unusable. The batch being retried is held intact, so events that arrive during a backoff are sent separately rather than folded into it.

Retries are bounded so a short-lived runtime cannot be held open by a throttled endpoint. The budget belongs to one flush, not to each batch: maxRetryAttempts defaults to 3 and maxRetryDelayMs caps each wait at 5000 ms, including a Retry-After longer than the cap, so a flush adds at most 15 s no matter how deep the queue is. When the budget is spent the current batch is dropped with the usual sampled warning and the drain stops; whatever is behind it stays queued for the next flush, which gets a fresh budget. maxRetryAttempts: 0 disables waiting entirely: a 429 drops the batch on its first rejection, as it did before this behaviour existed. The drain still stops there rather than dropping every batch behind it, so the rest stays queued. Tune both:

const trackAiTraffic = createDevTuneAiTrafficMiddleware({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  maxRetryAttempts: 2,
  maxRetryDelayMs: 2_000,
});

Because backing off lets the queue drain slower than traffic arrives, the queue itself is bounded. maxQueuedEvents defaults to 1000; past it the oldest events are shed with a sampled warning, so a sustained throttle cannot grow memory without limit.

Non-429 responses and network errors are unchanged: the batch is dropped after a single attempt, so a hard outage never keeps the queue alive.

Status Semantics

Express and Node adapters observe the completed response and report its actual status. Next.js proxy and middleware cannot observe the final route status after pass-through, so the convenience middleware omits status instead of guessing a 200.

When a Next.js proxy returns a response directly, use the client and pass the known status:

import { createDevTuneAiTraffic } from "@devtune/ai-traffic";
import type { NextFetchEvent, NextRequest } from "next/server";

const aiTraffic = createDevTuneAiTraffic({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  defaultStatus: null,
});

export function proxy(request: NextRequest, event: NextFetchEvent) {
  const response = new Response("Forbidden", { status: 403 });

  aiTraffic.trackRequest(request, event, response.status);

  return response;
}

Use withDevTuneAiTrafficRoute() where you own a Fetch-compatible route handler and want its exact response status captured automatically. notFoundPathPatterns and statusResolver remain available for applications that can provide additional status knowledge.

Registry Refresh and Fallback

Version 0.5.0 adds Amazon and DuckDuckGo registry support, along with explicit Claude Code, Gemini CLI and OpenCode retry detection. Recognition is a self-reported client claim. Client-run tools can use the user's network, so an address outside a hosted crawler list does not by itself prove an identity mismatch. DevTune performs identity checks on receipt. Generic browser or curl traffic cannot reliably identify an AI client or its model provider.

From 0.5.0, new platform names and traffic classes can arrive through the remote registry without a package upgrade. DevTune handles classification and identity verification on its servers. The SDK keeps validating match patterns before using them.

Clients start with the bundled snapshot. The first request, then the first request after the one-hour cache expires, triggers a background refresh from https://devtune.ai/api/v1/llm-traffic/registry. This includes ordinary browser requests, but the registry GET contains no visitor data and only matching machine requests are sent to ingest. Node, Express and middleware use the same policy. Concurrent requests share one refresh, and ETag avoids downloading an unchanged registry.

A failed refresh keeps the last working registry, or the bundled snapshot before the first successful refresh. Retries wait for the next cache interval. Requests use the currently cached rules while a refresh is in progress, so a newly added bot may be missed until that refresh completes. Create the client or middleware once per process or runtime instance to reuse its cache.

Upgrades can still be needed for address extraction, hosting adapters, security fixes or event-format changes. Standalone detectAiCrawler() uses its bundled snapshot unless you supply registryEntries; it does not refresh automatically.

User-agent matching is a conservative signal. Some agents spoof ordinary browsers or require network-level signals, so reported counts are a floor rather than exact bot truth. Cloudflare-proxied sites should prefer DevTune's Cloudflare pull integration when available because it can combine request, bot-score, and network signals without running middleware.

Forwarded Origins

Express and Node adapters build event URLs from the request protocol and host, preferring the first value in standard X-Forwarded-Proto and X-Forwarded-Host chains. Only accept those headers from a trusted proxy. If they are unavailable or not trustworthy in your deployment, pass a fixed origin:

const trackAiTraffic = createDevTuneAiTrafficNodeHandler({
  ingestKey: process.env.DEVTUNE_AI_TRAFFIC_INGEST_KEY!,
  origin: "https://www.example.com",
});

Failure Behavior

Registry refresh and ingest sends are fire-and-forget and guarded. Network failures may drop telemetry, but they do not block, throw into, or alter the customer response. The ingest endpoint and batched wire format are unchanged.

Documentation

• Install the edge middleware
• Choose the right AI Demand sensors

License

MIT © 2026 PLGeek Ltd.

CloudFront

When CloudFront connects directly to the application running the sensor, add trustedProxy: "cloudfront" to your existing middleware options. No CloudFront origin request policy change is needed.

CloudFront appends the viewer address to X-Forwarded-For, after any entries the viewer supplied. The sensor reads that rightmost entry and ignores earlier ones. See AWS's documented custom-origin behavior.

Only use this setting when requests reach the app through your trusted CloudFront distribution. If an ALB or another proxy between CloudFront and the app appends addresses, use a numeric trustedProxy offset instead: 0 selects the rightmost entry, 1 skips one appended address, and so on. Confirm the actual header chain before selecting an offset. Code executing inside CloudFront itself is a different integration; this setting is for middleware at the origin.