@talivia/bot-traffic

Server-side classification and best-effort reporting of AI crawler traffic to Talivia Cloud. The package supports standard Fetch APIs, Next.js, Express, Cloudflare Workers, and Cloudflare Pages.

Install

npm install @talivia/bot-traffic

Keep the website-scoped tlv_bot_... token in server-side secrets. Never expose it in browser code.

Next.js proxy or middleware

import { trackBotRequestInBackground } from "@talivia/bot-traffic/next";
import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";

export function proxy(request: NextRequest, event: NextFetchEvent) {
  trackBotRequestInBackground(
    request,
    { token: process.env.TALIVIA_BOT_TOKEN },
    event,
  );
  return NextResponse.next();
}

Pass the runtime context so delivery can finish through waitUntil after the page response continues. Request-only middleware cannot see the final page status, so Talivia stores it as unknown.

The default filter ignores API routes, framework assets, static files, normal browser traffic, and Next.js RSC/prefetch requests. Crawler-facing files such as robots.txt, llms.txt, sitemaps, and Markdown remain trackable. Set trackNextInternals: true only if those internal requests are intentionally part of your analytics.

Response-aware Fetch handlers

import { withBotTracking } from "@talivia/bot-traffic/cloudflare";

export default {
  fetch: withBotTracking(
    async (request: Request, env: Env) => fetch(request),
    (_request, env) => ({ token: env.TALIVIA_BOT_TOKEN }),
  ),
};

The wrapper records the final response status and uses a waitUntil context when one is present in the handler arguments. Pass an options resolver when credentials come from a per-request environment object.

You can also schedule tracking directly:

trackBotRequestInBackground(
  request,
  {
    token: env.TALIVIA_BOT_TOKEN,
    status: response.status,
  },
  context,
);

Express

import express from "express";
import { createExpressBotMiddleware } from "@talivia/bot-traffic/express";

const app = express();
app.use(createExpressBotMiddleware({
  token: process.env.TALIVIA_BOT_TOKEN,
}));

The middleware calls next() immediately and reports the final status after the response finish event.

Trusted source IP verification

Talivia can compare the original crawler IP with provider-published ranges. The package never reads forwarded IP headers automatically because clients can spoof them unless your infrastructure overwrites them.

Configure getSourceIp only for a source your deployment explicitly trusts:

import { getHeader } from "@talivia/bot-traffic";

trackBotRequestInBackground(request, {
  token: process.env.TALIVIA_BOT_TOKEN,
  // Safe only when your reverse proxy overwrites this header.
  getSourceIp: request => getHeader(request.headers, "x-real-ip"),
}, event);

For Cloudflare-only origins, cf-connecting-ip is appropriate when direct origin access is blocked. For Express receiving traffic directly, use request.socket.remoteAddress; when Express trust proxy is correctly configured, you may use request.ip.

The raw IP is used by Talivia Cloud for verification and is not stored. Talivia stores only a keyed hash for operational deduplication. Missing or untrusted IPs remain explicitly unverified.

Reverse proxies and internal hostnames

If the runtime URL contains an internal host such as http://app:3000, provide the public origin:

{
  token: process.env.TALIVIA_BOT_TOKEN,
  publicOrigin: "https://example.com",
}

publicOrigin must be an HTTP(S) origin without credentials, a path, query, or fragment. Talivia validates the resulting hostname against the website that owns the token.

Classification

import {
  classifyBotRequest,
  classifyBotUserAgent,
  shouldTrackBotRequest,
} from "@talivia/bot-traffic";

The registry covers answer fetchers, search/indexing crawlers, model-training crawlers, and other AI-related crawlers from major providers. Local classification is a bandwidth pre-filter and developer utility. Talivia Cloud always classifies the User-Agent again from the canonical server-side registry and ignores client-supplied provider/category claims.

Direct HTTP integration

The JavaScript package is optional. Other server runtimes can send the same minimal event:

POST https://talivia.com/v1/bot-traffic
Authorization: Bearer <website-bot-token>
Content-Type: application/json

{
  "schemaVersion": 1,
  "hostname": "example.com",
  "path": "/docs/getting-started",
  "method": "GET",
  "userAgent": "GPTBot/1.0",
  "status": 200,
  "sourceIp": "203.0.113.10"
}

status and sourceIp are optional. Send sourceIp only when it came from the socket or a trusted proxy. Strip query strings and fragments. Never send cookies, authorization headers, request bodies, visitor identifiers, referrers, or unrelated headers.

Configuration

• token: website-scoped server token; required to schedule delivery.
• publicOrigin: public HTTP(S) origin when the runtime exposes an internal host.
• timeoutMs: delivery timeout; defaults to 1500 ms.
• status: optional final response status from 100 through 599.
• getSourceIp: explicit trusted source-IP resolver; disabled by default.
• methods: defaults to GET and HEAD.
• includePaths / excludePaths: additional string prefixes or regular expressions.
• trackNextInternals: include Next.js RSC/prefetch requests; defaults to false.
• fetch: custom Fetch implementation, primarily for tests.

Queries and fragments are removed from tracked paths. Delivery is best-effort, uses the fixed Talivia Cloud endpoint, and returns structured results instead of throwing into application response handling.

Package contents

The npm release contains only compiled ESM/CommonJS code, TypeScript declarations, this README, the license, and package metadata. Repository source, tests, examples, and local files are not published.

License

MIT