@assertkit/bot-analyzer

Pure, dependency-free building blocks for bot detection, posture analysis, and remediation snippet generation. The same engine that powers AssertKit's dashboard and public scanner, packaged for use in your own apps.

Runs anywhere JS does: Node, Bun, Edge runtimes, Cloudflare Workers, browsers.

Install

npm install @assertkit/bot-analyzer

Three things it does

1. Classify an inbound User-Agent

import { classifyUserAgent, decideVerdict } from "@assertkit/bot-analyzer";

const { classifier, botName } = classifyUserAgent(req.headers.get("user-agent"));
// classifier: "ai" | "search" | "social" | "tool" | "crawler" | "human"
// botName:    "GPTBot" | "Googlebot" | ... | null

const verdict = decideVerdict(classifier);
// "allow" | "block" | "observe"
if (verdict === "block") return new Response("blocked", { status: 403 });

2. Analyze a site's bot posture

import { buildPostureReport } from "@assertkit/bot-analyzer";

const [html, robotsTxt] = await Promise.all([
  fetch("https://example.com").then((r) => r.text()),
  fetch("https://example.com/robots.txt").then((r) => r.text()),
]);

const report = buildPostureReport({
  target: "https://example.com",
  reachable: true,
  html,
  robotsTxt,
  xRobotsHeader: null,
});

console.log(report.score);     // 0–100
console.log(report.headline);  // "Strong posture — …"
for (const f of report.familySummaries) {
  console.log(f.classifier, f.status, f.summary);
}

The report carries:

• score — 0–100, weighted toward AI-bot blocking
• headline — one-line summary
• familySummaries — per-classifier verdicts (ai / search / social)
• robots.decisions — every recognized User-agent line and what it does
• metaRobotsDirectives / xRobotsTagDirectives — noai / noimageai etc.
• beaconInstalled — whether the AssertKit beacon is on the page

3. Generate remediation snippets

import { renderSnippet, DEFAULT_POLICY } from "@assertkit/bot-analyzer";

const robotsTxt = renderSnippet("robots-txt", {
  ...DEFAULT_POLICY,
  allowSearch: true,
  allowSocial: true,
  allowAi: false,
  allowTools: false,
});

const middleware = renderSnippet("next-middleware", DEFAULT_POLICY, {
  checkEndpoint: "https://your-app.example/api/check",
  apiKeyPlaceholder: process.env.ASSERTKIT_API_KEY,
});

Available kinds: robots-txt, meta-tag, x-robots-tag, next-middleware, cloudflare-worker, client-side.

Tree-shaking

Sub-path imports skip the barrel so unused modules drop out:

import { classifyUserAgent } from "@assertkit/bot-analyzer/classifier";
import { parseRobotsTxt } from "@assertkit/bot-analyzer/robots";
import { renderSnippet } from "@assertkit/bot-analyzer/snippets";

Determinism

Every public function is pure: no I/O, no clocks (buildPostureReport uses new Date().toISOString() for the report's fetchedAt — supply your own timestamp by overwriting if you need a deterministic test), no global state. Same input → same output. Means you can run the analyzer in benchmarks, snapshot tests, or batch jobs without isolation tricks.

License

MIT © AssertKit