@rebilder/agent-detect

Agent identification from request headers. Given a request's headers (and optionally its URL), detect() classifies who is asking — an AI agent, a human in a browser, a search crawler, or a protocol client — and, where possible, which platform it is. The package also ships verifyWebBotAuth() — cryptographic Web Bot Auth verification as a separate async step (see § Verification below).

Pure compute. Zero runtime dependencies. Framework-agnostic. No network calls. detect() does no crypto and is safe on the edge hot path (well under the 50ms p95 budget; it is a handful of string scans); verifyWebBotAuth() is pure crypto (ambient WebCrypto) over an injected key registry — still zero network, zero deps.

What this package does — and does not — do

• detect() does: parse and classify. Accept negotiation, Web Bot Auth header parsing (Signature-Agent, Signature, Signature-Input), protocol-route matching, User-Agent heuristics. It is synchronous and NEVER verifies: its verified field is always false.
• verifyWebBotAuth() does: cryptographically verify a Web Bot Auth signature chain (RFC 9421 Ed25519 message signatures) against a caller-injected key registry — a distinct async step for the paths that need cryptographic identity (preference payloads, protocol transactions).
• Neither does: network I/O. There is no key-directory fetching anywhere in this package — the registry is data the operator injects (see § Verification for why it ships empty). Do not gate anything security-sensitive on an unverified platform claim.

API

import { detect } from '@rebilder/agent-detect'

const result = detect({
  headers: req.headers,          // case-insensitive lookup; string[] values OK
  url: req.url,                  // pathname used for protocol-route hits
  method: req.method,            // accepted for contract stability; unused today
})
// {
//   kind: 'agent' | 'human' | 'crawler' | 'protocol',
//   platform: 'claude-code' | 'opencode' | 'claude' | 'chatgpt' | 'gemini'
//           | 'perplexity' | 'googlebot' | 'bingbot' | 'microsoft-preview'
//           | 'unknown' | null,
//   verified: boolean,          // ALWAYS false from detect() — verification is the separate
//                               // async verifyWebBotAuth() step (§ Verification)
//   acceptsMarkdown: boolean,   // Accept explicitly lists text/markdown with q > 0
//   signals: string[],          // ordered checks that fired, e.g. ['accept:text/markdown', 'ua:claude-code']
//   confidence: 'high' | 'medium' | 'low',
// }

acceptsMarkdownHeader(accept)

The Accept predicate detect() itself uses, over a raw header string:

import { acceptsMarkdownHeader } from '@rebilder/agent-detect'

acceptsMarkdownHeader('text/markdown')          // true
acceptsMarkdownHeader('text/html,text/markdown') // true
acceptsMarkdownHeader('text/markdown;q=0')      // false — a refusal
acceptsMarkdownHeader('*/*')                    // false — see below
acceptsMarkdownHeader(null)                     // false

A wildcard range is not a request for markdown, and that is deliberate rather than an oversight in the parser. Under ordinary HTTP content negotiation */* accepts everything; this package reads only an EXPLICIT range, because the interesting fact about agent traffic is precisely how few requests name the format they want.

It is exported so that anything REPORTING on negotiation asks the question with the same code that ANSWERED it. The Console's content-negotiation panel counts how many agent requests asked for markdown; computing that with a lookalike (a SQL LIKE, a second regex) would let the report drift from the serving decision and describe a gateway that does not exist. Also re-exported from @rebilder/gateway, so a consumer needs only the public package.

Verification (Web Bot Auth) — verifyWebBotAuth()

Agent verification, before accepting preference payloads or protocol transactions. A separate async step — detect() stays synchronous, parse-only, and unchanged:

import { verifyWebBotAuth, KNOWN_AGENT_DIRECTORY, type AgentKeyRegistry } from '@rebilder/agent-detect'

const result = await verifyWebBotAuth(
  { headers: req.headers, url: req.url, method: req.method }, // same DetectInput shape
  { keys: registry },                                          // INJECTED keys — no network, ever
)
// { verified: true,  platform: 'chatgpt', keyid: '…' }
// { verified: false, reason: 'no-signature' | 'unknown-agent' | 'unknown-key' | 'expired'
//                          | 'created-in-future' | 'bad-signature' | 'malformed' | 'unsupported-alg',
//   keyid?: '…' }   // keyid included once parsing got that far

What it checks (Web Bot Auth profile over RFC 9421):

• Signature + Signature-Input parse as structured fields; the first member with a matching signature entry is used → else no-signature / malformed
• covered components include @authority and signature-agent (anti-replay: the signature is pinned to this host and this identity) → else malformed
• alg, when present, is ed25519 → else unsupported-alg
• Signature-Agent origin resolves to a registry entry → else unknown-agent; keyid resolves to one of that entry's keys → else unknown-key
• created / expires are present, ordered, and within ±300s clock skew (CLOCK_SKEW_SECONDS) of options.now → else created-in-future / expired
• the Ed25519 signature verifies (WebCrypto subtle) over the RFC 9421 signature base, whose "@signature-params" line reuses the agent's exact Signature-Input serialization → else bad-signature

Constant behavior on hostile input: it never throws. Every malformed, truncated, or adversarial header resolves to { verified: false, reason }.

The key registry — honesty first

type AgentKeyRegistry = {
  [origin: string]: {           // Signature-Agent origin, e.g. 'https://chatgpt.com' (bare hosts accepted)
    platform: AgentPlatform
    keys: { keyid: string; alg: 'ed25519'; publicKeyJwk: { kty: 'OKP'; crv: 'Ed25519'; x: string } }[]
  }
}

KNOWN_AGENT_DIRECTORY (src/directory.ts) is the built-in registry and it ships empty — deliberately. We do not embed "known" production keys for OpenAI/Anthropic/Perplexity etc., because we have not fetched and pinned them from the platforms' published key directories, and invented key material would make verification silently meaningless. With the directory empty, every signed request resolves to unknown-agent — the honest answer when no trusted key is on file.

Operators populate the registry: platforms publish signing keys at well-known HTTP message signature directories (e.g. /.well-known/http-message-signatures-directory). Fetch those offline — never on the request hot path — pin the Ed25519 public JWKs into your registry (any AgentKeyRegistry object works), and refresh on key rotation. A refresh script that snapshots published directories is future work.

Tests use self-generated Ed25519 keypairs (WebCrypto, generated at test time) as vectors — pass/fail paths, tampered payloads, expiry windows, wrong keys, unknown keyids are all asserted deterministically against a fixed clock (tests/verify.test.ts).

Detection order (cheapest first)

Every check that fires is recorded in signals; the highest-precedence one classifies.

1. Accept: text/markdown (explicit in the list, q-values honored, q=0 excluded) → agent, acceptsMarkdown: true, high confidence. Claude Code and OpenCode send this. Platform attribution comes from Signature-Agent or UA if present, else 'unknown'.
2. Web Bot Auth headers — Signature-Agent (e.g. "https://chatgpt.com" → chatgpt) → agent, high confidence; a bare Signature + Signature-Input pair → agent/unknown, medium confidence. Parsed only by detect(); verified stays false here — cryptographic verification is the separate async verifyWebBotAuth() step (§ Verification).
3. Protocol route on the URL pathname — /.well-known/ucp, /.well-known/acp, /mcp, /acp/* (each also matching subpaths) → protocol.
4. UA heuristics — fallback only, capped at medium confidence. Known agent UAs: claude-code, opencode, ChatGPT-User, OAI-SearchBot, GPTBot, PerplexityBot, Perplexity-User, Gemini/Google-Extended, anthropic-ai/claude-web (plus ClaudeBot/Claude-User). Crawler UAs: Googlebot → crawler/googlebot; bingbot and adidxbot → crawler/bingbot; MicrosoftPreview → crawler/microsoft-preview.
5. Default → human. High confidence when a browserish Accept (text/html) or UA (Mozilla/) is present; low otherwise (empty or garbage headers).

One deliberate exception to the ordering (cloaking guardrail): a known crawler UA always classifies as crawler, even if other signals fired. Googlebot must receive canonical HTML, never markdown — even if a request claiming to be Googlebot sends Accept: text/markdown. (acceptsMarkdown still reports the header fact; kind: 'crawler' pins the serving path.)

Microsoft, Bing, and Copilot

There is no copilot platform, and that is the accurate answer rather than a gap. Microsoft Copilot answers from Bing's index and has no user agent of its own; its fetches arrive as bingbot, which is why Microsoft's own guidance for being reachable by Copilot is "allow bingbot". A copilot entry keyed on a UA token would be a rule that never fires, backed by a fixture nobody ever observed — and this corpus is worth something only because every sample in it is real.

Copilot does exist as a referrer platform in @rebilder/events (copilot.microsoft.com → copilot), because an AI-referred human genuinely does arrive from it. So a merchant sees Copilot on the arrivals side and bingbot on the fetch side, and neither number is invented.

Three Microsoft tokens are detected, all as crawler:

| Token | Platform | What it is | |---|---|---| | bingbot | bingbot | Bing's crawler — and the Copilot path | | adidxbot | bingbot | Bing Ads landing-page quality crawler | | MicrosoftPreview | microsoft-preview | Link unfurling in Microsoft products (Teams, Outlook) |

All three sit in the crawler table rather than the agent table, and the asymmetry is deliberate: misclassifying a crawler as an agent serves it markdown, which is the cloaking fact pattern; misclassifying an agent as a crawler only costs that caller a markdown response it can still request with Accept. adidxbot audits ad landing pages for policy compliance, so serving it anything but the canonical page is the worst outcome available here.

MicrosoftPreview gets its own platform rather than folding into bingbot because it is not Bing and not an assistant — attributing it to bingbot would inflate a merchant's Bing figure with traffic that never touched the index.

The UA-heuristics caveat (Hard Rule 3)

UA sniffing alone never justifies substantive content differences — different prices, claims, or availability by requester is cloaking. UA-derived classifications are therefore capped at medium confidence, and downstream consumers (the gateway) may use them only to pick a format transformation of the same substance (e.g. markdown rendering of the canonical page). Anything substantive — preference-payload personalization in particular — requires cryptographic identity: a verified: true result from verifyWebBotAuth() (§ Verification), never a parsed claim.

Fixture discipline (the moat)

fixtures/*.json is a corpus of real observed header samples, one file per agent:

{
  "name": "claude-code",
  "description": "…",
  "samples": [
    { "headers": { "...": "..." }, "url": "…", "expected": { "kind": "agent", "platform": "claude-code", "acceptsMarkdown": true } }
  ]
}

• Every newly observed agent gets a fixture. New headers, new signatures, new UA strings — capture them here first, then teach the detector. Treat this corpus like data, not test scaffolding.
• The test suite (tests/fixtures.test.ts) automatically loads every fixture file and asserts every sample, so adding a fixture is adding a regression test.
• Fixtures grow with observation; the current set was seeded from known agent header shapes and will be replaced/extended with captures from live traffic. Where a sample is transcribed from a platform's published crawler documentation rather than captured from our own traffic, the fixture's description says so — bingbot.json is the current example. A UA string nobody has published and nobody has observed does not go in this directory, however plausible it looks.

Scripts

pnpm --filter @rebilder/agent-detect lint        # eslint (flat config)
pnpm --filter @rebilder/agent-detect typecheck   # tsc --noEmit (strict)
pnpm --filter @rebilder/agent-detect test        # vitest (fixture-driven + unit)

Roadmap

• Classification is parse-only and synchronous; detect()'s verified is always false.
• Verification is verifyWebBotAuth() — pure Ed25519 crypto over an injected registry (§ Verification). Not yet shipped: key-directory refresh tooling (offline snapshot script) and verified-bot IP corroboration for crawlers.