@vercel/agent-readability

Detect AI agents. Serve them markdown. Audit your site against the Agent Readability Spec.

Install

npm install @vercel/agent-readability

Or run the audit CLI directly without installing:

npx @vercel/agent-readability audit https://vercel.com/docs

Quick Start

Add AI agent detection to your Next.js middleware in three lines:

// middleware.ts
import { withAgentReadability } from '@vercel/agent-readability/next'

export default withAgentReadability({
  rewrite: (pathname) => `/api/docs-md${pathname}`,
})

export const config = {
  matcher: ['/docs/:path*'],
}

AI agents hitting /docs/* now receive markdown instead of HTML.

What It Does

Three-layer detection identifies AI agents from HTTP request headers:

1. Known UA patterns. 40+ agents including ClaudeBot, GPTBot, Cursor, and Perplexity.
2. Signature-Agent header. Catches ChatGPT agent (RFC 9421).
3. sec-fetch-mode heuristic. Catches unknown bots that lack browser fingerprints.

The detection optimizes for recall over precision. Serving markdown to a non-AI bot is low cost. Missing an AI agent means a worse experience.

Core API

isAIAgent(request)

Detect AI agents from request headers.

import { isAIAgent } from '@vercel/agent-readability'

const result = isAIAgent(request)
// { detected: true, method: 'ua-match' }

Accepts any object with a headers.get() method: Request, NextRequest, or a custom wrapper.

Returns a discriminated union:

• { detected: true, method: 'ua-match' | 'signature-agent' | 'heuristic' }
• { detected: false, method: null }

When detected is true, method is always non-null. TypeScript narrows automatically.

acceptsMarkdown(request)

Check if the request prefers markdown via the Accept header.

import { acceptsMarkdown } from '@vercel/agent-readability'

if (acceptsMarkdown(request)) {
  return new Response(markdown, {
    headers: { 'Content-Type': 'text/markdown', 'Vary': 'Accept' },
  })
}

shouldServeMarkdown(request)

Combines agent detection and content negotiation into one call.

import { shouldServeMarkdown } from '@vercel/agent-readability'

const { serve, reason } = shouldServeMarkdown(request)
// serve: true, reason: 'agent' | 'accept-header'

generateNotFoundMarkdown(path, options?)

Generates a markdown body for missing pages. Return this with a 200 status (not 404) because agents discard 404 response bodies.

import { generateNotFoundMarkdown } from '@vercel/agent-readability'

const md = generateNotFoundMarkdown('/docs/missing', {
  baseUrl: 'https://example.com',
})
// Return as 200 so agents read the body
return new Response(md, {
  headers: { 'Content-Type': 'text/markdown' },
})

Pattern Exports

AI_AGENT_UA_PATTERNS, TRADITIONAL_BOT_PATTERNS, SIGNATURE_AGENT_DOMAINS, and BOT_LIKE_REGEX are exported for consumers who need to extend or inspect them.

Next.js Adapter

withAgentReadability(options, handler?)

Middleware wrapper that detects AI agents and rewrites matching requests to markdown routes. Compatible with Next.js 14 and 15 (Pages and App Router).

import { withAgentReadability } from '@vercel/agent-readability/next'

export default withAgentReadability({
  docsPrefix: '/docs',
  rewrite: (pathname) => `/en/llms.mdx/${pathname.replace('/docs/', '')}`,
  onDetection: async ({ path, method }) => {
    await trackMdRequest({ path, detectionMethod: method })
  },
})

The onDetection callback runs via event.waitUntil() and does not block the response.

Composing with other middleware

Pass your existing middleware as the second argument:

export default withAgentReadability(
  {
    rewrite: (p) => `/md${p}`,
  },
  (req, event) => i18nMiddleware(req, event),
)

Options

| Option | Type | Default | Description | |--------|------|---------|-------------| | docsPrefix | string | '/docs' | URL prefix to intercept | | rewrite | (pathname: string) => string | required | Maps request path to markdown route | | onDetection | (info) => void \| Promise<void> | - | Analytics callback (runs in waitUntil) |

agentReadabilityMatcher

A pre-built matcher that excludes Next.js internals and static files. Use this for site-wide agent detection instead of scoping to a prefix:

import { withAgentReadability, agentReadabilityMatcher } from '@vercel/agent-readability/next'

export default withAgentReadability({
  docsPrefix: '/',
  rewrite: (pathname) => `/md${pathname}`,
})

export const config = {
  matcher: agentReadabilityMatcher,
}

Audit CLI

Check your site against the Agent Readability Spec.

npx @vercel/agent-readability audit https://sdk.vercel.ai

Runs 16 weighted checks across three categories. Returns a score from 0 to 100. Failed checks include fix suggestions that can be copy-pasted into your coding agent.

CI Integration

- name: Audit agent readability
  run: npx @vercel/agent-readability audit ${{ env.SITE_URL }} --min-score 70 --json

Exits with code 1 if the score is below the threshold.

CLI Options

| Flag | Description | |------|-------------| | --json | Output as JSON | | --min-score <n> | Exit with error if score < n |

Caching

When the same URL serves HTML to browsers and markdown to AI agents, CDN caching must include the Accept header in the cache key.

Set Vary: Accept on all markdown responses. Without it, CDNs may serve cached HTML to agents or cached markdown to browsers.

The withAgentReadability middleware handles detection and rewriting. Your markdown route handler is responsible for setting response headers including Vary: Accept and Content-Type: text/markdown.

Edge Runtime

The core library and Next.js adapter use only Web APIs. They work in Vercel Edge Runtime, Cloudflare Workers, and any standard Request/Response environment.

The CLI runs on Node.js 20+.

License

MIT