astro-markdown-for-agents

An Astro integration that automatically serves Markdown to AI agents — including on Cloudflare's free plan.

Cloudflare's built-in Markdown for Agents feature (launched February 2026) converts HTML pages to Markdown when AI agents request them, significantly reducing token usage. However, it requires a paid Cloudflare plan. This integration replicates that behaviour inside Astro's middleware layer, so any Astro site can benefit regardless of hosting plan.

How It Works

When a request arrives with either:

• an Accept: text/markdown header, or
• a User-Agent that matches a known AI crawler (GPTBot, ClaudeBot, PerplexityBot, etc.)

…the middleware intercepts the normal HTML response, converts it to clean Markdown, and returns it with Content-Type: text/markdown.

All other requests are passed through unchanged.

Installation

npm install astro-markdown-for-agents

Usage

// astro.config.mjs
import { defineConfig } from 'astro/config';
import cloudflare from '@astrojs/cloudflare';
import markdownForAgents from 'astro-markdown-for-agents';

export default defineConfig({
  adapter: cloudflare(),
  integrations: [markdownForAgents()],
});

The integration works with any Astro adapter, not just Cloudflare.

Testing

# Should return text/markdown
curl -H "Accept: text/markdown" http://localhost:4321/

# Should return normal HTML
curl http://localhost:4321/

# Simulate a known AI crawler
curl -H "User-Agent: GPTBot/1.0" http://localhost:4321/

Token Savings

Serving Markdown instead of HTML dramatically reduces the number of tokens an AI model needs to process. For example, a typical documentation page:

| Format | Tokens | |----------|---------| | HTML | ~16,000 | | Markdown | ~3,100 |

That is roughly an 80% reduction.

Supported AI Agents

The following User-Agent strings are detected out of the box:

• anthropic-ai / Claude-Web / ClaudeBot
• GPTBot / ChatGPT-User
• Google-Extended
• cohere-ai
• PerplexityBot
• YouBot
• Applebot-Extended
• Bytespider, CCBot, DataForSeoBot, FacebookBot, facebookexternalhit
• ImagesiftBot, Omgilibot, Omgili, PiplBot, Seekr
• Timpibot, VelenPublicWebCrawler, WebzIO-Extended

Any request sending Accept: text/markdown is also matched, regardless of User-Agent.

Edge Runtime Compatibility

The converter is implemented with pure regex — no DOM, no Node.js APIs. It runs natively in Cloudflare Workers, Deno Deploy, and any other edge runtime.

License

GPL-3.0 — see LICENSE.