@caprail-dev/agent-pages

Markdown-first pages for AI agents in Next.js: co-locate a page.md next to each page.tsx and serve it via Accept-header content negotiation — plus generated /llms.txt and /llms-full.txt.

A GET/HEAD request is served the markdown twin when its Accept header lists text/markdown with q > 0 and at least as strongly as text/html (a bare */* is not a preference — browsers send it). Everything else gets HTML. A negotiated response Varys on Accept so caches key the two representations separately. A direct .md URL (/terms.md) and the /llms.txt / /llms-full.txt aggregates are served unconditionally.

Install

npm i @caprail-dev/agent-pages   # or bun add / pnpm add / yarn add

Setup

1. Wrap your next.config.js — runs codegen on every next dev / next build (and watches page.md files in dev):

import { withAgentPages } from "@caprail-dev/agent-pages/config";

export default withAgentPages(nextConfig, {
  siteUrl: process.env.BASE_URL ?? "http://localhost:3000"
});

2. Co-locate page.md twins next to the pages you want agent-readable:

src/app/page.tsx        →  src/app/page.md        (served at /index.md)
src/app/terms/page.tsx  →  src/app/terms/page.md  (served at /terms.md)

Optional frontmatter feeds the /llms.txt link list:

---
title: Example
description: One-liner shown in /llms.txt.
---

# Example — the markdown twin

3. Add the middleware — spread the generated AGENT_PAGES bundle straight into the helper. The file is proxy.ts (Next's current naming; middleware.ts on older Next), exporting proxy / middleware respectively:

// proxy.ts
import { createAgentPagesMiddleware } from "@caprail-dev/agent-pages/next";
import { AGENT_PAGES } from "@/app/_agent-pages/manifest";

export const proxy = createAgentPagesMiddleware({ ...AGENT_PAGES });
export const config = {
  matcher: ["/((?!api|_next/static|_next/image|favicon.ico).*)"],
};

The helper serves every doc inline from the manifest — a direct .md URL (/terms.md), an HTML page whose twin an agent wants as markdown (/terms), and the /llms.txt / /llms-full.txt aggregates — and HTML (with Vary) otherwise.

Composing with analytics or other middleware

observe runs developer-provided middleware for its side effects alongside serving. It's any Next-middleware-shaped function — analytics, a logger, a feature-flag tap — and its return value is ignored (the helper owns the response). Each observer receives the request whose pathname is what was actually served (the .md twin on the markdown branch), so a beacon records the served path without you cloning the URL by hand:

import { createAgentPagesMiddleware } from "@caprail-dev/agent-pages/next";
import { createCaprailMiddleware } from "@caprail-dev/analytics/next";
import { AGENT_PAGES } from "@/app/_agent-pages/manifest";

export const proxy = createAgentPagesMiddleware({
  ...AGENT_PAGES,
  observe: createCaprailMiddleware(), // or [collector, logger, …]
});

Composing with a routing middleware (next-intl, …)

observe is for middleware that only watches. A routing middleware like next-intl's createMiddleware(routing) owns the response (locale redirect/rewrite, NEXT_LOCALE cookie), so it goes in the html slot — it runs instead of NextResponse.next() on the HTML branch:

import { createAgentPagesMiddleware } from "@caprail-dev/agent-pages/next";
import { createCaprailMiddleware } from "@caprail-dev/analytics/next";
import createMiddleware from "next-intl/middleware";
import { routing } from "@/i18n/routing";
import { AGENT_PAGES } from "@/app/_agent-pages/manifest";

export const proxy = createAgentPagesMiddleware({
  ...AGENT_PAGES,
  observe: createCaprailMiddleware(), // side effects (analytics, logging)
  html: createMiddleware(routing),    // owns the HTML response (i18n routing)
});

Agent-pages decides first, on the unprefixed path, so AI agents get the one canonical markdown twin while humans fall through to localized HTML — a router that rewrites /terms → /en/terms never hides a twin. The negotiation Vary is appended to the router's response, so its own Vary / Link survive.

Need full control? Compose by hand with the decideAgentPages / serveAgentPages primitives — createAgentPagesMiddleware is just their batteries-included form.

What gets generated

A single file inside your app dir (commit it — deterministic output keeps diffs meaningful and shows exactly what agents will read):

• _agent-pages/manifest.ts — the full doc registry with each page.md's content inlined. Exports AGENT_DOCS (the docs), SITE_URL / LLMS_INTRO, and the AGENT_PAGES bundle ({ docs, llms }) you spread into the middleware. There are no per-page .md route handlers and no separate rewrites file — everything is served inline from this manifest.

The file is marker-headed; the codegen refuses to overwrite a manifest without the marker and removes generated entries whose page.md was deleted.

Options

withAgentPages(nextConfig, {
  siteUrl: "https://example.com",
  appDir: "src/app", // default: src/app, else app
  llms: {
    intro: "# Example\n\n> …", // /llms.txt preamble before "## Docs"
  },
  extraDocs: [
    // Docs whose markdown lives in app code (route stays hand-written);
    // included in /llms.txt + /llms-full.txt via an emitted import.
    {
      mdPath: "/install.md",
      title: "Install guide",
      description: "Agent-readable install instructions.",
      source: { module: "@/lib/install-doc", export: "INSTALL_DOC" },
    },
  ],
});

Notes & limitations

• Dynamic segments ([slug]/page.md) are unsupported in v1 — skipped with a warning.
• Interpolations are not supported in page.md — flatten values to literals and guard them with a test (e.g. assert the file contains your TERMS_VERSION constant).
• The dev watcher uses fs.watch(…, { recursive: true }) (Node ≥ 20 on Linux). Deleting a whole directory may not fire the page.md filter — restart next dev to clean up (a restart with no changes writes nothing).
• next.config.js consumes the compiled dist/config.js — in monorepos, build this package before next dev / next build.
• A blanket /* eslint-disable */ heads every generated file; with reportUnusedDisableDirectives enabled you may see warnings on clean files — harmless.

License

MIT