next-ai-discovery

Next.js 16 (App Router) helpers to make your site AI-discoverable by serving Markdown variants and advertising them via standard <link rel="alternate" type="text/markdown" ...> tags.

What this package provides:

• A proxy.ts factory to rewrite Markdown requests to a single internal route handler
• A route handler factory to serve per-page Markdown with correct headers (Content-Type, Vary: Accept)
• Metadata helpers to advertise per-page Markdown alternates
• A llms.txt route handler factory (and optional llms-full.txt)

Core behavior (explicit contract)

• Only these requests are rewritten to the internal Markdown endpoint:
  • URLs ending in .md (example: /docs.md)
  • Requests whose Accept header contains text/markdown (example: Accept: text/markdown, text/html;q=0.9)
• Other dot-paths are never rewritten (assets). Example: /logo.png is not rewritten.
• Root mapping is intentional: / is normalized to /index, so the home page Markdown twin is /index.md.

Install

npm i next-ai-discovery
# or
pnpm add next-ai-discovery
# or
bun add next-ai-discovery

Minimal setup (Next.js 16 App Router)

1) Add proxy.ts

Create proxy.ts at the project root:

import { createMarkdownProxy } from 'next-ai-discovery';

export default createMarkdownProxy();

export const config = {
  // Recommended matcher:
  // - runs on "normal" routes (no dot)
  // - also runs on explicit `.md` routes
  matcher: ['/((?!_next/|api/|.*\\..*).*)', '/(.*\\.md)'],
};

Notes:

• Your matcher controls where Next runs the proxy. The proxy itself still has its own rewrite rules.
• Internal endpoint default is DEFAULT_ENDPOINT_PATH = '/__aid/md'.
• The internal endpoint is automatically excluded to avoid rewrite loops.

2) Add the internal Markdown endpoint

Create app/__aid/md/route.ts:

import { createMarkdownRoute } from 'next-ai-discovery';
import type { NextRequest } from 'next/server';

const handler = createMarkdownRoute({
  async getMarkdown(pathname, request: NextRequest) {
    // IMPORTANT: `getMarkdown()` is your policy boundary.
    // Enforce the same auth/policy as your HTML routes.

    // Root is normalized to `/index`.
    if (pathname === '/index') {
      return {
        frontmatter: {
          title: 'home',
          canonical: 'https://example.com/',
        },
        body: '# home\n\nhello.',
      };
    }

    return null;
  },
});

export const GET = handler;
export const HEAD = handler;

Notes:

• Responses include Content-Type: text/markdown; charset=utf-8 and Vary: Accept.
• HEAD is supported; the handler returns the same status/headers but no body.

3) Add llms.txt

/llms.txt is a proposed convention for publishing a short, curated Markdown index to help LLMs and agents understand your site. Reference: https://llmstxt.org/

Create app/llms.txt/route.ts:

import { createLlmsTxtRoute } from 'next-ai-discovery';

export const GET = createLlmsTxtRoute({
  config: {
    site: {
      name: 'Example.com',
      description: 'This site publishes articles about X.',
      url: 'https://example.com',
    },
    sections: [{ title: 'Key sections', items: ['/blog', '/docs', '/about'] }],
    markdown: {
      appendDotMd: true,
      acceptNegotiation: true,
      fullIndexPath: '/llms-full.txt',
    },
  },
});

Optional full variant:

• Create app/llms-full.txt/route.ts with variant: 'full'.
• “full” is not auto-generated by this package; it just selects a variant so you can return a larger inventory if you want.

Advertising llms.txt:

• This package does not auto-inject a global <link ... href="/llms.txt">.
• If you want it, add it in your root app/layout.tsx metadata manually.

What crawlers will see (HTTP examples)

# content negotiation
curl -i -H 'Accept: text/markdown' https://example.com/docs

# explicit .md
curl -i https://example.com/docs.md

Expected headers (both):

• Content-Type: text/markdown; charset=utf-8
• Vary: Accept

Per-page Markdown auto-discovery

To advertise a Markdown twin from HTML using the Next.js Metadata API, use withMarkdownAlternate().

import { withMarkdownAlternate } from 'next-ai-discovery';
import type { Metadata } from 'next';

export async function generateMetadata(): Promise<Metadata> {
  return withMarkdownAlternate({ title: 'Docs' }, '/docs');
}

This emits:

<link rel="alternate" type="text/markdown" href="/docs.md" />

Home page note:

• pathnameToMd('/') yields /index.md (because / is normalized to /index).

Configuration

createMarkdownProxy(options)

• endpointPath (default: /__aid/md)
• enableDotMd (default: true)
• enableAcceptNegotiation (default: true)
• acceptHeader (default: text/markdown)
• exclude(pathname): boolean (optional)
• excludePrefixes (default: ["/_next", "/api"])
• excludeExact (default: ["/robots.txt", "/sitemap.xml"])
• onRewrite({ type: 'accept' | 'dotmd', pathname }) (optional)

Precedence / terminology:

• config.matcher decides which requests execute the proxy at all.
• exclude* decides which requests the proxy will rewrite.

Content negotiation semantics:

• Any Accept header value that contains a comma-separated entry starting with text/markdown triggers Markdown (q-values are ignored).

Dot-path behavior:

• For Accept negotiation, the proxy will not rewrite “asset-like” URLs containing a dot after the last slash (example: /logo.png).
• Explicit .md URLs are always eligible (example: /docs.md).

createMarkdownRoute({ getMarkdown, includeFrontmatter, onServed })

• getMarkdown(pathname, request) returns { body, frontmatter? } or null
• includeFrontmatter (default: true)
• onServed({ pathname, status }) (optional)

Path normalization

Internally, paths are normalized to make matching predictable:

• / -> /index
• Trailing slash is removed (/docs/ -> /docs)
• .md suffix is removed (/docs.md -> /docs)

Compatibility / runtime

• Next.js: App Router only
• Proxy (proxy.ts): Edge runtime (Next.js proxy)
• Route handlers: can be Edge or Node depending on how you configure your Next route file; this library itself does not require Node-only APIs.

Why not HTML -> Markdown conversion?

This package avoids HTML rewriting/conversion on purpose: it is hard to do reliably, and it risks leaking content. Instead, you provide an explicit Markdown representation via getMarkdown().

Auth parity patterns

This package intentionally makes getMarkdown() your policy boundary.

1) Reuse an existing access check

import { createMarkdownRoute } from 'next-ai-discovery';
import type { NextRequest } from 'next/server';

async function canViewPath(request: NextRequest, pathname: string) {
  // Example only. Wire this to your auth/session logic.
  // Return false for protected routes.
  return !pathname.startsWith('/admin');
}

const handler = createMarkdownRoute({
  async getMarkdown(pathname, request) {
    if (!(await canViewPath(request, pathname))) {
      // Recommended default: return null (404) to avoid leaking existence.
      return null;
    }

    return { body: `# ${pathname}\n` };
  },
});

export const GET = handler;
export const HEAD = handler;

2) 404 vs 401/403

If your HTML route would redirect unauthenticated users, decide whether your Markdown variant should:

• Return 404 (recommended for private areas)
• Return 401/403

Because Next.js route handlers return Response, you can also wrap the handler and map outcomes.

Roadmap

• frontmatter format options (yaml vs json vs none)
• llms-full helpers (sitemap-driven inventory)
• observability hooks (onServed, onRewrite) docs + examples
• codemod/snippets for adding withMarkdownAlternate()

License

MIT