astro-mark-don
v0.1.2
Published
Astro integration that generates a .md version of every page at build time, optimized for LLM crawlers.
Downloads
203
Maintainers
paultursuruReadme
astro-mark-don
An Astro integration that generates a .md version of every static page at build time — optimized for LLM crawlers and AI agents.
Part of the mark-don family — see also mark-don, the original Ruby gem that inspired this integration.
Why
LLMs consume your pages as raw text. A clean markdown file is cheaper (fewer tokens), easier to parse, and more accurate than a noisy HTML-to-text conversion. This integration does the conversion once at build time so every page has a .md companion at a matching URL ready to serve.
Install
npm install astro-mark-donUsage
// astro.config.mjs
import { defineConfig } from 'astro/config';
import markDon from 'astro-mark-don';
export default defineConfig({
integrations: [
markDon()
]
});Each page in your dist/ folder gets a .md file at a URL that mirrors the page itself. The root becomes index.md, and every other page becomes <page>.md (instead of <page>/index.md), so the markdown URL is identical to the page URL with a .md suffix:
dist/
├── index.html
├── index.md ← generated (/ → /index.md)
├── about/
│ └── index.html
├── about.md ← generated (/about → /about.md)
└── projects/
├── my-project/
│ └── index.html
└── my-project.md ← generated (/projects/my-project → /projects/my-project.md)Options
markDon({
// Pages to skip (matched against pathname)
exclude: ['404.html', 'drawing-board'],
// Options passed to Turndown (html → markdown converter)
turndownOptions: {
headingStyle: 'atx', // default
codeBlockStyle: 'fenced', // default
bulletListMarker: '-' // default
},
// Post-process the markdown before writing
cleanupFn: (markdown, pagePath) => {
markdown = markdown.replace(/\n{3,}/g, '\n\n');
if (pagePath.includes('index.html')) {
markdown = `> LLM-optimized version.\n\n` + markdown;
}
return markdown;
}
})Letting crawlers discover the markdown
Add a <link rel="alternate"> in your layout's <head> pointing to the .md file:
---
// Layout.astro
const pathname = Astro.url.pathname;
const normalized = pathname.replace(/\/$/, '');
const mdPath = normalized === '' ? '/index.md' : `${normalized}.md`;
const mdUrl = new URL(mdPath, Astro.site).href;
---
<head>
<link rel="alternate" type="text/markdown" href={mdUrl} />
</head>
<body>
<div class="hidden" aria-hidden="true">
A markdown version of this page optimized for LLMs is available at:
<a href={mdUrl}>{mdUrl}</a>
</div>
<slot />
</body>The hidden <div> acts as a plain-text signal for crawlers that don't follow <link> tags — it makes the markdown URL discoverable as an anchor in the HTML.
Serving .md files with the correct Content-Type
By default, some hosts serve .md files as application/octet-stream, which triggers a file download instead of displaying the content. This doesn't affect LLMs that fetch URLs directly via HTTP — they read the body regardless — but it can break browser-based tools or headless browser crawlers.
No action is needed unless you observe this issue. The examples below are starting points — they haven't all been tested on every platform version or configuration. If something doesn't work, check your host's documentation for MIME type or response header configuration.
Netlify — add a _headers file at the root of your dist/ or public/ folder:
/*.md
Content-Type: text/markdown; charset=utf-8Vercel — add to vercel.json:
{
"headers": [
{
"source": "/(.*)\\.md",
"headers": [{ "key": "Content-Type", "value": "text/markdown; charset=utf-8" }]
}
]
}fly.io / Generic nginx | tested and approved - add a location block for .md files inside your server block. This avoids conflicts with the existing include /etc/nginx/mime.types; directive:
http {
# ... other http blocks ...
server {
# ... other server blocks ...
location ~* \.md$ {
default_type "text/markdown; charset=utf-8";
}
}
}
}Do not add a top-level types {} block alongside include /etc/nginx/mime.types; — the two conflict and will break your server config.
For other platforms, the fix is equivalent: map the .md extension to text/markdown or text/plain in the server's MIME type configuration.
How it works
Uses the astro:build:done hook to read each generated HTML file and convert it to markdown via Turndown. Scripts, styles, and noscript tags are stripped. The result is written to a .md file whose URL mirrors the page (/about → /about.md, / → /index.md), with a YAML frontmatter header.
License
MIT