llmfood

Generate LLM-friendly Markdown from Docusaurus HTML builds, implementing the llms.txt convention.

Overview

llmfood converts a Docusaurus static HTML build into clean Markdown files optimized for LLM consumption. It:

1. Discovers all pages in a Docusaurus build directory
2. Resolves client-side content that doesn't exist in static HTML (GitHub code references, remote content, mermaid diagrams)
3. Converts each HTML page to Markdown, stripping Docusaurus chrome (breadcrumbs, pagination, TOC, footers)
4. Generates llms.txt — a structured index linking to all converted .md files
5. Generates custom files — aggregated Markdown files matching URL patterns (e.g., llms-full.txt)

Installation

npm install llmfood
# or
bun add llmfood

Usage

Docusaurus Plugin (recommended)

Add llmfood as a Docusaurus plugin for zero-config integration. It runs automatically after docusaurus build:

// docusaurus.config.js
module.exports = {
  plugins: [
    [
      "llmfood/docusaurus",
      {
        sectionOrder: ["guides", "api", "concepts"],
        sectionLabels: { guides: "Guides", api: "API Reference" },
        customFiles: [
          {
            filename: "llms-full.txt",
            title: "Full Documentation",
            description: "Complete documentation in a single file",
            includePatterns: [/.*/],
          },
        ],
      },
    ],
  ],
};

The plugin automatically derives baseUrl, buildDir, siteTitle, and siteDescription from your Docusaurus config. It also sets docsDir to {siteDir}/docs by default, enabling source file scanning for mermaid diagrams and remote content resolution.

Standalone

import { generateLlmsMarkdown } from "llmfood";

await generateLlmsMarkdown({
  baseUrl: "https://docs.example.com",
  buildDir: "./build",
  siteTitle: "My Docs",
  siteDescription: "Documentation for my project",
  docsDir: "./docs", // optional: enables source file scanning
  sectionOrder: ["guides", "api", "concepts"],
  sectionLabels: { guides: "Guides", api: "API Reference" },
  ignorePatterns: [/\/blog\//],
  customFiles: [
    {
      filename: "llms-full.txt",
      title: "Full Documentation",
      description: "Complete documentation in a single file",
      includePatterns: [/.*/],
    },
  ],
});

Standalone HTML to Markdown

You can also use the converter directly:

import { htmlToMarkdown } from "llmfood";

const markdown = htmlToMarkdown(docusaurusHtmlString);

Content Resolution

Some Docusaurus plugins render content client-side, so the static HTML contains placeholders instead of real content. When docsDir is set, llmfood scans MDX source files and resolves these automatically:

| Pattern | Source detection | Resolution | | ---------------------- | ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- | | GitHub code references | CodeBlock JSX, fenced ```lang reference, and children/src/srcUrl/source attributes | Fetches code from raw.githubusercontent.com with line ranges | | Remote content | url="..." or url={expr} in MDX | Fetches remote markdown (JSX expressions via resolveRemoteUrl) | | Mermaid diagrams | ```mermaid blocks in MDX | Injects mermaid source into HTML (client-side renders leave none) | | YouTube embeds | <iframe> with YouTube URL in HTML | Converts to [title](youtube-url) markdown link |

Source scanning also resolves imported MDX snippets (import Foo from "./_snippet.mdx"), substitutes ${props.x} expressions using caller prop values, and matches files by frontmatter id when the slug differs from the filename.

All external fetches run in parallel with a concurrency limit of 6.

API

generateLlmsMarkdown(config)

Processes an entire Docusaurus build and generates llms.txt plus any custom files.

LlmfoodConfig

| Property | Type | Required | Description | | --------------------- | --------------------------- | -------- | --------------------------------------------------------------------------- | | baseUrl | string | Yes | Base URL for generated links (e.g., https://docs.example.com) | | buildDir | string | Yes | Path to the Docusaurus build output directory | | customFiles | CustomLlmFile[] | No | Custom aggregated output files to generate | | docsDir | string | No | Path to docs source directory (enables mermaid + remote content resolution) | | ignorePatterns | RegExp[] | No | URL patterns to exclude (root / is always excluded) | | postProcessHtml | (html, context) => string | No | Hook to transform HTML before markdown conversion | | postProcessMarkdown | (md, context) => string | No | Hook to transform markdown after conversion | | resolveRemoteUrl | (expr) => string | No | Resolve JSX expressions (e.g., getBenchmarkURL(...)) to fetch URLs | | rootContent | string | No | Additional content to include at the top of llms.txt | | sectionLabels | Record<string, string> | No | Custom display labels for URL sections | | sectionOrder | string[] | No | Ordering for sections in llms.txt | | siteDescription | string | No | Site description shown in llms.txt | | siteTitle | string | No | Site title shown in llms.txt | | verbose | boolean | No | Log individual skipped pages with reasons |

Both hooks receive a ProcessContext with { urlPath: string } and may return a Promise.

CustomLlmFile

| Property | Type | Required | Description | | ----------------- | ---------- | -------- | ---------------------------------------- | | filename | string | Yes | Output filename (e.g., llms-full.txt) | | includePatterns | RegExp[] | Yes | URL patterns to include in this file | | description | string | No | Description shown at the top of the file | | title | string | No | Title shown at the top of the file |

htmlToMarkdown(html)

Converts a Docusaurus HTML string to clean Markdown. Expects the content to be wrapped in an <article> tag.

Returns an empty string if no <article> element is found.

Supported Docusaurus Elements

The converter handles these Docusaurus-specific elements:

• Prism code blocks — preserves language and syntax highlighting structure
• Admonitions — converts to :::type [title] syntax (tip, warning, info, caution, danger, note, important)
• Tabs — renders each tab panel with its label as a bold heading
• Details/Summary — preserves as HTML <details> elements
• KaTeX math — converts to $$...$$ (block) and $...$ (inline) syntax
• Images — converts to standard Markdown, skipping data URIs
• Tables — converts to GFM table syntax with alignment support (:---:, ---:)
• Strikethrough — converts <del> and <s> to ~~text~~
• YouTube iframes — converts to markdown links with video title
• Mermaid code blocks — preserves as fenced mermaid code blocks (when source is available)

Pages that can't be converted are tracked and summarized. Set verbose: true to see individual skipped pages with reasons (redirects, empty pages, missing files, errors).

License

MIT