llms-txt

Generate llms.txt file and .md file for your content.

Install

bun add -D @vahor/llms-txt

Usage

Example with contentlayer but you can use any other source of data.

import { allDocuments } from "contentlayer/generated";
import {
	generate,
	LLMS_TXT_FILENAME,
	type PluginOptions,
} from "@vahor/llms-txt";

const options = {
	outputPath: (path) => {
		if (path === LLMS_TXT_FILENAME) {
			return "./public/llms.txt";
		}
		// path is "./content/posts/[slug].mdx"
		const slug = path.split("/").slice(3).join("/");
		const withoutExtension = slug.split(".").slice(0, -1).join(".");
		return `./public/${withoutExtension}.md`;
	},
	formatFrontmatter: (frontmatter) => ({
		title: frontmatter.title,
		description: frontmatter.description,
	}),
	sections: [
		{
			title: "MySuperBlog",
			description: "This is a super cool blog",
			details: "In this blog I will write about stuff",
		},
		{
			title: "Blog",
			links: allDocuments.map((doc) => ({
				title: doc.title,
				url: `https://vahor.fr/${doc.pageType}/${doc.slug}.md`,
				description: doc.description,
			})),
		},
	],
	content: allDocuments.map((doc) => ({
		path: `./content/${doc._raw.sourceFilePath}`,
	})),
} satisfies PluginOptions;

generate(options);

And update your package.json to run this script:

{
	"scripts": {
		"build": "next build && bun generate:llms.txt",
		"generate:llms.txt": "bun ./scripts/generate-llms.txt.ts"
	}
}

This will generate a llms.txt file in the public folder and a .md file for each post.

# MySuperBlog

> This is a super cool blog

In this blog I will write about stuff

## Blog

- [Rehype D2 Plugin](https://vahor.fr/project/rehype-d2.md): Un plugin Rehype pour convertir des diagrammes D2 en SVG ou PNG.

---
title: Rehype D2 Plugin
description: Un plugin Rehype pour convertir des diagrammes D2 en SVG ou PNG.
---

tldr: [https://github.com/Vahor/rehype-d2](https://github.com/Vahor/rehype-d2)

...

Options

• outputPath: Function that determines where files will be written.

  • Takes the current MDX file path as argument and returns the output path.
  • Return null to ignore a file.
  • Use LLMS_TXT_OUTPUT_DIR_INPUT constant for the llms.txt file.

• fs: Custom filesystem implementation.

  • Must include writeFileSync, mkdirSync, and readFileSync methods.
  • Defaults to Node's built-in fs module if not provided.

• formatFrontmatter: Function to filter or transform frontmatter properties before writing to .md files.

  • Defaults to an identity function if not provided.

• content: Array of content paths to process.

  • Each item should be an object with a path property.

• sections: Array containing a header and sections for the llms.txt file.

  • First item is the header and contains { title: string, description?: string, details?: string }.
  • Subsequent items are sections with { title: string, links: { title: string, url: string, description?: string }[] }.
  • Structured as [LLMSTxtHeader, ...LLMSTxtSection[]].

• remarkPlugins: Array of remark plugins to transform markdown content.

  • See Unified.js guide for more information.
  • Each plugin function is called with the following arguments:
    • frontmatter: The frontmatter of the current file. (Same input as formatFrontmatter)
  • Optional parameter.

Check examples in tests files.

Who is using this?

• vahor.fr: my personal website