LLMSTXT-GEN

Generate LLMS.txt for your website - A universal library for creating AI-friendly site documentation.

🚀 Features

• Framework Agnostic: Works with Astro, Next.js, and static sites
• Multiple Entry Points: Core library, framework integrations, and CLI
• TypeScript Support: Full type safety and IntelliSense
• Flexible Configuration: Customizable content extraction and formatting
• CLI Tool: Generate LLMS.txt from command line
• Dynamic & Static: Support for both build-time and runtime generation

📦 Installation

npm install llmstxt-gen

🔧 Quick Start

CLI Usage

# Initialize configuration
npx llms init

# Generate LLMS.txt
npx llms generate --site https://your-site.com

# Generate with custom config
npx llms generate --config llms.config.js

Astro Integration

// astro.config.mjs
import { defineConfig } from 'astro/config';
import llms from 'llmstxt-gen/astro';

export default defineConfig({
  site: 'https://your-site.com',
  integrations: [
    llms({
      pages: ['src/pages/**/*.md', 'src/pages/**/*.astro'],
      exclude: ['src/pages/admin/**/*'],
      customFields: {
        title: (page) => page.frontmatter?.title,
        description: (page) => page.frontmatter?.description,
      },
    }),
  ],
});

Next.js Integration

Using Webpack Plugin

// next.config.js
const { withLLMS } = require('llmstxt-gen/nextjs');

module.exports = withLLMS({
  // Your Next.js config
  llms: {
    site: 'https://your-site.com',
    pages: ['pages/**/*.md', 'content/**/*.md'],
  },
});

API Route (Pages Router)

// pages/api/llms.txt.js
import { nextjsUtils } from 'llmstxt-gen/nextjs';

export default nextjsUtils.createApiRoute({
  site: 'https://your-site.com',
  pages: ['pages/**/*.md', 'content/**/*.md'],
});

Route Handler (App Router)

// app/llms.txt/route.js
import { nextjsUtils } from 'llmstxt-gen/nextjs';

export const GET = nextjsUtils.createRouteHandler({
  site: 'https://your-site.com',
  pages: ['app/**/*.md', 'content/**/*.md'],
});

Core Library Usage

import { LLMS } from 'llmstxt-gen';

const llms = new LLMS({
  site: 'https://your-site.com',
  pages: ['**/*.md', '**/*.html'],
  exclude: ['node_modules/**/*'],
  customFields: {
    title: (page) => page.frontmatter?.title || page.metadata?.title,
    description: (page) => page.frontmatter?.description,
    content: (page) => page.content,
  },
});

// Generate LLMS.txt content
const output = await llms.generate();
console.log(llms.formatAsString(output));

// Generate and save to file
const filePath = await llms.generateAndSave();
console.log(`Generated: ${filePath}`);

⚙️ Configuration

Configuration File (llms.config.js)

module.exports = {
  site: 'https://your-site.com',
  pages: [
    'src/pages/**/*.md',
    'src/pages/**/*.astro',
    'content/**/*.md',
  ],
  exclude: [
    'src/pages/admin/**/*',
    'src/pages/api/**/*',
    'node_modules/**/*',
  ],
  customFields: {
    title: (page) => page.frontmatter?.title || page.metadata?.title,
    description: (page) => page.frontmatter?.description,
    content: (page) => page.content,
    author: (page) => page.frontmatter?.author,
  },
  output: 'static', // 'static' or 'dynamic'
  outputPath: 'public/llms.txt',
  siteName: 'Your Site Name',
  maxContentLength: 2000,
  includeMetadata: true,
};

Configuration Options

| Option | Type | Default | Description | |--------|------|---------|-------------| | site | string | required | Base URL of your site | | pages | string[] | auto-detected | Glob patterns for pages to include | | exclude | string[] | [] | Glob patterns for pages to exclude | | customFields | object | {} | Custom field extractors | | output | 'static' \| 'dynamic' | 'static' | Generation mode | | outputPath | string | 'llms.txt' | Output file path | | siteName | string | extracted from site | Site name for LLMS.txt header | | maxContentLength | number | 2000 | Maximum content length per page | | includeMetadata | boolean | true | Include page metadata |

🎯 CLI Commands

llms generate

Generate LLMS.txt file.

npx llms generate [options]

Options:

• -c, --config <path> - Configuration file path (default: llms.config.js)
• -o, --output <path> - Output file path (default: llms.txt)
• -s, --site <url> - Site URL
• -d, --dir <path> - Base directory to scan (default: current directory)
• --dry-run - Print output without writing file

llms init

Create a configuration file.

npx llms init [options]

Options:

• -f, --framework <framework> - Target framework (astro|nextjs|static)

llms validate

Validate configuration file.

npx llms validate [options]

Options:

• -c, --config <path> - Configuration file path

📄 LLMS.txt Format

The generated LLMS.txt follows this structure:

# LLMS.txt for Your Site

Site: https://your-site.com
Generated: 2023-01-01T00:00:00.000Z

## Pages

### Page Title
URL: https://your-site.com/page
Description: Page description
Content: Extracted page content...

---

### Another Page
URL: https://your-site.com/another
Description: Another page description
Content: More extracted content...

🔌 Framework-Specific Features

Astro

• Automatic integration with Astro's build process
• Support for .astro, .md, and .mdx files
• Frontmatter extraction
• Build-time generation

Next.js

• Support for both Pages Router and App Router
• Dynamic API routes for runtime generation
• Webpack plugin for build-time generation
• MDX and Markdown support
• Automatic framework detection

Static Sites

• Universal file format support
• Flexible glob patterns
• CLI-based generation
• Custom content extractors

🧪 Testing

# Run tests
npm test

# Watch mode
npm run test:watch

# Coverage
npm test -- --coverage

🛠️ Development

# Clone the repository
git clone https://github.com/yourusername/llms.git
cd llms

# Install dependencies
npm install

# Build the project
npm run build

# Run in development mode
npm run dev

# Lint code
npm run lint

📚 Examples

Check the /examples directory for complete implementation examples:

• Astro Example
• Next.js Example
• Static Site Example

🤝 Contributing

Contributions are welcome! Please read our Contributing Guide for details.

📄 License

MIT License - see LICENSE file for details.

🔗 Links

• Documentation
• NPM Package
• GitHub Repository
• Issues

💡 What is LLMS.txt?

LLMS.txt is a proposed standard for helping AI systems understand website content structure. It provides a structured way to present your site's content to large language models, making it easier for AI to understand and work with your content.

Learn more about the LLMS.txt standard at llmstxt.org.