llms-forge

v1.0.0

Published

7 days ago

llm.energy - Extract documentation for AI agents from any site with llms.txt support

0High
0Medium
0Low

nirholas

llm.energy

📖 Overview

llm.energy is a web application and MCP server that fetches, parses, and organizes documentation from websites implementing the llms.txt and install.md standards. It transforms raw documentation into structured, agent-ready formats optimized for large language models, AI assistants, and developer tooling.

| Standard | Description | Learn More | |----------|-------------|------------| | llms.txt | Machine-readable documentation for AI systems | llmstxt.org | | install.md | LLM-executable installation instructions | installmd.org |

✨ Key Features

| Feature | Description | |---------|-------------| | 🔍 Smart Detection | Automatically finds llms.txt, llms-full.txt, and install.md | | 📄 Organized Output | Splits content into individual markdown files by section | | 🤖 Agent-Ready | Includes AGENT-GUIDE.md optimized for AI assistants | | 📦 Multiple Formats | Export as Markdown, JSON, YAML, or ZIP archive | | 🔌 MCP Server | Integrate with Claude Desktop, Cursor, and more | | ⚡ Batch Processing | Process multiple URLs simultaneously | | 📚 Site Directory | Browse 19+ curated llms.txt-enabled websites | | ✏️ llms.txt Generator | Create your own llms.txt files with a guided wizard | | 🚀 install.md Generator | AI-powered: generate from GitHub repos, docs URLs, or manually |

🎯 Use Cases

─────────────────────────────────────────────────────────────────
                                                                 
   📝 Feed docs to AI coding assistants (Cursor, Windsurf)       
   🤖 Build context-aware AI agents with up-to-date docs         
   🔄 Create documentation pipelines for RAG systems            
   📦 Aggregate docs from multiple sources automatically        
   ✏️  Generate llms.txt/install.md for your own projects       
   🚀 Auto-generate install.md from any GitHub repo             
                                                                 
─────────────────────────────────────────────────────────────────

🚀 install.md Generator

Generate LLM-executable installation instructions from any source:

| Mode | Description | Use Case | |------|-------------|----------| | From GitHub | Analyze any public GitHub repo and generate install.md | Perfect for creating install.md for existing projects | | From URL | Extract from any documentation page | Convert existing docs to install.md format | | Manual | Build from scratch with guided wizard | Full control over every detail |

How It Works

GitHub Mode: Analyzes README, package.json/pyproject.toml/Cargo.toml, GitHub Actions, and releases
URL Mode: Scrapes documentation pages, detects platforms (Mintlify, Docusaurus, GitBook, etc.)
AI Synthesis: Uses Claude to generate a properly formatted install.md

Quick Example

# Generate install.md for any GitHub project
curl -X POST https://llm.energy/api/generate-install \
  -H "Content-Type: application/json" \
  -d '{"url": "https://github.com/anthropics/anthropic-sdk-python", "type": "github"}'

Visit llm.energy/install-generator to use the web interface.

📚 Documentation

Full documentation is available at llm.energy/docs

🚀 Installation

Web Application

Visit llm.energy to use the hosted version.

Local Development

# Clone the repository
git clone https://github.com/nirholas/extract-llms-docs.git
cd extract-llms-docs

# Install dependencies
pnpm install

# Start development server
pnpm dev

🌐 The application runs on http://localhost:3001

MCP Server

Add to your MCP client configuration (Claude Desktop, Cursor, etc.):

{
  "mcpServers": {
    "llm-energy": {
      "command": "npx",
      "args": ["-y", "@llm-energy/mcp-server"]
    }
  }
}

See MCP Server Documentation for detailed setup.

🔌 API Reference

curl -X POST https://llm.energy/api/extract \
  -H "Content-Type: application/json" \
  -d '{"url": "docs.anthropic.com"}'

Response includes parsed sections, metadata, and download URLs.

curl -X POST https://llm.energy/api/validate \
  -H "Content-Type: application/json" \
  -d '{"url": "docs.example.com"}'

curl -X POST https://llm.energy/api/batch \
  -H "Content-Type: application/json" \
  -d '{"urls": ["docs.anthropic.com", "docs.stripe.com"]}'

# Formats: markdown, json, yaml, zip
curl "https://llm.energy/api/download?url=docs.anthropic.com&format=zip"

# Generate from a GitHub repository
curl -X POST https://llm.energy/api/generate-install \
  -H "Content-Type: application/json" \
  -d '{"url": "https://github.com/anthropics/anthropic-sdk-python", "type": "github"}'

# Generate from a documentation URL
curl -X POST https://llm.energy/api/generate-install \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.anthropic.com/en/docs/quickstart", "type": "docs"}'

# Analyze only (preview mode)
curl "https://llm.energy/api/generate-install?url=https://github.com/anthropics/anthropic-sdk-python&type=github"

📖 Full API Reference →

🤖 MCP Server Tools

The MCP server exposes the following tools for AI agents:

| Tool | Description | |------|-------------| | extract_docs | Extract documentation from a URL with llms.txt support | | validate_url | Check if a URL has llms.txt available | | verify_llms_txt | Verify llms.txt exists and get file info | | discover_documentation_urls | Find documentation URLs for a domain | | list_sites | Get directory of known llms.txt-enabled sites | | search_sites | Search the site directory by category or keyword |

🏗️ Architecture

┌──────────────────────────────────────────────────────────────────┐
│                        llm.energy                                │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐               │
│  │   Web App   │  │  REST API   │  │ MCP Server  │               │
│  │  (Next.js)  │  │  /api/*     │  │   (stdio)   │               │
│  └──────┬──────┘  └──────┬──────┘  └──────┬──────┘               │
│         │                │                │                      │
│         └────────────────┼────────────────┘                      │
│                          │                                       │
│                   ┌──────▼──────┐                                │
│                   │    Core     │                                │
│                   │  - Parser   │                                │
│                   │  - Extractor│                                │
│                   │  - Cache    │                                │
│                   └──────┬──────┘                                │
│                          │                                       │
│         ┌────────────────┼────────────────┐                      │
│         ▼                ▼                ▼                      │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐               │
│  │  llms.txt   │  │ install.md  │  │   Sitemap   │               │
│  └─────────────┘  └─────────────┘  └─────────────┘               │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

extract-llms-docs/
├── src/
│   ├── app/              # Next.js pages and API routes
│   │   ├── api/          # REST API endpoints
│   │   │   ├── extract/  # Documentation extraction API
│   │   │   ├── generate-install/  # AI-powered install.md generation
│   │   │   └── ...       # Other API endpoints
│   │   ├── extract/      # Extraction interface
│   │   ├── batch/        # Batch processing page
│   │   ├── directory/    # Site directory browser
│   │   ├── generator/    # llms.txt generator wizard
│   │   └── install-generator/  # install.md generator with tabs
│   ├── components/       # React UI components
│   │   └── install-generator/  # GitHubTab, UrlTab, Preview
│   ├── lib/              # Core utilities
│   │   ├── github-analyzer.ts  # GitHub repo analysis
│   │   ├── docs-analyzer.ts    # Documentation URL scraping
│   │   └── ...           # Parser, extractor, cache
│   ├── hooks/            # React hooks
│   └── types/            # TypeScript definitions
├── packages/core/        # Shared parser and types
├── mcp-server/           # MCP server package
└── docs-site/            # MkDocs documentation source

🛠️ Technology Stack

⚙️ Configuration

Environment variables (optional):

RATE_LIMIT_REQUESTS=100     # Max requests per window
RATE_LIMIT_WINDOW_MS=60000  # Window duration in ms
CACHE_TTL=3600              # Cache time-to-live in seconds
ADMIN_KEY=your-secret-key   # Admin API key (required for cache management)

💻 Development

pnpm dev          # 🚀 Start development server
pnpm build        # 📦 Production build
pnpm start        # ▶️  Start production server
pnpm test         # 🧪 Run tests (163 tests)
pnpm test:coverage # 📊 Tests with coverage report
pnpm typecheck    # ✅ TypeScript validation
pnpm lint         # 🔍 ESLint check

🔗 Related Projects

| Project | Description | |---------|-------------| | llms.txt Standard | Machine-readable documentation format | | install.md Standard | LLM-executable installation format | | Model Context Protocol | Protocol for AI tool integration | | MCP Servers Directory | Community MCP servers |

🌐 Sites with llms.txt Support

Browse 19+ curated websites with verified llms.txt support at llm.energy/directory

Featured sites include:

🤖 Anthropic Documentation
⚡ Vercel Documentation
💳 Stripe API Reference
🗄️ Supabase Docs
📘 Mintlify Documentation
...and more!

🤝 Contributing

Contributions are welcome! Please follow these steps:

🍴 Fork the repository
🌿 Create a feature branch (git checkout -b feature/amazing-feature)
💾 Commit your changes (git commit -m 'Add amazing feature')
📤 Push to the branch (git push origin feature/amazing-feature)
🔀 Open a Pull Request

Adding a site to the directory? Edit src/data/sites.ts and submit a PR.

📄 License

MIT License - see LICENSE for details

🔗 Links

👤 Author

Built by nich

Author

Built by nich - https://x.com/nichxbt

Keywords: llms.txt, llms-full.txt, documentation extraction, AI documentation, LLM context, Model Context Protocol, MCP server, AI agents, documentation parser, markdown extraction, API documentation, developer tools, AI coding assistant, RAG, retrieval augmented generation, context injection, AI-friendly documentation, vibe coding, cursor, windsurf, claude, chatgpt, copilot