llms-generator-cli
v1.0.3
Published
Generate llms.txt and advanced LLM contexts from a URL
Maintainers
asad-swdReadme
llms-gen
A fast, lightweight CLI tool to generate
llms.txtandllms-full.txtfrom any website URL — making your content instantly readable by AI language models.
What is llms.txt?
llms.txt is an emerging standard that helps AI language models better understand your website. Similar to how robots.txt guides search engine crawlers, llms.txt provides structured, clean Markdown context that LLMs can efficiently read and reference.
llms-gen automates this process — just point it at a URL and it handles the rest.
Features
- 🔍 Auto-scrapes any public URL using Axios + Cheerio
- 📝 Converts HTML → clean Markdown via Turndown
- 📄 Generates
llms.txt— a concise, LLM-optimized summary - 🚀 Generates
llms-full.txt— full-page deep context (with--advancedflag) - ⚡ Zero config — works out of the box
- 🛠️ Globally installable as a CLI command
Installation
Option 1: Run directly with npx (no install needed)
npx llms-generator-cli <url>Option 2: Install globally
npm install -g llms-generator-cliOption 3: Clone and use locally
git clone https://github.com/Asadsheblu/llms-generator-node.git
cd llms-generator-cli
npm install
npm linkUsage
Basic — Generate llms.txt
llms-gen https://example.comThis fetches the page, extracts the main content, and writes a clean llms.txt file in your current directory.
Advanced — Generate llms-full.txt
llms-gen https://example.com --advancedor with the shorthand flag:
llms-gen https://example.com -aThis generates both llms.txt (concise) and llms-full.txt (complete page context).
Output Format
llms.txt (Standard)
# Page Title
> Meta description of the page
## Info
- URL: https://example.com
## Core Content
... (first ~2000 characters of clean Markdown content)llms-full.txt (Advanced)
# Full Context: Page Title
... (complete page content in clean Markdown, no truncation)Examples
# Generate llms.txt for a documentation site
llms-gen https://docs.example.com
# Generate full context for a product landing page
llms-gen https://myproduct.com -a
# Generate for a blog post
llms-gen https://myblog.com/my-article --advancedOptions
| Option | Alias | Description |
|--------|-------|-------------|
| --advanced | -a | Also generate llms-full.txt with full page content |
| --version | -V | Show version number |
| --help | -h | Show help information |
Requirements
- Node.js v14.0.0 or higher
- npm v6.0.0 or higher
Dependencies
| Package | Purpose |
|---------|---------|
| axios | HTTP requests to fetch web pages |
| cheerio | Fast, jQuery-like HTML parsing |
| turndown | HTML to Markdown conversion |
| commander | CLI argument and option parsing |
How It Works
- Fetch — Downloads the HTML of the target URL via Axios
- Parse — Loads the HTML into Cheerio and extracts metadata (title, description) and the main content (
<main>or<body>) - Convert — Transforms the HTML into clean Markdown using Turndown
- Format — Structures the output into the
llms.txtstandard format - Save — Writes the file(s) to your current working directory
Contributing
Contributions are welcome! Feel free to open an issue or submit a pull request.
- Fork the repository
- Create your feature branch:
git checkout -b feature/my-feature - Commit your changes:
git commit -m 'Add my feature' - Push to the branch:
git push origin feature/my-feature - Open a Pull Request
License
MIT © 2024 llms-generator-cli contributors