LLMs Text Parser

A lightweight Node.js library to parse curated llms.txt (or Markdown) documents and optionally crawl linked llms.txt files. It converts a simple Markdown structure into an easily consumable JSON object.

# Title of page
> Overview line 1
> Overview line 2

## Section Name
- [Link Title](https://example.com): Description (optional)

Features

• Parse Markdown/llms.txt into structured JSON (title, overview, and a flat links array with an optional section).
• Crawl remote or local llms.txt files up to a configurable depth, following only links that look like additional llms.txt resources.
• Zero dependencies in runtime code. Jest used for tests only.

Installation

npm install llms-txt-parser

If you are working with the repository directly (cloning or forking) run:

npm install

Usage

Parse a string

const LLMSTextParser = require('llms-txt-parser');

const parser = new LLMSTextParser();
const markdown = `# My Page\n\n> Short overview.\n\n## Resources\n- [Docs](https://example.com)\n`;

const result = parser.parse(markdown);
console.log(result);
/*
{
  title: 'My Page',
  overview: 'Short overview.',
  links: [
    {
      title: 'Docs',
      url: 'https://example.com',
      description: '',
      section: 'Resources'
    }
  ]
}
*/

Parse a string (ESM)

import LLMSTextParser from 'llms-txt-parser';

const parser = new LLMSTextParser();
const markdown = `# My Page\n\n> Short overview.\n\n## Resources\n- [Docs](https://example.com)\n`;

const result = parser.parse(markdown);
console.log(result);

Crawl linked llms.txt files

const parser = new LLMSTextParser({ maxDepth: 3 });

(async () => {
  const data = await parser.crawl('https://example.com/llms.txt');
  console.log(data); // Array of parsed pages within depth limit
})();

CLI

A convenient CLI is included and exposed as llms-txt-parser when the package is installed (locally or globally).

# Parse a local file and crawl one level deep
npx llms-txt-parser ./tests/tailwind.md --depth 1

# Parse a remote llms.txt and crawl to depth 3
llms-txt-parser https://example.com/llms.txt --depth 3

Options

• --depth <n> – Depth to crawl linked llms.txt files (default 1).

The CLI prints the JSON to stdout, making it easy to pipe into tools like jq or redirect to a file.

Scripts

• npm test – Run Jest unit tests.
• npm publish – Publish the package to npm (make sure you bump the version first).

Running tests

npm test

Tests live in __tests__/ and cover parsing logic and edge-cases.

Contributing

1. Fork the repo & create a feature branch.
2. Run npm install to get dev dependencies.
3. Add / update tests and ensure npm test passes.
4. Submit a PR.

License

MIT © 2025 JT