llmstxt-kit

Framework-agnostic generator + validator for llms.txt, llms-full.txt, JSON-LD structured data, and AI-crawler robots.txt rules.

One command reads your content directory and produces every AI-discovery artifact your site needs — spec-compliant, validated, and honest about what these files actually do.

npm install -D llmstxt-kit
npx llmstxt init && npx llmstxt build

Honest positioning — read this first

This tool is not an "AI SEO ranking booster", and you should be suspicious of anything marketed as one:

• Google has publicly stated it does not support llms.txt. It is not a Google ranking factor.
• Measured citation-bot fetch rates for llms.txt are currently negligible; no study shows a citation lift today.
• The real, defensible value is the agentic/IDE layer: Cursor, Windsurf, Claude Code, GitHub Copilot, Cline, and Aider fetch /llms.txt and /llms-full.txt when pointed at a site. The AI-crawler robots rules and JSON-LD are what actually influence AI retrieval and rich results today.

What this package guarantees is correctness: strict spec-compliant output, a current (dated, sourced) crawler registry, and a validator you can gate CI on — so your AI files never silently break agent parsing.

30-second quickstart

npx llmstxt init      # scaffolds llmstxt.config.mjs by scanning your project
# edit site.name / site.url
npx llmstxt build     # writes llms.txt, llms-full.txt, schema/*.json, robots.ai.txt
npx llmstxt validate  # exits 1 on any spec violation (CI gate)

Configuration

llmstxt.config.ts / .mjs / .json:

import { defineConfig } from 'llmstxt-kit'

export default defineConfig({
  site: {
    name: 'Acme Docs',
    url: 'https://acme.dev',          // absolute base; used to resolve all links
    summary: 'One-sentence description of the site.',
    organization: {                   // -> JSON-LD Organization
      legalName: 'Acme Inc.',
      sameAs: ['https://x.com/acme', 'https://github.com/acme'],
    },
  },
  source: {
    type: 'markdown',                 // 'markdown' | 'routes'
    dir: './content',
    include: ['**/*.md', '**/*.mdx'],
    exclude: ['**/drafts/**'],
  },
  sections: [                          // llms.txt H2 grouping + order
    { title: 'Guides', match: 'guides/**' },
    { title: 'API', match: 'api/**' },
    { title: 'Optional', match: 'changelog/**', optional: true },
  ],
  outputs: {
    llmsTxt: true,
    llmsFullTxt: { enabled: true, maxKb: 200 },
    schema: { types: ['WebSite', 'Organization', 'BreadcrumbList', 'Article', 'FAQPage', 'Product'] },
    robots: { preset: 'allow-all' },  // 'allow-all' | 'search-only' | 'block-all'
    outDir: './public',
  },
})

Frontmatter fields consumed: title, description, section, optional, priority, type (article | faq | product), datePublished, author, faq (array of {q, a}), product (object).

CLI

llmstxt init                    # scaffold a config by scanning the project
llmstxt build                   # generate all enabled outputs into outDir
llmstxt build --only llms,schema
llmstxt preview                 # print llms.txt to stdout without writing
llmstxt validate                # check existing files; exit 1 on error
llmstxt validate --json         # machine-readable report

Flags: --config <path>, --out <dir>, --base-url <url>, --silent, --json.

Programmatic API

import { generate, loadPages, validateLlmsTxt } from 'llmstxt-kit'

const pages = await loadPages({ type: 'markdown', dir: './content' }, site)

const result = generate({
  site: { name: 'Acme', url: 'https://acme.dev' },
  pages,
  outputs: { llmsTxt: true, schema: { types: ['WebSite'] } },
})
// result.files -> [{ path, contents }]  (you decide how to write / inject)

const report = validateLlmsTxt(existingLlmsTxt) // { ok, errors, warnings }

String-returning helpers are exported too — buildLlmsTxt, buildLlmsFullTxt, buildRobotsBlock, buildWebSiteSchema, buildArticleSchema, buildFaqSchema, buildProductSchema, buildBreadcrumbSchema, toJsonLdScript — so you can inject output at request time or in any build pipeline without touching disk.

What gets validated

llms.txt (llmstxt.org format, strict on write, lenient on read):

1. Exactly one H1, first in the file (the only required element).
2. Blockquote summary directly under the H1 (warning if missing).
3. Only H1/H2 headings; no headings in context blocks.
4. Every list item under a section matches - [Title](https://absolute.url): Description.
5. Absolute URLs everywhere.
6. UTF-8, LF line endings, trailing newline.
7. Warns about the common fatal typo llm.txt.

robots.txt: flags deprecated AI user agents (anthropic-ai, claude-web — retired Jul 2024), empty user-agent groups, and an accidentally blocked Googlebot.

schema/*.json: required fields per type, absolute URLs, ISO 8601 dates.

AI-crawler registry

Registry last verified: 2026-07-04 (exported as REGISTRY_UPDATED_AT). Each entry carries its vendor documentation source URL in code (AI_CRAWLER_REGISTRY).

| Vendor | User-agent | Category | |---|---|---| | OpenAI | GPTBot | training | | OpenAI | OAI-SearchBot | search | | OpenAI | ChatGPT-User | live-fetch | | Anthropic | ClaudeBot | training | | Anthropic | Claude-SearchBot | search | | Anthropic | Claude-User | live-fetch | | Perplexity | PerplexityBot | search | | Perplexity | Perplexity-User | live-fetch | | Google | Google-Extended | training opt-out token | | Apple | Applebot-Extended | training opt-out token | | Meta | meta-externalagent | training | | Amazon | Amazonbot | mixed | | Common Crawl | CCBot | training | | ByteDance | Bytespider | training (documented robots.txt non-compliance) |

Presets: allow-all (default recommendation), search-only (allow retrieval/search bots, disallow training bots), block-all. Googlebot is never touched by any preset — blocking it deindexes you from Google Search and AI Overviews.

The robots block is written to robots.ai.txt so your existing robots.txt is never clobbered — append the block to it (or serve it as-is if you had none).

CI gate (GitHub Actions)

- uses: actions/setup-node@v4
  with: { node-version: 22 }
- run: npx llmstxt validate --out ./public

Updating the crawler registry

AI crawler user agents change a few times per year. The registry lives in src/generators/robots.ts with a source URL per entry — PRs that update it against vendor docs (and bump REGISTRY_UPDATED_AT) are very welcome.

License

MIT © Javaid Ahmad