AgRe CLI

Make your site crawlable by AI agents in minutes.

npx agre-cli

That's it. No install, no config, no account. Works on any machine with Node.js 20+.

Quick Start

npx agre-cli --site-url https://example.com
npx agre-cli generate all --site-url https://example.com

Prefer npx for one-off usage, or install globally:

npm i -g agre-cli
agre-cli --help

What it does

Run npx agre-cli in any web project root and it:

1. Detects your framework — Next.js, Nuxt, Astro, SvelteKit, Remix, Vite, plain HTML
2. Scans all routes and extracts metadata (title, description, content type)
3. Prints a scored checklist of AI-crawlability issues (0–100, A–F grade)
4. Generates llms.txt and llms-full.txt in the correct public directory
5. Analyzes robots.txt and patches it to allow all known AI bots
6. Detects CSR-only pages invisible to AI crawlers
7. Generates Schema.org JSON-LD snippets for key page types
8. Shows an agent preview — what an AI bot actually sees from a route file

Example output

┌─────────────────────────────────────────────────┐
│                                                 │
│   agre-cli v1.0.0                             │
│   Make your site crawlable by AI agents         │
│                                                 │
└─────────────────────────────────────────────────┘

◆ Detected: Next.js (App Router) — next.config.ts
◆ Found 23 routes

  CRITICAL ──────────────────────────────────────────
  ✗  llms.txt missing           Not found in public/
  ✗  llms-full.txt missing      Not found in public/

  HIGH ──────────────────────────────────────────────
  ✗  ClaudeBot allowed          ClaudeBot: Disallow: /
  ✓  GPTBot allowed             GPTBot: Allow: /

  MEDIUM ────────────────────────────────────────────
  ◐  Schema.org — 0/23 pages    No JSON-LD found
  ✓  sitemap.xml exists
  ✓  Meta descriptions — 19/23 pages

  Score: 44/100  Grade: D

Commands

npx agre-cli                          # full interactive scan (default)
npx agre-cli scan                     # scan only, print checklist, no writes
npx agre-cli generate llms            # generate llms.txt
npx agre-cli generate llms-full       # generate llms-full.txt
npx agre-cli generate robots          # patch robots.txt to allow AI bots
npx agre-cli generate schema          # generate Schema.org JSON-LD snippets
npx agre-cli generate all             # generate everything
npx agre-cli preview [file]           # show what an AI bot sees from a route
npx agre-cli init                     # create agre-cli.config.ts

Flags

--cwd <path>          Working directory (default: cwd)
--site-url <url>      Production URL (e.g. https://myapp.com)
--out <dir>           Output directory override
--yes, -y             Skip confirmation prompts
--dry-run             Show output without writing files
--json                Output as JSON (for piping/scripting)
--ci                  CI mode: non-interactive, exit 1 on critical failures
--quiet, -q           Suppress non-error output
--verbose             Debug-level output
--config <path>       Path to agre-cli.config.ts

Configuration

Create an agre-cli.config.ts in your project root (or run agre-cli init):

import type { agreConfig } from 'agre-cli';

const config: agreConfig = {
  siteUrl: 'https://myapp.com',
  siteName: 'MyApp',
  siteDescription: 'The fastest way to ship AI-ready web apps.',
  excludeRoutes: ['/internal/**', '/preview/**'],
  includeApiRoutes: false,
  includeDynamicRoutes: false,
  tokenBudget: 80000,
};

export default config;

Supported Frameworks

| Framework | Detection | Public Dir | Routes | |---|---|---|---| | Next.js (App Router) | next.config.* + app/ | public/ | app/**/page.{tsx,ts,jsx,js} | | Next.js (Pages Router) | next.config.* + pages/ | public/ | pages/**/*.{tsx,ts,jsx,js} | | Nuxt 3 | nuxt.config.* | public/ | pages/**/*.vue | | Astro | astro.config.* | public/ | src/pages/**/*.{astro,md,mdx} | | SvelteKit | svelte.config.* | static/ | src/routes/**/+page.svelte | | Remix | remix.config.* | public/ | app/routes/**/*.{tsx,ts,jsx,js} | | Vite SPA | vite.config.* | public/ | index.html | | Plain HTML | index.html in root | root | **/*.html |

CI / GitHub Actions

Quick CI check

- name: Check AI crawlability
  run: npx agre-cli scan --ci --site-url https://myapp.com

Exit code 1 if any critical checklist items fail.

GitHub Action

# If this project is published as a GitHub Action:
- uses: <owner>/<repo>@v1
  with:
    site-url: https://myapp.com
    generate: all
    fail-on-critical: true

JSON output for scripting

npx agre-cli scan --json | jq '.checklist.score'

Checklist Items

| ID | Severity | What it checks | |---|---|---| | llms-txt-exists | Critical | llms.txt exists in public dir | | llms-full-txt-exists | High | llms-full.txt exists | | llms-txt-valid | High | llms.txt has title, sections, links | | robots-allows-gptbot | High | GPTBot not blocked | | robots-allows-claudebot | High | ClaudeBot not blocked | | robots-allows-perplexity | Medium | PerplexityBot not blocked | | no-csr-only-pages | High | No client-side-only pages | | sitemap-exists | Medium | sitemap.xml found | | homepage-has-schema | Medium | JSON-LD on homepage | | meta-descriptions | Medium | ≥80% pages have descriptions |

Programmatic API

import {
  detect,
  scanRoutes,
  runChecklist,
  generateLlmsTxt,
  analyzeRobots,
  generateSchema,
  previewAgent,
} from 'agre-cli';

const framework = await detect(process.cwd());
const routes = await scanRoutes(process.cwd(), framework);
const result = await runChecklist(process.cwd(), framework, routes);

console.log(`Score: ${result.score}/100 (${result.grade})`);

Development

pnpm install
pnpm lint            # Run ESLint
pnpm build           # Build with tsup
pnpm test            # Run vitest
pnpm test:coverage   # Run tests with coverage
pnpm test:watch      # Watch mode
pnpm typecheck       # TypeScript type checking
pnpm dev             # Build in watch mode

How scoring works

Each checklist item has a severity weight:

• Critical: 35 points
• High: 25 points
• Medium: 15 points
• Low: 5 points

Pass = full points, Warn = half, Fail = 0, Skip = excluded.

| Grade | Score | |---|---| | A | 90–100 | | B | 75–89 | | C | 55–74 | | D | 35–54 | | F | 0–34 |

Non-goals (v1)

• Does not deploy or host anything
• Does not make HTTP requests (fully offline static analysis)
• Does not run a headless browser
• Does not require an account or API key
• Does not modify source code — generates new files and snippets only

License

MIT © agre-cli