Crawlens

SEO validation before Googlebot crawls.

Crawlens is a build-time SEO quality gate for prerendered static sites. Point it at your framework's build output and it crawls every HTML page plus sitemap.xml, robots.txt, _redirects, and local assets — runs 169 checks across 12 categories, builds an internal link graph with PageRank, scores each page, and writes a split-file JSON report.

Pure static analysis — zero network requests. No headless browser, no live HTTP checks.

Upload the report to the Crawlens UI for interactive dashboards, fix cards, and drill-down by page or category. Learn more at crawlens.ts-saas.com.

Requires Node.js ≥ 24.

Install

npm install -g crawlens

Or run without installing:

npx crawlens <dist-folder> [options]

Canary releases:

npm install -g crawlens@canary

Quick start

# After your framework build (e.g. next build && next export, astro build, etc.)
crawlens ./dist --base-url https://example.com --yes

# Upload the report folder to the Crawlens UI
open https://crawlens.ts-saas.com/upload

Reports are written to ./crawlens-report by default (override with --out).

How it works

1. Detect framework — Crawlens reads your dist folder layout and sampled HTML to identify TanStack Start, React Router, Next.js, Astro, SvelteKit, Nuxt, SolidStart, Angular, or falls back to a generic HTML crawl.
2. Resolve HTML output — Each framework emits prerendered HTML to a different path (client/, dist/, public/, etc.). Crawlens finds it automatically.
3. Confirm and run — On an interactive terminal, press Enter to proceed. In CI or with --yes, this step is skipped.
4. Pipeline — extract → graph → audit → score → report
   • Extract — Parse every HTML file into structured page data; read sitemap, robots, redirects, and asset sizes from disk.
   • Graph — Build the internal link graph, compute PageRank, depth, and near-duplicate clusters.
   • Audit — Run 33 auditor modules (16 per-page, 17 site-wide) producing findings with severity, route, and context.
   • Score — Deduct points by severity; compute per-page, per-category, and site scores (0–100).
   • Report — Write category JSON files plus index.json, pages.json, fixes.json, and optional graph.json.

What you get

| | | | --------------------- | ------------------------------------------------------- | | Checks | 169 unique finding ids across 12 categories | | Auditors | 33 modules (16 per-page · 17 site-wide) | | Scoring | Per-page, per-category, and site score (0–100) | | Link intelligence | Internal graph, PageRank, depth, topical clusters | | Fix guidance | Prioritized topFixes plus fixes.json action cards | | Regression | Baseline snapshots and diffs (regression.json) | | Network | None — analysis runs entirely against your build output |

Categories: On-page · Links · Technical · Content · Media · Performance · Structured Data · Accessibility · AI/GEO · E-E-A-T · Outbound · Ecommerce (opt-in).

Supported frameworks

Pass the build output root — the folder your framework's build command writes to.

| Framework | Pass | HTML resolved from | | --------------------------- | -------------------------- | --------------------------- | | TanStack Start | dist/ | client/ | | React Router v7 | dist/ | client/ | | Next.js (static export) | .next/ or project root | .next/ | | Astro | dist/ | dist root | | SvelteKit | .svelte-kit/ or build/ | output/prerendered/pages/ | | Nuxt | .output/ or dist/ | public/ | | SolidStart | dist/ | public/ | | Angular (SSR/prerender) | dist/<project>/ | browser/ | | Unknown | any dist with HTML | shallowest index.html dir |

Overrides: --framework <id>, --client-dir <path>, --base-url <url>.

Use --base-url when canonical, Open Graph, sitemap, or hreflang checks need your production host.

Options

Run crawlens --help for the full list. CLI flags override .crawlensrc.json in your project root.

| Option | Meaning | | --------------------- | -------------------------------------------------------------- | | --out <dir> | Report output directory (default ./crawlens-report) | | --base-url <url> | Production host for canonical / OG / sitemap checks | | --client-dir <path> | Override resolved HTML output directory | | --framework <id> | Force framework (tanstack-start, nextjs, astro, …) | | --yes, -y | Skip the framework-confirmation prompt | | --no-cache | Force a full re-parse (ignore parse cache) | | --compact | Minified JSON report output | | --zip | Also write <out>.zip beside the report folder (upload-ready) | | --no-graph | Omit graph.json from the report | | --quiet | Minimal output (site score + exit code) | | --verbose | Per-phase timing, cache stats, framework signals | | --json-summary | Print a JSON summary to stdout (CI-friendly) | | --watch | Re-run on dist changes; show finding deltas | | --top <n> | Number of top fixes to show (default 8) | | --route <pattern> | Filter findings by route glob (e.g. /blog/*) | | --category <name> | Filter findings by category (e.g. on-page) | | --ecommerce | Enable the ecommerce module | | --min-score <n> | Exit 1 if site score is below n | | --fail-on <mode> | Exit 1 on regression, error, or score:<n> (repeatable) | | --save-baseline | Save a regression baseline | | --compare [path] | Diff this run against a baseline | | --help, -h | Show usage |

Config file (.crawlensrc.json)

Optional JSON in your project root. Same keys as the flags above — useful in CI:

{
  "baseUrl": "https://example.com",
  "yes": true,
  "compact": true,
  "minScore": 90,
  "failOn": ["error", "regression"]
}

Agent setup prompt

Copy the prompt below into Cursor, Copilot, Claude Code, or any coding agent. It will inspect your repo and wire up Crawlens (config, scripts, and optional CI).

Set up Crawlens (https://crawlens.ts-saas.com) in this project.

Crawlens is a build-time SEO auditor for prerendered static sites. It crawls the framework build output (HTML + sitemap + robots + assets), runs ~169 checks, and writes JSON reports to `crawlens-report/`. Zero network requests.

Requirements:
- Node.js ≥ 24
- Install as a dev dependency: `crawlens` (or `crawlens@canary` for pre-releases)
- Do NOT rely on a global install — scripts must use the local binary from `node_modules/.bin`

Tasks:
1. Detect the framework (TanStack Start, React Router, Next.js static export, Astro, SvelteKit, Nuxt, SolidStart, Angular, etc.) and the correct **build output root** to pass to the CLI.
2. Find the production site URL for `--base-url` / `baseUrl` (env var, site config, or ask me if unclear). Needed for canonical, Open Graph, sitemap, and hreflang checks.
3. Create `.crawlensrc.json` in the project root with sensible defaults, for example:
   - `baseUrl` — production origin (no trailing slash)
   - `yes: true` — skip interactive confirmation (required for CI)
   - `failOn: ["error"]` — fail the command on error-severity findings
   - `minScore` — optional quality gate (e.g. 85)
   - `framework` / `clientDir` — only if auto-detection is wrong for this repo
   - `ecommerce: true` — only if the site has product pages
4. Add package scripts, e.g.:
   - `"seo:audit": "crawlens <dist-folder> --yes"`
   - `"seo:audit:ci": "crawlens <dist-folder> --yes --json-summary --fail-on error"`
   Run the audit **after** the production build (`build && seo:audit` or a dedicated script).
5. Crawlens appends `crawlens-report/` to your project `.gitignore` automatically (creates the file if missing).
6. Optionally add a CI step after the build that runs `pnpm seo:audit:ci` (or npm/yarn equivalent) and fails on errors or score regression (`--compare --fail-on regression` if we keep a baseline).
7. Tell me the exact commands to run locally and what to upload to https://crawlens.ts-saas.com/upload

Framework dist hints (pass the build root, Crawlens resolves HTML automatically):
- TanStack Start / React Router → `dist/` (HTML in `client/`)
- Next.js static export → `.next/`
- Astro → `dist/`
- SvelteKit → `.svelte-kit/` or `build/` (HTML in `output/prerendered/pages/`)
- Nuxt → `.output/` or `dist/` (HTML in `public/`)
- SolidStart → `dist/` (HTML in `public/`)
- Angular → `dist/<project>/` (HTML in `browser/`)

Reference: all config keys match CLI flags (`out`, `baseUrl`, `yes`, `compact`, `noGraph`, `quiet`, `verbose`, `jsonSummary`, `watch`, `top`, `route`, `category`, `clientDir`, `framework`, `minScore`, `failOn`, `saveBaseline`, `label`, `compare`, `ecommerce`). CLI flags override `.crawlensrc.json`.

CI example

crawlens ./dist --base-url https://example.com --yes --json-summary --fail-on error

Gate on score or regressions:

crawlens ./dist --yes --min-score 85 --compare --fail-on regression

Report output (<out>/)

JSON only — upload the folder to the Crawlens UI. Every category file is always emitted (empty findings: [] when clean).

| File | Contents | | ----------------------------------- | --------------------------------------------------------------------- | | index.json | Run meta, site score, categoryScores, topFixes, summary | | pages.json | Per-page rows: score, indexable, words, in/out links, depth, PageRank | | graph.json | Link graph nodes and edges (omit with --no-graph) | | on-page.json | Title, description, headings, canonical, robots, OG, Twitter | | links.json | Internal links, PageRank, clusters, equity recommendations | | technical.json | Sitemap, robots, redirects, pagination, hreflang, indexability | | content.json | Thin content, duplicates, cannibalization, quality signals | | media.json | Images: alt text, dimensions, formats, lazy-load | | performance.json | Route weight, chunks, render-blocking, CWV proxies | | structured-data.json | JSON-LD validation, schema opportunities, breadcrumbs | | accessibility.json | SEO-relevant a11y signals (lang, labels, roles, …) | | ai-search.json | llms.txt, AI-bot directives, citation readiness | | e-e-a-t.json | Author, trust pages, HTTPS canonical | | outbound.json | External link hygiene | | fixes.json | Actionable fix cards keyed by finding id | | ecommerce.json | Product schema (only with --ecommerce) | | baseline.json / regression.json | Regression snapshot / diff when used |

Scoring

Each page starts at 100 per category. Findings deduct by severity:

| Severity | Penalty | | -------- | ------------------------ | | error | −15 | | warn | −6 | | info | 0 (surfaced, no penalty) |

Page score = weighted average of category scores. Site score = average over indexable pages only (noindex and robots-disallowed pages are excluded). topFixes ranks issues by severity × pages affected × PageRank weight.

License

Proprietary software. See UNLICENSED in package.json. This package is not open source.