sitemap-scrapper

Audits a website by comparing sitemap.xml entries against pages reachable through site navigation. Flags orphaned pages (in the sitemap but not linked anywhere), pages linked on the site but missing from the sitemap, broken URLs, and redirects.

Installation

npm install -g @mraichelson/sitemap-scrapper
npx playwright install chromium

Usage

sitemap-audit <url> [options]

Arguments

| Argument | Description | |---|---| | <url> | Base URL of the website to audit (required) |

Options

| Option | Default | Description | |---|---|---| | --max-pages <number> | 500 | Maximum number of pages to crawl. Increase for large sites. | | --output <dir> | ./output | Directory to save the CSV report |

Examples

# Basic audit
sitemap-audit https://example.com

# Audit a large site with no crawl limit
sitemap-audit https://example.com --max-pages 9999

# Save output to a custom directory
sitemap-audit https://example.com --output ~/Desktop

Output

Results are saved to output/{hostname}-{YYYY-MM-DD}.csv with the following columns:

| Column | Description | |---|---| | URL | The page URL | | Linked on site? | Yes if found via a link on any crawled page, No if orphaned | | HTTP status | HTTP status code returned when the page was fetched | | Redirect target | Destination URL if the page redirects. HTTP redirects show the destination URL; client-side redirects (meta refresh or JS) show [destination] (Client-side redirect) | | Robots.txt | Allowed or Disallowed per the site's robots.txt | | Sitemap file | Which sitemap file this URL came from, or Not in sitemap for pages discovered during crawl that don't appear in any sitemap | | Notes | Fetch errors or other flags | | Next steps | Blank — for manual notes during audit review |

Rows are sorted: orphaned sitemap pages first, then pages not in the sitemap, then linked sitemap pages.

Development

npm install
npx playwright install chromium
npm start -- https://example.com

Note: The -- between npm start and your arguments is required. Without it, npm intercepts flags like --max-pages and they never reach the script.