SiteCrawler

A CLI tool to crawl websites and extract metadata from the pages.

Installation

To install it globally

npm install -g sitecrawler

Or you can run it via npx

npx sitecrawler

Usage

sitecrawler <url> [options]

| Option | Description | Default | | ----------------------- | -------------------------------------------------------------------------------------------------------------- | ------- | | -d, --depth <number> | Maximum crawl depth from the starting URL | 3 | | -p, --progress | Log each URL as it is crawled | | | -f, --format <format> | Output format: json, md, csv | json | | -o, --output <file> | Write output to a file instead of stdout. Format is inferred from the file extension if --format is not set. | |

Crawling Scope

Only pages at or below the provided URL path will be crawled. Links that go above the starting path are ignored.

Given sitecrawler https://example.com/docs:

| URL | Crawled | | ------------------------------------------ | ------- | | https://example.com/docs | ✅ | | https://example.com/docs/getting-started | ✅ | | https://example.com/docs/api/reference | ✅ | | https://example.com/about | ❌ | | https://other.com/docs | ❌ |

[!NOTE] External links are ignored.

Output Format

{
  "url": "https://example.com",
  "pages": [
    {
      "path": "/about",
      "statusCode": 200,
      "visitedAt": "2026-06-30T10:10:10.000Z",
      "data": {
        "canonical": "https://example.com/about",
        "title": "About Us",
        "description": "Learn more about us",
        "icon": "/favicon.ico",
        "sitemap": "/sitemap.xml",
        "og:url": "https://example.com/about",
        "og:type": "website",
        "og:site_name": "Example",
        "og:title": "About Us",
        "og:description": "Learn more about us",
        "og:image": "https://example.com/og.png",
        "og:image:alt": "Example"
      },
      "links": ["/", "/contact"]
    }
  ]
}

[!NOTE] All data fields are optional and only included when present on the page.