Cockroach Crawler

Cockroach Crawler is an open-source public-web crawler for agent workflows. It turns crawlable public pages into clean JSON or JSONL with titles, metadata, readable text, markdown, links, status codes, content types, and timestamps.

It is designed for documentation indexing, RAG ingestion, QA crawling, content inventory, public research, and agent toolchains that need a local crawler with a simple CLI and JavaScript API.

What it does

• Crawls public HTTP/HTTPS pages from one or more seed URLs.
• Respects robots.txt by default.
• Discovers sitemaps from robots.txt and /sitemap.xml.
• Extracts readable text and markdown.
• Outputs JSON or JSONL for agent pipelines.
• Supports concurrency, per-origin delay, max pages, max depth, include/exclude filters, URL files, and same-origin controls.
• Uses a clear user agent and supports contact details.
• Avoids likely private account/admin/cart/login URLs by default.
• Ships zero hosted dependencies: no account, API key, browser session, or LLM call is required.

What it does not do

Cockroach Crawler is not a stealth scraper. It does not include bypass tooling for login walls, paywalls, CAPTCHA, anti-bot systems, authorization boundaries, or robots.txt. If a site owner requires permission, get permission and crawl within their terms.

When to use it

Use Cockroach Crawler when an agent or data pipeline needs a repeatable public-web fetcher that returns compact, structured page records. It is intentionally smaller than full crawler frameworks and intentionally simpler than browser agents.

| Use case | Cockroach Crawler fit | | --- | --- | | Crawl public docs, blogs, help centers, or marketing pages | Strong | | Export crawl results as JSONL or markdown for RAG | Strong | | Respect robots.txt and crawl-delay style politeness controls | Strong | | Crawl JavaScript-only apps that require clicks, login, or a live browser | Use Browser Use, Playwright, Puppeteer, Crawlee, or another browser stack | | Bypass paywalls, CAPTCHA, anti-bot, auth, or owner restrictions | Not supported |

Comparison

Cockroach Crawler is not a universal replacement for Browser Use, Crawlee, Scrapy, Firecrawl, Crawl4AI, or hosted browser tools. It is better for one narrow job: low-friction public HTML crawling into agent-friendly JSON/JSONL/Markdown from Node.js.

| Tool | Best at | Tradeoff compared with Cockroach Crawler | | --- | --- | --- | | Browser Use | LLM-driven browser control and interactive workflows | Heavier stack; better for clicking/forms, not a lightweight batch crawler | | Crawlee | Production-grade crawling with browser and queue primitives | More powerful, larger API surface, more setup | | Scrapy | Mature Python crawling framework | Excellent framework, but Python-first and less direct for Node agent pipelines | | Firecrawl | Hosted/API-first page extraction and crawling | Great API product; self-hosting/licensing/deployment is heavier | | Crawl4AI | LLM-oriented Python crawling and markdown extraction | Strong AI extraction focus; Python-first | | Cockroach Crawler | Local public-web crawl to JSONL/Markdown for agents | Smaller scope; no JS browser rendering, no stealth, no hosted extraction |

The goal is not to beat every crawler at every job. The goal is to be the simplest useful crawler an agent can call locally when it needs public pages converted into clean records.

Install

npm install -g cockroach-crawler

Or run without global install:

npx cockroach-crawler https://example.com --max-pages 20 --jsonl

CLI examples

Crawl one public site:

cockroach-crawl https://example.com --max-pages 50 --jsonl --output crawl.jsonl

Use sitemaps and include only docs URLs:

cockroach-crawl https://example.com --sitemaps --include "/docs/" --max-pages 200 --output docs.json

Read many seed URLs from a file:

cockroach-crawl --url-file urls.txt --max-pages 100 --jsonl

Add a contact-aware user agent:

cockroach-crawl https://example.com --contact "mailto:[email protected]"

CLI options

• --url-file <file>: read seed URLs from a text file, one URL per line.
• --max-pages <n>: maximum pages to return. Default: 50.
• --max-depth <n>: maximum link depth from seeds. Default: 2.
• --concurrency <n>: concurrent workers. Default: 4.
• --delay <ms>: minimum delay per origin. Default: 250.
• --timeout <ms>: request timeout. Default: 15000.
• --sitemaps: discover URLs from robots.txt sitemaps and /sitemap.xml.
• --all-origins: allow crawling across discovered origins.
• --include <regex>: only crawl URLs matching regex. Can be repeated.
• --exclude <regex>: skip URLs matching regex. Can be repeated.
• --allow-non-public: allow likely login/account/admin/cart URLs.
• --jsonl: output JSON Lines instead of a JSON array.
• --output <file>: write output to a file.
• --user-agent <ua>: custom user agent.
• --contact <email/url>: add contact detail to the default user agent.
• --version: show package version.

Library API

import { crawl } from "cockroach-crawler";

const pages = await crawl({
  seeds: ["https://example.com"],
  maxPages: 25,
  maxDepth: 2,
  concurrency: 4,
  includeSitemaps: true,
  include: ["/docs/"],
  exclude: ["/login", "/account"],
  onPage(page) {
    console.log(page.url, page.title);
  }
});

console.log(pages[0].markdown);

Output shape

{
  "url": "https://example.com/",
  "canonical": "https://example.com/",
  "title": "Example",
  "description": "Example description",
  "h1": "Example",
  "text": "Readable text...",
  "markdown": "# Example\n\nReadable markdown...",
  "links": ["https://example.com/about"],
  "fetchedAt": "2026-06-27T00:00:00.000Z",
  "status": 200,
  "contentType": "text/html; charset=utf-8"
}

Development

npm install
npm test
npm run check
npm run bench

License

MIT