npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2026 – Pkg Stats / Ryan Hefner

@xberg-io/opencode-crawlberg

v1.13.1

Published

Web crawling and scraping with HTML→Markdown and headless-Chrome fallback.

Readme

crawlberg

Crawl, scrape, and convert websites to Markdown using the local crawlberg CLI in your agent.

Install

From the marketplace (recommended)

Pending review for official Claude marketplace.

Self-host:

/plugin marketplace add xberg-io/crawlberg
/plugin install crawlberg@crawlberg

Kimi installation and compatibility

From a checkout of this repository, install with the Python Kimi CLI version used for native plugin validation:

uvx --from kimi-cli==1.51.0 kimi plugin install ./plugin

Native plugin installation was validated with Python Kimi CLI 1.51.0. The replacement @moonshot-ai/kimi-code 2.1.1 does not provide a plugin CLI subcommand; its interactive plugin loading has not been verified. These results do not establish compatibility with newer Kimi releases.

Pi

From a checkout of this repository:

pi install ./plugin

After the next plugin release, install the same package used by OpenCode:

pi install npm:@xberg-io/opencode-crawlberg

Pi loads the bundled skills and prompt templates. Use /crawlberg-scrape or /crawlberg-crawl with the input as arguments. Install the crawlberg CLI separately for shell access. This package does not register MCP servers, hooks, or native Pi extensions.

Binary requirement

The MCP server runs through an auto-installing launcher (scripts/mcp-launch.sh): on first use it reuses any working crawlberg binary already on PATH, then tries npx, Homebrew, or a prebuilt download. No manual install is required for MCP.

To install the CLI yourself (recommended for direct CLI use, and reused by the launcher):

brew install xberg-io/tap/crawlberg
# or run it without a persistent install (the CLI proxy package self-installs the binary):
pnpm dlx @xberg-io/crawlberg-cli --help
# or build from source (the mcp/api subcommands are non-default features):
cargo install crawlberg-cli --features all

The published npm (@xberg-io/crawlberg) and PyPI (crawlberg) packages are language library bindings, not the CLI — install via Homebrew or the from-source build above for the crawlberg binary.

Headless fallback requires Chrome/Chromium on your system. The CLI launches it on demand; skip the binary if you only plan to use --browser-mode never.

Skills shipped

| Skill | Trigger | |-------|---------| | crawlberg | Crawl, scrape, and convert websites to Markdown using the local crawlberg CLI and its MCP server. Use when the user wants to fetch a page, follow links across a domain, enumerate URLs, or drive a real browser. Covers installation, the subcommands (scrape, crawl, map, interact, batch-scrape, batch-crawl, download, citations, version, mcp, serve), output formats (JSON + Markdown), browser fallback, and when to prefer the MCP server over shelling out. | | crawlberg-crawling-a-site | Use when the user wants to follow links across a domain and capture every reachable page as Markdown. Covers crawlberg crawl with depth, page caps, concurrency, rate limiting, domain scoping, robots, and output selection. | | crawlberg-scraping-html-to-markdown | Use when the user wants a single page rendered as clean Markdown plus structured metadata. Covers crawlberg scrape <url>, JSON vs Markdown output, what metadata is returned, and how to handle JS-heavy pages. | | crawlberg-mapping-urls | Use when the user wants the list of URLs on a site rather than the page content — sitemap analysis, link planning, or seeding another tool. Covers crawlberg map <url> with --limit, --search, robots, output, and how it differs from a full crawl. | | crawlberg-automating-the-browser | Use when extracting a page needs scripted interaction first — click, type, press a key, scroll, wait, screenshot, or run JS before capturing the DOM. Covers crawlberg interact <url> --actions with the real action schema, result shape, limits, and external-CDP options. | | crawlberg-serving-the-api | Use when the user wants a long-running HTTP service for scrape/crawl/map instead of one-shot CLI calls or the MCP server. Covers crawlberg serve, the Firecrawl-v1-compatible endpoints, --host/--port, and when to prefer it. | | crawlberg-headless-fallback | Use when a static fetch returns nothing useful and the page needs a real browser. Covers --browser-mode auto\|always\|never, external CDP via --browser-endpoint, symptoms of JS-only pages and WAF blocks, and the performance cost. |

MCP / CLI

MCP-capable integrations wire up the crawlberg MCP server via scripts/mcp-launch.sh, which resolves an existing binary or installs the latest available CLI, then runs crawlberg mcp over stdio. Override binary resolution with CRAWLBERG_LAUNCHER=auto|download.

The MCP server exposes nine tools:

  • scrape — fetch and convert a single URL to Markdown or JSON.
  • crawl — follow links across a domain, bounded by depth and page count.
  • map — enumerate URLs from sitemaps and link extraction.
  • batch_scrape — scrape multiple URLs concurrently.
  • batch_crawl — crawl multiple seed URLs concurrently.
  • download — download a document from a URL and report its file metadata.
  • interact — drive a headless browser with click, type, scroll actions.
  • generate_citations — rewrite markdown links as numbered citations with a reference list.
  • get_version — report the crawlberg library version.

The CLI offers the same operations as subcommands — scrape, crawl, map, interact, batch-scrape, batch-crawl, download, citations, version — plus serve (a Firecrawl-v1-compatible REST API server) and mcp (the stdio MCP server). See the crawlberg skill for the full per-subcommand flag surface, and the crawlberg-serving-the-api skill for when to run the server instead of the CLI or MCP.

Configuration

Pass flags or use inline JSON via --config:

crawlberg scrape https://example.com \
  --format markdown \
  --browser-mode auto \
  --timeout 30000

For complex configs, use JSON:

crawlberg crawl https://example.com \
  --config '{"max_depth":3,"max_pages":200,"max_concurrent":8,"respect_robots_txt":true}'

See the crawlberg and crawlberg-crawling-a-site skills for the full flag surface.

Examples

Fetch a single page and print Markdown:

crawlberg scrape https://example.com/article --format markdown

Crawl a site at depth 3 with rate limiting:

crawlberg crawl https://example.com --depth 3 --max-pages 200 --concurrent 8 --stay-on-domain --format markdown

Enumerate URLs from a sitemap:

crawlberg map https://example.com --limit 500

Versioning

The plugin version tracks the crawlberg core release (Cargo.toml). See CHANGELOG.md for release notes.

License

MIT.

See also

  • Marketplace: this repo — /plugin marketplace add xberg-io/crawlberg
  • Upstream: xberg-io/crawlberg

Generated runtime bundles

plugin/.ai-rulez/config.toml loads config.plugins.toml through the plugins layer; skills, commands and runtime adapters are authored alongside it. The latest AI-Rulez release (5.7.0 or newer) generates all eleven supported runtimes: Claude, Cursor, Codex, Gemini, Kimi, OpenCode, Factory, Hermes, Pi, Agent Plugins and Copilot. Codex ships both manifest layouts; repository-root Claude and Codex marketplace indexes point to the plugin bundle. Cursor also has a marketplace index.

Pi packages skills and prompts without MCP integration. Copilot packages skills and MCP configuration, but does not package slash commands or hooks. Hermes includes a Python package under .hermes/package; OpenCode and Pi share the npm package. The Agent Plugins manifest uses the stable 1.0 specification. Runtime capabilities differ; install the CLI before using skills in a runtime without MCP registration.

Run ai-rulez generate --plugin from the repository root to regenerate and ai-rulez verify --plugin to compare current source output and provenance. The release workflow packages every runtime into a deterministic archive, checksums, and marketplace metadata; npm and PyPI retain their existing publishing jobs.

Hermes runtime setup

Install the product CLI with MCP support and put it on Hermes' PATH before using the plugin. The Hermes adapter registers every bundled skill under the product namespace and provides crawlberg_call, which calls the selected native MCP tool with tool_arguments through a local stdio session. Select tool: "list_tools" first to inspect the server's exact request schemas. Hermes skills are selected explicitly; they are not automatically appended to the system prompt. Enable the plugin in Hermes configuration; project plugins also require HERMES_ENABLE_PROJECT_PLUGINS=true.

The bridge uses the installed CLI directly. The auto-installing launchers used by other MCP clients are not included in the Hermes Python wheel. Provider credentials are inherited from the environment, and are needed for provider-backed requests. Missing CLIs, unavailable tools and model preparation fail with an actionable request error. CLI result data remains untrusted document content.

Public Agent Skills

The repository-root skills/ directory contains the same generated skills and supporting resources as plugin/skills/. Install them independently of a plugin runtime:

npx skills add xberg-io/crawlberg --skill crawlberg

Edit plugin/.ai-rulez/skills, then run mise run plugin:generate to regenerate runtime bundles and synchronize the public export. mise run plugin:verify checks both projections; mise run plugin:skills:test verifies resource freshness, ownership and symlink protection. The public ownership manifest records SHA-256 hashes of every exported file.

Download integrity: the launcher currently relies on HTTPS for direct binary downloads. Verify release checksums separately when available before manually installing an archive.

Launcher and visual assets are authored in plugin/.ai-rulez/scripts/ and plugin/.ai-rulez/assets/; mise run plugin:generate copies them into the runtime bundle.