kreuzcrawl

WebAssembly bindings for kreuzcrawl — a high-performance Rust web crawling engine. Built with wasm-bindgen for use in browsers and Node.js. Single-threaded; features requiring std::thread are not available.

What This Package Provides

• Same crawler as every binding — one Rust engine behind Python, Node.js, Ruby, Go, Java, .NET, PHP, Elixir, Dart, Kotlin Android, Swift, Zig, WASM, and C FFI.
• Structured scrape output — HTML, Markdown, metadata, links, assets, response headers, and extraction warnings with consistent field names.
• Crawl controls — depth, page limits, concurrency, URL filters, robots/sitemap handling, rate limits, and partial failure reporting.
• Rendering path — optional browser rendering for JavaScript-heavy pages; direct HTTP path for fast static pages.
• WASM package — browser and edge-compatible binding for environments where native libraries are unavailable.

Installation

pnpm add @kreuzberg/kreuzcrawl-wasm

Agent plugin

The kreuzcrawl plugin is available via the kreuzberg-dev/plugins marketplace.

/plugin marketplace add kreuzberg-dev/plugins
/plugin install kreuzcrawl@kreuzberg

Works with Claude Code, Codex, Cursor, Gemini CLI, Factory Droid, GitHub Copilot CLI, and opencode. See the marketplace README for harness-specific install instructions.

Quick Start

import { CrawlConfig, crawl, createEngine, scrape } from "@kreuzberg/kreuzcrawl-wasm";

async function main() {
  // Simplest case: scrape a single page with default settings.
  const engine = createEngine();
  const result = await scrape(engine, "https://example.com/");
  console.log(`Title: ${result.metadata?.title ?? ""}`);
  console.log(`Status: ${result.statusCode}`);
  console.log(`Links found: ${result.links?.length ?? 0}`);

  // Crawl from a seed URL, limited to one hop and a handful of pages.
  const config = new CrawlConfig();
  config.maxDepth = 1;
  config.maxPages = 5;
  const crawlEngine = createEngine(config);
  const crawlResult = await crawl(crawlEngine, "https://en.wikipedia.org/wiki/Web_scraping");
  console.log(`Pages crawled: ${crawlResult.pages?.length ?? 0}`);
}

main().catch((error) => console.error(error));

API Reference

Full API documentation is available at docs.kreuzcrawl.kreuzberg.dev.

Key functions:

• create_engine(config?) — Create a crawl engine with optional configuration
• scrape(engine, url) — Scrape a single URL
• crawl(engine, url) — Crawl a website following links
• map_urls(engine, url) — Discover all pages on a site
• batch_scrape(engine, urls) — Scrape multiple URLs concurrently
• batch_crawl(engine, urls) — Crawl multiple seed URLs concurrently

Contributing

Contributions are welcome! Please see our Contributing Guide for details.

Part of Kreuzberg.dev

• Kreuzberg — document intelligence: text, tables, metadata from 91+ formats with optional OCR.
• Kreuzberg Cloud — managed extraction API with SDKs, dashboards, and observability.
• kreuzcrawl — web crawling and scraping with HTML→Markdown and headless-Chrome fallback.
• html-to-markdown — fast, lossless HTML→Markdown engine.
• liter-llm — universal LLM API client with native bindings for 14 languages and 143 providers.
• tree-sitter-language-pack — tree-sitter grammars and code-intelligence primitives.
• alef — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos.
• Discord — community, roadmap, announcements.

License

This project is licensed under Elastic License 2.0.

Links

• Documentation
• GitHub Repository
• Issue Tracker