Crawlingo Node.js SDK is a next-generation web data extraction, crawling, and website monitoring library. It wraps a high-performance Rust core in an elegant React-inspired developer-first Node.js API with complete TypeScript support, allowing you to build scraping workflows that survive page design shifts.

📚 Read the full guide and API references at crawlingo.dev/docs

🎥 30-Second Demo

Watch Crawlingo's self-healing DOM selector engine dynamically recover element references when a website's layout/DOM structure drifts:

How Self-Healing Works Under the Hood:

1. Drift Detection: When the target element (e.g., button#submit.btn-primary) undergoes styling or structure updates (e.g., renamed to button#send-btn.btn-primary-new), traditional scrapers fail and return empty results.
2. Dynamic DOM Parsing: Crawlingo's Rust engine intercepts the mismatch, loads the active DOM, and isolates candidates within the parent node coordinates.
3. Jaro-Winkler Similarity Comparison: The engine ranks candidates by checking tag names, surrounding attributes, text contents, and deep structural fingerprints.
4. Auto-Match Recovery: The candidate with the highest similarity score exceeding the threshold (e.g., 94% confidence) is automatically bound, updating the cache without breaking your production data pipeline.

📦 Installation

Install the package directly from npm:

npm install crawlingo

🚀 Why Crawlingo?

Traditional scrapers break when websites change their class names, IDs, or HTML structures (selector drift). Crawlingo solves this by caching element layout fingerprints and using similarity matching heuristics to self-heal and find drifted elements on the fly.

Comparison Matrix

| Feature | Crawlingo | Scrapy | Crawl4AI | |----------|------------|---------|---------| | Rust Core | ✅ | ❌ | ❌ | | Python SDK | ✅ | ✅ | ✅ | | Node SDK | ✅ | ❌ | ❌ | | AI Agent Ready | ✅ | ⚠️ | ✅ | | Change Monitoring | ✅ | ❌ | ❌ | | Dataset Extraction | ✅ | ⚠️ | ⚠️ | | Cross Language | ✅ | ❌ | ❌ |

🛠️ Core Features

Crawlingo packs all components required to scrape, watch, and pipe modern web pages under JavaScript/TypeScript:

• 🧠 Self-Healing DOM Fingerprinting: Tracks layout changes and leverages Jaro-Winkler calculations dynamically. Learn more.
• 🛡️ Stealth Browser Impersonation: Bypasses bot verification systems (Cloudflare, etc.) using high-performance HTTP/2 TLS fingerprint rotation. Learn more.
• ⚡ SIMD-Accelerated Text Anchors: CSS/XPath is great, but anchoring relative to text values using vector calculations is faster. Learn more.
• 🔄 High-Speed Proxy Rotation: Automatically rotates proxy configurations inside background crawling loops. Learn more.
• ⏰ Reactive Watch Monitors: Run background threads that poll websites and notify handlers upon layout shifts or price changes. Learn more.
• 🤖 Built-in MCP Server: Native server that connects scraping tool functions straight to Claude Code or Cursor. Learn more.
• 📦 Schema-Driven Datasets: Map results and export them straight to JSON, CSV, Apache Arrow, or Pandas DataFrames. Learn more.

⚡ Quick Start

1. Basic Web Scrape

import { Session, Page } from 'crawlingo';

const session = new Session();
session.autoMatch(true);

const page = await Page.create("https://example.com", { session });
console.log("Page Title:", page.title());

const headings = page.css("h1");
console.log("Header text:", headings.text.join(", "));

2. Multi-Page Crawling & Webhooks

import { Session, Crawl } from 'crawlingo';

const session = new Session();
session.proxyPool([
  "http://proxy1.example.com:8080",
  "http://proxy2.example.com:8080"
]);

const crawl = new Crawl("https://example.com/products", session);
crawl.follow("a.next-page");
crawl.field("title", "h1");

// Deliver items to a webhook endpoint in real-time
crawl.webhook("https://my-api.com/webhooks/crawl");

// Run background crawl loops every hour
crawl.schedule(3600);

🤖 AI LLM Ingestion & Benchmarking

For web parsing pipelines feeding LLM context or RAG indices, Crawlingo provides structured inputs. The table below outlines how different AI models compare on processing raw scraped web pages for automated RAG/extraction tasks:

| Model | Context Window | Speed (tok/s) | Avg. Cost / 1M Tok | Markdown Parsing Accuracy | Native MCP Support | |-----------|----------------|---------------|--------------------|---------------------------|---------------------| | Claude 3.5 Sonnet | 200k | ~80 | $3.00 / $15.00 | 👑 98% (Best for tables/JSON) | ✅ Native | | GPT-4o | 128k | ~90 | $2.50 / $10.00 | 95% (Excellent formatting) | ✅ Via Gateway | | Gemini 1.5 Pro | 2M | ~60 | $1.25 / $5.00 | 92% (Huge content ingestion) | ⚠️ Experimental | | Llama 3.1 70B | 128k | ~45 | $0.60 / $0.60 | 88% (Great open-source alternative) | ❌ Needs wrapper |

📝 License

MIT License. See LICENSE file.