AgentCrawl

The High-Performance TypeScript Web Scraper for LLM Agents.

AgentCrawl is built to be the "eyes" of your AI Agents. It fetches web content, strips away the noise (ads, scripts, styles), and returns clean, token-optimized Markdown ready for your LLM context window.

It features a Hybrid Engine that starts with extremely fast static scraping and automatically falls back to a headless browser (Playwright) only when necessary for dynamic content or authentication.

Features

• 🚀 Hybrid Engine: Instant static fetch by default, auto-switch to Headless Browser for dynamic sites.
• ⚡ Token Optimized: Returns clean Markdown, stripping 80-90% of tokens (ads, navs, footers).
• 🧠 Agent-First: Detects Main Content, removes boilerplate, and extracts semantic structure.
• 🔌 Plug-and-Play: Simple API designed for agent runtimes (scrape + crawl).
• 🛡️ Production Ready: Built-in caching, retry logic, user-agent rotation, and resource blocking.
• 🕵️ Stealth Mode: Optional best-effort browser hardening to reduce common bot-detection fingerprints.
• ✅ Predictable Errors: Non-2xx HTTP responses are surfaced as errors instead of silently parsed as success.
• 🇹 Type-Safe: 100% TypeScript with Zod validation.

Installation

npm install agent-crawl
# OR
bun add agent-crawl

Quick Start

Basic Usage

import { AgentCrawl } from 'agent-crawl';

// Simplest usage - returns clean properties
const page = await AgentCrawl.scrape("https://example.com");

console.log(page.title);   // "Example Domain"
console.log(page.content); // "Example Domain\n\nThis domain is for use..."
console.log(page.links);   // Array of same-origin links found on the page

Advanced Usage (Optimized for LLMs)

const page = await AgentCrawl.scrape("https://news.ycombinator.com", {
  mode: "hybrid",            // "static" | "browser" | "hybrid" (default)
  extractMainContent: true,  // Extract only the article body
  optimizeTokens: true,      // Compress excessive whitespace (default: true)
  stealth: true,             // Enable browser stealth hardening when browser is used
  stealthLevel: "balanced",  // "basic" | "balanced" (default: "balanced")
  waitFor: ".main-content",  // CSS selector to wait for (browser mode)
});

Crawling Multiple Pages

Crawl an entire website with configurable depth, page limits, and concurrency:

const result = await AgentCrawl.crawl("https://docs.example.com", {
  maxDepth: 2,        // How many link-hops from start URL (default: 1)
  maxPages: 20,       // Stop after N pages (default: 10)
  concurrency: 4,     // Parallel requests (default: 2)
  extractMainContent: true,
});

console.log(`Crawled ${result.totalPages} pages`);
console.log(`Max depth reached: ${result.maxDepthReached}`);
result.pages.forEach(page => {
  console.log(`- ${page.title}: ${page.url}`);
});

Configuration

Scrape Options

| Option | Type | Default | Description | |--------|------|---------|-------------| | mode | 'hybrid' | 'static' | 'browser' | 'hybrid' | Strategy to use. hybrid tries static first, then browser. | | extractMainContent | boolean | false | Extract only the main article body using Readability-like algorithm. | | optimizeTokens | boolean | true | Remove extra whitespace and empty links for token efficiency. | | stealth | boolean | false | Apply best-effort browser stealth hardening (browser mode only). | | stealthLevel | 'basic' | 'balanced' | 'balanced' | Stealth profile strength when stealth is enabled. | | waitFor | string | undefined | CSS selector to wait for (browser mode only). | | maxResponseBytes | number | undefined | Best-effort cap for static fetch response size in bytes. | | httpCache | boolean \| { dir?, ttlMs?, maxEntries? } | undefined | Opt-in disk HTTP cache for static fetch (ETag/Last-Modified). | | cache | boolean \| { dir?, ttlMs?, maxEntries? } | undefined | Opt-in disk cache for processed scrape results (ScrapedPage). | | chunking | boolean \| { enabled?, maxTokens?, overlapTokens? } | undefined | Opt-in token-aware chunking (page.chunks) with citation anchors. |

Disk Cache Example (HTTP + Processed Result)

const page = await AgentCrawl.scrape("https://example.com", {
  mode: "static",
  httpCache: { dir: ".cache/agent-crawl/http", ttlMs: 60_000, maxEntries: 1000 },
  cache: { dir: ".cache/agent-crawl", ttlMs: 5 * 60_000, maxEntries: 1000 },
});

Chunking Example (For Agent RAG/Tools)

const page = await AgentCrawl.scrape("https://example.com", {
  chunking: { enabled: true, maxTokens: 1200, overlapTokens: 100 },
});

// page.chunks: [{ id, text, approxTokens, headingPath, citation: { url, anchor } }, ...]

Crawl Options

Crawl options include all scrape options plus:

| Option | Type | Default | Description | |--------|------|---------|-------------| | maxDepth | number | 1 | Maximum link depth to crawl from the start URL. | | maxPages | number | 10 | Maximum number of pages to crawl. | | concurrency | number | 2 | Number of pages to fetch in parallel. | | perHostConcurrency | number | concurrency | Maximum concurrent requests per host. | | minDelayMs | number | 0 | Minimum delay between requests to the same host. | | includePatterns | string[] | [] | Only crawl URLs containing any of these substrings. | | excludePatterns | string[] | [] | Do not crawl URLs containing any of these substrings. | | robots | boolean \| { enabled?, userAgent?, respectCrawlDelay? } | undefined | Opt-in robots.txt compliance (Disallow/Allow + Crawl-delay). | | sitemap | boolean \| { enabled?, maxUrls? } | undefined | Opt-in sitemap seeding from /sitemap.xml. | | crawlState | boolean \| { enabled?, dir?, id?, resume?, flushEvery?, persistPages? } | undefined | Opt-in resumable crawl state persisted to disk. |

Polite Crawl Example (Robots + Sitemap + Throttling)

const result = await AgentCrawl.crawl("https://docs.example.com", {
  maxDepth: 2,
  maxPages: 100,
  concurrency: 6,
  perHostConcurrency: 2,
  minDelayMs: 250,
  robots: { enabled: true, userAgent: "agent-crawl", respectCrawlDelay: true },
  sitemap: { enabled: true, maxUrls: 1000 },
});

Resumable Crawl Example

const result = await AgentCrawl.crawl("https://docs.example.com", {
  maxDepth: 3,
  maxPages: 500,
  concurrency: 6,
  crawlState: {
    enabled: true,
    dir: ".cache/agent-crawl/state",
    id: "docs-example",
    resume: true,
    flushEvery: 5,
    persistPages: true,
  },
});

Return Values

scrape() → ScrapedPage

{
  url: string;           // The final URL (after redirects)
  content: string;       // Clean markdown content
  title?: string;        // Page title
  links?: string[];      // Same-origin links found on the page
  chunks?: Array<{       // Present only when chunking is enabled
    id: string;
    text: string;
    approxTokens: number;
    headingPath: string[];
    citation: { url: string; anchor?: string };
  }>;
  metadata?: {
    status: number;      // HTTP status code
    contentLength: number;
    error?: string;      // Populated when scrape fails
    structured?: {       // Structured metadata from HTML (when present)
      canonicalUrl?: string;
      openGraph?: Record<string, string>;
      twitter?: Record<string, string>;
      jsonLd?: unknown[];
    };
    // ... other headers
  }
}

scrape() returns an empty content plus metadata.error for non-2xx responses or fetch/browser failures.

When browser rendering is used, metadata also includes:

• stealthApplied: boolean
• stealthLevel?: "basic" | "balanced" (when stealth is enabled)

Stealth Mode (Best-Effort)

• Stealth is opt-in via stealth: true.
• It is applied only for browser rendering (mode: "browser" and hybrid browser fallback).
• It hardens common automation fingerprints (navigator.webdriver, language/plugins/platform hints, permission query behavior, and browser headers/profile).
• It is best-effort: some anti-bot systems may still block requests.

crawl() → CrawlResult

{
  pages: ScrapedPage[];  // Array of all scraped pages
  totalPages: number;    // Total number of pages crawled
  maxDepthReached: number; // Deepest level reached
  errors: Array<{        // Any errors encountered
    url: string;
    error: string;
  }>;
}

License

MIT © silupanda