ag-webscrape

A generic TypeScript web scraper with an intelligent fallback strategy. It attempts a direct HTTP fetch first, then falls back to a headless browser (puppeteer-core) for anti-scraping protection.

This package is intentionally site- and domain-agnostic. It provides fetching, browser rendering, XPath/DOM helpers, content sanitisation and anti-bot detection only; product/plant semantics belong in the consumer.

Features

• Dual Strategy: Direct fetch first, headless-browser fallback
• Anti-Scraping Detection: Automatically detects Cloudflare, Akamai, DataDome, PerimeterX, human-verification pages and generic 403/429 blocks
• Persistent Browser: Maintains a browser instance for faster subsequent scrapes
• Generic Helpers: XPath over node-html-parser, content cleaning/validation, DOM navigation
• Error Handling: Typed result with block provider and content classification
• TypeScript Support: Full type safety and IntelliSense

Installation

npm install ag-webscrape

Quick Start

import { WebScraper } from 'ag-webscrape';

const scraper = new WebScraper();

const result = await scraper.scrape('https://example.com');
console.log(result.html, result.contentType, result.blockedBySecurity);

await scraper.dispose();

API Reference

WebScraper

new WebScraper(options?: ScrapingOptions)

interface ScrapingOptions {
  timeout?: number;                 // Whole-scrape timeout in ms (default: 30000)
  userAgent?: string;               // Custom user agent
  headers?: Record<string, string>; // Additional headers
  forceBrowser?: boolean;           // Start with Chromium instead of direct fetch
  includeResponseBody?: boolean;    // Include the main response body as rawBody
  maxResponseBytes?: number;        // Bound a response body when it is read
  retries?: number;                 // Number of retries (default: 3)
  waitForSelector?: string;         // Selector to wait for in the browser
  waitForTimeout?: number;          // Browser wait in ms (default: 5000)
  executablePath?: string;          // Explicit browser executable
  validateUrl?: (url: URL) =>       // Guard every direct/browser destination
    void | Promise<void>;
  validationTimeoutMs?: number;     // Per-destination guard timeout
  deadlineAt?: number;              // Optional absolute epoch-ms cutoff
}

interface ScrapingResult {
  url: string;
  html: string;
  rawBody?: string;                 // Original main response body when requested
  status: number;
  method: 'fetch' | 'visual';
  error?: string;
  redirected?: boolean;
  finalUrl?: string;
  contentType: 'target' | 'challenge' | 'empty' | 'error' | 'unknown';
  blockedBySecurity: boolean;
  blockProvider?: 'cloudflare' | 'akamai' | 'datadome' | 'perimeterx' | 'unknown';
  blockReason?: string;
  challengeSnippet?: string;
}

Browser fallback uses the configured user agent and applies extra headers only to same-origin main-frame navigations, so credentials are not copied to third-party subresources. With includeResponseBody, callers can access the original navigation payload in rawBody (including Atom/XML that Chromium wraps in an HTML viewer). Set maxResponseBytes to bound that payload. forceBrowser skips the initial direct fetch. Browser selection prefers an explicit executablePath or executable path environment variable, then Chromium installed under PLAYWRIGHT_BROWSERS_PATH, then a detected system browser, and finally the bundled serverless Chromium runtime.

Methods: scrape(url, options?), scrapeMultiple(urls, options?), dispose().

validateUrl runs before the initial request, before every direct-fetch redirect, and before Chromium continues each HTTP(S) request. Throw to block the destination. A rejection becomes UrlValidationError and never falls through to another request strategy. validationTimeoutMs bounds each guard call and defaults to the request or navigation timeout. Consumers performing SSRF protection should resolve the hostname and reject loopback, private, link-local, reserved, credential-bearing, and unapproved protocol/port destinations in this hook.

timeout is one budget shared by direct fetch (including the response body), browser fallback, browser launch, navigation, request validation, and retry. deadlineAt can shorten that budget for callers with an external execution deadline. goToPage applies the same whole-call semantics and rejects with ScrapingTimeoutError when it expires.

Helpers

• DOM: goToPage, launchBrowser, closeBrowser, GoToPageOptions, PageResult, ScrapingTimeoutError
• Request safety: UrlValidator, UrlValidationError, continueValidatedBrowserRequest
• XPath: selectXPath, nodeInnerText, nodeDirectInnerText, nodeAttribute, nodeElement, nodeStringValue, XPathNode
• Content: cleanHtml, cleanText, decodeHtmlEntities, isValidContent, ELEMENTS_TO_REMOVE, ATTRIBUTES_TO_REMOVE, PREVENTION_SELECTORS, CONTEXTUAL_PATTERNS

How It Works

1. Direct Fetch: HTTP request with a strict timeout
2. Anti-Scraping Detection: Scans the response for challenge/block markers
3. Browser Fallback: On failure or a detected block, renders with a headless browser
4. Resource Management: Reuses the browser instance across scrapes

License

MIT