Web structure

A powerful and flexible web scraping library built with TypeScript and Puppeteer. It supports concurrent scraping, recursive crawling, and intelligent content extraction with DOM hierarchy awareness.

Features

• Concurrent Processing: Parallel processing of multiple selectors and pages
• DOM Hierarchy Aware: Smart content extraction that respects DOM structure
• Recursive Crawling: Ability to crawl through child pages with depth control
• Flexible Selectors: Support for both single and multiple CSS selectors
• Retry Mechanism: Built-in retry with exponential backoff for reliability
• Deduplication: Automatic deduplication of content and URLs
• Structured Output: Clean, structured JSON output with metadata

Installation

npm install web-structure

Quick Start

import { scraping } from 'web-structure';

// Basic usage
const result = await scraping('https://example.com');

// Advanced usage with options
const result = await scraping('https://example.com', {
  maxDepth: 2,
  selectors: {
    headings: ['h1', 'h2', 'h3'],
    content: '.article-content',
    links: 'a.important-link'
  },
  excludeChildPage: (url) => url.includes('login'),
  withConsole: true
});

Configuration Options

| Option | Type | Default | Description | |--------|------|---------|-------------| | maxDepth | number | 0 | Maximum depth for recursive crawling | | excludeChildPage | (url: string) => boolean | () => false | Function to determine if a URL should be skipped | | selectors | { [key: string]: string \| string[] } | See below | Selectors to extract content | | withConsole | boolean | true | Whether to show console information | | breakWhenFailed | boolean | false | Whether to break when a page fails | | retryCount | number | 3 | Number of retries when scraping fails | | waitForSelectorTimeout | number | 12000 | Timeout for waiting for a selector (ms) | | waitForPageLoadTimeout | number | 12000 | Timeout for waiting for page load (ms) |

Default Selectors

{
  headings: ['h1', 'h2', 'h3', 'h4', 'h5'],
  paragraphs: 'p',
  articles: 'article',
  spans: 'span',
  orderLists: 'ol',
  lists: 'ul'
}

Output Structure

interface ScrapingResult {
  url: string;          // URL of the scraped page
  title: string;        // Page title
  data: {              // Extracted content
    [key: string]: string | string[];
  };
  timestamp: string;    // ISO timestamp of when the page was scraped
  childPages?: ScrapingResult[]; // Results from child pages (if maxDepth > 0)
}

Advanced Features

DOM Hierarchy Awareness

The library intelligently handles nested elements to prevent duplicate content. If a parent element is selected, its child elements won't be included separately in the results.

Concurrent Processing

• Multiple selectors are processed concurrently
• Array selectors (e.g., ['h1', 'h2', 'h3']) are processed in parallel
• Child pages are processed sequentially to prevent overwhelming the target server

Retry Mechanism

Built-in retry mechanism with exponential backoff:

• Retries failed operations with increasing delays
• Configurable retry count
• Includes random jitter to prevent thundering herd problems

Error Handling

The library provides robust error handling:

• Failed selector extractions don't stop the entire process
• Each selector and page has independent error handling
• Detailed error logging when withConsole is enabled
• Option to break on failures with breakWhenFailed

Limitations

• Maximum crawling depth is limited to 10 levels
• Maximum of 5 child links per page are processed
• Respects robots.txt and rate limiting by default
• Requires JavaScript to be enabled on target pages

License

MIT