Simple Web Scraper

Live Documentation

A lightweight and efficient web scraping package for JavaScript/TypeScript applications. This package helps developers fetch HTML content, parse web pages, and extract data effortlessly.

✨ Features

• ✅ Fetch Web Content – Retrieve HTML from any URL with ease.
• ✅ Parse and Extract Data – Utilize integrated parsing tools to extract information.
• ✅ Configurable Options – Customize scraping behaviors using CSS selectors.
• ✅ Headless Browser Support – Optionally use Puppeteer for JavaScript-rendered pages.
• ✅ Lightweight & Fast – Uses Cheerio for static HTML scraping.
• ✅ TypeScript Support – Fully typed for robust development.
• ✅ Data Export Support – Export scraped data in JSON or CSV formats.
• ✅ CSV Import Support – Read CSV files and convert them to JSON.

📚 Installation

Install via npm:

npm install simple-web-scraper

or using Yarn:

yarn add simple-web-scraper

🚀 Why Use Cheerio and Puppeteer?

This package leverages Cheerio and Puppeteer for powerful web scraping capabilities:

🔹 Cheerio (Fast and Lightweight)

• Ideal for static HTML parsing (like jQuery for the backend).
• Extremely fast and lightweight – perfect for pages without JavaScript rendering.
• Provides easy CSS selector querying for extracting structured data.

🔹 Puppeteer (Headless Browser Automation)

• Handles JavaScript-rendered pages – essential for scraping dynamic content.
• Can interact with pages, click buttons, and fill out forms.
• Allows screenshot capturing, PDF generation, and full-page automation.

✅ Best of Both Worlds

• Use Cheerio for speed when scraping static pages.
• Switch to Puppeteer for JavaScript-heavy sites requiring full rendering.
• Provides flexibility to choose the best approach for your project.

✅ API Reference

WebScraper Class

new WebScraper(options?: ScraperOptions)

📊 Props

| Parameter | Type | Description | | -------------- | ------------------------ | --------------------------------------------------------------- | | usePuppeteer | boolean (optional) | Whether to use Puppeteer (default: true) | | throttle | number (optional) | Delay in milliseconds between requests (default: 1000) | | rules | Record<string, string> | CSS selectors defining data extraction rules |

Methods

scrape(url: string): Promise<Record<string, any>>

• Scrapes the given URL based on the configured options.

exportToJSON(data: any, filePath: string): void

• Exports the given data to a JSON file.

exportToCSV(data: any | any[], filePath: string): void

• Exports the given data to a CSV file.

🛠️ Basic Usage

1. Scraping Web Pages

You can scrape web pages using either Puppeteer (for JavaScript-heavy pages) or Cheerio (for static HTML pages).

import { WebScraper } from 'simple-web-scraper';

const scraper = new WebScraper({
  usePuppeteer: false, // Set to true for dynamic pages
  rules: {
    title: 'h1',
    description: 'meta[name=\"description\"]',
  },
});

(async () => {
  const data = await scraper.scrape('https://example.com');
  console.log(data);
})();

2. Using Puppeteer for JavaScript-heavy Pages

To scrape pages that require JavaScript execution:

const scraper = new WebScraper({
  usePuppeteer: true, // Enable Puppeteer for JavaScript-rendered content
  rules: {
    heading: 'h1',
    price: '.product-price',
  },
});

(async () => {
  const data = await scraper.scrape('https://example.com/product');
  console.log(data);
})();

3. Exporting Data

• Scraped data can be exported to JSON or CSV files using utility functions.

Export to JSON

import { exportToJSON } from 'simple-web-scraper';

const data = { name: 'Example', value: 42 };
exportToJSON(data, 'output.json');

Export to CSV

import { exportToCSV } from 'simple-web-scraper';

const data = [
  { name: 'Example 1', value: 42 },
  { name: 'Example 2', value: 99 },
];
exportToCSV(data, 'output.csv');

// Preserve null and undefined values as null
exportToCSV(data, 'output.csv', { preserveNulls: true });

🖥 Backend Example - Module (import)

This example demonstrates how to use simple-web-scraper in a Node.js backend:

import express from 'express';
import { WebScraper, exportToJSON, exportToCSV } from 'simple-web-scraper';

const app = express();
const scraper = new WebScraper({
  usePuppeteer: true,
  rules: { title: 'h1', content: 'p' },
});

app.get('/scrape-example', async (req, res) => {
  try {
    const url = 'https://github.com/The-Node-Forge';
    const data = await scraper.scrape(url);

    exportToJSON(data, 'output.json'); // export JSON
    exportToCSV(data, 'output.csv', { preserveNulls: true }); // export CSV

    res.status(200).json({ success: true, data });
  } catch (error) {
    res.status(500).json({ success: false, error: error.message });
  }
});

🖥 Backend Example - CommonJS (require)

This example demonstrates how to use simple-web-scraper in a Node.js backend:

const {
  WebScraper,
  exportToJSON,
  exportToCSV,
} = require('@the-node-forge/simple-web-scraper/dist');

const scraper = new WebScraper({
  usePuppeteer: true,
  rules: {
    fullHTML: 'html', // Entire page HTML
    title: 'head > title', // Page title
    description: 'meta[name="description"]', // Meta description
    keywords: 'meta[name="keywords"]', // Meta keywords
    favicon: 'link[rel="icon"]', // Favicon URL
    mainHeading: 'h1', // First H1 heading
    allHeadings: 'h1, h2, h3, h4, h5, h6', // All headings on the page
    firstParagraph: 'p', // First paragraph
    allParagraphs: 'p', // All paragraphs on the page
    links: 'a', // All links on the page
    images: 'img', // All image URLs
    imageAlts: 'img', // Alternative text for images
    videos: 'video, iframe[src*="youtube.com"], iframe[src*="vimeo.com"]', // Video sources
    tables: 'table', // Capture table elements
    tableData: 'td', // Capture table cells
    lists: 'ul, ol', // Capture all lists
    listItems: 'li', // Capture all list items
    scripts: 'script', // JavaScript file sources
    stylesheets: 'link[rel="stylesheet"]', // External CSS files
    structuredData: 'script[type="application/ld+json"]', // JSON-LD structured data
    socialLinks:
      'a[href*="facebook.com"], a[href*="twitter.com"], a[href*="linkedin.com"], a[href*="instagram.com"]', // Social media links
    author: 'meta[name="author"]', // Author meta tag
    publishDate: 'meta[property="article:published_time"], time', // Publish date
    modifiedDate: 'meta[property="article:modified_time"]', // Last modified date
    canonicalURL: 'link[rel="canonical"]', // Canonical URL
    openGraphTitle: 'meta[property="og:title"]', // OpenGraph title
    openGraphDescription: 'meta[property="og:description"]', // OpenGraph description
    openGraphImage: 'meta[property="og:image"]', // OpenGraph image
    twitterCard: 'meta[name="twitter:card"]', // Twitter card type
    twitterTitle: 'meta[name="twitter:title"]', // Twitter title
    twitterDescription: 'meta[name="twitter:description"]', // Twitter description
    twitterImage: 'meta[name="twitter:image"]', // Twitter image
  },
});

app.get('/test-scraper', async (req, res) => {
  try {
    const url = 'https://github.com/The-Node-Forge';
    const data = await scraper.scrape(url);

    exportToJSON(data, 'output.json'); // export JSON
    exportToCSV(data, 'output.csv'); // export CSV

    res.status(200).json({ success: true, data });
  } catch (error) {
    res.status(500).json({ success: false, error: error.message });
  }
});

🛠️ Full Usage Example

import { WebScraper } from 'simple-web-scraper';

const scraper = new WebScraper({
  usePuppeteer: true, // Set to false if scraping static pages
  rules: {
    fullHTML: 'html', // Entire page HTML
    title: 'head > title', // Page title
    description: 'meta[name="description"]', // Meta description
    keywords: 'meta[name="keywords"]', // Meta keywords
    favicon: 'link[rel="icon"]', // Favicon URL
    mainHeading: 'h1', // First H1 heading
    allHeadings: 'h1, h2, h3, h4, h5, h6', // All headings on the page
    firstParagraph: 'p', // First paragraph
    allParagraphs: 'p', // All paragraphs on the page
    links: 'a', // All links on the page
    images: 'img', // All image URLs
    imageAlts: 'img', // Alternative text for images
    videos: 'video, iframe[src*="youtube.com"], iframe[src*="vimeo.com"]', // Video sources
    tables: 'table', // Capture table elements
    tableData: 'td', // Capture table cells
    lists: 'ul, ol', // Capture all lists
    listItems: 'li', // Capture all list items
    scripts: 'script', // JavaScript file sources
    stylesheets: 'link[rel="stylesheet"]', // External CSS files
    structuredData: 'script[type="application/ld+json"]', // JSON-LD structured data
    socialLinks:
      'a[href*="facebook.com"], a[href*="twitter.com"], a[href*="linkedin.com"], a[href*="instagram.com"]', // Social media links
    author: 'meta[name="author"]', // Author meta tag
    publishDate: 'meta[property="article:published_time"], time', // Publish date
    modifiedDate: 'meta[property="article:modified_time"]', // Last modified date
    canonicalURL: 'link[rel="canonical"]', // Canonical URL
    openGraphTitle: 'meta[property="og:title"]', // OpenGraph title
    openGraphDescription: 'meta[property="og:description"]', // OpenGraph description
    openGraphImage: 'meta[property="og:image"]', // OpenGraph image
    twitterCard: 'meta[name="twitter:card"]', // Twitter card type
    twitterTitle: 'meta[name="twitter:title"]', // Twitter title
    twitterDescription: 'meta[name="twitter:description"]', // Twitter description
    twitterImage: 'meta[name="twitter:image"]', // Twitter image
  },
});

(async () => {
  const data = await scraper.scrape('https://example.com');
  console.log(data);
})();

📊 Rule Set Table

| Rule | CSS Selector | Target Data | | -------------------- | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------- | | fullHTML | html | The entire HTML of the page | | title | head > title | The <title> of the page | | description | meta[name="description"] | Meta description for SEO | | keywords | meta[name="keywords"] | Meta keywords | | favicon | link[rel="icon"] | Website icon | | mainHeading | h1 | The first <h1> heading | | allHeadings | h1, h2, h3, h4, h5, h6 | All headings (h1-h6) | | firstParagraph | p | The first paragraph (<p>) | | allParagraphs | p | All paragraphs on the page | | links | a | All anchor <a> links | | images | img | All image <img> sources | | imageAlts | img | All image alt texts | | videos | video, iframe[src*="youtube.com"], iframe[src*="vimeo.com"] | Video sources (<video>, YouTube, Vimeo) | | tables | table | All <table> elements | | tableData | td | Individual <td> elements | | lists | ul, ol | All ordered <ol> and unordered <ul> lists | | listItems | li | All list <li> items | | scripts | script | JavaScript files included (<script src="...">) | | stylesheets | link[rel="stylesheet"] | Stylesheets (<link rel="stylesheet">) | | structuredData | script[type="application/ld+json"] | JSON-LD structured data for SEO | | socialLinks | a[href*="facebook.com"], a[href*="twitter.com"], a[href*="linkedin.com"], a[href*="instagram.com"] | Facebook, Twitter, LinkedIn, Instagram links | | author | meta[name="author"] | Page author (meta[name="author"]) | | publishDate | meta[property="article:published_time"], time | Date article was published | | modifiedDate | meta[property="article:modified_time"] | Last modified date | | canonicalURL | link[rel="canonical"] | Canonical URL (avoids duplicate content) | | openGraphTitle | meta[property="og:title"] | OpenGraph metadata for social sharing | | openGraphDescription | meta[property="og:description"] | OpenGraph description | | openGraphImage | meta[property="og:image"] | OpenGraph image URL | | twitterCard | meta[name="twitter:card"] | Twitter card type (summary, summary_large_image) | | twitterTitle | meta[name="twitter:title"] | Twitter title metadata | | twitterDescription | meta[name="twitter:description"] | Twitter description metadata | | twitterImage | meta[name="twitter:image"] | Twitter image metadata |

💡 Contributing

Contributions are welcome! Please submit issues or pull requests.