NetScrape

Web scraping for Node.js made efficient, simple, and compliant. NetScrape complies with the Robots Exclusion Protocol.

Installation

npm install netscrape

Usage

Netscrape is designed to be simple, but also extensible enough for advanced use cases. The following example demonstrates how to make a simple request to a website.

import Bot from 'netscrape';

const exampleBot = new Bot({ name: 'ExampleBot', version: '1.0' });

try {
  const response = await exampleBot.makeRequest('https://www.example.com/path');
  console.log(response.body);
} catch (error) {
  console.error(error);
}

Bot#constructor

import Bot from 'netscrape';

type BotOptions = {
  name: string;
  version: string;
  minimumRequestDelay?: number;
  maximumRequestDelay?: number;
  disableCaching?: boolean;
  policyURL?: string;
  hideLibraryAgent?: boolean;
  userAgent?: string;
};

const exampleBot = new Bot({
  name: 'ExampleBot' /* required, Name of your bot */,
  version: '1.0' /* required, Version of your bot */,
  minimumRequestDelay: 1000 /* optional, Minimum delay between requests in milliseconds */,
  maximumRequestDelay: 5000 /* optional, Maximum delay between requests in milliseconds (default 10000) */,
  disableCaching:
    true /* optional, Disable caching of responses (default false) */,
  policyURL:
    'https://www.example.com/robots.txt' /* optional, URL to robots.txt file (default https://npm.im/netscrape) */,
  hideLibraryAgent:
    true /* optional, Hide the library agent from the user agent (default false) */,
  userAgent:
    'ExampleBot/1.0' /* optional, Custom user agent, overrides all other user agent fields */,
});

Bot#makeRequest

import Bot from 'netscrape';

const exampleBot = new Bot({ name: 'ExampleBot', version: '1.0' });

try {
  /* Note: Bot#makeRequest automatically requests /robots.txt in the background */
  const response = await exampleBot.makeRequest(
    'https://www.example.com/path' /* required, well-formatted URL to make request to */,
    false /* optional, should you return a byte stream instead of utf8 text */,
  );

  /* Bot#makeRequest returns the raw npm.im/got package request response */
  console.log(response.body);
} catch (error) {
  /* Robots.txt rejection, robots.txt 500 error, etc. */
  console.error(error);
}

Bot#makeRequestWithBody

import Bot from 'netscrape';

const exampleBot = new Bot({ name: 'ExampleBot', version: '1.0' });

try {
  /* Note: Bot#makeRequestWithBody automatically requests /robots.txt in the background */
  /* Note: Bot#makeRequestWithBody automatically sets the
    Content-Type header to application/json or text/plain based on the body type */

  const response = await exampleBot.makeRequestWithBody(
    'https://www.example.com/path' /* required, well-formatted URL to make request to */,
    {
      example: 'body',
    } /* required, body to send to server (string or object) */,
    {
      'x-example-header': 'example header',
    } /* optional, headers to send to server */,
    'POST' /* optional, HTTP method to use (default POST) */,
  );

  /* Bot#makeRequestWithBody returns the raw npm.im/got package request response */
  console.log(response.body);
} catch (error) {
  /* Robots.txt rejection, robots.txt 500 error, etc. */
  console.error(error);
}

License

MIT (C) 2023 Russell Steadman. See LICENSE file. Visit Google deps.dev for dependency information.