npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

scraper-api

v1.0.0

Published

Interface for calling ScraperAPI from Node

Downloads

19

Vulnerabilities

Powered byPowered by Snyk
  • 0High
  • 0Medium
  • 0Low

Links

Git

Maintainers

kylerosskyleross

Keywords

scraperapiscraperapiwebservice

Readme

scraper-api

npm npm David Travis Coveralls license Beerpay

Interface to call ScraperAPI.com easily from Node. All current API endpoints and features are implemented in this module. Requires Node 8+.

Install

$ npm install --save scraper-api

Usage

const scraperAPI = require('scraper-api')({
    // ... options
});

scraperAPI.get('http://httpbin.org/ip')
    .then(result => {
        // result => '<!doctype html> ...'
    })
    .catch(error => {
        console.error(error);
        //=> 'Internal server error'
    });

API Documentation

scraper-api

scraperAPI(options) ⇒ ScraperAPI ⏏

Creates new instance of ScraperAPI with the provided options.

Kind: Exported function
Returns: ScraperAPI - New instance of ScraperAPI.

| Param | Type | Default | Description | | --- | --- | --- | --- | | options | Object | | Optional configuration options to pass into ScraperAPI. | | [options.apiKey] | String | process.env.SCRAPER_API_KEY | API key for Scraper API. Defaults to pulling API Key from environment variable. | | [options.renderJs] | Boolean | false | Render JavaScript on the page before scraping the HTML for the page. | | [options.keepHeaders] | Boolean | false | Keep headers sent in the request to Scraper API in subsequent request(s) when scraping the provided url. You must set your headers in options.gotOptions.headers. | | [options.geoCode] | String | 'us' | Geo code in which to use proxies for when scraping. See documentation for more information. | | [options.premium] | Boolean | false | Whether to use premium proxies. Caution: This will cost 10-25 times more than standard proxies. | | [options.sessionId] | Number | | A numeric session id to use to maintain the same proxy. See ScraperAPI.session() for more information. | | [options.gotOptions] | Object | {} | Additional options to pass into got for requests to ScraperAPI. |

Example

const scraperAPI = require('scraper-api')({
    // options...
});

scraperAPI.ScraperAPI

Access to the uninstantiated ScraperAPI class.

Kind: static property of scraperAPI
Example

const ScraperAPI = require('scraper-api').ScraperAPI;
const scraperAPI = new ScraperAPI({
    // options...
});

ScraperAPI

Kind: global class

new ScraperAPI([options])

Constructor for the ScraperAPI class.

| Param | Type | Default | Description | | --- | --- | --- | --- | | [options] | Object | {} | Options for the ScraperAPI class. Options may be overridden on all methods for a single request. See options above for more information. |

scraperAPI.session(id, [options]) ⇒ ScraperAPI

Creates a new instance of ScraperAPI with the specified session id. Sessions allow subsequent requests with the same session id to go through the same proxy. See documentation for more information.

Kind: instance method of ScraperAPI
Returns: ScraperAPI - A new instance of ScraperAPI with sessionId set to the provided id.

| Param | Type | Default | Description | | --- | --- | --- | --- | | id | Number | | Session ID in which to use for the given session. Must only contain numbers. | | [options] | Object | {} | Options to override for all subsequent requests to Scraper API. Same as the global options. |

Example

const session = scraperAPI.session(1234);

let result = await scraperAPI.get('https://google.com');
// result -> '<!doctype html> ...'

scraperAPI.get(url, [options]) ⇒ Promise.<String>

Calls Scraper API with a GET request to the provided url.

Kind: instance method of ScraperAPI
Returns: Promise.<String> - Promise that resolves with HTML source from requested URL.

| Param | Type | Default | Description | | --- | --- | --- | --- | | url | String | | The URL in which to scrape. | | [options] | Object | {} | Options to override for this specific request. |

Example

let result = await scraperAPI.get('https://google.com');
// result -> '<!doctype html> ...'

scraperAPI.post(url, data, [options]) ⇒ Promise.<Object>

Calls Scraper API with a POST request to the provided url with the provided data.

Kind: instance method of ScraperAPI
Returns: Promise.<Object> - Promise that resolves with object response from Scraper API. See documentation.

| Param | Type | Default | Description | | --- | --- | --- | --- | | url | String | | The URL in which to scrape. | | data | Object | form-data | | Data in which to post to the provided URL. Must be either a plain object or instance of form-data. | | [options] | Object | {} | Options to override for this specific request. May be any of the global options and any additional options below. | | [options.form] | Boolean | false | Set true if provided data is form data and should be sent as such. By default, data will be sent as JSON. |

Example

let result = await scraperAPI.post('https://example.com/endpoint', {
   hello: 'world',
   some: 'data'
});

More Information

keepHeaders Option

If you would like to pass custom headers through Scraper API to the destination, you may do so by setting your custom headers in options.gotOptions.headers and enabling this option.

let result = await scraperAPI.get('https://google.com', {
    keepHeaders: true,
    gotOptions: {
        headers: {
            'My-Custom-Header': 'some value'
        }
    }
});

Tests

Tests are written and provided as part of the module. You may run the tests by calling:

$ npm run test

License

MIT License. See License in the repository.