Google HTTP Scraper

This module lets you scrape google using HTTP requests. This don't needs Google's api to work.

Features

• Proxy support: http and socks5 with and without authentication.
• Language support: Choose which language to get SERPs from.
• Useragent support: Can set your own user agent.
• Number of results from page support.
• Can have multiple instances, each with their own cookies/proxies on the same script.

Installing

npm install google-http-scraper

Usage

The module exports the GoogleSession prototype. Use it to create multiple instances each with their own cookies/proxies/languages etc.

The constructor accepts a settings object as parameter:

  settings = {
    saveSearches: true, //If true it will save all your searches in an array for saving later.
    proxy: proxyObject, //An object defining the proxy to use, if not set will not use any proxy.
    useragent: 'string', //An string defining the useragent to use with the requests, if not set will use the defaults.
    maxResults: 100, //The max search results you want, searches will use this if they don't have their own settings.
    language: 'lang_en', //The language to use with the searches, searches will use this if they don't have their own language set.
  }

  proxyObject = {
    type: 'socks or http', //The type of proxy.
    host: 'hostname', //The proxy host/ip address.
    port: 123, //The proxy's port.
    username: 'username', //The proxy's authentication username if needed.
    password: 'password', //The proxy's authentication password if needed.
  }

Default user agent: 'Mozilla/5.0 (X11; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0'

A GoogleSession has an async method called search, it queries google using the given parameters.

  searchParams = {
    query: 'your search query', //A string containing your query
    maxResults: 10, //The number of search results you want
    language: 'lang_en', //The language to perform the search in
  }

Example

  const GoogleSession = require('google-http-scraper');

  async function main() {
    var session = new GoogleSession({language: 'lang_en', maxResults: 10});
    var results1 = await session.search({query: 'searching something'});
    var results2 = await session.search({query: 'searching in another language', language: 'lang_pt'});
    var results3 = await session.search({query: 'getting more results', maxResults: 100});
    session.save('./session.json');
  }

  main();

Author

Ícaro Augusto

• https://icaroaugusto.com
• Discord: IcaroAugusto#3303

Changelogs

1.0.0

Created.