README is available in the following languages: Russian | English

Proxy Checker

proxy-checker - The most versatile proxy parser and proxy checker.

Why ?

I, and therefore many people, need to get a proxy to bypass the ban on IP with a large number of requests. This module allows you to perform almost all tasks related to proxy preparation. I, and therefore many people, need to build proxies individually for some sites, I also took care of this. I hope my contribution will make it easier for someone life.

Get started

Install the npm module @prohetamine/proxy-checker

$ npm install @prohetamine/proxy-checker

or

$ yarn add @prohetamine/proxy-checker

Examples and description

Connecting the module

const proxy = require('@prohetamine/proxy-checker')

load

The load function collects proxies and easily returns them via all and random.

object

| key | value | default value | required | information| | ------ | ------ | ------ | ------ | ------ | | port | number | false | none | used for the port filter | | debug | boolean | false | none | used for debugging |

const proxy = require('@prohetamine/proxy-checker')

;(async () => {
  const count = await proxy.load({
    port: 8888,
    debug: true
  })

  console.log(count) // Number of collected proxies with port 8888
})()

loadInterval

The loadInterval function collects proxies with an interval of N milliseconds and easily returns them via all and random.

| options | default | info | | ------ | ------ | ------ | | function | () => { /* ... */} | callback function returns in a single argument the number of IP loaded | | int | 5000 | interval in milliseconds | | object | { started, port, debug } | additional options |

object

| key | value | default value | mandatory | information | | ------ | ------ | ------ | ------ | ------ | | started | boolean | false | no | use for instant download | | port | boolean | false | no | used to filter ports | | debug | boolean | false | none | used for debugging |

const proxy = require('@prohetamine/proxy-checker')

;(async () => {
  const kill = await proxy.loadInterval(count => {
    console.log(`count parse: ${count}`) // Number of collected proxies with port 8080
  }, 60000, { started: true, debug: false, port: 8080 })

  console.log(proxy.all({ port: 8888 })) // All downloaded IP addresses with port 8888
  console.log(proxy.random({ port: 80 })) // One random IP address with port 80

  setInterval(() => {
    if (proxy.all().length > 1488) {
      console.log(proxy.all())
      kill() // kills the current loadInterval
    }
  }, 5000)
})()

all

The all function returns an array of IP addresses when it is loaded load or loadInterval and false when the load has not yet occurred.

object

| key | value | default value | required | information| | ------ | ------ | ------ | ------ | ------ | | port | number | false | none | used for the port filter |

const proxy = require('@prohetamine/proxy-checker')

;(async () => {
  const count = await proxy.load({
    debug: true
  })

  console.log(proxy.all({ port: 1488 })) // All downloaded IP addresses with port 1488
})()

random

The random function returns a single random IP address when it is loaded load or loadInterval and false when the load has not yet occurred.

object

| key | value | default value | required | information| | ------ | ------ | ------ | ------ | ------ | | port | number | false | none | used for the port filter |

const proxy = require('@prohetamine/proxy-checker')

;(async () => {
  const count = await proxy.load({
    debug: true
  })

  console.log(proxy.random({ port: 8888 })) // One random IP address with port 8888
})()

checkerInterval

The checkerInterval function checks proxies with an interval of N milliseconds for a specific site and easily returns them via get.all and get.random.

| parameters | default value | information| | ------ | ------ | ------ | | string | null | unique identifier | | object | { url, isBrowser, browserConfig, timeout, stream, debug, indicators, session, onData } | additional options |

object

| key | value | default value | mandatory | information | | ------ | ------ | ------ | ------ | ------ | | url | string | null | no | website link | | isBrowser | boolean | false | none | used for checking through the browser | | trashIgnore | boolean | false | none | used for optimization, removes loading of media, xhr, fetch, websocket, manifest, image, stylesheet, font, script | | browserConfig | object | { ... } | none | used to configure the launcher | | onData | function | () => { ... } | none | used to get the body of a successful request (works only for browser) | | timeout | int | 10000 | none | interval in milliseconds | | stream | int | 2 | none | number of concurrent threads | | debug | boolean | false | none | used for debugging | | indicators | array | [] | none | indicators of correctly loaded data | | session | string | false | none | path to save valid IP addresses after completion or saving | | quarantineMode | boolean | false | none | used to quarantine non-working proxies |

array[object]

| key | value | default value | required | information| | ------ | ------ | ------ | ------ | ------ | | keyword | string | no | yes | used to search the page by keyword | | selector | string | no | yes | used for searching on a page by dom node |

const proxy = require('@prohetamine/proxy-checker')

;(async () => {
  await proxy.loadInterval(count => {
    console.log(`count parse: ${count}`) // Number of collected proxies
  }, 60000, { started: true, debug: false })

  const { key, kill, save, clean } = await proxy.checkerInterval('chaturbate', {
    url: 'https://chaturbate.com/diana_smiley/',
    isBrowser: true,
    trashIgnore: true,
    browserConfig: proxy => ({
      headless: true,
      ignoreHTTPSErrors: true,
      args: [`--proxy-server=${proxy}`],
    }),
    timeout: 60000,
    stream: 10,
    session: __dirname + '/chaturbate.json',
    debug: true,
    indicators: [{
      //keyword: 'close_entrance_terms',
      selector: '#close_entrance_terms'
    }] // List other metrics in the objects if they are required
  })

  setInterval(() => {
    const proxys = proxy.get(key).all()
    console.clear()
    console.log(proxys, proxys.length)

    if (proxys.length > 20) {
      kill() // Kills the current checkerInterval and saves the session
      save() // Only saves the session
    }
  }, 3000)
})()

const proxy = require('@prohetamine/proxy-checker')

;(async () => {
  await proxy.loadInterval(count => {
    console.log(`count parse: ${count}`) // The number of proxies collected with the port
  }, 60000, { started: true, debug: false })

  const keyName = null // A unique identifier for IP accumulation, if set to null, will be generated automatically.

  const { key, kill, save, clean } = await proxy.checkerInterval(keyName, {
    url: 'https://github.com/prohetamine',
    timeout: 30000,
    stream: 20,
    session: __dirname + '/github.json',
    debug: true,
    indicators: [{
      keyword: 'Prohetamine',
      //selector: '#close_entrance_terms'
    }, {
      keyword: 'Stas'
    }] // List other metrics in the objects if they are required
  })

  setInterval(() => {
    const proxys = proxy.get(key).all()
    console.clear()
    console.log(proxys, proxys.length)

    if (proxys.length > 20) {
      kill() // Kills the current checkerInterval and saves the session
      save() // Only saves the session
    }
  }, 3000)
})()

get.all

The get.all function returns an array of checked checkerInterval IP addresses by a unique identifier and false when the download has not yet occurred.

get(key)

| parameters | default value | required | information| | ------ | ------ | ------ | ------ | | string | null | yes | unique identifier |

object

| key | value | default value | required | information| | ------ | ------ | ------ | ------ | ------ | | port | number | false | none | used for the port filter |

const proxy = require('@prohetamine/proxy-checker')

;(async () => {
  await proxy.loadInterval(count => {
    console.log(`count parse: ${count}`) // Number of collected proxies
  }, 60000, { started: true, debug: false })

  const { key, kill, save, clean } = await proxy.checkerInterval('chaturbate', {
    url: 'https://chaturbate.com/diana_smiley/',
    isBrowser: true,
    debugBrowser: false,
    timeout: 60000,
    stream: 10,
    session: __dirname + '/chaturbate.json',
    debug: true,
    indicators: [{
      //keyword: 'close_entrance_terms',
      selector: '#close_entrance_terms'
    }] // List other metrics in the objects if they are required
  })

  setInterval(() => {
    const proxys = proxy.get(key).all()
    console.clear()
    console.log(proxys, proxys.length)

    if (proxys.length > 20) {
      kill() // Kills the current checkerInterval and saves the session
      save() // Only saves the session
    }
  }, 3000)
})()

get.random

The function get.random returns one random of the checked checkerInterval IP addresses by a unique identifier and false when the download has not yet occurred.

get(key)

| parameters | default value | required | information| | ------ | ------ | ------ | ------ | | string | null | yes |unique identifier |

object

| key | value | default value | required | information| | ------ | ------ | ------ | ------ | ------ | | port | number | false | none | used for the port filter |

const proxy = require('@prohetamine/proxy-checker')

;(async () => {
  await proxy.loadInterval(count => {
    console.log(`count parse: ${count}`) // Number of collected proxies
  }, 60000, { started: true, debug: false })

  const { key, kill, save, clean } = await proxy.checkerInterval('chaturbate', {
    url: 'https://chaturbate.com/diana_smiley/',
    isBrowser: true,
    debugBrowser: false,
    timeout: 60000,
    stream: 10,
    session: __dirname + '/chaturbate.json',
    debug: true,
    indicators: [{
      //keyword: 'close_entrance_terms',
      selector: '#close_entrance_terms'
    }] // List other metrics in the objects if they are required
  })

  setInterval(() => {
    console.log(proxy.get(key).random())
  }, 3000)
})()

Other examples from this project can be found here: examples

Contacts

My Telegram: @prohetamine, channel

Email: [email protected]

Donat money: patreon

If you have any questions and/or suggestions, please email me in telegram, if you find any bugs also let me know, I will be very grateful.