node-multi-curl

A performant multi-curl class in Node.js that supports working with proxies and concurrent requests.

Features

• Parallel HTTP requests with concurrency control
• Proxy support with automatic rotation
• Docker integration for HTTP/2 support
• Axios-like API for ease of use
• Request/response interceptors
• Automatic retry mechanism
• HTTP/2 support** via Docker curl integration
• Cloudflare capabilities with advanced header management
• Brotli, Gzip and Deflate** compression support

Installation

npm install node-multi-curl

Quick Start

const { MultiCurl } = require('node-multi-curl');

// Simple GET request
const response = await MultiCurl.get('https://example.com/api');

// Multiple concurrent requests
const client = new MultiCurl({ concurrency: 5 });
const responses = await client.execute([
  { url: 'https://example.com/api/users' },
  { url: 'https://example.com/api/products' },
  { url: 'https://example.com/api/orders' }
]);

Advanced Features

HTTP/2 and Compression Support

const client = new MultiCurl({
  curlConfig: {
    useDocker: true,
    dockerImage: 'badouralix/curl-http2:latest'
  }
});

// This request will use HTTP/2 if the server supports it
const response = await client.get('https://http2.pro/api/v1');

// Brotli, Gzip and other compression methods are automatically handled

Cloudflare Protection Bypass

// Configure with specialized headers
const client = new MultiCurl();

client.interceptors.request.use(request => {
  request.headers = {
    ...request.headers,
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Accept-Encoding': 'gzip, deflate, br',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1',
    'Sec-Fetch-Dest': 'document',
    'Sec-Fetch-Mode': 'navigate',
    'Sec-Fetch-Site': 'none',
    'Sec-Fetch-User': '?1'
  };
  return request;
});

// Now you can access Cloudflare-protected sites
const response = await client.get('https://cloudflare-protected-site.com');

Query Parameters

// Using query parameters
const response = await MultiCurl.get('https://httpbin.org/get', {
  queryParams: {
    param1: 'value1',
    param2: 123,
    param3: true,
    nullParam: null, // This will be skipped
  }
});
// Makes a request to: https://httpbin.org/get?param1=value1&param2=123&param3=true

// Or with the execute method:
const client = new MultiCurl();
const responses = await client.execute([
  { 
    url: 'https://httpbin.org/get', 
    queryParams: { search: 'test', page: 1 } 
  }
]);

Documentation

For detailed documentation and examples, check the examples directory.

License

MIT