@entergreat/worker-pool

Concurrent task processing with retry logic and rate limit handling.

Installation

npm install @entergreat/worker-pool

Usage

import { WorkerPool } from '@entergreat/worker-pool';

const pool = new WorkerPool({
  concurrency: 3,          // parallel workers (default: 3)
  delayBetweenTasks: 3000, // ms between tasks (default: 3000)
  maxRetries: 5,           // retry attempts (default: 5)
  retryOn: [429],          // HTTP status codes to retry (default: [429])
});

const results = await pool.process({
  tasks: [
    { id: 1, data: { url: 'https://api.example.com/1' } },
    { id: 2, data: { url: 'https://api.example.com/2' } },
  ],
  fetch: async (task) => {
    const response = await fetch(task.data.url);
    return response.json();
  },
  save: async (task, result) => {
    await saveToDatabase(result);
    return 1; // return count of items processed
  },
  logPrefix: 'API Sync:', // optional prefix for log messages
});

pool.logReport(results, 'API SYNC');

API

new WorkerPool(config?)

Creates a new worker pool instance.

Config options:

| Option | Type | Default | Description | |--------|------|---------|-------------| | concurrency | number | 3 | Number of parallel workers | | delayBetweenTasks | number | 3000 | Milliseconds to wait between tasks | | maxRetries | number | 5 | Maximum retry attempts per task | | retryOn | number[] | [429] | HTTP status codes that trigger retry |

pool.process({ tasks, fetch, save, logPrefix? })

Processes tasks concurrently with the configured worker pool.

Parameters:

| Parameter | Type | Description | |-----------|------|-------------| | tasks | array | Array of task objects with id and data | | fetch | async function | (task) => result - fetches/processes the task | | save | async function | (task, result) => count - saves result, returns item count | | logPrefix | string | Optional prefix for log messages |

Returns: Array of results with { success, taskName, resultCount, error? }

pool.logReport(results, operationType, logPrefix?)

Logs a summary report of the processing results.

Features

• Concurrent processing - Multiple workers process tasks in parallel
• Exponential backoff - Retries with increasing delays on rate limits
• Adaptive throttling - Automatically slows down when rate limited
• Progress logging - Real-time updates on worker status
• Error handling - Graceful handling of failures with detailed reporting

License

UNLICENSED