Smart Worker Pool

A smart worker pool for Node.js that allows using worker threads without separate files and supports direct library usage.

Features

• 🚀 No separate worker files: Run functions in worker threads without creating separate JS files
• 📚 Direct library usage: Use libraries directly in your worker threads
• ⚙️ Configurable concurrency: Control the number of concurrent worker threads
• 🔄 Automatic retries: Retry failed tasks automatically
• ⏱️ Task timeouts: Set timeouts for long-running tasks
• 🧹 Proper cleanup: Automatically terminates workers when done

Installation

npm install smart-worker-pool

Usage

Basic Example

import { SmartWorkerPool } from 'smart-worker-pool';

// Create a worker pool
const pool = new SmartWorkerPool();

// Run a function in a worker thread
const result = await pool.run(
  (data) => {
    // This code runs in a worker thread
    return data.x + data.y;
  },
  { x: 10, y: 20 }
);

console.log(result); // 30

// Shutdown the pool when done
await pool.shutdown();

Using Libraries

import { SmartWorkerPool } from 'smart-worker-pool';

// Create a worker pool with preloaded libraries
const pool = new SmartWorkerPool({
  preload: {
    // Format: alias: package-name
    lodash: 'lodash',
    moment: 'moment'
  }
});

// Use the libraries in the worker thread
const result = await pool.run(
  (data, libs) => {
    // libs contains the preloaded libraries
    const { lodash, moment } = libs;
    
    // Use lodash in the worker thread
    const items = lodash.range(1, data.count + 1);
    const sum = lodash.sum(items);
    
    // Use moment in the worker thread
    const now = moment().format('YYYY-MM-DD');
    
    return { sum, now };
  },
  { count: 5 }
);

console.log(result); // { sum: 15, now: '2023-04-01' }

// Shutdown the pool when done
await pool.shutdown();

Advanced Configuration

import { SmartWorkerPool } from 'smart-worker-pool';

// Create a worker pool with advanced options
const pool = new SmartWorkerPool({
  // Libraries to preload
  preload: {
    axios: 'axios',
    dayjs: 'dayjs'
  },
  // Maximum number of concurrent workers
  maxConcurrency: 4,
  // Maximum number of retries for failed tasks
  maxRetry: 3,
  // Timeout in milliseconds (5 seconds)
  taskTimeout: 5000,
  // Whether to terminate workers after each task
  terminateAfterTask: true
});

// ... use the pool

// Shutdown the pool when done
await pool.shutdown();

API Reference

SmartWorkerPool

Constructor Options

interface SmartWorkerPoolOptions {
  // Libraries to preload in the format { alias: 'package-name' }
  preload?: Record<string, string>;
  // Maximum number of concurrent workers (default: 4)
  maxConcurrency?: number;
  // Maximum number of retries for failed tasks (default: 0)
  maxRetry?: number;
  // Timeout in milliseconds for each task (default: 0, no timeout)
  taskTimeout?: number;
  // Whether to terminate workers after each task (default: true)
  terminateAfterTask?: boolean;
}

Methods

• run<T, R>(fn: (data: T, libs: Record<string, any>) => R | Promise<R>, data: T): Promise<R>

  Runs a function in a worker thread.

  • fn: The function to run in the worker thread
  • data: The data to pass to the function
  • Returns a promise that resolves with the result of the function

• shutdown(): Promise<void>

  Shuts down the worker pool, terminating all workers and rejecting queued tasks.

License

MIT