async-pool

@sesamecare-oss/async-pool is a TypeScript/JavaScript library that provides a utility function for processing items from an (async) generator with a configurable concurrency limit. It allows you to efficiently handle IO-bound tasks (like network requests, file operations, etc.) in parallel, while controlling how many tasks run simultaneously.

Features

• Limit the number of concurrently running async operations.
• Works with both async and sync generators.
• Simple, promise-based API.
• Written in TypeScript & fully typed.
• Propagates errors just like native Promise.all.

Install

npm install @sesamecare-oss/async-pool

Usage

Basic Example

import { asyncPool } from '@sesamecare-oss/async-pool';

async function* generator() {
  yield 1;
  yield 2;
  yield 3;
  yield 4;
  yield 5;
}

const results: number[] = [];

await asyncPool(2, generator(), async (item) => {
  // Only 2 iterator functions run concurrently
  await doSomethingAsync(item);
  results.push(item);
});

Using with Array

const items = [1, 2, 3, 4, 5];
function* generator() {
  for (const item of items) {
    yield item;
  }
}

await asyncPool(3, generator(), async (item) => {
  // Processing up to 3 items in parallel
  await doSomethingAsync(item);
});

Collecting Results

asyncPool returns an array of all the resolved values from the iterator function, in the order items were yielded.

const items = [1, 2, 3, 4, 5];

const doubles = await asyncPool(2, items.values(), async (item) => {
  return item * 2;
});

console.log(doubles); // [2, 4, 6, 8, 10]

Error Handling

If any iterator function throws (or rejects), asyncPool will reject with that error and stop processing new items.

try {
  await asyncPool(2, [1, 2, 3][Symbol.iterator](), async (item) => {
    if (item === 2) throw new Error('fail!');
    return item;
  });
} catch (err) {
  console.error(err); // Error: fail!
}

API

async function asyncPool<T, R>(
  concurrency: number,
  iterable: Iterable<T> | AsyncIterable<T>,
  iteratorFn: (item: T, index: number) => Promise<R> | R
): Promise<R[]>;

• concurrency: Maximum number of concurrently running tasks.
• iterable: Any iterable or async iterable source.
• iteratorFn: Async/sync function to execute for each item.

License

MIT