@manyducks.co/queue-each

A tool for asynchronously working through large lists while keeping things a little more controlled than await Promise.all().

Example

A hypothetical example showing a lookup of user ratings for 50,000 movies. Concurrency is set to 50 so we don't get rate limited or run into timeouts.

import queueEach from "@manyducks.co/queue-each";

const lotsOfItems = [
  /* 50,000 item array */
];

const movieScores = await queueEach({
  items: lotsOfItems,
  concurrency: 50,
  process: async (item) => {
    // A maximum of 50 requests will be pending at once.
    // When one finishes the next is queued up.
    const results = await http.get(
      `https://movie-api.com/api/stuff?t=${item.name}`,
    );
    return {
      name: item.name,
      score: results.userRating,
    };
  },
});

for (const movie of movieScores) {
  console.log(`${movie.name}: ${movie.score}`);
}