@parallel-park/run-jobs

Parallel async work with an optional concurrency limit, like Bluebird's Promise.map.

Usage

@parallel-park/run-jobs exports one function: runJobs.

runJobs

runJobs is kinda like Promise.all, but instead of running everything at once, it'll only run a few Promises at a time (you can choose how many to run at once). It's inspired by Bluebird's Promise.map function.

To use it, you pass in an iterable (array, set, generator function, etc) of inputs and a mapper function that transforms each input into a Promise. You can also optionally specify the maximum number of Promises to wait on at a time by passing an object with a concurrency property, which is a number. The concurrency defaults to 8.

When using an iterable, if the iterable yields a Promise (ie. iterable.next() returns { done: false, value: Promise }), then the yielded Promise will be awaited before being passed into your mapper function. Additionally, async iterables are supported; if iterable.next() returns a Promise, it will be awaited.

import { runJobs } from "parallel-park";

const inputs = ["alice", "bob", "carl", "dorsey", "edith"];
const results = await runJobs(
  inputs,
  async (name, index, inputsCount) => {
    // Do whatever async work you want inside this mapper function.
    // In this case, we use a hypothetical "getUser" function to
    // retrieve data about a user from some web API.
    console.log(`Getting fullName for ${name}...`);
    const user = await getUser(name);
    return user.fullName;
  },
  // This options object with concurrency is an optional argument.
  // If unspecified, it defaults to { concurrency: 8 }
  {
    // This number specifies how many times to call the mapper
    // function before waiting for one of the returned Promises
    // to resolve. Ie. "How many promises to have in-flight concurrently"
    concurrency: 2,
  }
);
// Logs these two immediately:
//
// Getting fullName for alice...
// Getting fullName for bob...
//
// But, it doesn't log anything else yet, because we told it to only run two things at a time.
// Then, after one of those Promises has finished, it logs:
//
// Getting fullName for carl...
//
// And so forth, until all of them are done.

// `results` is an Array of the resolved value you returned from your mapper function.
// The indices in the array correspond to the indices of your inputs.
console.log(results);
// Logs:
// [
//   "Alice Smith",
//   "Bob Eriksson",
//   "Carl Martinez",
//   "Dorsey Toth",
//   "Edith Skalla"
// ]

License

MIT