handle-that

handles pieces of work in parallel. A small wrapper around child_process.fork.

Features:

• chunkifys the work in pieces to minimize communication
• shuffles the workpieces for a better load balance

Install

npm install --save handle-that

Usage

handleThat = require("handle-that")

// handleThat(work:Array, options:Object)
// note, that only serializable data can be passed to workers.
handleThat(["work1","work2"],{
  worker: "./someWorker.js"
}).then(=>
  // finished
)

// ./someWorker.js
// expects a function which is called with an array of work pieces
// should return a Promise
module.exports = (work, current, altogetherWorkCount) => { return Promise.all(work.map(async (piece,i) => {
  currentWorkIndex = current+i
  // piece is either "work1" or "work2"
  return await doSomething(piece)
}))}

Options

Name | type | default | description ---:| --- | ---| --- worker | String | - | (required) path to worker source file shuffle | Boolean | true | should the work get shuffled flatten | Boolean | true | the work array will be flattened concurrency | Number | #CPUS | how many workers should get spawned onFork | Function | - | will be called on process forking with the worker unit onProgress | Function | - | will be called on progress with the remaining work count onFinish | Function | - | will be called once all work is done onText | Function | - | will set children to silent and instead call this function with all output

The options object will also get passed down to child_process.fork.

Example with ora

path = require("path")
ora = require("ora")
handleThat = require("handle-that")

spinner = ora(work.length + " workpieces remaining...").start()

handleThat(work,{
  worker: path.resolve(__dirname, "_worker"),
  onText: (lines, remaining) => {
    spinner.stop()
    console.log(lines.join("\n"))
    spinner.start(remaining + " workpieces remaining...")
  },
  onProgress: (remaining) => { spinner.text = remaining  + " workpieces remaining..." }
  onFinish: => { spinner.succeed("finished") }
})

License

Copyright (c) 2017 Paul Pflugradt Licensed under the MIT license.