@keeex/worker-node

Management library for workers and workers pool.

Installation

npm install @keeex/worker-node

Concept

This library allows using workers, which can be seen as separate instances of Node, to perform long tasks without locking the main thread. Communication between the main thread and the workers is done by passing JSON objects around.

A worker is expected to be either a separate script file or a JavaScript string. The default behavior is to load the main script in each worker, and have a different behavior. This library provide tools to implement this in a straightforward way.

Script entry point

When using the same script for both the main script and the workers, we need to use the workerSplitter() function. It takes two parameters: the main application function, and the worker command handling callback (more about this below).

Worker code

Workers are built around the concept of receiving a command then returning a result. To this end, the communication mechanism uses a simple JSON message system. To implement a worker in this fashion, this library provides two options:

• Use workerSplitter(), which will call the worker command handler appropriately
• Use runWorker() directly (if you're sure you're in a worker script) with your command callback.

In both case the worker code should solely consist of a main command handling function and optionally an initialization function that will be called before the worker is ready to process commands.

Manual worker implementation

It may be required to write the worker without using the helper code provided in this library; in this case, the following must be implemented so that it can work correctly with WorkerInstance and WorkerPool:

• Send the message {"type":"ready"} when the worker is ready to process commands
• Accept messages in the form of {"type": "command": "payload": /* command payload */}
• Always reply one of these after a command has been processed:
  • {"type": "done", "payload": /* result */}
  • {"type": "fail", "payload": /* error */}
• If the above is used to report an error, then the worker will continue execution. If an error is thrown and uncatched in the worker, it will be terminated.

Worker Pool

Instead of spawning individual workers, a WorkerPool can be created to handle many commands at once. In addition to being able to handle mutliple commands, a pool will also queue commands that can't be processed immediately for later processing.

Basic example

The following example spawn workers that just double their input value as their result. It is all in a single script.

import {workerSplitter} from "@keeex/worker-node/lib/splitter.js";
import {WorkerPool} from "@keeex/worker-node/lib/workerpool.js";

const main = async () => {
  const pool = await WorkerPool.create();
  const cmds = [];
  for (let i = 0; i < 100; ++i) cmds.push(pool.runCommand(i));
  for await (const cmd of cmds) {
    console.log("Result:", cmd);
  }
  await pool.stop();
}

let tid;

const workerInit = async (threadId) => {
  tid = threadId;
}

const workerHandler = async (data) => {
  return {
    input: data,
    result: data * 2,
    from: tid,
  };
}

workerSplitter(main, workerHandler, workerInit);

Debugging

This library uses the debug package, with the top-level namespace worker-node. To enable all debugging, export DEBUG=worker-node:*.