distribution

This is the distribution library.

Environment Setup

We recommend using the prepared container image.

Installation

After you have setup your environment, you can start using the distribution library. When loaded, distribution introduces functionality supporting the distributed execution of programs. To download it:

$ npm i '@brown-ds/distribution'

This command downloads and installs the distribution library.

Testing

There are several categories of tests:

• Regular Tests (*.test.js)
• Scenario Tests (*.scenario.js)
• Extra Credit Tests (*.extra.test.js)
• Student Tests (*.student.test.js) - inside test/test-student

Running Tests

By default, all regular tests are run. Use the options below to run different sets of tests:

1. Run all regular tests (default): $ npm test or $ npm test -- -t
2. Run scenario tests: $ npm test -- -c
3. Run extra credit tests: $ npm test -- -ec
4. Run the non-distribution tests: $ npm test -- -nd
5. Combine options: $ npm test -- -c -ec -nd -t

Usage

To try out the distribution library inside an interactive Node.js session, run:

$ node

Then, load the distribution library:

> let distribution = require("@brown-ds/distribution")();
> distribution.node.start(console.log);

Now you have access to the full distribution library. You can start off by serializing some values.

> s = distribution.util.serialize(1); // '{"type":"number","value":"1"}'
> n = distribution.util.deserialize(s); // 1

You can inspect information about the current node (for example its sid) by running:

> distribution.local.status.get('sid', console.log); // null 8cf1b (null here is the error value; meaning there is no error)

You can also store and retrieve values from the local memory:

> distribution.local.mem.put({name: 'nikos'}, 'key', console.log); // null {name: 'nikos'} (again, null is the error value) 
> distribution.local.mem.get('key', console.log); // null {name: 'nikos'}

> distribution.local.mem.get('wrong-key', console.log); // Error('Key not found') undefined

You can also spawn a new node:

> node = { ip: '127.0.0.1', port: 8080 };
> distribution.local.status.spawn(node, console.log);

Using the distribution.all set of services will allow you to act on the full set of nodes created as if they were a single one.

> distribution.all.status.get('sid', console.log); // {} { '8cf1b': '8cf1b', '8cf1c': '8cf1c' } (now, errors are per-node and form an object)

You can also send messages to other nodes:

> distribution.local.comm.send(['sid'], {node: node, service: 'status', method: 'get'}, console.log); // null 8cf1c

Most methods in the distribution library are asynchronous and take a callback as their last argument. This callback is invoked when the method completes, with the first argument being an error (if any) and the second argument being the result. The following runs the sequence of commands described above inside a script (note the nested callbacks):

let distribution = require("@brown-ds/distribution")();
// Now we're only doing a few of the things we did above
const out = (cb) => {
  distribution.local.status.stop(cb); // Shut down the local node
};
distribution.node.start(() => {
  // This will run only after the node has started
  const node = {ip: '127.0.0.1', port: 8765};
  distribution.local.status.spawn(node, (e, v) => {
    if (e) {
      return out(console.log);
    }
    // This will run only after the new node has been spawned
    distribution.all.status.get('sid', (e, v) => {
      // This will run only after we communicated with all nodes and got their sids
      console.log(v); // { '8cf1b': '8cf1b', '8cf1c': '8cf1c' }
      // Shut down the remote node
      distribution.local.comm.send([], {service: 'status', method: 'stop', node: node}, () => {
        // Finally, stop the local node
        out(console.log); // null, {ip: '127.0.0.1', port: 1380}
      });
    });
  });
});

Results and Reflections

...