Scientist

A JavaScript library for carefully refactoring critical paths, port of GitHub Scientist

Installation

npm install @stuarthadfieldglobality/scientist

Usage

import Scientist from '@stuarthadfieldglobality/scientist'

const experiment = new Scientist('experiment #1');
experiment.use(() => 12); // old way
experiment.try(() => 10 - 8); // new way
experiment.publish(results => console.log(results)); // publish
const result = experiment.run();

// Sometimes we need to go Async
const experiment = new Scientist('experiment #1', { async: true });
experiment.use(async () => 12); // old way (async or promise)
experiment.try(async () => 10 - 8); // new way (async or promise)
experiment.publish(results => console.log(results)); // publish
const result = await experiment.run();

How to Science

Let's pretend you are changing how the result of a calculation is obtained. Tests can help you refactor, but really you want to compare the current and refactored behaviours under load.

Pass your original code to the use function, and pass your new code / behaviour to the try function. experiment.run will always return whatever the use function returns, but it does some stuff behind the scenes:

• It decided whether or not to run the try block
• Randomizes the order in which use and try functions are run
• Measures the duration of all behaviours
• Compares the result of try and the result of use
• Swallow and record errors thrown in the try block
• Publishes all of this information

The use function is called the control. The try function is called the candidate

If you do not declare a candidate, then the control is always ran and returned.

Creating useful experiments

The examples above are rather basic, the try blocks don't run yet, and none of the results get published, lets improve upon that:

const experiment = new Scientist('experiment #1', { async: true });

experiment.enabled(() => {
    // see "Ramping up experiments" below
    return true;
})

experiment.publish(results => {
   // see "Publishing results" below
   console.log(results)
});

experiment.use(async () => 12);
experiment.try(async () => 10 - 8);

const result = await experiment.run();

Controlling comparison

Scientist compares control and candidate values using lodash.isequal. To override this behaviour, use compare to define how to compared observed values instead:

const experiment = new Scientist('experiment #1', { async: true });

experiment.use(async () => 12);
experiment.try(async () => 10 - 8);

experiment.compare((control, candidate) => {
    return control.id === candidate.id
})

const result = await experiment.run();

Keeping it clean

Sometimes you dont want to store the full value for later analysis. For example an experiment may return a whole object instance, but you only care about a specific property. You can define how to clean these values in an experiment:

const experiment = new Scientist('experiment #1', { async: true });

experiment.use(async () => 12);
experiment.try(async () => 10 - 8);

experiment.clean((value) => {
    return value.map(n => n.id).sort()
})

const result = await experiment.run();

And this cleaned value is available in observations in the final published result:

const experiment = new Scientist('experiment #1', { async: true });

// ...

experiment.publish((result) => {
    console.log(result.control.value)           // [<User Bob>, <User alice>]
    console.log(result.control.cleaned_value)   // [alice, Bob]
})

Ramping up experiments

As a scientist, you know its always important to be able to turn your experiment off, lest it run wild. In order to control whether or not an experiment is enabled, you must include the enabled method in your implementation:

let percentEnabled = 100;

const experiment = new Scientist('experiment #1', { async: true });

// ...

experiment.enabled(() => {
    return percentEnabled > 0 && (Math.random() * 100) < percentEnabled
})

const result = await experiment.run();

This code will be invoked for every method with an experiment every time, so be sensitive about its performance

Publishing results

Whats the point of being a Scientist if you can't publish your results?

You must implement the publish(result) method, and you can publish the data however you like. For example timing data could be sent to graphite, and mismatches coule be sent to a debugging service.

The structure of result is:

{
    "name": "<Experiment Name>",
    "matched": true,
    "execution_order": ["control", "candidate"],
    "error": false,
    "control": {
        "duration": 100,
        "value": 20,
        "cleaned_value": 20,
        "startTime": 100002320,
        "endTime": 1000021000,
        "error": "error ...",
    },
    "candidate": {
        "duration": 100,
        "value": 20,
        "cleaned_value": 20,
        "startTime": 100002320,
        "endTime": 1000021000,
        "error": "error ...",
    },
}

Handling errors

Scientist handles and tracks all errors raised in a try or use block. Scientist also calls publish which contains all of the information about any errors.

In control code

If an error is thrown in a use block, it will eventually be thrown by the Scientist module (in order to mimick real world), therefore you will want to handle this properly.

In candidate code

If an error is thrown in a try block, it will be swallowed and not thrown, you will want to handle this in your publish block.

Breaking the rules

Sometimes it can be useful to break the rules.

Run Control or Candidate only

There will be times when you cannot have both the control and candidate running sequentially (such as performing a database update, but l would not advise using this library for that).

In order to get around that, we can use the run_only method, this will either run the candidate or control, and return the result from whatever function you ran:

const experiment = new Scientist('experiment #1', { async: true });

experiment.use(async () => 12);
experiment.try(async () => 10 - 8);

const candidateResult = await experiment.run_only('candidate');

// OR

const controlResult = await experiment.run_only('control');

In both scenarios above, the publish method will still be invovked, but the control or candidate properties will be null depending on which one you ran.

A/B Testing

Similar to the above, we can utilize the run_only method to perform A/B testing:

const experiment = new Scientist('experiment #1', { async: true });

experiment.use(async () => 12);
experiment.try(async () => 10 - 8);

const result = experiment.run_only(['control', 'candidate'][Math.round(Math.random())]);

Maintainers

@olliejennings