firebase-queue-reducer

This is a simple wrapper around firebase-queue that abstract a data manipulation pipeline from one tasks queue to another.

The problem we try to solve here:

Say you have some raw json documents that you want to refine and you want to split the manipulation logic among different processes:

// raw documment:
{ name: 'Marco', surname: 'Peg', birthDate: 19810630 }

Now I want to compute the information age from the raw document and remove the birthdate because is not really relevant to my project:

// resulting document
{ name: 'Marco', surname: 'Peg', age: 34 }

At this point I want to compute the fullName property instead of the two distinct values, the resulting document will look like:

// resulting document
{ fullName: 'Marco Peg', age: 34 }

The idea is to split each transformation logic into a simple script and use Firebase as message bus to coordinate the transformation.

Define the reducers:

You need 3 queues of documents to handle that:

• raw (original documents)
• step1 (age)
• step2 (name)

Each document will move from raw -> step1 and from step1 -> step2 and each process can be represented as a reducing function:

function reduceAge(data) {
	return new Promise(function(resolve, reject) {
		resolve({
			name: data.name,
			surname: data.surname,
			age: computeCurrentAge(data.birthDate)
		});
	});
}

function reduceName(data) {
	return new Promise(function(resolve, reject) {
		resolve({
			fullName: [data.name, data.surname].join(' '),
			age: data.age
		});
	});
}

reducing functions must return a Promise so to be able to handle asynchronous tasks.

Setup the services

At this point we can start playing with NodeJS and define the two services that will use the already defined reducers:

// step1.js:
var fb = require('./my-local-firebase-instance');
require('firebase-queue-reducer')({
	source: fb.child('raw'),
	target: fb.child('step1'),
	reducer: reduceAge
});

// step2.js
var fb = require('./my-local-firebase-instance');
require('firebase-queue-reducer')({
	source: fb.child('step1'),
	target: fb.child('step2'),
	reducer: reduceName
});

Running the pipeline

At this point you can run those two services and start feeding the raw/tasks data endpoint with documents. The pipeline will work out the reducing logic and the final documents will land into step1/tasks.

You can set up as many reducing steps you want keeping each setp as a standalone application. Each app focuses on the reducer function which should be carefully tested.

Because each reducing service will subscribe to the source queue and push to the target queue thei are completely independent from each other. Each process can fail and be rebooted withouth affect the others.

Install

npm install firebase-queue-reducer

Config Options

source, dest

FirebaseQueue end points.

reducer

A function that receive a task data from the source queue and return a Promise.

If the promise resolve then the reduced document is pushed to the dest/tasks data endpoint.

If the promise reject the original document is pushed to source/errors for further analisys and debug.

workers

Provide an integer number to tell the queue how many workers to run.