filepile

file backed work queues

use

var filepile = require('filepile');

// create a new pile called 'emails'
var pile = filepile('emails', function(details, done) {
    // details is whatever job data you provide
    // call done() when done
});

// add things to the pile
// the will be processed by the function above
pile({ to: '[email protected]', body: 'hello' });

multiple producers single consumer

Each filepile will only allow for one consumer but multiple producers. Any named pile which already has a consumer on the same system will not allow another consumer to process jobs. If one of the consumers dies, then another consumer will start.

var cluster = require('cluster');

if (cluster.isMaster) {
    for (var i=0 ; i<4 ; ++i) {
        cluster.fork();
    }
    return;
}

var filepile = require('filepile');

// only one of the processes will invoke this function
var pile = filepile('emails', function(details, done) {
    console.log(details);
    done();
});

// both processes will generate "work"
setInterval(function() {
    pile({ foo: 'bar' });
}, 1000);

how it works

All producers write json files to a folder in /tmp. These files are read by the single consumer and processed. A lockfile is used to ensure that there is only one consumer.

The consumer listens using fs.watch for new files and processes them.

gotchas

filepile is not meant to work across machines at this time and will only ensure one consumer with multiple processes on the same machine.

install

npm install filepile