delaysqs

A simple node module to extend the SQS delay over the 15 minutes hard-limit imposed by Amazon. This is done by effectively reading and relaying the message at 15-minute intervals until the remaining delay time is below the 15-minute limit. In other words, if at 01:00:00 I want to send a message to the Queue with a delay of 1 hr 50 minutes, the delaysqs library will first send the message with a delay of 900 seconds, and it will repeat at 01:15:00, 01:30:00 and so on until 02:45:00, at that point it will send the message once more to the queue but with a DelaySeconds attribute of 300 seconds, so that the message will appear in queue at 02:50:00.

The relaying process uses a custom MessageAttribute named deliveryTimestamp which contains the UNIX timestamp at which the message should finally be consumed. Specifying the explicitly the timestamp makes it such that even if the queue is not polled for periods of time before the delivery time, the message still has a chance of being delivered in time.

Reliability

As SQS is an "At-least-once" deliver system, there is always a slight chance of the message being delivered to two message consumers. At each "relay" step therefore we have a possibility of the message being duplicated, with each of the copy of the message continuing its own independent existence. Therefore it is very important that the processors that will finally handle the message check whether the action indicated by the message has already been carried out or not.

Costs

Polling, sending and receiving messages to/from SQS has an associated cost. Delaysqs uses long polling to minimize the roundtrips from the server in case of no traffic. The minimum costs for monitoring an SQS with long polling (prices are region specific and subject to change, these one reported here are for region us-west-1 and are correct as of Apr 12th, 2016)

4320 request/days * $0.00000050 $/request = 0.0216 $/day (assuming you have already consumerd your first 1 milion requests) + data transfer fees

Due to the possibility of duplicated requests, it is advisable to use delayed messages just as triggers for specific actions, carrying only the id of the action (and not the entire action payload), making the data transfer costs a non issue.

Error handling

The aim of this library is to continue processing the messages in the queue as much as possible, despite of errors occuring. Errors are forwarded verbatim to the error callback function. It is up to the client of this library to analyze the error and decide whether it is the cast to invoke stopPolling and interrupt the processing of messages.

Installation and usage

This package tries to be as little intrusive as possible. For this reason the AWS SDK dependency is declared as peer dependency, and the methods expect to receive an instance of an AWS.SQS object, thus leaving all the AWS configuration details to the caller (chances are that if you want to use this module you are already using the AWS SDK and already have the credentials set up).

To install do

npm install delaysqs

Then require the module in your code

var delaysqs = require('delaysqs');

The main method is the startPolling(sqs, queueUrl, messageCallback, errorCallback) method which takes 4 parameters:

• sqs is a configured instance of AWS.SQS
• queueUrl is a String containing the URL of the SQS queue used to have delayed messages. The URL can be either found in the AWS Console, or queried from the AWS SDK using AWS.SQS.getQueueUrl method
• messageCallback is a function that will be invoked with the body of a message when the deliveryTimestamp for that message has been reached
• errorCallback is a function that will receive errors generated by the library while accessing the SQS queue

Example

var AWS = require('aws-sdk');

var DelaySqs = require('./index');

sqs = new AWS.SQS({
    region: "<my region>",
    accessKeyId: "<my access key id>",
    secretAccessKey: "<my secret access key>",
});

var queueUrl = 'https://sqs.region.amazonaws.com/account/queue_name'


var delayer = DelaySqs(sqs, queueUrl,
function (message) {
    console.log("Message expired");
    console.log(message);

}, function (error) {
    console.log("Error occurred");
    console.log(error);
});

delayer.startPolling();