lambda-dynamodb-router

A module for AWS Lambda functions to create routers that handle changes to a DynamoDB table and route to other Lambda functions via SNS or direct invocation

AWS Lambda is a compute platform for running code that responds to events without provisioning or managing the underlying infrastructure. One such event that Lambda functions can be developed to respond to are changes to a DynamoDB table in the forms of streams. With each insert/modify/delete event in DynamoDB, functions configured to listen to a table's stream will fire. Particuarly busy tables that have a growing number of functions triggered will waste Lambda execution time determining if the change is relevant.

This module allows for the creation of a single Lambda function that acts as an intermediary to the DynamoDB changes, and when configured routes match the chnges, can fire subsequent Lambda functions via SNS or via direct invocation depending on the configuration. Lambda invocation uses 'Event' for the invocation type parameter, and the route configuration can be set to include a property from the record to be used as the qualifier parameter.

Usage

A simple lambda function configured with a single route responding to an insert in a table with criteria

var router = require('lambda-dynamodb-router')
var aws = require('aws-sdk')
var sns = new aws.SNS()
var lambda = new aws.Lambda()

routes = {
	insert: [
		{
			name: 'basic route',
			is: [
				{
					'state': 'open'
				}
			],
			topics: [
				'arn:aws:sns:[region]:[account]:[topic-name]'
			],
			lambdas: [
				'arn:aws:lambda:[region]:[account]:function:[function-name]'
			]
		}
	]
}

exports.handler = function(event, context, callback) {
    router.routeChanges(routes, sns, lambda, event, (err, failedRecords) => {
		// Do something with failed records
        callback()
    })
}

Configuration

The route configuration has a number of rules to help match criteria in your database updates. The routes below demonstrate the various rules available, which you can mix and match as appropriate. Insert routes require 'is' rules, remove routes require 'was' rules and modify routes can use a combination of 'is', 'was' and 'changed' rules. 'is' and 'was' rules compare configuration new and old records respectively. 'changed' rules look for variance in the values of keys in new and old records.

By default, the payload sent to SNS or Lambda is the normalised record from DynamoDB that triggered the router to fire. The 'payload' and 'includeTableName' options for each route can alter the payload structure.

var _ = require('lodash')

routes = {
	insert: [
		{
			name: 'insert route',
			// Property from the record to send as the payload instead of the entire record, optional
			payload: 'record property',
			// Property from the record to use as the qualifier for the lambda invocation, eg, 'environment' - optional
			qualifier: 'qualifier property',
			// Property from the record to use as the prefix for the lambda name, eg, ['environment', 'test'] - optional
			prefix: ['prefix property', 'default prefix'],
			// Property to send the tableName that triggered the route, useful when multiple routers trigger the same function
			// This changes the payload that is sent to SNS/Lambda to the following structure which the configured lambda or SNS
			//  {
			//    payload: 'payload object'
			//    context: {
			//      tableName: 'table name from eventSourceARN'
			//    }
			//  }
			// Optional
			includeTableName: true | false
			// Compare new record
			is: [
				// compare key:value pairs, must be defined and of the same type
				{
					'property_one': 'a',
					'property_two': 'b'

				},
				// compare if property three is one of the values in the array
				{
					'property_three': ['x', 'y', 'z']
				},
				// check key:value pairs do not match, must be defined and of the same type
				{
					not: {
						'property_three': 'z'
					}
				},
				// check keys have defined values
				{
					defined: ['property_four', 'map.map_property']
				}
				// check keys are undefined (note: DynamoDB can't have null values)
				{
					undefined: ['property_five']
				},
				// Use functions to match more complex rules - using the find function from lodash
				{
					matches: [
						function (record) {
							return(_.find(record.array, function(obj) {
								return (obj.property_one == 'one' && obj.property_two !== undefined)
							}) !== undefined)
						}
					]
				}
			],
			// One or more topics can be set for each route
			topics: [
				'arn:aws:sns:[region]:[account]:[topic-name]',
				'arn:aws:sns:[region]:[account]:[topic-name-two]'
			],
			lambdas: [
				'arn:aws:lambda:[region]:[account]:function:[function-name]'
			]
		}
	],
	modify: [
		{
			name: 'modify route',
			// Property from the record to send as the payload instead of the entire record, optional
			payload: 'record property',
			// Property from the record to use as the qualifier for the lambda invocation, eg, 'environment' - optional
			qualifier: 'qualifier property',
			// Property from the record to use as the prefix for the lambda name, eg, ['environment', 'test'] - optional
			prefix: ['prefix property', 'default prefix'],
			// Property to send the tableName that triggered the route
			// This changes the payload that is sent to SNS/Lambda to the following structure
			//  {
			//    payload: 'payload object'
			//    context: {
			//      tableName: 'table name from eventSourceARN'
			//    }
			//  }
			// Optional
			includeTableName: true | false
			// Accepts 'new', 'old' or 'both'. Default to 'new' if not set. Backwards compatible with 'useOldRecord'.
			// If set to 'both', record passed will include both new and old records in an object with properties
			// 'new' and 'old'
			useRecord: 'old'
			//Compare new record
			is: [
				// compare key:value pairs
				{
					'property_one': 's',
					'property_two': 't'

				},
				// check key:value pairs do not match
				{
					not: {
						'property_three': 'k'
					}
				},
				// check keys have defined values
				{
					defined: ['property_four']
				},
				// check keys are undefined (note: DynamoDB can't have null values)
				{
					undefined: ['property_five', 'map.map_property']
				},
				// Use functions to match more complex rules - using the find function from lodash
				{
					matches: [
						function (record) {
							return(_.find(record.array, function(obj) {
								return (obj.property_one == 'one' && obj.property_two !== undefined)
							}) !== undefined)
						}
					]
				}
			],
			//Compare old record
			was: [
				// compare key:value pairs
				{
					'property_one': 'a',
					'property_two': 'b'

				},
				// check key:value pairs do not match
				{
					not: {
						'property_three': 'l'
					}
				},
				// check keys have defined values
				{
					defined: ['property_four']
				},
				// check keys are undefined (note: DynamoDB can't have null values)
				{
					undefined: ['property_five']
				}
			],
			changed: ['property_with_a_different value'],
			// One or more topics can be set for each route
			topics: [
				'arn:aws:sns:[region]:[account]:[topic-name]'
			],
			lambdas: [
				'arn:aws:lambda:[region]:[account]:function:[function-name]'
			]
		}
	],
	remove: [
		{
			name: 'remove route',
			// Property from the record to send as the payload instead of the entire record, optional
			payload: 'record property',
			// Property from the record to use as the qualifier for the lambda invocation, eg, 'environment' - optional
			qualifier: 'qualifier property',
			// Property from the record to use as the prefix for the lambda name, eg, ['environment', 'test'] - optional
			prefix: ['prefix property', 'default prefix'],
			// Property to send the tableName that triggered the route
			// This changes the payload that is sent to SNS/Lambda to the following structure
			//  {
			//    payload: 'payload object'
			//    context: {
			//      tableName: 'table name from eventSourceARN'
			//    }
			//  }
			// Optional
			includeTableName: true | false
			// Compare old record
			was: [
				// compare key:value pairs
				{
					'property_one': 'w',
					'property_two': 'x'

				},
				// check key:value pairs do not match
				{
					not: {
						'property_three': 'y'
					}
				},
				// check keys have defined values
				{
					defined: ['map.map_property','can.go.deeper']
				}
			],
			// One or more topics can be set for each route
			topics: [
				'arn:aws:sns:[region]:[account]:[topic-name]',
				'arn:aws:sns:[region]:[account]:[topic-name-two]'
			],
			lambdas: [
				'arn:aws:lambda:[region]:[account]:function:[function-name]'
			]
		}
	]
}

Install

With npm installed, run

$ npm install lambda-dynamodb-router

License

ISC