AC SQS

This tool is a wrapper for AWS SDK's SQS function. It includes handling of big SQS messages using S3.

Breaking changes - version 2

This new class-based wrapper is not compatible with older versions.

Usage

yarn add ac-sqs

const acsqs = new ACSQS({
  account: '123456789', // AWS account id
  availableLists: [{
    name: 'listName'
    batchSize: 10
  }],
  useS3: {
    enabled: true,
    bucket: 'ac-sqs-messages' // s3 bucket
  }
})

await acsqs.sendSQSMessage({ name: listName, message: 'This is my message' })

Available methods/calls

Initiate class

Account The account id of your AWS account. This is required

Available lists Array of AWS SQS lists that will be used by this function. Every item in the list must be an object with the following properties:

• name -> the name of the list in AWS SQS. See below for more info.
• batchSize -> number of messages to fetch per call. Max 10, defaults to 1
• visibilityTimeout -> see AWS SQS for details, defaults to 30. If set, visibilityTimeout management is activated
• waitTime -> see AWS SQS for details, defaults to 20
• fifo -> set to true, if this is a fifo list
• localPrefix -> set your local prefix. See below for more info
• debug -> if true, all SQS payloads for that list will be logged (level debug)
• throwError -> if true, error will throw otherwise they will only be logged (default)

batchExtendInterval [optional] Interval in milliseconds at which the visibility timeout extension loop runs. Defaults to 5000 (5 seconds). Can be set to a lower value in test environments for faster shutdown.

Name should be the plain name of the list. Parameters like fifo or test (in test environment) or localPrefixes (for local development) should not be part of the list name. LocalPrefix can be used if multiple developers work on your project and you want to make sure they all work on their own SQS list without changing the name of all SQS lists in your main project.

Example for a FIFO list: Let the name be "mylist". This will automatically create "test_mylist" in test environment (NODE_ENV=test) and "local_LOCALPREFIX_mylist" if you set localPrefix.

useS3 [optional] AWS SQS only allows a certain message size (262kb at the time of the documentation). To process messages with a bigger payload, the actual message content will be stored in a file on AWS S3. The feature is enabled by default and you must make sure to set a bucket.

useS3: {
  enabled: true,
  bucket: 'ac-sqs-message'
}

Visibility Timeout Management If you set list config parameter visibilityTimeout, the automatic visibilityTimeout management will be activated. It will make sure that visibilityTimeout for messages is automatically extended.

Make sure your function can read, write and delete messages in the bucket.

sendSQSMessage

Create a messsage in a SQS list.

name The name of the list to send a SQS message to.

message The actual message (as plain test). If you want to send JSON, please stringify!

messageGroupId [optional] See AWS SQS for details

deDuplicationId [optional] See AWS SQS for details

delay [optional] See AWS SQS for details

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_SendMessage.html

receiveSQSMessages

Receive messages from SQS. By default up to 10 messages are fetched per call. You set a lower number using the batchSize parameter in the corresponding availableLists entry.

name The name of the list to receive a SQS messages from.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html

deleteSQSMessages

Deletes one or multiple SQS messages and also cleans up/deletes related S3 files.

name The name of the list to delete SQS messages from.

items An array of objects to delete. Every objects must a least have properties Id and ReceiptHandle. If the messages was originally stored on S3 the entry must also contain Key.

The Id parameter is the message id, ReceiptHandle is an identifier you get with the receive request.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_DeleteMessageBatch.html

getQueueAttributes

Get information about a list

name The name of the list to get information about.

attributes An array of metadata to get. By default only "ApproximateNumberOfMessages" is requested. Please see AWS SQS for other metadata options.

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_GetQueueAttributes.html

createQueues

Call with all lists and let the function check if the queues exist. If not, the missing queue is created.

Test

Run tests using yarn test or npm test.

No AWS credentials or real infrastructure required. All SQS and S3 calls are stubbed using sinon, so tests run fully offline and complete in under 2 seconds.

NODE_ENV=test yarn test

The test suite covers:

• send, receive and delete (single and batch)
• large message handling via S3 (PutObject / GetObject / DeleteObjects)
• automatic visibility timeout extension (processBatchExtensions)
• max extensions and invalid receipt handle cleanup
• createQueues (existing and missing queues)
• extendVisibility legacy method
• getVisibilityStats breakdown
• error handling at class and function level (throwError)

Misc

Links

• Website

License

MIT License Copyright © 2009-present, AdmiralCloud AG, Mark Poepping