sqs-ecs-jobs

A worker that consumes jobs from SQS and runs them on predefined ECS tasks.

Has the following qualities:

• Jobs run at least once (quality of SQS)
• Each task runs within its own docker container
• Supports arbitrary data to be passed
• Simple JSON protocol that can be used anywhere SQS can be called from

It does not:

• Configure ECS task definitions for you
• Handle scheduling (please see Scheduling)

Running

You'll need to start two processes:

npm run consume

Which will consume jobs in the SQS queue in the format:

{
  "name": "SomeECSTaskDefinitionName",
  "payload": {
    "arbitrary": 123,
    "data": "here"
  },
  "maxRetries": 1
}

Where payload are maxRetries are optional (will retry infinitely by default). You'll also need to run:

npm run watch

This process will ensure that jobs are run successfully or, if they fail, will ensure they are scheduled to be rerun or moved in to a Dead Letter Queue of your choosing.

Docker

To run in docker:

docker run -i -t nearst/sqs-ecs-jobs npm run consume

docker run -i -t nearst/sqs-ecs-jobs npm run watch

You can pass either command line arguments or provide an environment to docker.

Config

Node-config is used for a flexible config:

{
  "queues": {
    "main": SQS main queue URL,
    "processing": SQS processing queue URL,
    "deadLetter": SQS Dead Letter Queue URL
  },
  "retryBase": Retry backoff multiplier (default 60),
  "maxRetries": Total attempts to get job to run (default 6),
  "clusterName": Cluster to run ECS tasks on
}

Can be configured using command line overrides or environmental variables:

{
  "queues": {
    "main": "QUEUE_URL",
    "processing": "PROCESSING_QUEUE_URL",
    "deadLetter": "DEAD_LETTER_QUEUE_URL"
  },
  "retryBase": "RETRY_BASE",
  "maxRetries": "MAX_RETRIES",
  "clusterName": "CLUSTER_NAME"
}

AWS config

AWS credentials need to be provided in any of their standard ways, but is probably easiest to provide using AWS environmental variables.

Scheduling

Scheduling is not supported natively, however you can utilise Cloudwatch scheduled events, SNS notifications and SQS SNS subscriptions to achieve this without having to run a separate cron daemon. This will have the guarantees of running that Cloudwatch scheduled events, SNS and SQS SNS subscriptions have.

Roadmap

• Support aggregation of similar tasks within a given timeframe (addition to job's JSON format)
• Better logging and error notifications
• Simple support interface to augment AWS console

Contributing

Feel free to contribute by forking and providing a Pull Request. Feel free to raise an issue if you don't know how to tackle a particular problem.