Serverless framework to consume webhooks at scale
Nami is a serverless framework for consuming webhooks at scale.
When events occur in batches, a wave of webhooks may overwhelm consumers, resulting in dropped data. Always-on systems must be provisioned to handle worst-case scenarios, however, this is inefficient as these resources will be idling most of the time.
The Nami framework accommodates bursty webhook traffic with AWS Lambda Functions as a Service (FaaS), which dynamically scales computing resources to match the flow of incoming data. Nami then throttles the flow using an SQS queue as message broker to send the data to a MongoDB data store deployed using Docker on an EC2 instance. The framework abstracts away the complexities of cloud infrastructure and the effort required to deploy, configure, and choreograph all of these services.
Sachin Chandy Software Engineer London, UK
Wendy Kuhn Software Engineer Austin, TX
Nick Miller Software Engineer Los Angeles, CA
- AWS account
- AWS CLI
- Node.js >= 8.10
Nami requires that users have an account with AWS and have set up an AWS CLI configuration on their local machine. If you have not already done so, please visit Configuring the AWS CLI for instructions. Nami will use the default credentials and region specified within that profile in order to interact with AWS services.
npm install -g nami-serverless
Nami commands conform to the following structure:
nami <commandName> [<name>]
nami create <name>
optional: create local directories and files for pre-queue and post-queue Lambda functions for user to insert custom logic before deploying
nami deploy <name>
deploy API endpoint and accompanying scalable architecture. User can register endpoint with the webhook provider
deploy command will create instances of the below:
- API Gateway resource
- SQS Queue (with Dead Letter Queue attached)
- Pre- and post-queue Lambda functions
- EC2 instance with MongoDB deployed using Docker
The first time the
deploy command is run, Nami will also create:
- a hidden directory within the user's home directory that holds configuration files and serves as a staging directory for deploying the Lambda functions
- IAM roles for the Lambda functions
- an API Gateway
nami destroy <name>
delete API endpoint and accompanying scalable architecture
All instances of AWS services for a particular endpoint will be deleted, except for the Elastic Block Storage (EBS) volume that persists any webhook data written to the MongoDB database.
list active API endpoints
documentation of commands
Accessing AWS services
The Nami framework deploys instances of multiple AWS services. While these instances can be deleted using the
destroy command, they can also be accessed and modified using the AWS CLI or via the web console.
At-least once delivery
Due to the behavior of AWS services used by Nami, the nature of the HTTP request/response cycle, and retry policies of webhook providers, exactly once delivery of webhook data in not possible. Users should be aware of this and either write applications to be idempotent, or remove duplicate messages.
Webhook providers generally do not guarantee delivery of events in the order in which they are generated. Your endpoint should not expect delivery of events in order and should handle this accordingly. Event timestamps can be used to order messages once they are written to the database.
Security and SSH
To ensure that the data store is not accessible from the public internet, the post-queue Lambda function and EC2 instance are both within their own Virtual Private Cloud (VPC) and the EC2 instance allows inbound access only from the post-queue Lambda security group. Users can access their own AWS EC2 instances using SSH, however, TCP port 22 is closed by default when users deploy instances of Nami. This can be opened by the user either by using the
authorize-security-group-ingress aws cli command, or by editing the inbound rules for the
<name>EC2SecurityGroup to allow SSH connections from your desired source.
Once port 22 is opened, a user can SSH into the EC2 instance and access their webhook data directly from MongoDB.
- From the directory with the
nami.pemkeypair file, SSH into the EC2 instance. Connection details can be found by navigating to the EC2 section of the web console, and clicking the 'Connect' button once the EC2 instance is selected.
yeswhen prompted to finish initiating the connection.
sudo docker exec -it namiStore bashto create a new Bash session in the running
mongo namito start the Mongo shell with database
- Access your webhook data in
Common commands include:
db.namiCollection.find()- retrieves all documents from
db.namiCollection.count()- display the number of documents in