terraform-aws-sqs-ecs
Terraform module for provisioning an SQS queue paired with an ECS service to run containerized queue consumers.
Architecture
Why This Module?
Running containerized SQS consumers on AWS requires orchestrating many services:
- Complex scaling logic - You need to scale based on both CPU utilization and queue depth, coordinating ECS tasks and EC2 instances
- Capacity planning is tricky - Calculating how many tasks fit per instance requires accounting for CPU, memory, sidecar containers, and OS overhead
- Cost optimization - Using spot instances for queue consumers is ideal but adds configuration complexity
This module solves these problems by:
- Providing dual auto-scaling (CPU + SQS backlog) out of the box
- Automatically calculating task placement based on instance type and task resource quotas
- Supporting spot instances with a single variable (
consumer_on_demand_base_capacity)
- Including CloudWatch monitoring and alerting for queue health
The module is similar to terraform-aws-sqs-pod, but runs consumers from Docker images instead of deploying directly on EC2 instances.
Features
- Dual auto-scaling - Scales on CPU load and SQS backlog per task
- Automatic capacity planning - Calculates tasks per instance from CPU/memory quotas
- Spot instance support - Optional spot instances with configurable on-demand base
- EFS volume support - Mount EFS volumes in consumer containers
- Secret injection - Pass secrets from AWS Secrets Manager to containers
- FIFO queue support - Optional FIFO queue behavior
- Dead-letter queue - Optionally set aside messages consumers can't process, with an alarm (see below)
- CloudWatch monitoring - SQS message age alerting with email notifications
- ECR image tagging - Optionally protect the deployed image from lifecycle pruning (see below)
Quick Start
module "sqs_consumer" {
source = "registry.infrahouse.com/infrahouse/sqs-ecs/aws"
version = "3.2.0"
service_name = "my-consumer"
consumer_subnet_ids = var.private_subnet_ids
consumer_docker_image = "my-org/my-consumer:latest"
alert_notification_email = "[email protected]"
consumer_on_demand_base_capacity = 0 # Use spot instances
}
ECR image tagging
When the consumer image lives in ECR, the repository's lifecycle policy can
prune an image that is still actively deployed. If the consumer is not
redeployed for a while and newer images are pushed, the deployed image may be
deleted, which breaks recovery (a new EC2 host cannot pull it) and leaves stale,
vulnerable images around.
Set enable_ecr_image_tagging = true to deploy a small Lambda alongside the
consumer service. Each time the service reaches steady state, the Lambda tags
the actively deployed image with deployed-at-<timestamp>. Pair this with the
matching retention rule in
infrahouse/terraform-aws-ecr,
which preserves deployed-at-* tagged images.
module "sqs_consumer" {
source = "registry.infrahouse.com/infrahouse/sqs-ecs/aws"
version = "3.2.0"
service_name = "my-consumer"
consumer_subnet_ids = var.private_subnet_ids
consumer_docker_image = "123456789012.dkr.ecr.us-west-2.amazonaws.com/my-consumer:latest"
alert_notification_email = "[email protected]"
enable_ecr_image_tagging = true
}
consumer_docker_image must be an ECR image URI when this is enabled. Images
pulled from Docker Hub or public ECR are silently skipped.
Dead-letter queue
Without a dead-letter queue, a message that every consumer fails on returns to
the queue after each visibility timeout and is received again until it expires,
4 days by default.
Set enable_dead_letter_queue = true to create one, named after the queue with
-dlq (<name>-dlq.fifo for a FIFO queue). A message received
dead_letter_queue_max_receive_count times (5 by default) without being deleted
moves there. The dead-letter queue keeps messages 14 days, SQS's maximum, but a
standard queue's message keeps the time it was first sent: it expires 14 days
after that, so at least 10 days after the move, since the queue keeps messages
4. A FIFO queue's message gets 14 days from the move. An alarm emails
alert_notification_email when the dead-letter queue holds any message.
module "sqs_consumer" {
source = "registry.infrahouse.com/infrahouse/sqs-ecs/aws"
version = "3.2.0"
service_name = "my-consumer"
consumer_subnet_ids = var.private_subnet_ids
consumer_docker_image = "my-org/my-consumer:latest"
alert_notification_email = "[email protected]"
enable_dead_letter_queue = true
}
It is off by default, so an upgrade changes nothing for a module that doesn't
set it. If you already set a redrive policy on queue_url yourself, keep the
option off: the module's redrive policy would replace yours. Turning it off
again removes the queue's redrive policy and deletes the dead-letter queue, with
any messages still in it.
Documentation
Full documentation is available at infrahouse.github.io/terraform-aws-sqs-ecs.
Requirements
| Name | Version |
|---|
| aws | >= 6.0, < 7 |
Providers
Modules
| Name | Source | Version |
|---|
| asg | ./modules/asg | n/a |
| ecs | ./modules/ecs | n/a |
| scaling | ./modules/scaling | n/a |
Resources
Inputs
Outputs
Contributing
See CONTRIBUTING.md for development setup and guidelines.
License
Apache 2.0
