Containers · · 6 min read
Scaling an ECS Fargate worker on SQS queue depth, down to zero
By Rushit Chandegara, Founder & CTO, InfraCloudEdge
A background worker that reads from an SQS queue is one of the easiest workloads to run on ECS Fargate, and one of the easiest to overpay for. Scaled on CPU, it either runs idle tasks all night or falls behind when a batch arrives. Scaling on the queue itself fixes both.
Scale on backlog per task
The number of visible messages alone is not a good target, because it says nothing about how many tasks are already working on it. AWS recommends backlog per task: visible messages divided by running tasks. Decide how many messages one task can clear in an acceptable time, and use that as the target.
- Publish backlog per task as a custom CloudWatch metric, or build it with metric math from ApproximateNumberOfMessagesVisible and the service's running task count.
- Attach a target tracking policy with Application Auto Scaling.
- Set a sensible maximum so a flood of messages cannot exhaust your database connections.
Scaling to zero
An ECS service can run zero tasks, and Application Auto Scaling accepts a minimum of 0. The catch is that backlog per task cannot be calculated with no tasks running. Add a step scaling policy on queue depth that starts one task as soon as a message is visible; target tracking takes over from there.
Make shutdowns safe
- ECS sends SIGTERM before stopping a task, then SIGKILL after the stop timeout (30 seconds by default, up to 120 on Fargate). Stop taking new messages on SIGTERM and finish the current one.
- Set the queue's visibility timeout longer than your slowest message, so a message is not handed to a second worker mid-way.
- Configure a dead-letter queue with a maxReceiveCount, so one bad message cannot loop forever.
- Make processing idempotent: SQS standard queues can deliver a message more than once.
Ship it as code
Keep the service, scaling policies, alarms and queues in Terraform with the rest of the environment, and deploy the image from CI without stored keys. Then a copy of production, for testing or for a new region, is one plan away.
We built this pattern for a healthcare payments platform: production runs one to four workers with the backlog, and its QA and UAT environments scale to zero when idle.
