Scaling AWS ECS services with Alarms, Target tracking & CloudFormation

The topic of ECS autoscaling is a vast area of heated discussions and broken dreams.

Stan Yudin

ECS Autoscaling

It is quite hard to come up with efficient scaling policies for Amazon Elastic Container Services (ECS). The more distributed your architecture, the more issues with cascading load and increasing latency you are going to face.

But fear not, the promised salvation in the form of autoscaling for your services is here to save the day and distribute your computing load evenly across your micro services. So let’s examine what we have to work with to achieve that.

Scaling services

Autoscaling of ECS services is implemented as an automated action executed upon an event: scale in or scale out. The source of such event can be an alarm with either a StepScaling type of policy or a TargetTrackingScaling type. Usage of target tracking is very similar to the implementation for DynamoDB, with the options of ECSServiceAverageCPUUtilization and ECSServiceAverageMemoryUtilization metrics available for tracking.

Notice that ECS can track only average metrics of the service, so you need to make sure that tasks have load distributed evenly on the load balancer. Significant gaps between maximum and average consumption can lead to a termination of a task due to “out of memory” or out of CPU credits, and lead to 502 errors.

max vs avg

New and Old

ECS scaling policies can be combined to produce even greater efficiency in load distribution. Usage of the StepScaling policies can handle scale out events on Application Load Balancers (ALBs) or SQS metrics by estimating the load in the input source. ALB’s Target Group metrics such as AWS/ApplicationELB/RequestCountPerTarget are a good baseline to start policies. Size of the SQS queue is another example of a deterministic metric to estimate incoming load for service.

A combination of StepScaling and TargetTrackingScaling looking at ECSServiceAverageCPUUtilization or ECSServiceAverageMemoryUtilization can allow greater flexibility in how your service can react on load. If it is possible to determine whether the service in question is mostly CPU or memory bound, then selection of a threshold for one of these average metrics should be pretty easy by observing the service under generated test load.

CloudFormation support for ECS scaling

To define an ECS service with scaling policies in CloudFormation you need to have a cluster, instance role for EC2 hosts, and other essentials omitted from this example.

First we need a service role to perform scaling actions on our behalf.

ScalingRole: 
  Type: AWS::IAM::Role
  Properties: 
    RoleName: ScalingRole
    AssumeRolePolicyDocument: 
        Version: "2012-10-17"
        Statement: 
        - Effect: Allow
            Principal: 
            Service: 
            - application-autoscaling.amazonaws.com
            Action: 
            - sts:AssumeRole

ScalingRolePolicy:
  Type: AWS::IAM::Policy
  Properties:
    Roles:
    - !Ref ScalingRole
    PolicyName: ScalingRolePolicyPolicy
    PolicyDocument: 
        Version: '2012-10-17'
        Statement: 
        - Effect: Allow
            Action:
            - application-autoscaling:*
            - ecs:RunTask
            - ecs:UpdateSerice
            - ecs:DescribeServices
            - cloudwatch:PutMetricAlarm
            - cloudwatch:DescribeAlarms
            - cloudwatch:GetMetricStatistics
            - cloudwatch:SetAlarmState
            - cloudwatch:DeleteAlarms
            Resource: '*'

Now we’re going to have a look at a service definition, its target group for ALB, scaling targets and policies, and a CloudWatch alarm. For this example we are going to define ExampleCPUAutoScalingPolicy for a new capacity to grow to a value so that current usage ECSServiceAverageCPUUtilization accounts for 50% and ExampleRequestsAutoScalingPolicy when we have more than 1000 requests per target within a minute.

  ExampleTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Port: 80
      Protocol: HTTP
      VpcId: !Ref VpcId
      HealthCheckIntervalSeconds: 30
      HealthCheckPath: /status
      HealthCheckTimeoutSeconds: 15
      HealthyThresholdCount: 2
      UnhealthyThresholdCount: 6
      Matcher:
        HttpCode: 200
      TargetGroupAttributes:
        - Key: deregistration_delay.timeout_seconds
          Value: 30

  ExampleService:
    Type: AWS::ECS::Service
    Properties:
      TaskDefinition: !Ref ExampleTask # omitted
      PlacementStrategies:
        - Field: attribute:ecs.availability-zone
          Type: spread
      DesiredCount: 1
      Cluster: ExampleCluster # omitted
      LoadBalancers:
        - TargetGroupArn: !Ref ExampleTargetGroup
          ContainerPort: 8080
          ContainerName: example-service

  ExampleAutoScalingTarget:
    Type: AWS::ApplicationAutoScaling::ScalableTarget
    Properties:
      MaxCapacity: !Ref MaxServicesCount # parameters
      MinCapacity: !Ref MinServicesCount
      ResourceId:
        Fn::Sub:
          - service/ExampleCluster/${ServiceName}
          - ServiceName: !GetAtt ExampleService.Name
      RoleARN: !GetAtt ScalingRole.Arn
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs

  ExampleCPUAutoScalingPolicy:
    Type: AWS::ApplicationAutoScaling::ScalingPolicy
    Properties:
      PolicyName: ExampleCPUAutoScalingPolicy
      PolicyType: TargetTrackingScaling
      ScalingTargetId: !Ref ExampleAutoScalingTarget
      TargetTrackingScalingPolicyConfiguration:
        DisableScaleIn: True
        TargetValue: 50 
        ScaleInCooldown: 60
        ScaleOutCooldown: 60
        PredefinedMetricSpecification: 
          PredefinedMetricType: ECSServiceAverageCPUUtilization

  ExampleRequestsAutoScalingPolicy:
    Type: AWS::ApplicationAutoScaling::ScalingPolicy
    Properties:
      PolicyName: ExampleRequestsAutoScalingPolicy
      PolicyType: StepScaling
      ScalingTargetId: !Ref ExampleAutoScalingTarget
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      StepScalingPolicyConfiguration:
        AdjustmentType: ChangeInCapacity
        Cooldown: 60
        MetricAggregationType: Average
        StepAdjustments:
        - MetricIntervalLowerBound: 0
          ScalingAdjustment: 1
        - MetricIntervalUpperBound: 0
          ScalingAdjustment: -1
  
  ExampleRequestsAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      MetricName: RequestCountPerTarget
      Namespace: AWS/ApplicationELB
      Statistic: Sum
      Period: 60
      EvaluationPeriods: 1
      Threshold: 1000
      AlarmActions:
        - !Ref ExampleRequestsAutoScalingPolicy
      OKActions:
        - !Ref ExampleRequestsAutoScalingPolicy
      Dimensions:
      - Name: TargetGroup
        Value: !GetAtt ExampleTargetGroup.TargetGroupFullName
      ComparisonOperator: GreaterThanOrEqualToThreshold

Notice that the parameters section of the ExampleCPUAutoScalingPolicy resource contains DisableScaleIn: true for a specific reason. In order to guarantee that requests for target scaling events have priority over target tracking, the scale in logic of tracking can be disabled completely.

Stability, stability is the key

Ok, so now we have the service scaling up and down based on the number of requests per target in Elastic Load Balancers (ELB). However, you will notice that the threshold in StepAdjustments for scale up starts right after scale down. It means that your service’s desired count would oscillate around some value, going up and down with new tasks spun up.

To allow for a window of stability, you need to have a range with ScalingAdjustment: 0, whereby you have a boundary to increase and decrease desired count. That way it is possible to have an alarm to alert on the scale in boundary, and StepAdjustments to interpret the range.

Lets see an example, where we want to scale out on more than RequestsScaleOutThreshold requests per target, and scale in on less than RequestsScaleInThreshold:

  ExampleRequestsAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      MetricName: RequestCountPerTarget
      Namespace: AWS/ApplicationELB
      Statistic: Sum
      Period: 60
      EvaluationPeriods: 1
      Threshold: 500 # scale in boundary to trigger the alarm
      AlarmActions:
        - !Ref ExampleRequestsAutoScalingPolicy
      Dimensions:
      - Name: TargetGroup
        Value: !GetAtt ExampleTargetGroup.TargetGroupFullName
      ComparisonOperator: GreaterThanOrEqualToThreshold

  ExampleRequestsAutoScalingPolicy:
    Type: AWS::ApplicationAutoScaling::ScalingPolicy
    Properties:
      PolicyName: ExampleRequestsAutoScalingPolicy
      PolicyType: StepScaling
      ScalingTargetId: !Ref ExampleAutoScalingTarget
      ScalableDimension: ecs:service:DesiredCount
      ServiceNamespace: ecs
      StepScalingPolicyConfiguration:
        AdjustmentType: ChangeInCapacity
        Cooldown: 60
        MetricAggregationType: Average
        StepAdjustments:
        - MetricIntervalLowerBound: !Ref RequestsScaleOutThreshold
          ScalingAdjustment: 1
        - MetricIntervalLowerBound: !Ref RequestsScaleInThreshold
          MetricIntervalUpperBound: !Ref RequestsScaleOutThreshold
          ScalingAdjustment: 0
        - MetricIntervalUpperBound: !Ref RequestsScaleInThreshold
          ScalingAdjustment: -1

Here we have a range between MetricIntervalLowerBound=RequestsScaleInThreshold and MetricIntervalUpperBound=RequestsScaleOutThreshold where ScalingAdjustment=0 and no changes are made to desired count. This will ensure that oscillation of desired count does not happen to you.

Only one? Take two!

Another approach would be to define alarms, one to scale out and one to scale in. Each would have a specific range and specific policy associated. Such approach in fact is used quite a lot, but the problem is that CloudWatch alarms are not free, in fact they are pretty expensive.

Further reading and additional details are found here: AWS::ApplicationAutoScaling::ScalableTarget and AWS::CloudWatch::Alarm

Contact us

We will get back to you within 24 hours.