DynamodB Autoscaling with Cloudformation

Autoscaling DynamoDB capacity for fun and profit.

DYNAMODB AUTOSCALING

DynamoDB autoscaling is a feedback-loop based monitoring setup which can dynamically change provisioned capacity for the table or global secondary index. There are two ways one can define autoscaling policy for this resource:

• Step scaling policy
• Target tracking policy

STEP SCALING POLICY

The first kind of policy – step scaling – is based on CloudWatch alarms. This approach is the original autoscaling policy, the kind of which has been available to us from the beginning of days. EC2 autoscaling is based on this, and in general, CloudWatch alarms were the main way to trigger scaling activities.

This approach works well enough, but does not allow the user to allocate capacity larger or smaller than the configured step (scale in or out); in addition it doesn’t allow us to request certain capacity above the level needed currently in this instance of time.

TARGET TRACKING POLICY

The target tracking policy is here to help with the issues highlighted above. The new policy kind uses generic scalable target API application-autoscaling:RegisterScalableTarget and allows dynamic changes to the provisioned capacity. But the most important aspect is that you can ask for capacity as a percentage of current or projected use. This way, each scale out event can take the capacity to a level with a some room to grow, while the next tick of feedback is analysed.

CLOUDFORMATION SUPPORT FOR DYNAMODB

Naturally everything around scalability is covered by the AWS::ApplicationAutoScaling::ScalingPolicy and AWS::ApplicationAutoScaling::ScalableTarget resources. Let’s examine how one can define a policy for a AWS::DynamoDB::Table resource.

We start with defining the service role to perform scaling actions on our behalf.

ScalingRole: 
    Type: AWS::IAM::Role
    Properties: 
    RoleName: ScalingRole
    AssumeRolePolicyDocument: 
        Version: "2012-10-17"
        Statement: 
        - Effect: Allow
            Principal: 
            Service: 
            - application-autoscaling.amazonaws.com
            Action: 
            - sts:AssumeRole

ScalingRolePolicy:
    Type: AWS::IAM::Policy
    Properties:
        Roles:
        - !Ref ScalingRole
        PolicyName: ScalingRolePolicyPolicy
        PolicyDocument: 
            Version: '2012-10-17'
            Statement: 
            - Effect: Allow
                Action:
                - application-autoscaling:*
                - dynamodb:DescribeTable
                - dynamodb:UpdateTable
                - cloudwatch:PutMetricAlarm
                - cloudwatch:DescribeAlarms
                - cloudwatch:GetMetricStatistics
                - cloudwatch:SetAlarmState
                - cloudwatch:DeleteAlarms
                Resource: '*'

Next is the definition of our test table. To define it, we need minimum and maximum capacity values. Here we define a scalable target, the object used to hold a reference to scalable dimension, and range of possible values. In this example we will look into dynamodb:table:WriteCapacityUnits scaling. The read capacity setup is identical with a different scalable dimension.

TestTable:
    Type: AWS::DynamoDB::Table
    Properties:
        TableName: TestTable
        AttributeDefinitions:
            - AttributeName: id
            AttributeType: S
            - AttributeName: external
            AttributeType: S
        KeySchema:
            - AttributeName: id
            KeyType: HASH
        ProvisionedThroughput:
            ReadCapacityUnits: !Ref MinReadCapacityUnits
            WriteCapacityUnits: !Ref MinWriteCapacityUnits
        GlobalSecondaryIndexes:
            - IndexName: TestIndex
            KeySchema:
                - AttributeName: external
                KeyType: HASH
            Projection:
                ProjectionType: ALL
            ProvisionedThroughput:
                ReadCapacityUnits: !Ref MinReadCapacityUnits
                WriteCapacityUnits: !Ref MinWriteCapacityUnits

TableWriteCapacityScalableTarget:
    Type: AWS::ApplicationAutoScaling::ScalableTarget
    Properties: 
        MaxCapacity: !Ref MaxWriteCapacityUnits
        MinCapacity: !Ref MinWriteCapacityUnits
        ResourceId: table/TestTable
        RoleARN: !GetAtt ScalingRole.Arn
        ScalableDimension: dynamodb:table:WriteCapacityUnits
        ServiceNamespace: dynamodb

IndexWriteCapacityScalableTarget:
    Type: AWS::ApplicationAutoScaling::ScalableTarget
    Properties: 
        MaxCapacity: !Ref MaxWriteCapacityUnits
        MinCapacity: !Ref MinWriteCapacityUnits
        ResourceId: table/TestTable/index/TestIndex
        RoleARN: !GetAtt ScalingRole.Arn
        ScalableDimension: dynamodb:index:WriteCapacityUnits
        ServiceNamespace: dynamodb

Now we have two scalable targets; one for primary index and one for global secondary index. Each has a target resource ID, scalable dimension and namespace, as well as a range of possible values.

Now it’s time to define a policy to move these values. Here we’re going to specify the required capacity of the scalable target to be a certain percent of current consumption. For this example we are going to ask for a new capacity to grow to a value so that current usage DynamoDBWriteCapacityUtilization accounts for 70%:

TableWriteScalingPolicy:
    Type: AWS::ApplicationAutoScaling::ScalingPolicy
    Properties:
        PolicyName: TableWriteScalingPolicy
        PolicyType: TargetTrackingScaling
        ScalingTargetId: !Ref TableWriteCapacityScalableTarget
        TargetTrackingScalingPolicyConfiguration: 
            TargetValue: 70
            ScaleInCooldown: 60
            ScaleOutCooldown: 60
            PredefinedMetricSpecification: 
                PredefinedMetricType: DynamoDBWriteCapacityUtilization

Setup of index is identical with scalingTargetId pointed at the IndexWriteCapacityScalableTarget.

CLOUDFORMATION SUPPORT LIMITATIONS

This post would not be complete without a warning of certain implementation-specific limitations of DynamoDB support in CloudFormation.

First of all DynamoDB supports a fixed number of tables in the CREATING state with global secondary indexes. This means you need to serialise creation of a large number of tables with DependsOn attribute. This can be a very annoying issue since rollback of a failed creation will take a significant amount of time to clean up.

Second, the most important catch is the fact that AWS::DynamoDB::Table with associated AWS::ApplicationAutoScaling::ScalableTarget will always fail to update, and then fail to rollback. This may make your stack completely unusable for a long period of time. The best approach would be to implement a Lambda-based custom resource to deregister scalable targets BEFORE table updates.

Additional information on these subjects can be found in AWS documentation for table and target resources.