Autoscaling DynamoDB capacity for fun and profit.
DynamoDB autoscaling is a feedback-loop based monitoring setup which can dynamically change provisioned capacity for the table or global secondary index. There are two ways one can define autoscaling policy for this resource:
- Step scaling policy
- Target tracking policy
STEP SCALING POLICY
The first kind of policy – step scaling – is based on CloudWatch alarms. This approach is the original autoscaling policy, the kind of which has been available to us from the beginning of days. EC2 autoscaling is based on this, and in general, CloudWatch alarms were the main way to trigger scaling activities.
This approach works well enough, but does not allow the user to allocate capacity larger or smaller than the configured step (scale in or out); in addition it doesn’t allow us to request certain capacity above the level needed currently in this instance of time.
TARGET TRACKING POLICY
The target tracking policy is here to help with the issues highlighted above. The new policy kind uses generic scalable target API
application-autoscaling:RegisterScalableTarget and allows dynamic changes to the provisioned capacity. But the most important aspect is that you can ask for capacity as a percentage of current or projected use. This way, each scale out event can take the capacity to a level with a some room to grow, while the next tick of feedback is analysed.
CLOUDFORMATION SUPPORT FOR DYNAMODB
Naturally everything around scalability is covered by the
AWS::ApplicationAutoScaling::ScalableTarget resources. Let’s examine how one can define a policy for a
We start with defining the service role to perform scaling actions on our behalf.
ScalingRole: Type: AWS::IAM::Role Properties: RoleName: ScalingRole AssumeRolePolicyDocument: Version: "2012-10-17" Statement: - Effect: Allow Principal: Service: - application-autoscaling.amazonaws.com Action: - sts:AssumeRole ScalingRolePolicy: Type: AWS::IAM::Policy Properties: Roles: - !Ref ScalingRole PolicyName: ScalingRolePolicyPolicy PolicyDocument: Version: '2012-10-17' Statement: - Effect: Allow Action: - application-autoscaling:* - dynamodb:DescribeTable - dynamodb:UpdateTable - cloudwatch:PutMetricAlarm - cloudwatch:DescribeAlarms - cloudwatch:GetMetricStatistics - cloudwatch:SetAlarmState - cloudwatch:DeleteAlarms Resource: '*'
Next is the definition of our test table. To define it, we need minimum and maximum capacity values. Here we define a scalable target, the object used to hold a reference to scalable dimension, and range of possible values. In this example we will look into
dynamodb:table:WriteCapacityUnits scaling. The read capacity setup is identical with a different scalable dimension.
TestTable: Type: AWS::DynamoDB::Table Properties: TableName: TestTable AttributeDefinitions: - AttributeName: id AttributeType: S - AttributeName: external AttributeType: S KeySchema: - AttributeName: id KeyType: HASH ProvisionedThroughput: ReadCapacityUnits: !Ref MinReadCapacityUnits WriteCapacityUnits: !Ref MinWriteCapacityUnits GlobalSecondaryIndexes: - IndexName: TestIndex KeySchema: - AttributeName: external KeyType: HASH Projection: ProjectionType: ALL ProvisionedThroughput: ReadCapacityUnits: !Ref MinReadCapacityUnits WriteCapacityUnits: !Ref MinWriteCapacityUnits TableWriteCapacityScalableTarget: Type: AWS::ApplicationAutoScaling::ScalableTarget Properties: MaxCapacity: !Ref MaxWriteCapacityUnits MinCapacity: !Ref MinWriteCapacityUnits ResourceId: table/TestTable RoleARN: !GetAtt ScalingRole.Arn ScalableDimension: dynamodb:table:WriteCapacityUnits ServiceNamespace: dynamodb IndexWriteCapacityScalableTarget: Type: AWS::ApplicationAutoScaling::ScalableTarget Properties: MaxCapacity: !Ref MaxWriteCapacityUnits MinCapacity: !Ref MinWriteCapacityUnits ResourceId: table/TestTable/index/TestIndex RoleARN: !GetAtt ScalingRole.Arn ScalableDimension: dynamodb:index:WriteCapacityUnits ServiceNamespace: dynamodb
Now we have two scalable targets; one for primary index and one for global secondary index. Each has a target resource ID, scalable dimension and namespace, as well as a range of possible values.
Now it’s time to define a policy to move these values. Here we’re going to specify the required capacity of the scalable target to be a certain percent of current consumption. For this example we are going to ask for a new capacity to grow to a value so that current usage
DynamoDBWriteCapacityUtilization accounts for 70%:
TableWriteScalingPolicy: Type: AWS::ApplicationAutoScaling::ScalingPolicy Properties: PolicyName: TableWriteScalingPolicy PolicyType: TargetTrackingScaling ScalingTargetId: !Ref TableWriteCapacityScalableTarget TargetTrackingScalingPolicyConfiguration: TargetValue: 70 ScaleInCooldown: 60 ScaleOutCooldown: 60 PredefinedMetricSpecification: PredefinedMetricType: DynamoDBWriteCapacityUtilization
Setup of index is identical with
scalingTargetId pointed at the
CLOUDFORMATION SUPPORT LIMITATIONS
This post would not be complete without a warning of certain implementation-specific limitations of DynamoDB support in CloudFormation.
First of all DynamoDB supports a fixed number of tables in the CREATING state with global secondary indexes. This means you need to serialise creation of a large number of tables with
DependsOn attribute. This can be a very annoying issue since rollback of a failed creation will take a significant amount of time to clean up.
Second, the most important catch is the fact that
AWS::DynamoDB::Table with associated
AWS::ApplicationAutoScaling::ScalableTarget will always fail to update, and then fail to rollback. This may make your stack completely unusable for a long period of time. The best approach would be to implement a Lambda-based custom resource to deregister scalable targets BEFORE table updates.