Background
Terraform is great at managing Infrastructure. Things like Subnets, Ec2 Instances and IAM policies are easy to set up with the declarative nature of the platform.
Unfortunately, AWS does not offer all its functionality via API calls – with some more complex actions being exclusive to AWS Console and / or CloudFormation.
The Problem
One of the most common problems Terraform encounters is managing rolling updates to AWS AutoScaling Groups.
Did you know that you can use CloudFormation templates within Terraform?
Terraform has an “aws_cloudformation_stack” resource that is actually recommended by Terraform to take advantage of CloudFormation native functionality [1].
This works [2] however, how do the ECS Instances stay up to date with the latest AMIs without constantly updating the parameters / resource? Once the resource is created Terraform only triggers an “Update CloudFormation Stack” if there is change to the stack within Terraform.
The Answer
AWS AMI IDs in SSM Parameter Store, Cloudwatch Event Rules and Lambda.
Specifically, in this post we will be going through:
- Using AWS CloudFormation within Terraform
- Using AWS AMI Aliases via Systems Manager Parameter Store
- Triggering Lambda Functions based on a CloudWatch Event Rule schedule
AWS AMI ID Aliases via SSM Parameter Store and CloudFormation in Terraform
AWS now populates SSM Parameters Store values with their latest optimized images for EC2 [3], which can be referenced as a parameter in CloudFormation Templates.
Eg.
resource “aws_cloudformation_stack” “autoscaling_group_cloudformation” {
name = “My CloudFormation Stack”
template_body = <<EOF
Description: “CloudFormation stack for AutoScalingGroup as deployed by Terraform”
Parameters:
LatestAmiId:
Type: ‘AWS::SSM::Parameter::Value’
Default: “/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2”
Resources:
LaunchTemplate:
Type: AWS::EC2::LaunchTemplate
Properties:
LaunchTemplateName: “My LaunchTemplate”
LaunchTemplateData:
BlockDeviceMappings:
–
DeviceName: “/dev/sda1”
Ebs:
DeleteOnTermination: True
Encrypted: True
VolumeSize: “${var.instance_storage}”
VolumeType: “gp2”
IamInstanceProfile:
Name: “${var.instance_profile_name}”
InstanceType: “${var.instance_type}”
ImageId: !Ref LatestAmiId
SecurityGroupIds: [“${var.security_group_ids}”]
TagSpecifications:
– ResourceType: “volume”
Tags:
– Key: “MyTags”
Value: ‘${var.tags}’
AutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
VPCZoneIdentifier: [${var.vpc_zone_identifier}]
LaunchTemplate:
LaunchTemplateId: !Ref LaunchTemplate
Version: !GetAtt LaunchTemplate.LatestVersionNumber
…
The “LatestAmiId” parameter is referencing an SSM Parameter Ec2 Image ID Value [4,5] and populated in the LaunchTemplate block.
As mentioned above though, this configuration will populate the launch template with the latest AMI at the time of the initial ‘terraform apply’ only. After that point the code never changes so ‘terraform plan’ will never trigger an ‘Update Stack’ to get the latest SSM Parameter value.
Creating the CloudWatch Event Rules / EventBridge and Lambda
AWS updates their AMIs on a regular basis and in turn, will update the SSM Parameter Store value with the latest AMI ID as well.
This update may occur during business hours when changes to production infrastructure are not acceptable, so in this scenario a controlled mechanism to check for updates is required.
A CloudWatch Event Rule (now known as EventBridge) can be used to set a regular schedule during a known maintenance window to update the CloudFormation Stack.
An example of a CloudWatch Event Rule running every Sunday at 12:00 with Event Target and Permissions –
resource “aws_cloudwatch_event_rule” “trigger_cloudformation_update” {
name = “Update Cloudformation Stack”
description = “Trigger the Update Cloudformation Stack Lambda”
schedule_expression = “cron(12 0 ? * 1 *)”
}
The target for the CloudWatch Event Rule is a simple Lambda Function that attempts to update the CloudFormation Stack – if the SSM Parameter value has changed, then the update is applied and rolling policy enacted.
If there are no changes to the SSM Parameter, then the Lambda Function returns no changes required.
resource “aws_cloudwatch_event_target” “trigger_cloudformation_update_event_target” {
rule = aws_cloudwatch_event_rule.trigger_cloudformation_update.name
target_id = “update_cloudformation_stack_lambda”
arn = aws_lambda_function.update_cloudformation_stack_lambda.arn
}
resource “aws_lambda_permission” “allow_cloudwatch_to_call_update_stack_lambda” {
statement_id = “AllowExecutionFromCloudWatch”
action = “lambda:InvokeFunction”
function_name = aws_lambda_function.update_cloudformation_stack_lambda.function_name
principal = “events.amazonaws.com”
source_arn = aws_cloudwatch_event_rule.trigger_cloudformation_update.arn
}
And the Python Lambda Function called “update_cloudformation_stack_lambda” that will trigger the CloudFormation Stack
import boto3
Import os
stack_name = os.environ[‘MY_STACK_NAME’]
cfn_resource = boto3.resource(‘cloudformation’)
def lambda_handler(event, context):
print(f”Stack name to update is: {stack_name}”)
response = update_cfn(stack_name)
def update_cfn(stack_name):
stack = cfn_resource.Stack(stack_name)
response = stack.update(UsePreviousTemplate=True)
return response
This is a simple example and can be extended out to meet further business requirements
Summary
In summary – we tackled the following problems:
- How to apply an AWS Rolling Update Policy to your Terraform AutoScaling Group
- How to automate your Terraform code to use the latest AWS AMIs
- Updating AutoScaling Groups during a known maintenance window with CloudWatch Event Rules and Lambda
Sources –
- https://github.com/hashicorp/terraform/issues/1552
- https://github.com/travis-ci/terraform-config/blob/b7584146cfd2b4978def7a87c5f034994cc94766/modules/aws_asg/main.tf#L134-L221
- https://aws.amazon.com/about-aws/whats-new/2020/05/amazon-ec2-now-supports-aliases-for-amis/
- https://aws.amazon.com/blogs/compute/query-for-the-latest-amazon-linux-ami-ids-using-aws-systems-manager-parameter-store/
- https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-optimized_AMI.html
