Performing Rolling Updates in Terraform with CloudFormation

BLOG ARTICLE

Background

Terraform is great at managing Infrastructure. Things like Subnets, Ec2 Instances and IAM policies are easy to set up with the declarative nature of the platform.

Unfortunately, AWS does not offer all its functionality via API calls – with some more complex actions being exclusive to AWS Console and / or CloudFormation.

The Problem

One of the most common problems Terraform encounters is managing rolling updates to  AWS AutoScaling Groups.

Did you know that you can use CloudFormation templates within Terraform?

Terraform has an “aws_cloudformation_stack” resource that is actually recommended by Terraform to take advantage of CloudFormation native functionality [1]. 

This works [2] however, how do the ECS Instances stay up to date with the latest AMIs without constantly updating the parameters / resource? Once the resource is created Terraform only triggers an “Update CloudFormation Stack” if there is change to the stack within Terraform.

The Answer

AWS AMI IDs in SSM Parameter Store, Cloudwatch Event Rules and Lambda.

Specifically, in this post we will be going through:

  • Using AWS CloudFormation within Terraform
  • Using AWS AMI Aliases via Systems Manager Parameter Store
  • Triggering Lambda Functions based on a CloudWatch Event Rule schedule
AWS AMI ID Aliases via SSM Parameter Store and CloudFormation in Terraform

AWS now populates SSM Parameters Store values with their latest optimized images for EC2 [3], which can be referenced as a parameter in CloudFormation Templates.

Eg.

resource “aws_cloudformation_stack” “autoscaling_group_cloudformation” {

  name = “My CloudFormation Stack”

  template_body = <<EOF

Description: “CloudFormation stack for AutoScalingGroup as deployed by Terraform”

Parameters:

  LatestAmiId:

Type: ‘AWS::SSM::Parameter::Value’

Default: “/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2”

 

Resources:

  LaunchTemplate:

Type: AWS::EC2::LaunchTemplate

Properties:

LaunchTemplateName: “My LaunchTemplate”

LaunchTemplateData:

     BlockDeviceMappings:

    

     DeviceName: “/dev/sda1”

     Ebs:

     DeleteOnTermination: True

     Encrypted: True

     VolumeSize: “${var.instance_storage}”

     VolumeType: “gp2”

     IamInstanceProfile:

     Name: “${var.instance_profile_name}”

     InstanceType: “${var.instance_type}”

     ImageId: !Ref LatestAmiId

     SecurityGroupIds: [“${var.security_group_ids}”]

     TagSpecifications:

     – ResourceType: “volume”

     Tags:

     – Key: “MyTags”

     Value: ‘${var.tags}’

 

  AutoScalingGroup:

Type: AWS::AutoScaling::AutoScalingGroup

Properties:

VPCZoneIdentifier: [${var.vpc_zone_identifier}]

LaunchTemplate:

LaunchTemplateId: !Ref LaunchTemplate

     Version: !GetAtt LaunchTemplate.LatestVersionNumber

The “LatestAmiId” parameter is referencing an SSM Parameter Ec2 Image ID Value [4,5] and populated in the LaunchTemplate block.

As mentioned above though, this configuration will populate the launch template with the latest AMI at the time of the initial ‘terraform apply’ only. After that point the code never changes so ‘terraform plan’ will never trigger an ‘Update Stack’ to get the latest SSM Parameter value.

Creating the CloudWatch Event Rules / EventBridge and Lambda

AWS updates their AMIs on a regular basis and in turn, will update the SSM Parameter Store value with the latest AMI ID as well. 

This update may occur during business hours when changes to production infrastructure are not acceptable, so in this scenario a controlled mechanism to check for updates is required.

A CloudWatch Event Rule (now known as EventBridge) can be used to set a regular schedule during a known maintenance window to update the CloudFormation Stack. 

An example of a CloudWatch Event Rule running every Sunday at 12:00 with Event Target and Permissions –

resource “aws_cloudwatch_event_rule” “trigger_cloudformation_update” {

  name    = “Update Cloudformation Stack”

  description = “Trigger the Update Cloudformation Stack Lambda”

  schedule_expression = “cron(12 0  ? * 1 *)”

}

The target for the CloudWatch Event Rule is a simple Lambda Function that attempts to update the CloudFormation Stack – if the SSM Parameter value has changed, then the update is applied and rolling policy enacted.

If there are no changes to the SSM Parameter, then the Lambda Function returns no changes required.

resource “aws_cloudwatch_event_target” “trigger_cloudformation_update_event_target” {

  rule  = aws_cloudwatch_event_rule.trigger_cloudformation_update.name

  target_id = “update_cloudformation_stack_lambda”

  arn   = aws_lambda_function.update_cloudformation_stack_lambda.arn

}

resource “aws_lambda_permission” “allow_cloudwatch_to_call_update_stack_lambda” {

  statement_id  = “AllowExecutionFromCloudWatch”

  action    = “lambda:InvokeFunction”

  function_name = aws_lambda_function.update_cloudformation_stack_lambda.function_name

  principal = “events.amazonaws.com”

  source_arn = aws_cloudwatch_event_rule.trigger_cloudformation_update.arn

}

And the Python Lambda Function called “update_cloudformation_stack_lambda” that will trigger the CloudFormation Stack

import boto3
Import os

stack_name = os.environ[‘MY_STACK_NAME’]

cfn_resource = boto3.resource(‘cloudformation’)

def lambda_handler(event, context):

print(f”Stack name to update is: {stack_name}”)

response = update_cfn(stack_name)

def update_cfn(stack_name):

stack = cfn_resource.Stack(stack_name)

response = stack.update(UsePreviousTemplate=True)

return response

This is a simple example and can be extended out to meet further business requirements

Summary