Introduction
My teammate, Matt Merriel, has recently written a series of blogs around creating CloudFormation templates and deploying Infrastructure-as-Code (IaC). While deploying your resources as code is excellent, it’s also easy to tear down those resources. Quickly replacing your resources is ideal in a development environment or sandbox, but you might not want to delete your resources in production. So, how do you combat that?
Resource Termination Protection
The first thing to look at is protecting the resources. If the resource has termination protection in the console, there is generally an option to enable that within CloudFormation. This type of protection means that your infrastructure won’t be deleted, even if the CloudFormation stack is terminated. Similarly, CloudFormation will create a new resource if an update replaces the resource, but the old one will remain.
For EC2, you can see that in the console:
In CloudFormation, this is configured via:
The curious thing here is the name, DisableApiTermination. The “Api” part could be a little confusing, but this is what you need to protect your EC2 instances.
Looking at the documentation confirms this is the property you want.
Some other resources with termination protection include:
AWS::RDS::DBCluster – DeletionProtection
AWS::ElasticLoadBalancingV2::LoadBalancer – LoadBalancerAttributes – deletion_protection.enabled
CloudFormation Termination Protection
One thing that many people aren’t aware of is that termination protection can be enabled on stacks. Given that stacks contain multiple resources, production environments’ security is essential.
Viewing this from the console, the configuration is as follows:
You can’t use a CloudFormation template to protect a CloudFormation stack, so how is that done? The AWS CLI is the answer. At Cevo, we often use a Makefile when CLI commands are used, for example, deploying a stack to build a CodePipeline or connecting using SSM. In this case, I use a Makefile target to either set or clear the protection.
# Protect stacks from being deleted
protect-stack:
aws cloudformation update-termination-protection \
--stack-name ${STACK_NAME} \
--enable-termination-protection
unprotect-stack:
aws cloudformation update-termination-protection \
--stack-name ${STACK_NAME} \
--no-enable-termination-protectionHaving an easy way to back out the protection is also necessary. You want to make sure that you don’t accidentally delete stacks you don’t intend to, but you also want a way to allow you to delete that doesn’t include Click-Ops.
Protecting Resources In The Stack
While you might have termination protection on your EC2 resource, if you make an update that replaces your resource, there are potentially a lot of other headaches. Maybe you are trying to increase the size of a volume configured within the EC instance. Typically, increasing an EBS volume is non-destructive. Applying a change to a BlockDevice within an AWS::EC2::Instance will cause a replacement. So, instead of having a bigger drive, you now have a new instance with potentially no data or configuration. Not something you want in production. The AWS::EC2::Instance documentation has a warning about this.
You can apply a Stack policy to protect against this sort of thing. A Stack policy can apply an Update:Modify, Update:Delete, Update:Replace protections to all or specific resources.
I deploy this via a Makefile with separate policies to protect the appropriate resources and remove stack termination protection. Again, there needs to be an easy way to remove the protection if you want to change a protected resource.
In the makefile:
# Protect specified stack resources from destructive updates
lock-stack:
aws cloudformation set-stack-policy \
--region ap-southeast-2 \
--stack-name ${STACK_NAME} \
--stack-policy-body file://update-policies/lock-policy-${POLICY}.json
unlock-stack:
aws cloudformation set-stack-policy \
--region ap-southeast-2 \
--stack-name ${STACK_NAME}
--stack-policy-body file://update-policies/full-access-policy.json
full-access-policy.json:
{
"Statement" : [
{
"Effect" : "Allow",
"Action" : "Update:*",
"Principal": "*",
"Resource" : "*"
}
]
}
lock-policy-database.json:
{
"Statement" : [
{
"Effect" : "Allow",
"Action" : "Update:*",
"Principal": "*",
"Resource" : "*"
},
{
"Effect" : "Deny",
"Action" : "Update:*",
"Principal": "*",
"Resource" : "LogicalResourceId/ProductionDatabase"
}
]
}You can view the Stack Policy in the CloudFormation console.
You can find further information on Stack Policies in the AWS documentation: Prevent updates to stack resources
Conclusion
CloudFormation is a powerful tool to create resources, but it’s also easy to delete or modify resources you don’t intend to. AWS gives you the above three options to protect your infrastructure. I would recommend the Defence-in-depth principle for production workloads and apply as many of these protections as possible.
