Deploying resources to a multi-account AWS environment can be a complex and challenging task. At times, you may find yourself attempting to manage tens, if not hundreds, of AWS accounts. Managing the resources within these accounts in a centralised and standardised manner can be complicated and time-consuming. Likewise, deploying resources in a multi-account environment requires a thorough understanding of AWS services and their interdependencies.
Given these issues, we at Cevo took the time to trial the AWS Customisations for Control Tower (CfCT) solution, which seeks to alleviate some of these difficulties, so you don’t have to. This blog outlines how to deploy and use the solution, as well as our experience trialling the solution.
Note: This blog assumes basic familiarity with AWS Organizations and AWS Control Tower.
What is CfCT?
CfCT is an AWS-provided solution that allows you to deploy Service Control Policies (SCPs) and CloudFormation StackSets into an AWS Organisation via Infrastructure as Code (IaC), while maintaining Control Tower best practices. It’s an out-of-the-box GitOps solution for managing a multi-account environment, without having to get hands-on with individual accounts.
CfCT provides two standard ways of getting your IaC code into the pipeline, both via S3 and CodeCommit. The first allows you to upload a zipped folder containing the manifest and associated templates/policies to an S3 bucket, which is connected to a CodePipeline and Step Functions that deploy those templates/policies across accounts as specified via the manifest file.
The alternative uses CodeCommit to store a repo which is automatically connected to CodePipeline and Step Functions to help with developing a GitOps pipeline from scratch.
Solution Architecture
Deploying the Solution
Following the steps provided by AWS, it is relatively trivial to deploy the solution. But for completeness these are the steps we followed:
- You must already have a Control Tower environment set up, and you will need to deploy the AWS provided CloudFormation stack to your Control Tower’s Management Account.
- You’ll be prompted with a number of configuration options, including the option to choose whether to use the CodeCommit or S3 configurations. You’ll also be able to add an approval stage to the pipeline if you wish to have an additional manual review step in the process.
- We chose to use the CodeCommit option to take full advantage of GitOps out-of-the-box in our testing. If you already have an established GitOps method, it may be best to use the S3 option – you’ll just need to zip and upload the manifest, template, and policy files as part of your existing procedure.
- The solution will run the CodePipeline whenever new code is pushed to the CodeCommit repo or when the manifest file is updated in the S3 bucket, depending on which configuration you chose to go with.
- The solution will also monitor for Lifecycle Events in Control Tower and deploy appropriate changes. For example, if an account is added to an Organisational Unit, which is the deployment target for a StackSet in the manifest file, CfCT will deploy that StackSet to the account automatically.
Using the Solution
Once deployed, CfCT provides you the below folder structure in the provisioned S3 bucket or CodeCommit repo (depending on your selection at deployment time). This includes an empty manifest.yaml file, ready for you to work with, as well as an example structure for how you might make use of the solution.
The example configuration folder provides a simple structure for how you may want to go about organising your CfCT code. It also provides demonstrations of the major functionality of the solution, providing examples of both StackSet and SCP deployments, as well as how to target individual accounts for deployment or Organisational Units. The manifest.yaml in the root folder is where you’ll do the actual work to configure your CfCT setup.
As is implied from the example structure, we make use of the manifest file to call templates or policies from the relative folders. This is how CfCT allows us to deploy policies/templates to multiple accounts.
The manifest itself acts as a pseudo CloudFormation template.
Here’s an example of a simple manifest file:
This manifest deploys the template from templates/custom-resource.template to all accounts in the Test Organisational Unit via StackSets. Alternatively, you can target individual accounts by either entering an Account ID or the Account Name. Demonstrations for this configuration are provided in the example manifest.
While the out-of-the-box deployment of the solution does not enable the deletion of StackSets via CfCT, as of version 2.5 we can now enable this functionality by setting the enable_stack_set_deletion parameter to true in the manifest.yaml file. Once set, when resources are removed from the manifest.yaml CfCT will remove them from any accounts and/or Organisational Units that it deployed them to. For more on enabling StackSet deletion, see the AWS Documentation.
The Verdict
As the earlier deployment guide exemplifies, the CfCT solution is quite simple to deploy and it gets up and running very quickly. This makes it a good option where there’s a lack of an existing alternative procedure to manage cloud resources/policies within an AWS Organisation or where there exists a lack of resources to implement a more bespoke solution. However, there are a number of caveats that it would be best to be aware of when considering CfCT.
One of the major issues we identified in testing CfCT was that when a CloudFormation Stack fails to deploy, it’s often incredibly difficult to find useful error logging about the cause of the failure. This is because the error occurs at the account level, and does not always report the failure reason back to the Step Function at the management account level before removing the failed stack. In one of the test examples, we identified that the stack was failing due to a malformed template. However, it was difficult to identify this until we tried deploying it as a straight StackSet. In this regard, CfCT works best with tried and tested StackSets that you’re confident are already well-formed and you are confident will deploy successfully in the accounts being targeted.
Furthermore, the CfCT solution does not currently support the deployment of CloudFormation Stacks. Given that StackSets are unable to deploy CloudFormation Stacks to an Organisation’s Management Account, this means that CfCT does not allow the management of the Organisation itself, nor the Control Tower. While there may be reasons for this, we would have liked the ability to maintain our Organisation (and in particular be able to control Organisational Units) in one place. Hopefully a feature is added that allows this, or something similar, in the future.
With those caveats out of the way, having an out-of-the-box GitOps solution for Control Tower is promising and something we’re excited to see expanded upon. CfCT is quick to set up and relatively easy to integrate with your current CI/CD pipelines whether using it via CodeCommit or uploading to S3. This makes it easy to gain the benefits that a proper review process and version control system provides to managing a multi-account AWS Organisation.
It is also a benefit that AWS provides the source code for CfCT. If you really want to see what’s happening under the hood, or even further customise your own deployment, this makes those options available.
Conclusion
Given the current limitations of CfCT, it’s difficult to see how it outperforms alternative solutions, such as Terraform or even straight StackSet deployments. However, from a conceptual stand-point, the idea is promising. It will be interesting to see where AWS takes this in the future and whether we will see CfCT becoming the standard for managing resources across an AWS Control Tower managed Organisation.
For some information about what might be to come in the future, it’s worth taking a look at the CfCT GitHub Issues Page.
