I recently had some deep experience in deploying AWS Control Tower into an existing AWS Organization structure. The experience went well – until I broke it. This blog is all about the ways to break (and fix) Control Tower. It would not be a great technical blog without covering the basics of what Control Tower is, does, can do and finally, how I broke it, fixed it, and learned a lot more about it than what is widely documented.
In Part 1 of this two-part series, we explored Control Tower components and structures. In Part 2 of this series, we explore how to break and fix Control Tower and some guidance when integrating existing accounts in an Organization into Control Tower.
Deploying Control Tower Into An Existing Organization
While most Control Tower deployments occur as greenfield landing zones, there may be times when a client requires us to deploy a new or upgrade an existing Control Tower in a pre-established AWS Organization. In my most recent experience, the client had deployed Control Tower and managed a few accounts with it. The remaining accounts and Organizational Units (OUs) were all manually managed. The Control Tower had not been upgraded in a substantial amount of time and the accounts managed under it had drifted from the original deployment.
As is usually a best practice recommendation, the client had a development Control Tower Landing Zone built in a development Organization. Naturally, we would assume most things would align to the production Control Tower, however this too had been abandoned over time. As a development environment built some time ago, there was a lot of experimentation that had taken place and drifted away from the old Control Tower. The first task we had was to unpack the environment with a goal to align it as closely to the production Control Tower as possible so we could mimic the environment characteristics and plan detailed changes for Change Managment (implementation, verification and backout plans).
Existing AWS Config Setup
Where there is an existing AWS Config setup, there is a good chance that accounts will fail to enroll. This is due to Control Tower expecting to control AWS Config recorder, delivery channels and the central aggregator. Before we delve into what we did to overcome challenges, it is important to understand the context of these AWS Config configuration items and their assorted options for configuration.
AWS Config Options
AWS Config comes with its very own configuration. There are several configuration items that modify the behaviour of AWS Config and how it tracks and reports on resources against Config Rules (that determine what the resource is measured against its configuration).
Config Rules
The AWS Config Rules are derived from security standards via Conformance Packs for industry standards and AWS Well-Architected best practices (enabled by default and closely aligned to CIS benchmarks). Conformance Packs (such as APRA CPG-234 for Information Security in the Financial Services Industry) are not managed by Control Tower.
Delivery Channels
The role of Delivery Channels in AWS Config is to collate and inform about changes to resources against the Config Rules conditions. While these non-compliant resources can be seen in the console, it is often helpful to have proactive notification such as an SNS topic subscribed to a monitored email mailbox and/or an S3 bucket where findings can be ingested into an event reporting tool. We can only have a single SNS Topic, and a single S3 bucket destination configured for Delivery Channels at any given time.
Control Tower manages the Delivery Channels and configures these to a central S3 bucket in the Audit account with a managed bucket policy permitting the Root Organization ID to allow writes to it. It sets an SNS topic as a second Delivery Channel to the managed Control Tower SNS topic. This is done through a CloudFormation StackSet created and deployed by Control Tower from the Management Account to Member Accounts during enrollment.
To help protect the Config Delivery Channels, Control Tower implements a Control Tower managed guardrail SCP applied to the OU for a Member Account that prevents turning off the delivery channels unless the assumed role is the ControlTowerExecutionRole. This maintains the Config Delivery Channels state for compliance.
Config Authorizations
To allow accounts permission to send Config recordings to an Aggregator, they first need to be authorised to send them. To assist with this, the Aggregator creates Authorizations for a given account and region, and these are sent as in invitation to Member Accounts to accept. Once the Authorization is accepted by the Member Account, the Aggregator can then see the Config recorder state from the Member Account.
Control Tower manages and maintains both the creation of the Authorizations in the Audit Account as well as the acceptance of the invitation in the Member Accounts. This is done through a CloudFormation StackSet created and deployed by Control Tower from the Management Account to Member Accounts during enrollment.
To help protect the Config Delivery Channels, Control Tower implements a Control Tower managed guardrail SCP applied to the OU for a Member Account that prevents turning off the delivery channels unless the assumed role is the ControlTowerExecutionRole. This maintains the Config Delivery Channels state for compliance.
Breaking and Remediating Control Tower
Here we dive into what you need to do to remediate failed enrollments for Member Accounts with existing configurations and what not to do with Control Tower managed SCP’s. Strap in as it is a wild ride.
Managed SCPs
The managed guardrails SCPs deployed by Control Tower are super important to understand. They are designed to prevent messing with resources deployed by Control Tower. More importantly though – they are not equal in terms of impacts to Control Tower. If the managed guardrails SCPs are removed from the Security OU (that contains the Audit and Log Archive accounts), Control Tower loses access to the CloudTrail. This then causes Control Tower to fail and the only way to recover is to reset the Landing Zone.
Config Authorizations
Any existing Config Authorizations breaks Member Account Enrollment. It is important to note the Authorizations are crucial for the Audit Account to manage. A Member Account containing existing Authorization invitations cannot be enrolled. You can delete invitations using the console in a Member Account before enrollment. However, once the OU has been successfully registered, a Control Tower managed guardrails SCP applied to any parent OU will prevent you from doing so. You first must detach the guardrails SCP applied to the Member Account OU (or the parent OU when using child OU’s as it is inherited by the parent).
Removing Config Authorizations
Any existing Config Authorizations in the Member Account will fail the pre-check for account enrollment. Before enrollment, check and delete any existing Authorizations which can be done via console.
- Check any existing Authorizations
Go to Aggregators > Authorizations - Delete existing Authorizations
Removing Authorizations in Audit Account
In my case, there was a second deployed CloudFormation Stack that had been deployed to configure the Authorizations in a specific Member Account. The account was failing to enroll due to existing Config Authorizations. Confused about what was causing this, I removed the existing Authorizations in the member account however it persistently complained it could not enroll due to existing Authorizations. Realising this was coming from the Audit account, I manually removed the Authorizations from the Audit account. Huge mistake.
Removing the Config Authorizations from the Audit Account breaks Control Tower. I was forced to reset the Control Tower and redeploy the Landing Zone. This did not impact existing resources, but it also failed to re-register existing OU’s where accounts resided due to missing Authorizations. To resolve this, I deleted the stack that was trying to create the Authorizations. I was then able to delete the StackSet-AWSControlTowerBP-BASELINE-CONFIG instance against the Audit Account in each region where I had deleted the Authorizations manually. This allowed Control Tower to continue the reset, and it recreated the stack instances as part of the onboarding.
AWS Config Recorder and Delivery Channel
The limitations of AWS Config in only permitting a single Recorder per region, per account in addition to single S3 bucket and SNS Topic for Delivery Channels conflicts with Control Tower wanting to own and configure them. This causes Control Tower to fail to enroll the member accounts with an existing AWS Config configuration as it does not have the authority to modify existing configuration.
When AWS Config has been previously deployed with any configuration – it breaks Member Account enrollment. This requires removing the existing Config Recorder and Delivery Channels before enrolling accounts into Control Tower. It is important to note this does not break Control Tower – only the ability to enroll Member Accounts.
While these setting can be disabled in the console – it does not remove the Recorder and Delivery Channels. You cannot remove the Recorder and Delivery Channels against AWS Config through any console action!
Removing Config Recorder and Delivery Channel
To remove the Config Recorder and Delivery Channel you must use the AWS CLI. There is no other way (unless the previous Config deployment was deployed by a CloudFormation StackSet or other IaC solution). If you fall into the unfortunate category of manually deployed AWS Config configuration, these are the AWS CLI commands you need (assuming defaults and the ap-southeast-2 – Sydney region below – yours may differ):
- Verify the existing Config Recorder (the default name is called “default”)
“aws configservice describe-configuration-recorders” - Delete the Config Recorder
“aws configservice delete-configuration-recorder –configuration-recorder-name default –region ap-southeast-2” - Verify the existing Config Delivery Channels (the default name is called “default”)
“aws configservice describe-delivery-channels” - Delete the Delivery Channels
“aws configservice delete-delivery-channel –delivery-channel-name default –region ap-southeast-2″
Adding Back Customer Managed SCPs
During the reset, Control Tower removed customer managed SCPs from all OU’s. This is expected as customer managed SCPs are not owned and managed by Control Tower itself. It simply added the guardrails SCPs onto the root and individual OU’s. The custom SCPs had to be manually added back to the respective OUs after the Control Tower reset.
Enrolling Accounts in Root Organization
To enroll an existing account in an Organization, it must be a member of an Organization Unit (OU). When an existing account is already a member of an OU, the account is enrolled when that OU is registered into Control Tower. In Control Tower > Organization:
- Select any parent OU (if applicable) and register the parent first
- Continue with any child OU (if applicable) registration
- Select the OU the Member Account is a member of
- Go to Actions > Register
Register Member Account in Root Organization
If your chosen Member Account is not already a member of an OU and resides in the Root of the Organization, the Member Account does not contain the “ControlTowerExecutionRole” and you are asked to first provision these manually in the account prior to enrollment in the Control Tower console. The correct way to enroll a Member Account into an OU is as follows:
- Go to AWS Organizations service (not in Control Tower)
- Select the Member Account in the Root Organization
- Move the Member Account to the desired OU
- Go back to Control Tower > Organizations
- Select the OU
- Go to Actions > Re-Register
This sequence will create the necessary roles for you by deploying a CloudFormation Stack using the pre-existing AWSOrganizationsFullAccess role.