Introduction
As a Principal Consultant and AWS Ambassador with Cevo, I get the awesome privilege of working with many customers. I carry over 20 years of IT experience across networking, security, infrastructure, governance and compliance, and 7 years of AWS experience. Even so, there is always so much to learn and sometimes things do not go as planned. In those times, leaning into a trusted AWS Partner and people such as myself, we can help where a customer may struggle to help themselves.
In this blog post, we will dive deep into AWS Control Tower Errors and AWS Organizations Account Migrations with lessons learned from a Premier Tier AWS Consulting Partner – so you do not have to!
I would also like to take the opportunity to recognise that AWS Support people are incredible at what they do. They are the foundation of what made this possible.
Overview
When deploying Control Tower into a new AWS account, setting up the Landing Zone is usually a straightforward process. There are times when errors can and do occur, however these challenges are usually well-documented in the official Troubleshooting guide found here: https://docs.aws.amazon.com/controltower/latest/userguide/troubleshooting.html
In a recent customer deployment, however, things did not go quite to plan.
Situation
My customer was in a difficult situation with a previous set of accounts and an AWS Organizations structure which placed the Management Account under an Organizational Unit (OU). This was not specifically an issue in this customer as there were no restrictive Service Control Policies (SCP’s) being applied to any of the accounts or OU’s.
As a best practice, it is always recommended to place the Management Account for an AWS Organization into the root of the AWS Organizations. Accounts in the root of the Organization are exempt from the effects of SCP’s. This is super important when applying restrictive policies as you may lose the ability to manage SCP’s if applied incorrectly. Any SCP’s with explicit deny policies override any explicit allow policies.
SCP Evaluation Logic
When a user belongs to an account that is a member of an organisation and accesses a resource that doesn’t have a resource-based policy configured, the resulting permissions are the intersection of the user’s policies, service control policies (SCPs), and resource control policy (RCP). This means that an action must be allowed by all three policy types. An explicit deny in the identity-based policy, an SCP, or an RCP overrides the allow.
Control Tower Errors
During initial deployment of the Control Tower Landing Zone, the deployment failed to complete. In most cases, this is due to a race condition, failed CFN stack or StackSet or just not waiting long enough for the account to “settle” after creation and usually fixed with a “Retry”. Being experienced in this process, I checked the CFN stacks etc – nothing was wrong. In fact, it appeared to have failed very early as it had not yet built the Organization CloudTrail. I waited a little while and then tried the obvious thing – I hit “Retry” and confirmed all the previous Landing Zone settings and ran it again. It failed again.
The error was one I was not familiar with – “AWS Control Tower failed to set up your landing zone completely: AWS Control Tower setup failed. Be sure your account is subscribed to the AWS EC2 service, then try again. If this error persists, contact AWS Support. Learn more.”
I had not seen this before and ironically enough, the link to “Learn more” took me to the official Control Tower troubleshooting guide – however the error was not listed in the guide for any prescriptive guidance on resolving it.
The Control Tower Account Factory had built the Audit and Log Archive Accounts in the Security OU. To find out what was going on, I assumed the “AWSControlTowerExecution” role to switch roles into the Audit and Log Archive accounts. Both accounts presented me with the following welcome page:
A valid payment method had been added to the Management Account upon creation and had already been verified successfully.
A google search returned many interesting results. I followed some breadcrumbs from AWS re:Post posts and stumbled across this entry:
Following the advice of the post, I raised a support case with AWS. As we needed an urgent response, we enabled the Business Support Plan. This enabled access to Technical Support and faster response times. The team confirmed that the payment method was valid and tried launching an EC2 instance. However, when I hit the EC2 landing page, I was immediately thrown back into the Complete sign-up page again. Bizarre, to say the least.
Getting it fixed
Through escalation with AWS Support, we were able to confirm indeed the EC2 had not yet been activated in the Audit account. We did not have access to the backend API’s to do this ourselves. AWS enabled this behind the scenes. I was able to successfully create a VPC and launch an EC2 instance in the Audit Account. I retried the Control Tower Landing Zone and it failed again for the same reason. Switching back to the Audit and Log Archive Accounts – I was again faced with the Complete Sign-Up page.
Understanding I was in fresh territory at this point, I attempted to close the Audit and Log Archive accounts to “reset” the Landing Zone. This in hindsight was the wrong thing to do, however, as the Control Tower Landing Zone manifest was still looking for the accounts. A quick support ticket to AWS Billing was raised and accounts were reinstated.
NOTE: The request for reinstatement must be done by the root user in the account – it cannot be done using an IAM or federated user such as Identity Center.
Despite all this work, however the same error remained:
“AWS Control Tower failed to set up your landing zone completely: AWS Control Tower setup failed. Be sure your account is subscribed to the AWS EC2 service, then try again. If this error persists, contact AWS Support. Learn more”
Another support ticket was raised. AWS Technical Support reactivated the Audit and Log Archive accounts. AWS Support mentioned it was necessary in our case to undertake both actions as when first attempted to reset, the EC2 service was not enabled.
When I ran the retry operation again, it then gave me the following error:
“AWS Control Tower failed to set up your landing zone completely: AWS Control Tower could not baseline VPC in the enrolled account because of existing resource dependencies. Learn more”
This time the error was included in the troubleshooting guide:
To resolve this, I terminated the EC2 instance and deleted the VPC from the Audit Account. I hit the retry and this time it went through successfully. Now we had an established Control Tower Landing Zone, the work of migrating the accounts from the old AWS Organization over to the new one.
Migrating member accounts between AWS Organizations
Procedure followed was the following from each member account:
- Sign in to the AWS Organizations console. You must sign in as an IAM user, assume an IAM role, or sign in as the root user (not recommended) in the organisation’s management account.
- On the AWS accounts page, find and choose the check boxnext to each member account that you want to remove from your organisation. You can navigate the OU hierarchy or enable View AWS accounts only to see a flat list of accounts without the OU structure. If you have a lot of accounts, you might have to choose Load more accounts in ‘ou-name’ at the bottom of the list to find all of those you want to move.
- On the AWS accounts page, find and choose the name of the member account that you want to remove from your organisation. You might have to expand OUs to find the account that you want.
- Choose Actions, then under AWS account, choose Remove from organisation.
- In the Remove account ‘account-name’ (#account-id-num) from organization? dialog box, choose Remove account.
- If AWS Organizations fails to remove one or more of the accounts, it’s typically because you have not provided all the required information for the account to operate as a standalone account. Perform the following steps:
- Sign in to the failed accounts. We recommend that you sign in to the member account by choosing ‘Copy link’, and then pasting it into the address bar of a new incognito browser window. If you do not see ‘Copy link’, use this link to go the Sign up for AWS page and complete the missing registration steps. If you don’t use an incognito window, you’ll be signed out of the management account and won’t be able to navigate back to this dialog box.
- The browser takes you directly to the sign-up process to complete any steps that are missing for this account. Complete all the steps presented. They might include the following:
- Provide contact information
- Provide a valid payment method
- Verify the phone number
- Select a support plan option
6. After you complete the last sign-up step, AWS automatically redirects your browser to the AWS Organizations console for the member account. Choose ‘Leave organization’, and then confirm your choice in the confirmation dialog box. You are redirected to the ‘Getting Started’ page of the AWS Organizations console, where you can view any pending invitations for your account to join other organisations.
7. Remove the IAM roles that grant access to your account from the organisation.
Pre-existing roles
If your account was created in the organization, then Organizations automatically created an IAM role in the account that enabled access by the organization’s management account. If the account was invited to join, then Organizations did not automatically create such a role, but you or another administrator might have created one to get the same benefits. In either case, when you remove the account from the organization, any such role is not automatically deleted. If you want to terminate this access from the former organization’s management account, then you must manually delete this IAM role. For information about how to delete a role, see Deleting roles or instance profiles in the IAM User Guide.
A default payment method must be added to enable leaving an Organization as a Standalone Account. Once the standalone account is removed from the old Organizations added to the new Organization through acceptance of the previously sent invitation.
Default payment method
A default payment is unable to be deleted from an account even after joining an Organization using consolidated billing. We confirmed with AWS Billing, through a support ticket, that the default payment method on a member account is never charged while it remains a member of an Organization. Pablo updated the payment method post-change window to add the corporate credit card and removed his card as a payment method.
Challenges enrolling Into Control Tower
When enrolling the previous Management Account into Control Tower, we encountered an error relating to pre-existing AWS Config Recorder and Delivery Channel. The account had a pre-existing AWS Config recorder and delivery-channel. This was deleted using the following method:
View commands:
- aws configservice describe-delivery-channels
- aws configservice describe-delivery-channel-status
- aws configservice describe-configuration-recorders
- The normal response is something like “name”: “default”
Delete commands:
- aws configservice stop-configuration-recorder –configuration-recorder-name default
- aws configservice delete-delivery-channel –delivery-channel-name default
- aws configservice delete-configuration-recorder –configuration-recorder-name default
Migration was then successful.
Challenges with GuardDuty
While the Delegated Administrator had been setup, GuardDuty had not been enabled into Organizations from the Control Tower Management Account. Further, the previous Management Account had a pre-existing member added for one of the other workload accounts. To resolve this, we had to:
- Login to the old Management Account
- Delete the member account from GuardDuty
- Login to the Management Account
- Enable Integration with Organizations (the banner at the top of the GuardDuty page)
- Login to the Audit account
- Add the member accounts to GuardDuty in the Audit account via Organizations (no need for invitations)
- Turn on Auto-Enable
- Enable the protection plans
Result
Do not attempt to resolve this problem yourself. The problem is not within your control to resolve. This problem may manifest at various stages in the deployment and depending on that stage of failure, may need additional rigor to resolve across multiple AWS Support streams. We were able to get the Control Tower successfully deployed through navigating and explaining clearly the actions undertaken and the excellent support from AWS Support.
Lastly, this has served as a great case study for engaging with the AWS Partner Network – like Cevo. The breadth and depth of experience coupled with our extensive hivemind and AWS reach is crucial to success. You do not have to do it alone. We have your back.
