Cookie Monster: AWS Account Provisioning

INDUSTRY

Insurance Services

CUSTOMER OVERVIEW

Our client is the parent company of a general insurance group with controlled operations in Australia, New Zealand, Thailand, Vietnam and Indonesia.

Case Study

BUSINESS CHALLENGE

Our client was transitioning from an experimentation to implementation phase of their Amazon Web Services (AWS) journey. As the demand for AWS accounts increased, a consistent and streamlined way of provisioning accounts was required. Being part of a highly regulated industry, it was imperative that new accounts complied with the organisation’s risk management framework.

Several cloud products had already been built internally to assist in meeting the compliance obligations of the organisation, including identity access management (Bakery), auditable log aggregation (Woodstock) and security control validation (Watchmen).

Given the size of the enterprise organisation, making sure that teams wishing to move workloads to the cloud were aware of these compliance products (and were running them on their accounts), was proving to be a challenge.

Case Study

SOLUTION

1. REQUIREMENTS GATHERING

Working with our client, the core requirements for the account provision process were identified. The three main requirements were to ensure that:

  • Root account passwords were not employee’s personal email addresses
  • All accounts were linked with a valid cost center for budget tracking
  • Accounts met the client’s security and compliance standards.

The first two of these requirements were quite clearly defined, while the third is subject to change over time, depending on what the security and compliance landscape looked like.

Being embedded with the team responsible for account provisioning also revealed an important non-functional requirement that needed to be addressed; user experience. In this instance, user experience from both a staff member requesting an account, and the team responsible for provisioning the account, are equally important.

From the account requestor point of view, providing a frictionless, easy-to-use service was critical. Previously it was not clear what information was required for a new account, or where to send a request to. Some requests came to individual team members, some went to the team manager.

Within the provisioning team, the steps to administer a new account were not clear to all team members. Additionally, one task for account creation – adding an AWS Direct Connect interface – was the responsibility of another team. Improving both the consistency of the account provisioning and the interfaces between teams was then also brought into scope for the project.

2. SOLUTION DESIGN

Adopting the approach of treating the project as a product (as described here: https://cevo.com.au/devops/2017/09/26/productise_me.html), the first goal was to ensure a consistent user experience when provisioning an a new account. To achieve this, the process was broken down into three stages:

  1. Employee initiating a request with the team
  2. Team provisioning the account
  3. Team sending account details to employee.

Stage one and three were the major touch points with our customer (the person requesting an account), so the primary focus was standardising these.

As the provisioning of the account was largely contained within the same team, what happened during that process initially wasn’t as critical, so long as the output matched what the customer was expecting. Those from a software development background may be familiar with this kind of pattern: keep service interfaces consistent and abstract away the implementation.

Every product needs a name, and calling this project “AWS account provisioning” wasn’t going to be a memorable or unique reference. The project goal was to produce an easy, repeatable way of provisioning accounts – similar to using a cookie cutter. The team had a history of naming other products after fictional characters and so this lead to ‘Cookie Monster’.

3. STANDARDISING INTERFACES
COMMUNICATION METHOD

Several communication options were assessed, with the final choice being a simple email template. There was temptation to create an integrated Jira form, however from a value perspective the team determined that as long as the interface to the other teams was consistent, the time would be better spent on the core account provisioning process. Additionally, the organisation was typically more comfortable with using email than Jira.

This also enabled utilisation of the same medium for stage one (submitting a request) as stage three (sending the account details to the team). Minimising the different types of interfaces presented by a system reduces the cognitive load on users when familarising themselves with its use.

DATA INPUTS

Three initial pieces of data are required from a customer to create a new account:

  • A new email address (restricting the use of personal emails) for the root account
  • Recording a valid cost center against an account
  • Determining whether an AWS Direct Connect interface was required.

When provisioning a direct connect interface, the number of IP addresses required must also be considered.

DATA OUTPUTS

The new account owner will then receive the following about the account:

  • Root account password
  • Sign in link for the account
  • The CIDR range allocated on the direct connect.
4. ACCOUNT PROVISIONING

Based on previous work with this customer, we knew that new accounts had to be provisioned with the following tooling:

  1. Woodstock – A tool for log forwarding to a secure S3 bucket.
  2. Watchmen – A tool for verifying organisation security controls on an account.
  3. General account settings – based on the organisational controls, certain account configuration needed to be set before any workloads could be run from it.

Woodstock and Watchmen had their own Jenkins deployment pipelines that handled the configuration for a new account, as well as the configuration of the central account used by the tools. The account settings however have to be manually configured for the time being.

To ensure the accounts were configured in a consistent manner, lightweight documentation on the process was generated. Manually executing the account provision steps was also the easiest way to validate that what had been done so far was correct. Similarly, there was a concious decision to avoid automating too early to allow the team to ascertain business needs were being met and identify any gaps.

5. ADDING AUTOMATION

After running through the process of provisioning accounts a couple of times, some clear candidates for automation arose. First was the creation of the account on AWS. AWS Organizations provides a simple API for account creation. After conducting a short proof of concept, we found that Oraganizations provided the features needed.

Next was the creation of new email addresses and validation of cost centers. Investigation found there were APIs that could be leveraged to automate both of these tasks as well.

As mentioned above, Woodstock and Watchmen already had pipelines in place to handle deployment to new accounts, and could simply hook into the creation process by passing the new account number into their configuration files.

At this point, all of the major processes for creating a new AWS account had automation wrapped around them. The only manual part was triggering the automated jobs in the correct order, a much less human-error prone task.

AUDIT SCOPE​

For this particular audit the focus areas were: procedure, implementation and verification. The existing policy at the customer was deemed sufficient.

Cevo’s major role involved design of the procedure with the Cloud Engineering team, and ensuring that implementation and verification were completed on time.

AUDIT SCOPE​

A key for quick resolution of audit items is having clearly defined responsibilities, especially regarding decision making. To achieve this, the number of stakeholders involved in addressing the audit was kept to a minimum. In this case, the main stakeholder groups and responsibilities were:

PROCEDURE DESIGN

When designing any security procedure, a trade off between certain factors must be considered:

  • Cost of implementation.
  • Reduction of security risk.
  • Impact on business operations.
    •  
    •  

A perfectly secure, zero risk system may be possible. For example a completely isolated network with no user access. However it may not be very useful for business operations.
The approach taken by Cevo when designing security procedure was to:

      • Use cloud native services where possible.
        • To reduce implementation time and ongoing maintenance.
      • Iteratively reduce security risk.
        • To reduce implementation risk and reduce business impact.
      • Automate where possible.
        • To reduce human error and procedural burden.

The benefits of this iterative, low risk, and automated approach were further reinforced by the recent announcements at AWS re:Invent. Many new cloud native security governance services were announced.

Case Study_1

BENEFITS

The final result from the Cookie Monster project is an account provisioning process that requires only a handful of manual touch points and is extremely consistent in its execution.

The major benefits delivered to our client by this project include:

CONSISTENCY

‘Cookie cutter’ consistency can be seen across the customer experience when requesting an account, the process conducted by the team when provisioning an account, and the final product being delivered to the customer.

COMPLIANCE

Automation provided a fully auditable method of deploying the client’s compliance tools, Woodstock and Watchmen.

The is now a record of who created the account, and who ran the pipelines to install the tools into the accounts. This also removed the need for a documentation-heavy manual process.

EFFICIENCY

The team responsible for account provisioning now has much less manual work to conduct when provisioning an account, improving efficiency and work productivity and reducing the chance of human error.