Industry

Online Retail

Customer Overview

Our client is one of Australia’s leading eCommerce organisations, with a track record of growth that has seen their two core brands become household names over the past 10 years; highly profitable in every year of operation. Competitive and driven, they provide customers with an innovative shopping experience everyday.

Business Challenge

The client had experienced an outage on the website due to performance issues. For an online retailer, any outage can have a significant impact on the business and its customers, so needed to be addressed as quickly as possible.

Further, due to the high demand of the website, the issue needed to be investigated and resolved whilst avoiding unscheduled outages. Our client had a theory as to the cause of the outage, but required external support to assist the team with verifying this cause and finding a solution.

Solution

A consultant from Cevo was brought in to evaluate the current state and theory behind the outage. Both the environment and application were reviewed in conjunction with the fixes that were scheduled to be implemented.

Based on the findings it was identified that the environment was not the cause, rather an issue within the caching. As a result, a suggested roadmap was delivered which identified a hierarchy of recommendations.

Benefits

By using an external party with a fresh view of the application and environment, our client was able to quickly verify their theory and look to a solution without causing interruption to their business. Within a short period of time, teams were able to collaborate and develop a roadmap that provided the most appropriate way to resolve the issue and avoid any future recurrences.

Industry

Insurance services

Customer Overview

Our client is the parent company of a general insurance group with controlled operations in Australia, New Zealand, Thailand, Vietnam and Indonesia.

Business Challenge

As the move to the cloud started to pick up pace across the organisation, the demand for new Amazon Web Services (AWS) accounts increased. There was a manual process in place for provisioning new accounts, but the effort involved was quite high. For each new account the following activities had to be performed:

1. Validate the cost centre provided.
2. Create a new shared mailbox to register the account.
3. Create the account under the AWS organisation.
4. Add the account to Watchmen and install the Citizen CloudFormation stack on the new account. (See previous case study for an overview of Watchmen)
5. Add the account to Woodstock and install the Lumberjack CloudFormation stack on the new account. (See previous case study for an overview of Woodstock)
6. Email the account requestor that the account was now ready for use.

These steps took some time for the team to complete, and they were also error prone.

Solution

The key considerations for building a solution to this problem were:

1. Reducing the potential for manual error.

2. Minimising the lead time; that is the time between the account being requested and the account details being sent to the requestor.

There were two major sources of manual error that were identified in the current account provisioning process:

1. Accuracy of data entry.

2. Missing process steps.

Working with our client we concluded that if most of the account provisioning process could be automated both of these issues would be addressed. Automation would also decrease the process lead time, and if the process was easy to execute then the team would be more willing to do so when the request came through, rather than delay until later.

The solution ended up being a Bamboo pipeline, with each of the tasks being a separate step in the pipeline. To remove the data entry errors, a simple account request form was created with several fields to complete. Someone in the team would review the form to see that all the required information had been provided, then the file would get checked into source control. The check-in would trigger the account creation pipeline to start.

The installation of the Citizen and Lumberjack CloudFormation stacks still requires manual intervention, we are working toward removing those manual touch points as well.

Benefits

Teams can now request an account and have it created in less than an hour, which is a great result for an enterprise organisation the size of our client.

The solution improves team efficiency, reducing the laborious process for those responsible for creating new accounts as well as the lead time for those waiting for access. The new account is provisioned with the necessary compliance tools, Watchmen and Woodstock, as well as having a valid cost centre associated with it. The accuracy of the process is now also very high, with little chance of manual mistakes being made.

The solution also allowed for future-proofing, so if more steps need to be taken when provisioning an account, the pipeline can be easily extended.

Industry

Online Employment Services

Customer Overview

Our client is a diverse group of companies in the Online Employment Services sector that have a unified purpose to help people live more fulfilling and productive working lives, and help organisations succeed on a global basis. They create world-class product technology solutions to address the needs and facilitate the matching between job seekers and hirers with exposure to 4.1 billion people globally and greater than 30% of Global GDP.

Across their employment marketplaces they receive over 375 million visits to their sites every month and have over 3 million job opportunities available at any given time with over 100 million jobseeker profiles.

Business Challenge

Jobs listed on our client’s website contain links to the company profiles. Recently, 20% of the traffic was routed to a brand-new solution for the homepage, search results and job pages that had more prominent showing of the company reviews (1-5 stars right next to the company name). The volume of traffic to company profiles increased by 40% within a week causing performance issues due to the Content Delivery Network (global cache) failing to remember what it had previously served.

Solution

A replica of the production Content Delivery Network settings was built in a development account and investigated to steadily narrow down the root cause. Once identified, a fix was trialled and tested, with the positive response allowing a deployment to production.

Benefits

The improved responses from the Content Delivery Network reduced the average page load time of job postings by 20% within hours, giving our client’s customers a better front end User Experience. The findings were also documented and shared with the other delivery teams in the organisation to ensure an improvement in future deployments.

Industry

Tertiary Education

Customer Overview

Our client is one of Australia’s largest and most international universities. As a truly global institution, there are five Australian campuses, a campus in Malaysia, a joint graduate school in China, a learning centre in Italy, and a research centre in India. A research-intensive university, they are known for some significant and lasting discoveries that have delivered impact beyond the academic community.

Business Challenge

Our client has a small team of infrastructure staff who manage the Amazon Web Services (AWS) environment through a support ticketing system. Any work or fix with a reliance on the infrastructure was submitted via a ticket, placed in a queue and worked on by this team on top of their daily workload.

As a result they soon became a bottleneck for change for the increasing number of teams within the university using the AWS environment.

Solution

To resolve this backlog and additional workload on the development team, the university worked with Cevo to devise a solution to improve infrastructure handling capability and understanding of cloud within other teams. To do this, the application operations team (previously denied access to the AWS environment), were upskilled in the key areas of AWS which they would require on a frequent basis. A master apprentice model was used to ensure that the team understood and could implement what they had learnt.

The focus of this engagement was split between how and why. Team members were able to learn the technical skills enabling their ability on HOW to solve the problem, but additional time was also spent in understanding WHY this approach not only solved the problem, but increased the manageability of the new solution.

Benefits

Both the teams gained confidence in the ability of the operations team to access and manage key components within the AWS account. This included feeling confident that they understood why they were undertaking particular actions and were able to collaborate more effectively. Overall this achieved the result of reducing the delay in environmental changes through logging tickets.

Industry

Financial Technology

Customer Overview

Our client is a next-gen universal payment platform that enables central payment networks and financial institutions to streamline merchant and consumer transactions. Using existing infrastructures, they leverage a new security paradigm that eliminates the need for exposing cardholder data during a transaction and delivers one app for all payments. Our client reduces the major causes of card fraud and breaches, increases mobile payment adoption and lowers fees and costs.

Business Challenge

The founders of this business found a need in the market for a more robust payment platform and assembled a clever team of developers to build and host the solution through Amazon Web Service (AWS). The development team, while creating a well thought out environment, had learnt about AWS on the job and were not yet experts in this area so required some help in validating and expanding the environment.

Solution

A consultant specialising in AWS was engaged over a two week period to review the current AWS setup and validate the infrastructure architecture and its ability to scale once the product was launched to a test market.

A roadmap was identified and presented to the team via a showcase prior to commencing implementation. This included developing CloudFormation templates as well as educating the team in monitoring their apps and shared AWS infrastructure for compliance with CloudWatch and CloudTrail. Some DevOps training was also conducted around automatic deployments of their workload in CloudFormation.

Benefits

The solution allowed our client to be more confident in their environment moving forward as well as upskill their team in AWS. Having an external consultant work on this allowed the rest of the team to focus their attention on their backlog of work giving them the best opportunity at a successful launch.

An auxiliary benefit came from the network of information which was fed into our client through other Cevo consultants and this left them with an additional skill set around AWS.

Industry

Insurance Services

Customer Overview

A large insurance organisation was looking for ways to minimise their cloud costs. The businesses underwrite over $11.4 billion of premium per annum, selling insurance under many leading brands. The company also has interests in general insurance joint ventures in Malaysia and India. Standard & Poor’s has assigned a ‘Very Strong’ Insurer Financial Strength Rating of ‘AA-’ to the Group’s core operating subsidiaries.

Business Challenge

To improve their software development speed, the firm were investigating using Amazon Web Services (AWS) to provide development environments for a selection of their delivery teams. This was great for solving the common and painful problem of environment availability, however it also created a new challenge: rapidly escalating cloud costs.

Solution

Our client ran some internal initiatives to rightsize AWS usage which were successful. To evaluate further cost reductions they asked Cevo to assist in evaluating:

• If using AWS spot instances would reduce the monthly hosting bill
• If there were any other, non-monetary expenses generated from running on spot

A experiment was set up to produce a cost analysis that would outline how different AWS instance types could be used to manage the monthly AWS spend more effectively.

Benefits

As a result of this investigation, our client realised there would be no cost saving for them in moving to spot. They were able to evaluate the effectiveness over a short period of time, and make a decision without affecting any of their environments or risking a spike in cost.

Industry

Property Management

Customer overview

This start-up is a world-first integrated property managment technology and services platform that transforms the way people in a geographic area connect and communicate. Their goal is to create activated communities, places that make residents and retailers of the area stickier and drive value for asset owners.

Business challenge

Operating as a start up, the digital platform emerged from a series of MVP experiments to identify which services provided value to customers. This also led to architectural challenges around scale, technical debt and reliability. With competitors entering the market, improving the customer experience and speed of the delivery team became a top priority.

Solution

Cevo worked with this client to significantly enhance the events system with a focus on an improved user experience. This process included story writing workshops and coaching the development team in agile methodologies and user acceptance testing.

Benefits

The execution of this value add project surfaced several interesting learnings for the start-up:

1. The flexibility of their technology stack was improved in terms of what could be achieved from a user experience view as well the amount of time it took to make changes.

2. Spending the time on polishing features was worth the effort, as the new events system helped seal a contract with a new client.

Industry

Engineering Software

Customer Overview

Our client is the most widely used online collaboration platform in the world for construction, infrastructure, energy and resources projects. They help manage more than US$1 trillion in capital projects across 70 countries, serving over 70,000 user organizations.

With over 850 employees dedicated to helping project teams finish early and under budget they are transforming how project teams work together, increasing accountability and extending control.

Business Challenge

Over the past four years our client has experienced rapid grown through acquisitions and expansions into new territories and markets. As an organisation that has been working in the Software as a Service (SaaS) market for many years, the breadth and complexity of their systems is both wide and deep. They currently service customers around the globe from a number of co-managed hosting centres.

To meet the demands of an increasing market, they required an ability to accelerate the rate at which they could bring on new clients, and the infrastructure required to support them. At the time, using traditional infrastructure provisioning processes, the organisation was able to bring online a new hosting centre but it would take multiple months of effort. The Cloud Engineering team was given the directive to slash the time it takes to provision new infrastructure while continuing to provide a high standard of quality.

Investigations by the in-house Cloud Engineering team identified both Amazon Web Services (AWS) and Azure as great candidates to provide the Infrastructure as a Service to meet these growing demands.

Solution

Cevo was engaged to join the Cloud Engineering team to work through replicating a production scale environment in AWS, ensuring that opportunities were taken to improve the consistency, resilience and repeatability of the process. The decision to start with AWS was based on efficiency, having already established workloads in the environment - however a key requirement for the solution was the ability for it to be applied across the Azure platform as and when required.

Bringing solid experience in Infrastructure as Code patterns, the Cevo team provided guidance on ways to improve the development and management of the infrastructure code base. Working alongside the client team, they were able to develop a set of automated delivery pipelines that enabled staff to focus on identifying the infrastructure requirements and delivering value, safe in the knowledge that if something was misconfigured the automated deployment process would rapidly highlight the issue and enable the team member to correct the error with minimal turnaround time.

Benefits

Through investing in an Infrastructure as Code approach, the Cloud Engineering team is now able to deploy all of the required infrastructure within an hour. Not only does this save a large amount of time, but also increases the consistency of the environments due to automation repeatedly delivering the solution.

By coupling a focus on automated quality validation and continuous delivery practice, the team now have a development process that will allow them to deliver the needed expansion to the organisation while still supporting a growing appetite for innovation.

Industry

Accounting Software

Customer Overview

MYOB is a leading provider of business management solutions in Australia and New Zealand. MYOB helps businesses of all kinds and sizes, delivering software and services that simplify accounting, payroll, client management, websites, and more. With a network of more than 25,000 accountants and other professional partners, MYOB provides the tools and support that help make business life easier.

Business Challenge

MYOB had a contract with its traditional Australian data centre due for renewal in early 2016. This provided the organisations with an opportunity to innovate and seek out infrastructure improvements to provide delivery teams more control of their environments. The vision was to reduce the cycle time of “concepts to production” through gaining configuration traceability, repeatability through automation and self service to reduce dependency on third parties. This vision ultimately tied into one of MYOB’s core business values to help businesses (their customers) be successful.
“We needed a solution that would enable greater flexibility and visibility in terms of infrastructure configuration, automation and performance” says Grant Tibbey, Infrastructure Lead.
In its current state, MYOB Essentials was running in a third-party data center, utilising SQL Server and a Java front end. The company decided to move its infrastructure to AWS.

Solution

From the time MYOB decided it wanted more flexibility and more customer benefits from the Essentials infrastructure (until the existing contract expired), it had only approximately three months to complete the cutover.

MYOB formed a team using existing in house DevOps skills but didn’t have enough capacity to meet the aggressive deadlines. MYOB utilises a wide range of technologies, and Cevo were able to offer skills and availability with minimal onboarding requirements. MYOB was looking for experience without having to bring additional people up to speed on tools and techniques that were being applied to automation.

The existing application had some automation around building and deploying to the application servers using Ansible and Gradle. There was however no automation around building MS SQL Server and no automation around building the network infrastructure. Given the time constraints, existing automation was re-used and augmented with additional tools. Chef was chosen because of its ability to configure Windows and Linux servers securely using Secure WinRM and SSH protocols. It also gave the team the ability to unit and integration test components of the server automation as isolated modules and provided rapid feedback to the development processes using KitchenCI (test-kitchen). Unlike other automation technologies code could be developed and tested with the physical server being stubbed. This reduced server development wait-times. CloudFormation was used for network automation and enhanced with complex logic for alarm calculations using ruby code. CloudFormation templates were generated using a ruby gem “cfndsl” to enhance its static nature.

Options for running MS SQL Server in AWS were considered and included RDS, Log Shipping and Always on Availability Groups to achieve High Availability. These options however were discarded due to cost, extended outage and data loss that would have occurred on failure or reconfiguration. Instead, Mirroring was chosen for HA with Log Shipping initially used to keep the data in AWS in sync with the traditional datacenter to within fifteen minutes so that the cutover would cause minimal impact to customers.

The entire process of building the SQL Cluster in AWS was scripted from scratch. This included formatting, moving, labeling disks, moving system databases and logs, creating users, backup jobs, installing certificates to establishing mirroring using partner and witness amongst many other things that would traditionally have been manually done.
“If something didn’t work, we can tear it down and build it again in hours,” says Grant. “Practice also makes perfect. We practiced rebuilding servers, and cutting over the system constantly. Even weeks before the actual cut over took place”.
By the time cutover took place the entire system could be rebuilt with a couple of commands.

Moving forward, the company plans to re-engineer the application from statically deployed to dynamic auto-scaling servers that come and go, and will look at ways to achieve zero downtime deploys to deal with fault tolerance. Through the ongoing use of Chef scripts and development practices applied to Ops, MYOB’s development team can rewrite the application in such a way that it can easily be deployed to AWS.

Benefits

Benefits of the solution include providing MYOB with:

• A safe, repeatable and easy means to release features to their customers as soon as they are capable of providing benefit.
• The ability to rebuild servers at will or when they fail.
• The ability to reconfigure or change the way the MYOB Essentials app is built; whether it’s to increase performance, reliability or change how they run deployments, infrastructure architecture can be changed and new services added.
• Cost optimisation – with the scalability offered by AWS, MYOB is able to have full visibility of what is running and can start reducing expenditure through ensuring they are not over or under provisioned.

Industry

Tertiary Education

Customer Overview

Our client is one of Australia’s largest and most international universities. As a truly global institution, there are five Australian campuses, a campus in Malaysia, a joint graduate school in China, a learning centre in Italy, and a research centre in India. A research-intensive university, they are known for some significant and lasting discoveries that have delivered impact beyond the academic community.

Business Challenge

One of the key benefits of undertaking a cloud transformation adoption strategy is unlocking the potential to move fast and reduce the overheads with respect to existing infrastructure provisioning processes.

In the University’s case, IT staff were spending a significant amount of time manually provisioning services within the Amazon Web Services (AWS) environment. The delays in getting code deployed was impacting the ability for the teams to deliver their value to the business and also increasing the cost of delivery.

Whilst the AWS adoption had already begun for our client, the real benefit was not being realised due to manual activities still being one of the biggest constraints.

Solution

Cevo consultants worked alongside the IT staff to create a semi-automated process to reduce the amount of repetition in building environments. This approach was not about ‘automate at all cost’, but focused on automating the highly repetitive actions, unlocking capacity within the existing teams to focus on higher value activities.

Through automation approaches, the actions required to create an environment became explicitly clear, enabling a whole new level of collaboration between teams to review and identify opportunities for reuse.

Benefits

A significant reduction in time was achieved, allowing the University to set up their servers and deploy code faster. A cost saving was also found through the reduction in duplicating manual processes.

In addition, the patterns identified during this approach are able to be repurposed for additional solutions. This outcome helps the University extend their existing patterns and toolkits into new application design patterns.

Industry

Education Software

Customer Overview

Factor5™ Software aims to disrupt the way universities approach academic and student administration by modernising curriculum management. They do this through a highly configurable, cloud-based curriculum lifecycle management solution, CourseLoop™ (built on the third-party ServiceNow platform). Purpose built for universities, CourseLoop automates manual processes, improving time to market for new offerings, reducing administration burden on staff and students, and increases quality of curriculum design.

Business Challenge

As a fast-moving start up, Factor5 has CourseLoop under active development, with frequent and continuous deployments. At the time, these deployments were very time consuming and did not always guarantee reliability. With Factor5 scaling to commence development on two more product offerings, the overhead strain was only going to become more exacerbated. Foreseeing this, the company wanted to implement a better and more sustainable way to perform and track deployments.

Solution

The team kicked off with a series of investigative workshops and quickly realised that the non-conventional nature of the development workflow running on top of the ServiceNow platform was going to require some creativity and a change in team practices. A hybrid solution which implemented bespoke and ServiceNow native elements was proposed. This paved the way for the team to implement a more automated deployment method, packaging the changes into a singular item which could also be extracted from the ServiceNow platform into a version control system.

Benefits

The Factor5 team is now able to conduct deployments of CourseLoop much quicker than before. Importantly, they have the confidence that deployments will occur without issue and no longer depend on human intervention. These improvements in the way CourseLoop is built and deployed can also be applied to future product offerings where applicable, helping maintain efficiencies and manage team workflow.

Industry

Insurance services

Customer Overview

IAG sell insurance under many leading brands including NRMA Insurance, CGU, SGIO, SGIC, Swann Insurance and WFI (Australia); NZI, State, AMI and Lumley Insurance (New Zealand); Safety and NZI (Thailand); AAA Assurance (Vietnam); and Asuransi Parolamas (Indonesia).

Business Challenge

IAG had approximately 100 applications sitting on physical servers with a strategic direction to migrate to Amazon Web Services (AWS). Before this could begin, they needed to ensure that all environments met APRA regulations. Of the tools available at the time, none were suitable to fully achieve these objectives.

Solution

IAG and Cevo worked with APRA to clarify and document a list of compliance requirements. The team then wrote a tool called ‘Watchmen’ that was governed by 40 unique security policies that would scan each environment (called ‘Citizens’) and report on compliance with these policies. Where an environment did not conform to the security policies, the tool would report this exception and where the exception occurred.

Features of Watchmen:

1. Isolated rule logic
   The logic used for determining compliance to each rule sits in an isolated account that non of the monitored (Citizen) accounts can access. This ensures that there is no tampering with the rule logic, and a result of ‘Compliant’ means what we think it means.

2. Serverless
   Watchmen runs on AWS serverless infrastructure. All of the rule logic exists in Lambda functions, while the compliance scans themselves are managed by AWS config. This means the ongoing maintenance and support required for Watchmen is very low.

3. Auto updating
   When new rules are added to Watchmen, the Citizens are able to be automatically updated though the use of SNS topics and Lambda. So once the Citizen CloudFormation stack has been installed on a Citizen account, the account owner never needs to touch it again.

4. Easy to extend
   As the rule logic sits in Lambda functions, you are able to write new rules in any Lambda supported function. The deployment pipeline created for Watchmen will pick up new rules and incorporate them into the deployment.

Benefits

Prior to this project, the migration had been delayed because of the time it would take for the security team to manually check the compliance of any given AWS account. Watchmen has allowed IAG to confidently approve the migration of applications to pre-authorised environments. Now IAG can move forward with their AWS migration knowing that Watchmen automatically checks the environments daily ensuring continual compliance with security procedures and APRA.

Both IAG and Cevo hope to open-source Watchmen over the coming weeks and are working towards that end.

Industry

Insurance services

Customer Overview

Our client is the parent company of a general insurance group with controlled operations in Australia, New Zealand, Thailand, Vietnam and Indonesia.

Business Challenge

As part of our client’s intrusion detection system (IDS), one commonly adopted security pattern is the forwarding of logs of various types into the company’s Security Information and Event Management (SIEM) system. Of particular interest to the SIEM team are the Amazon Web Services (AWS) CloudTrail logs, as they provide information about all AWS API calls that are run on an account, and are a good place to start when trying to detect suspicious activity.

There were several challenges that the client faced when getting CloudTrail logs into SIEM:

• The AWS maturity of teams across the organisation varies greatly. Not all teams were aware of the CloudTrail service and what it does, and this created variation in compliance.

• Each integration into SIEM required several non-trivial steps. Following this, the integration needed to be monitored for a few days to make sure it was working correctly.

• Identifying which teams still needed to complete the integration, and providing the necessary instruction on how to do this, was proving an additional management burden.

Solution

Working together with Cevo, a solution was devised to address these challenges in the simplest way possible. When broken down, the main goal was to design a process that:

1. Is easy for all AWS account owners to use

2. Only requires one integration with SIEM.

The solution created was entitled ‘Woodstock’ (ie a large pile of logs in one place!) Woodstock stores the CloudTrail logs of each AWS account. Each account writes to a specific directory that only it has access to, and SIEM can then read all of the logs from the one place.

For AWS account owners, a simple CloudFormation template we named Lumberjack was created (because, Lumberjacks send the logs). The template only takes five minutes to download and run and conducts the necessary CloudTrail configuration.

Leveraging other work done previously with our client, a Watchmen rule was added to check that accounts had their CloudTrail configured correctly.

Benefits

Simplifying the process of forwarding CloudTrail logs into SIEM means that teams are much more likely to undertake the process, thereby increasing compliance and visibility whilst reducing administrative burden on management.

The Woodstock solution also adds another layer of security monitoring to all AWS workloads with no room for human error, as all the configuration for required for an account is executed as code.

As a result of these benefits, the solution has now also been extended to cover logs from other AWS services within the organisation.