Our client is one of Australia’s leading eCommerce organisations, with a track record of growth that has seen their two core brands become household names over the past 10 years; highly profitable in every year of operation. Competitive and driven, they provide customers with an innovative shopping experience everyday.
Business Challenge
The client had experienced an outage on the website due to performance issues. For an online retailer, any outage can have a significant impact on the business and its customers, so needed to be addressed as quickly as possible.
Further, due to the high demand of the website, the issue needed to be investigated and resolved whilst avoiding unscheduled outages. Our client had a theory as to the cause of the outage, but required external support to assist the team with verifying this cause and finding a solution.
Solution
A consultant from Cevo was brought in to evaluate the current state and theory behind the outage. Both the environment and application were reviewed in conjunction with the fixes that were scheduled to be implemented.
Based on the findings it was identified that the environment was not the cause, rather an issue within the caching. As a result, a suggested roadmap was delivered which identified a hierarchy of recommendations.
Benefits
By using an external party with a fresh view of the application and environment, our client was able to quickly verify their theory and look to a solution without causing interruption to their business. Within a short period of time, teams were able to collaborate and develop a roadmap that provided the most appropriate way to resolve the issue and avoid any future recurrences.
ADFS automation with EC2 and CloudFormation
Industry
Wagering / Gaming
Customer Overview
Our client, an ASX100 company, is one of the largest publicly listed gaming and wagering organisations globally.
As one of the few integrated gambling and entertainment companies in the world, their iconic brands and stable of subscription television channels ignite passion and excitement in millions of customers each year.
Business Challenge
Our client runs Active Directory to manage all of their user permissions and profiles. The growing number of applications being developed and hosted externally in the cloud has brought to light the large amount overhead required to integrate each with Active Directory.
As a result, the organisation wanted an automated Active Directory Federation Services (ADFS) system for test environments. Their base requirements were:
Ease of maintenance
Speed of provisioning
Reproducibility for multiple test environments
Hosted by Amazon Web Services (AWS) in Sydney
In addition, they required the created Active Directory system to:
Be configured from a defined list of users, group, and claim rules, using a format and location to be determined
Be configured for Security Assertion Markup Language (SAML) authentication and direct Lightweight Directory Access Protocol (LDAP) connectivity
Solution
Working alongside our client, Cevo helped to refine the scope of requirements,identify appropriate implementation options, build, test, and document the solution using an iterative approach.
The original challenge was solved by creating a repeatable deployment and auto-configuration process, between Active Directory and external applications, which automated the authentication process through ADFS hosted on AWS.
During evaluation however, it was identified that because of the complexity of the Active Directory, only partial automation could be completed unless the solution was restructured. Within the original timeframe set, the entire process was re-engineered and additional workflows created to achieve end-to-end automation of the ADFS integration.
Benefits
By building and deploying a fully automated system, our client was able to reduce a manual process taking upwards of 2 weeks to a push button solution completing in under 30 minutes.
AWS Account Provisioning Automation: Improving lead time, efficiency and accuracy
Industry
Insurance services
Customer Overview
Our client is the parent company of a general insurance group with controlled operations in Australia, New Zealand, Thailand, Vietnam and Indonesia.
Business Challenge
As the move to the cloud started to pick up pace across the organisation, the demand for new Amazon Web Services (AWS) accounts increased. There was a manual process in place for provisioning new accounts, but the effort involved was quite high. For each new account the following activities had to be performed:
Validate the cost centre provided.
Create a new shared mailbox to register the account.
Create the account under the AWS organisation.
Add the account to Watchmen and install the Citizen CloudFormation stack on the new account. (See previous case study for an overview of Watchmen)
Add the account to Woodstock and install the Lumberjack CloudFormation stack on the new account. (See previous case study for an overview of Woodstock)
Email the account requestor that the account was now ready for use.
These steps took some time for the team to complete, and they were also error prone.
Solution
The key considerations for building a solution to this problem were:
Reducing the potential for manual error.
Minimising the lead time; that is the time between the account being requested and the account details being sent to the requestor.
There were two major sources of manual error that were identified in the current account provisioning process:
Accuracy of data entry.
Missing process steps.
Working with our client we concluded that if most of the account provisioning process could be automated both of these issues would be addressed. Automation would also decrease the process lead time, and if the process was easy to execute then the team would be more willing to do so when the request came through, rather than delay until later.
The solution ended up being a Bamboo pipeline, with each of the tasks being a separate step in the pipeline. To remove the data entry errors, a simple account request form was created with several fields to complete. Someone in the team would review the form to see that all the required information had been provided, then the file would get checked into source control. The check-in would trigger the account creation pipeline to start.
The installation of the Citizen and Lumberjack CloudFormation stacks still requires manual intervention, we are working toward removing those manual touch points as well.
Benefits
Teams can now request an account and have it created in less than an hour, which is a great result for an enterprise organisation the size of our client.
The solution improves team efficiency, reducing the laborious process for those responsible for creating new accounts as well as the lead time for those waiting for access. The new account is provisioned with the necessary compliance tools, Watchmen and Woodstock, as well as having a valid cost centre associated with it. The accuracy of the process is now also very high, with little chance of manual mistakes being made.
The solution also allowed for future-proofing, so if more steps need to be taken when provisioning an account, the pipeline can be easily extended.
Accelerating the Cloud Transformation Maturity Model
Industry
Financial services
Customer Overview
Our client exists to provide Australian Small and Medium Enterprises with innovative products and services to help improve and grow their business. The company started by providing integrated EFTPOS, and recently launched lending and deposit products, with plans to expand into other services in the future.
Business Challenge
Having not long started their Amazon Web Services (AWS) journey, our client was moving from the project stage to the foundation stage of the cloud transformation maturity model (CTMM - see note). They had several proof of concepts successfully deployed to AWS, and were defining processes and building tooling to assist with the migration of more applications.
A lack of in-house AWS experience resulted in the transition taking longer than desired as the engineering teams had to learn new concepts and technologies whilst developing new patterns, processes and tools.
As a result, two key areas were identified as requiring assistance:
Migrating to cloud and adopting sustainable cloud native patterns (to reduce reliance on physical data centres and increase business agility)
Reaching the next stage of the CTMM
Solution
Two engineers from Cevo were brought in to work with the client’s Platform Engineering team and assist with the development of shared tooling, deployment infrastructure and the design of AWS architectural patterns.
During this process they were able to share AWS knowledge and skills with the team to help them move forward through the CTMM. In addition, Cevo engineers validated a number of design decisions the team had made, and provided guidance on alternative approaches for several others.
Benefits
By building team capability and sharing of sustainable, scalable technical delivery practices, our client was able to accelerate their progress on the cloud transformation maturity model.
The transfer of experience and knowledge has given the Platform Engineering team the confidence, skills and ability to reduce the environment complexity, and set up tooling and processes. This has the knock-on effect of enabling rapid adoption of AWS cloud services by other delivery teams within the company.
Note: The AWS Cloud Transformation Maturity Model (CTMM) maps the maturity of an IT organisation’s process, people, and technology capabilities as they move through the four stages of the journey to the AWS Cloud: project, foundation, migration, and optimization. The objective of the CTMM is to help enterprise IT organizations understand the significant challenges they might face as they adopt AWS, learn best practices and activities to handle those challenges, and recognize the signs of maturity or expected outcomes to gauge their maturity and readiness at every stage.
Achieving a 20% decrease in Load Time
Industry
Online Employment Services
Customer Overview
Our client is a diverse group of companies in the Online Employment Services sector that have a unified purpose to help people live more fulfilling and productive working lives, and help organisations succeed on a global basis. They create world-class product technology solutions to address the needs and facilitate the matching between job seekers and hirers with exposure to 4.1 billion people globally and greater than 30% of Global GDP.
Across their employment marketplaces they receive over 375 million visits to their sites every month and have over 3 million job opportunities available at any given time with over 100 million jobseeker profiles.
Business Challenge
Jobs listed on our client’s website contain links to the company profiles. Recently, 20% of the traffic was routed to a brand-new solution for the homepage, search results and job pages that had more prominent showing of the company reviews (1-5 stars right next to the company name). The volume of traffic to company profiles increased by 40% within a week causing performance issues due to the Content Delivery Network (global cache) failing to remember what it had previously served.
Solution
A replica of the production Content Delivery Network settings was built in a development account and investigated to steadily narrow down the root cause. Once identified, a fix was trialled and tested, with the positive response allowing a deployment to production.
Benefits
The improved responses from the Content Delivery Network reduced the average page load time of job postings by 20% within hours, giving our client’s customers a better front end User Experience. The findings were also documented and shared with the other delivery teams in the organisation to ensure an improvement in future deployments.
Building confidence with Infrastructure Upskilling
Industry
Tertiary Education
Customer Overview
Our client is one of Australia’s largest and most international universities. As a truly global institution, there are five Australian campuses, a campus in Malaysia, a joint graduate school in China, a learning centre in Italy, and a research centre in India. A research-intensive university, they are known for some significant and lasting discoveries that have delivered impact beyond the academic community.
Business Challenge
Our client has a small team of infrastructure staff who manage the Amazon Web Services (AWS) environment through a support ticketing system. Any work or fix with a reliance on the infrastructure was submitted via a ticket, placed in a queue and worked on by this team on top of their daily workload.
As a result they soon became a bottleneck for change for the increasing number of teams within the university using the AWS environment.
Solution
To resolve this backlog and additional workload on the development team, the university worked with Cevo to devise a solution to improve infrastructure handling capability and understanding of cloud within other teams. To do this, the application operations team (previously denied access to the AWS environment), were upskilled in the key areas of AWS which they would require on a frequent basis. A master apprentice model was used to ensure that the team understood and could implement what they had learnt.
The focus of this engagement was split between how and why. Team members were able to learn the technical skills enabling their ability on HOW to solve the problem, but additional time was also spent in understanding WHY this approach not only solved the problem, but increased the manageability of the new solution.
Benefits
Both the teams gained confidence in the ability of the operations team to access and manage key components within the AWS account. This included feeling confident that they understood why they were undertaking particular actions and were able to collaborate more effectively. Overall this achieved the result of reducing the delay in environmental changes through logging tickets.
CloudFormation, CloudWatch and CloudTrail: building skills in AWS
Fully utilising the functionality of Amazon Web Services
Industry
Financial Technology
Customer Overview
Our client is a next-gen universal payment platform that enables central payment networks and financial institutions to streamline merchant and consumer transactions. Using existing infrastructures, they leverage a new security paradigm that eliminates the need for exposing cardholder data during a transaction and delivers one app for all payments. Our client reduces the major causes of card fraud and breaches, increases mobile payment adoption and lowers fees and costs.
Business Challenge
The founders of this business found a need in the market for a more robust payment platform and assembled a clever team of developers to build and host the solution through Amazon Web Service (AWS). The development team, while creating a well thought out environment, had learnt about AWS on the job and were not yet experts in this area so required some help in validating and expanding the environment.
Solution
A consultant specialising in AWS was engaged over a two week period to review the current AWS setup and validate the infrastructure architecture and its ability to scale once the product was launched to a test market.
A roadmap was identified and presented to the team via a showcase prior to commencing implementation. This included developing CloudFormation templates as well as educating the team in monitoring their apps and shared AWS infrastructure for compliance with CloudWatch and CloudTrail. Some DevOps training was also conducted around automatic deployments of their workload in CloudFormation.
Benefits
The solution allowed our client to be more confident in their environment moving forward as well as upskill their team in AWS. Having an external consultant work on this allowed the rest of the team to focus their attention on their backlog of work giving them the best opportunity at a successful launch.
An auxiliary benefit came from the network of information which was fed into our client through other Cevo consultants and this left them with an additional skill set around AWS.
Cost reductions & faster time to market with automated, on-demand test environments
Industry
Wagering / Gaming
Customer Overview
Our client, an ASX100 company, is one of the largest publicly listed gaming and wagering organisations globally.
As one of the few integrated gambling and entertainment companies in the world, their iconic brands and stable of subscription television channels ignite passion and excitement in millions of customers each year.
Business Challenge
Internal software development within our client’s organisation was constrained by a small number of fixed, manually-curated test environments. Software development teams were regularly blocked from delivering features and bug fixes due to an inability to test changes rapidly enough; it was not uncommon for a team to be delayed for over 150 days during a 12-month period, which drove up the cost of development and increased the time until the return on investment could be realised.
Solution
Cevo partnered with our client’s technology teams to identify, prioritise, migrate, and deliver test environment automation. This allowed independent teams to create and manage their own on-demand test systems in the Amazon Web Services (AWS) cloud environment.
The solution included features for cost control, version management, automated event notifications, and integration with existing build systems to ensure future changes were available with no additional effort from teams.
Benefits
The project delivered automated, on-demand test environments for two core systems, which removed blockers for many different streams of work.
As a result our client was able to realise the benefits from automation of manual processes, including:
Reduced software development cycle and testing times
Improved efficiencies and quality
Faster identification and response to any errors.
These benefits, combined with automation reducing the need for manual effort in future projects, also achieves overall cost reductions and faster delivery to market for our client.
Data driven decision making: Investigating the benefits of AWS spot instances
Industry
Insurance Services
Customer Overview
A large insurance organisation was looking for ways to minimise their cloud costs. The businesses underwrite over $11.4 billion of premium per annum, selling insurance under many leading brands. The company also has interests in general insurance joint ventures in Malaysia and India. Standard & Poor’s has assigned a ‘Very Strong’ Insurer Financial Strength Rating of ‘AA-’ to the Group’s core operating subsidiaries.
Business Challenge
To improve their software development speed, the firm were investigating using Amazon Web Services (AWS) to provide development environments for a selection of their delivery teams. This was great for solving the common and painful problem of environment availability, however it also created a new challenge: rapidly escalating cloud costs.
Solution
Our client ran some internal initiatives to rightsize AWS usage which were successful. To evaluate further cost reductions they asked Cevo to assist in evaluating:
If using AWS spot instances would reduce the monthly hosting bill
If there were any other, non-monetary expenses generated from running on spot
A experiment was set up to produce a cost analysis that would outline how different AWS instance types could be used to manage the monthly AWS spend more effectively.
Benefits
As a result of this investigation, our client realised there would be no cost saving for them in moving to spot. They were able to evaluate the effectiveness over a short period of time, and make a decision without affecting any of their environments or risking a spike in cost.
Improving team productivity by automating distributed application deployment
Industry
Wagering / Gaming
Customer Overview
Our client, an ASX100 company, is one of the largest publicly listed gaming and wagering organisations globally.
As one of the few integrated gambling and entertainment companies in the world, their iconic brands and stable of subscription television channels ignite passion and excitement in millions of customers each year.
Business Challenge
With millions of dollars worth of investment, our client is releasing a product unique to the Australian wagering market. This regulated software is a distributed application containing 15 services, running on a total of 32 servers with a combination of operating systems. In order to provide high availability to the entire platform, the infrastructure is architected with multiple clusters & multicast interfaces.
The deployment of these services has special requirements as they need to be run on both clustered & non-clustered platforms, while being deployed in a specific sequence & configuration. Prior to Cevo’s engagement the deployment was done manually with more than 10 hours of required effort.
The biggest challenge was deploying these applications across multiple environments like dev/beta, pre-production etc. Not all test environments were equipped with a production-like setup to keep the infrastructure cost low, and this made the deployment process even more challenging to meet the requirements of each environment.
Solution
Working alongside our client, Cevo helped to streamline the deployment strategy for each environment. With the help of tools like Docker, Ansible and Jenkins, a single deployment solution suitable for any environment type was created. This was achieved by using an ‘environment profiling’ technique where the specification of each environment was defined into a separate configuration file. Based on the type of target environment, the right config file is then supplied to the deployment pipeline.
With this strategy, our client eliminated the need for writing environment specific deployment scripts. Also, having a single code-base allowed reproducible deployments across all environments. This has also dropped the deployment window to less than 30 minutes.
While working closely with our client’s development team, Cevo identified other issues which were blocking the developer’s ability to test their work when software testers were engaging any environment. After analysing the situation, a solution was devised allowing each developer to spin-up their own test-stack in AWS on-demand. Leveraging the earlier work, they were also able to configure their test-stack to mimic any environment.
Benefits
By implementing an automated deployment strategy, our client managed to cut down a significant amount of time previously wasted on manual deployments (from 10+ hours to under 30 minutes). It also allowed them to release new patches more frequently on test environments. Since the deployment scripts were written in Ansible, it was easy to understand & maintain the code.
Further, the ability of spinning up dev-stacks on demand gives developers the freedom to continue their development without any interruptions. This improvement in infrastructure & deployment automation has boosted the productivity of each team.
Improving user experience: Technology Review
Customer-focused changes to the digital platform
Industry
Property Management
Customer overview
This start-up is a world-first integrated property managment technology and services platform that transforms the way people in a geographic area connect and communicate. Their goal is to create activated communities, places that make residents and retailers of the area stickier and drive value for asset owners.
Business challenge
Operating as a start up, the digital platform emerged from a series of MVP experiments to identify which services provided value to customers. This also led to architectural challenges around scale, technical debt and reliability. With competitors entering the market, improving the customer experience and speed of the delivery team became a top priority.
Solution
Cevo worked with this client to significantly enhance the events system with a focus on an improved user experience. This process included story writing workshops and coaching the development team in agile methodologies and user acceptance testing.
Benefits
The execution of this value add project surfaced several interesting learnings for the start-up:
The flexibility of their technology stack was improved in terms of what could be achieved from a user experience view as well the amount of time it took to make changes.
Spending the time on polishing features was worth the effort, as the new events system helped seal a contract with a new client.
Our client is the most widely used online collaboration platform in the world for construction, infrastructure, energy and resources projects. They help manage more than US$1 trillion in capital projects across 70 countries, serving over 70,000 user organizations.
With over 850 employees dedicated to helping project teams finish early and under budget they are transforming how project teams work together, increasing accountability and extending control.
Business Challenge
Over the past four years our client has experienced rapid grown through acquisitions and expansions into new territories and markets. As an organisation that has been working in the Software as a Service (SaaS) market for many years, the breadth and complexity of their systems is both wide and deep. They currently service customers around the globe from a number of co-managed hosting centres.
To meet the demands of an increasing market, they required an ability to accelerate the rate at which they could bring on new clients, and the infrastructure required to support them. At the time, using traditional infrastructure provisioning processes, the organisation was able to bring online a new hosting centre but it would take multiple months of effort. The Cloud Engineering team was given the directive to slash the time it takes to provision new infrastructure while continuing to provide a high standard of quality.
Investigations by the in-house Cloud Engineering team identified both Amazon Web Services (AWS) and Azure as great candidates to provide the Infrastructure as a Service to meet these growing demands.
Solution
Cevo was engaged to join the Cloud Engineering team to work through replicating a production scale environment in AWS, ensuring that opportunities were taken to improve the consistency, resilience and repeatability of the process. The decision to start with AWS was based on efficiency, having already established workloads in the environment - however a key requirement for the solution was the ability for it to be applied across the Azure platform as and when required.
Bringing solid experience in Infrastructure as Code patterns, the Cevo team provided guidance on ways to improve the development and management of the infrastructure code base. Working alongside the client team, they were able to develop a set of automated delivery pipelines that enabled staff to focus on identifying the infrastructure requirements and delivering value, safe in the knowledge that if something was misconfigured the automated deployment process would rapidly highlight the issue and enable the team member to correct the error with minimal turnaround time.
Benefits
Through investing in an Infrastructure as Code approach, the Cloud Engineering team is now able to deploy all of the required infrastructure within an hour. Not only does this save a large amount of time, but also increases the consistency of the environments due to automation repeatedly delivering the solution.
By coupling a focus on automated quality validation and continuous delivery practice, the team now have a development process that will allow them to deliver the needed expansion to the organisation while still supporting a growing appetite for innovation.
MYOB is a leading provider of business management solutions in Australia and New Zealand. MYOB helps businesses of all kinds and sizes, delivering software and services that simplify accounting, payroll, client management, websites, and more. With a network of more than 25,000 accountants and other professional partners, MYOB provides the tools and support that help make business life easier.
Business Challenge
MYOB had a contract with its traditional Australian data centre due for renewal in early 2016. This provided the organisations with an opportunity to innovate and seek out infrastructure improvements to provide delivery teams more control of their environments. The vision was to reduce the cycle time of “concepts to production” through gaining configuration traceability, repeatability through automation and self service to reduce dependency on third parties. This vision ultimately tied into one of MYOB’s core business values to help businesses (their customers) be successful.
“We needed a solution that would enable greater flexibility and visibility in terms of infrastructure configuration, automation and performance” says Grant Tibbey, Infrastructure Lead.
In its current state, MYOB Essentials was running in a third-party data center, utilising SQL Server and a Java front end. The company decided to move its infrastructure to AWS.
Solution
From the time MYOB decided it wanted more flexibility and more customer benefits from the Essentials infrastructure (until the existing contract expired), it had only approximately three months to complete the cutover.
MYOB formed a team using existing in house DevOps skills but didn’t have enough capacity to meet the aggressive deadlines. MYOB utilises a wide range of technologies, and Cevo were able to offer skills and availability with minimal onboarding requirements. MYOB was looking for experience without having to bring additional people up to speed on tools and techniques that were being applied to automation.
The existing application had some automation around building and deploying to the application servers using Ansible and Gradle. There was however no automation around building MS SQL Server and no automation around building the network infrastructure. Given the time constraints, existing automation was re-used and augmented with additional tools. Chef was chosen because of its ability to configure Windows and Linux servers securely using Secure WinRM and SSH protocols. It also gave the team the ability to unit and integration test components of the server automation as isolated modules and provided rapid feedback to the development processes using KitchenCI (test-kitchen). Unlike other automation technologies code could be developed and tested with the physical server being stubbed. This reduced server development wait-times. CloudFormation was used for network automation and enhanced with complex logic for alarm calculations using ruby code. CloudFormation templates were generated using a ruby gem “cfndsl” to enhance its static nature.
Options for running MS SQL Server in AWS were considered and included RDS, Log Shipping and Always on Availability Groups to achieve High Availability. These options however were discarded due to cost, extended outage and data loss that would have occurred on failure or reconfiguration. Instead, Mirroring was chosen for HA with Log Shipping initially used to keep the data in AWS in sync with the traditional datacenter to within fifteen minutes so that the cutover would cause minimal impact to customers.
The entire process of building the SQL Cluster in AWS was scripted from scratch. This included formatting, moving, labeling disks, moving system databases and logs, creating users, backup jobs, installing certificates to establishing mirroring using partner and witness amongst many other things that would traditionally have been manually done.
“If something didn’t work, we can tear it down and build it again in hours,” says Grant. “Practice also makes perfect. We practiced rebuilding servers, and cutting over the system constantly. Even weeks before the actual cut over took place”.
By the time cutover took place the entire system could be rebuilt with a couple of commands.
Moving forward, the company plans to re-engineer the application from statically deployed to dynamic auto-scaling servers that come and go, and will look at ways to achieve zero downtime deploys to deal with fault tolerance. Through the ongoing use of Chef scripts and development practices applied to Ops, MYOB’s development team can rewrite the application in such a way that it can easily be deployed to AWS.
Benefits
Benefits of the solution include providing MYOB with:
A safe, repeatable and easy means to release features to their customers as soon as they are capable of providing benefit.
The ability to rebuild servers at will or when they fail.
The ability to reconfigure or change the way the MYOB Essentials app is built; whether it’s to increase performance, reliability or change how they run deployments, infrastructure architecture can be changed and new services added.
Cost optimisation – with the scalability offered by AWS, MYOB is able to have full visibility of what is running and can start reducing expenditure through ensuring they are not over or under provisioned.
Reducing risk and cost of failure using the Potato Cannon pattern
Industry
Wagering / Gaming
Customer Overview
Our client, an ASX100 company, is one of the largest publicly listed gaming and wagering organisations globally.
As one of the few integrated gambling and entertainment companies in the world, their iconic brands and stable of subscription television channels ignite passion and excitement in millions of customers each year.
Business Challenge
Our client was, as is common, very focused on their waterfall approach to the delivery of software. Across multiple delivery teams, the expectation was to build and deliver new features frequently through a primarily manual process. A significant challenge with this software delivery method is that functionality is grouped together and released in large batches with the intention of mitigating the risks of change by reducing its frequency.
In a typical waterfall pipeline the following workflow generally occurs (or a variation of):
Our client’s software build and test process is highly customised for each component; builds are performed on independent physical systems, each configured differently, and copied manually from dev, to test, and then onward towards production. This introduces a high degree of variability in delivery times, adds system library-specific version dependencies on built artifacts, reduces visibility into the current state and history of each artifact, and promotes key-person risks thanks to the specific domain knowledge required to administer each build system.
As a result, they wanted to reduce the variability in the software artifact generation process, in order to achieve more predictable delivery lead times, improve software quality, improve confidence in the released software, and mitigate the risk of specific physical systems and key-person domain knowledge.
Solution
A management decision was made to hire consultants within the delivery teams, bringing in specific skill sets in a co-sourced delivery model. During this exercise, the Cevo technologists on site identified a pattern in the process which lead to an increased risk of failure causing an increased cost of failure.
In the current state, at each step in the process a separate team was expected to test the code, with any issues bringing an increase in the average time to fix as it moved through the chain. This increased cost had heavily influenced the decision to combine and more carefully plan releases as frequent releases became a less commercially viable option.
This is a common occurrence in organisations which is often overlooked. The Cevo team saw this as a valuable opportunity to introduce technology and process changes which would allow our client to release software more frequently with reduced risk and at a significantly reduced cost.
A solution was identified and tested to automate and standardise the build process. This would allow failure to be identified in the first step, reducing the amount of time required to fix it. Once identified and tested, an automated process based on a pattern known as Potato Cannon, would access the changes in the code repository, test them and deploy to the relevant servers for further testing (outlined below).
THE TECHNICAL PART
Our client was already using Jenkins for building software so it therefore made sense to work within the domain knowledge that already existed. As a result, the solution proposed included:
creation of a standardised set of CloudFormation templates to create Jenkins-based build pipelines in AWS (using the Potato Cannon pattern)
use of existing GitHub repositories for source code, with the addition of appropriate plugins to Jenkins to enable branch-based builds, defined in the Jenkinsfile pipeline syntax
use of a centralised artifact repository, to be deployed as a Docker container on an ECS cluster, with RDS for the metadata database and S3 as the persistent object storage.
Each Jenkins-based “Potato Cannon”-shaped pipeline consists of a master node, deployed as a Docker container in an ECS Cluster. It will create slave nodes on-demand in an ECS “Slave” Cluster (which could be a shared resource for multiple Cannons, in order to improve utilisation and reduce cost). The slave nodes will be Docker containers, enabling multiple styles of build from consistent baselines. Potato Cannons are disposable (any data persistence required will be provided by the package artifact repository).
Benefits
The results are significant. The Potato Cannon pattern, which can be applied across any organisation with the same challenges, reduced the previously manual 5-15 week process (developing the code through to deployment to a testing environment) to under a day’s work. The idea behind the process was to shift as much of the test planning and documentation to the start of the process to reduce the cost of failure. It was built in a way which makes all parts of the system replicable to mitigate the risk of redundancy.
The Potato Cannon pattern has been implemented in one area of the business and has proved to be extremely successful in raising efficiency and reducing potential costs of projects allowing he business to implement changes more frequently. It can be reused across all delivery teams to achieve the same result.
This project was also featured on AWS This is My Architecture series:
Reducing time and costs with Infrastructure Automation
Industry
Tertiary Education
Customer Overview
Our client is one of Australia’s largest and most international universities. As a truly global institution, there are five Australian campuses, a campus in Malaysia, a joint graduate school in China, a learning centre in Italy, and a research centre in India. A research-intensive university, they are known for some significant and lasting discoveries that have delivered impact beyond the academic community.
Business Challenge
One of the key benefits of undertaking a cloud transformation adoption strategy is unlocking the potential to move fast and reduce the overheads with respect to existing infrastructure provisioning processes.
In the University’s case, IT staff were spending a significant amount of time manually provisioning services within the Amazon Web Services (AWS) environment. The delays in getting code deployed was impacting the ability for the teams to deliver their value to the business and also increasing the cost of delivery.
Whilst the AWS adoption had already begun for our client, the real benefit was not being realised due to manual activities still being one of the biggest constraints.
Solution
Cevo consultants worked alongside the IT staff to create a semi-automated process to reduce the amount of repetition in building environments. This approach was not about ‘automate at all cost’, but focused on automating the highly repetitive actions, unlocking capacity within the existing teams to focus on higher value activities.
Through automation approaches, the actions required to create an environment became explicitly clear, enabling a whole new level of collaboration between teams to review and identify opportunities for reuse.
Benefits
A significant reduction in time was achieved, allowing the University to set up their servers and deploy code faster. A cost saving was also found through the reduction in duplicating manual processes.
In addition, the patterns identified during this approach are able to be repurposed for additional solutions. This outcome helps the University extend their existing patterns and toolkits into new application design patterns.
Understanding AWS infrastructure costs: a FinOps case study
Industry
Real Estate
Customer Overview
Our customer is a global online real estate advertising company, and operates one of Australia’s leading residential, commercial and share property websites, as well as a number of leading property portals in Asia.
Business Challenge
Within this large and very mature agile agency, multiple business units use and rely heavily on Amazon Web Services (AWS) to support their development. However the finance team had no breakdown of AWS infrastructure costs by department.
Solution
At that time there were no ‘off the shelf’ applications available manage organisation spend on AWS infrastructure. Cevo consultants were engaged to help identify and build a bespoke solution which would allow the data to be pulled through to an easy-to-read dashboard giving a full break down of the AWS costs by department.
AWS allows each resource to contain an individual tag. Setting this up correctly, by department, became a key requirement to ensure the dashboard displayed accurate figures. A consistent naming convention was agreed upon, and the business units were approached to ensure their resources were appropriately tagged (existing and new).
While managing the tagging process Cevo also assisted in developing a tool which reviewed all services in the AWS account, and helped the client teams decommission any that did not respect the tagging requirements. This created a standard procedure moving forward, ensuring all services were measurable and leaving the numbers as accurate as possible.
A management decision was made to forego the initial dashboard so long as the information could be viewed in a meaningful way. The last phase of the project involved creating a spreadsheet which fed in all financial information broken down by department and AWS resource.
Benefits
The solution provided met all the key requirements of the finance team, allowing for the first time a proper allocation of current costs as well as giving visibility over expected future spend.
Teams also have a better understanding of their infrastructure costs which allows for better decisions to be made around required servers and the most appropriate ways to configure code and environments.
Using DevOps to improve deployments
Industry
Education Software
Customer Overview
Factor5™ Software aims to disrupt the way universities approach academic and student administration by modernising curriculum management. They do this through a highly configurable, cloud-based curriculum lifecycle management solution, CourseLoop™ (built on the third-party ServiceNow platform). Purpose built for universities, CourseLoop automates manual processes, improving time to market for new offerings, reducing administration burden on staff and students, and increases quality of curriculum design.
Business Challenge
As a fast-moving start up, Factor5 has CourseLoop under active development, with frequent and continuous deployments. At the time, these deployments were very time consuming and did not always guarantee reliability. With Factor5 scaling to commence development on two more product offerings, the overhead strain was only going to become more exacerbated. Foreseeing this, the company wanted to implement a better and more sustainable way to perform and track deployments.
Solution
The team kicked off with a series of investigative workshops and quickly realised that the non-conventional nature of the development workflow running on top of the ServiceNow platform was going to require some creativity and a change in team practices. A hybrid solution which implemented bespoke and ServiceNow native elements was proposed. This paved the way for the team to implement a more automated deployment method, packaging the changes into a singular item which could also be extracted from the ServiceNow platform into a version control system.
Benefits
The Factor5 team is now able to conduct deployments of CourseLoop much quicker than before. Importantly, they have the confidence that deployments will occur without issue and no longer depend on human intervention. These improvements in the way CourseLoop is built and deployed can also be applied to future product offerings where applicable, helping maintain efficiencies and manage team workflow.
Watchmen: Collaborating with Security and APRA to migrate applications to AWS
Industry
Insurance services
Customer Overview
IAG sell insurance under many leading brands including NRMA Insurance, CGU, SGIO, SGIC, Swann Insurance and WFI (Australia); NZI, State, AMI and Lumley Insurance (New Zealand); Safety and NZI (Thailand); AAA Assurance (Vietnam); and Asuransi Parolamas (Indonesia).
Business Challenge
IAG had approximately 100 applications sitting on physical servers with a strategic direction to migrate to Amazon Web Services (AWS). Before this could begin, they needed to ensure that all environments met APRA regulations. Of the tools available at the time, none were suitable to fully achieve these objectives.
Solution
IAG and Cevo worked with APRA to clarify and document a list of compliance requirements. The team then wrote a tool called ‘Watchmen’ that was governed by 40 unique security policies that would scan each environment (called ‘Citizens’) and report on compliance with these policies. Where an environment did not conform to the security policies, the tool would report this exception and where the exception occurred.
Features of Watchmen:
Isolated rule logic
The logic used for determining compliance to each rule sits in an isolated account that non of the monitored (Citizen) accounts can access. This ensures that there is no tampering with the rule logic, and a result of ‘Compliant’ means what we think it means.
Serverless
Watchmen runs on AWS serverless infrastructure. All of the rule logic exists in Lambda functions, while the compliance scans themselves are managed by AWS config. This means the ongoing maintenance and support required for Watchmen is very low.
Auto updating
When new rules are added to Watchmen, the Citizens are able to be automatically updated though the use of SNS topics and Lambda. So once the Citizen CloudFormation stack has been installed on a Citizen account, the account owner never needs to touch it again.
Easy to extend
As the rule logic sits in Lambda functions, you are able to write new rules in any Lambda supported function. The deployment pipeline created for Watchmen will pick up new rules and incorporate them into the deployment.
Benefits
Prior to this project, the migration had been delayed because of the time it would take for the security team to manually check the compliance of any given AWS account. Watchmen has allowed IAG to confidently approve the migration of applications to pre-authorised environments. Now IAG can move forward with their AWS migration knowing that Watchmen automatically checks the environments daily ensuring continual compliance with security procedures and APRA.
Both IAG and Cevo hope to open-source Watchmen over the coming weeks and are working towards that end.
Woodstock: AWS Log Aggregation
Industry
Insurance services
Customer Overview
Our client is the parent company of a general insurance group with controlled operations in Australia, New Zealand, Thailand, Vietnam and Indonesia.
Business Challenge
As part of our client’s intrusion detection system (IDS), one commonly adopted security pattern is the forwarding of logs of various types into the company’s Security Information and Event Management (SIEM) system. Of particular interest to the SIEM team are the Amazon Web Services (AWS) CloudTrail logs, as they provide information about all AWS API calls that are run on an account, and are a good place to start when trying to detect suspicious activity.
There were several challenges that the client faced when getting CloudTrail logs into SIEM:
The AWS maturity of teams across the organisation varies greatly. Not all teams were aware of the CloudTrail service and what it does, and this created variation in compliance.
Each integration into SIEM required several non-trivial steps. Following this, the integration needed to be monitored for a few days to make sure it was working correctly.
Identifying which teams still needed to complete the integration, and providing the necessary instruction on how to do this, was proving an additional management burden.
Solution
Working together with Cevo, a solution was devised to address these challenges in the simplest way possible. When broken down, the main goal was to design a process that:
Is easy for all AWS account owners to use
Only requires one integration with SIEM.
The solution created was entitled ‘Woodstock’ (ie a large pile of logs in one place!) Woodstock stores the CloudTrail logs of each AWS account. Each account writes to a specific directory that only it has access to, and SIEM can then read all of the logs from the one place.
For AWS account owners, a simple CloudFormation template we named Lumberjack was created (because, Lumberjacks send the logs). The template only takes five minutes to download and run and conducts the necessary CloudTrail configuration.
Leveraging other work done previously with our client, a Watchmen rule was added to check that accounts had their CloudTrail configured correctly.
Benefits
Simplifying the process of forwarding CloudTrail logs into SIEM means that teams are much more likely to undertake the process, thereby increasing compliance and visibility whilst reducing administrative burden on management.
The Woodstock solution also adds another layer of security monitoring to all AWS workloads with no room for human error, as all the configuration for required for an account is executed as code.
As a result of these benefits, the solution has now also been extended to cover logs from other AWS services within the organisation.
