Wagering / Gaming
Our client, an ASX100 company, is one of the largest publicly listed gaming and wagering organisations globally.
As one of the few integrated gambling and entertainment companies in the world, their iconic brands and stable of subscription television channels ignite passion and excitement in millions of customers each year.
Our client was, as is common, very focused on their waterfall approach to the delivery of software. Across multiple delivery teams, the expectation was to build and deliver new features frequently through a primarily manual process. A significant challenge with this software delivery method is that functionality is grouped together and released in large batches with the intention of mitigating the risks of change by reducing its frequency. In a typical waterfall pipeline the following workflow generally occurs (or a variation of):
Our client’s software build and test process is highly customised for each component; builds are performed on independent physical systems, each configured differently, and copied manually from dev, to test, and then onward towards production. This introduces a high degree of variability in delivery times, adds system library-specific version dependencies on built artifacts, reduces visibility into the current state and history of each artifact, and promotes key-person risks thanks to the specific domain knowledge required to administer each build system.
As a result, they wanted to reduce the variability in the software artifact generation process, in order to achieve more predictable delivery lead times, improve software quality, improve confidence in the released software, and mitigate the risk of specific physical systems and key-person domain knowledge.
A management decision was made to hire consultants within the delivery teams, bringing in specific skill sets in a co-sourced delivery model. During this exercise, the Cevo technologists on site identified a pattern in the process which lead to an increased risk of failure causing an increased cost of failure.
In the current state, at each step in the process a separate team was expected to test the code, with any issues bringing an increase in the average time to fix as it moved through the chain. This increased cost had heavily influenced the decision to combine and more carefully plan releases as frequent releases became a less commercially viable option. This is a common occurrence in organisations which is often overlooked. The Cevo team saw this as a valuable opportunity to introduce technology and process changes which would allow our client to release software more frequently with reduced risk and at a significantly reduced cost.
A solution was identified and tested to automate and standardise the build process. This would allow failure to be identified in the first step, reducing the amount of time required to fix it. Once identified and tested, an automated process based on a pattern known as Potato Cannon, would access the changes in the code repository, test them and deploy to the relevant servers for further testing (outlined below).
THE TECHNICAL PART
Our client was already using Jenkins for building software so it therefore made sense to work within the domain knowledge that already existed. As a result, the solution proposed included:
creation of a standardised set of CloudFormation templates to create Jenkins-based build pipelines in AWS (using the Potato Cannon pattern)
use of existing GitHub repositories for source code, with the addition of appropriate plugins to Jenkins to enable branch-based builds, defined in the Jenkinsfile pipeline syntax
use of a centralised artifact repository, to be deployed as a Docker container on an ECS cluster, with RDS for the metadata database and S3 as the persistent object storage.
Each Jenkins-based “Potato Cannon”-shaped pipeline consists of a master node, deployed as a Docker container in an ECS Cluster. It will create slave nodes on-demand in an ECS “Slave” Cluster (which could be a shared resource for multiple Cannons, in order to improve utilisation and reduce cost). The slave nodes will be Docker containers, enabling multiple styles of build from consistent baselines. Potato Cannons are disposable (any data persistence required will be provided by the package artifact repository).
The results are significant. The Potato Cannon pattern, which can be applied across any organisation with the same challenges, reduced the previously manual 5-15 week process (developing the code through to deployment to a testing environment) to under a day’s work. The idea behind the process was to shift as much of the test planning and documentation to the start of the process to reduce the cost of failure. It was built in a way which makes all parts of the system replicable to mitigate the risk of redundancy.
The Potato Cannon pattern has been implemented in one area of the business and has proved to be extremely successful in raising efficiency and reducing potential costs of projects allowing he business to implement changes more frequently. It can be reused across all delivery teams to achieve the same result.
This project was also featured on AWS This is My Architecture series: