Improving ways of working with automated development and delivery cycles
Our customer is one of Australia’s largest supermarket, retail and consumer services chains.
This particular project was for a business-critical function within the organisation responsible for managing transactions from retailers and routing these to banks and card companies.
In recent years the team has experienced an increased demand for change, driven by the desire to improve internal change and release processes.
These processes supported development and delivery of a third-party application, but were outdated and highly manual, resulting in long delivery cycles. The steps involved were all managed by key employees with numerous online forms providing gates and checkpoints. This reduced the velocity of implementing changes, and prevented the ability of the team to own the changes they were making. Additionally, because of the highly manual nature of the release and change (R&C) process, no single source of truth existed to identify what was contained in long-lived environments.
Due to the level of governance required to access software and make changes, there was a high dependency on the central R&C team by the project teams. This caused a bottleneck, typically when the project needed, or wanted, to work quickly. Central governance functions took hours or even days to complete, mainly due to time spent waiting for relatively simple functions to be performed.
The technology supporting these changes was also limited; the software was not maintained in source control repositories and artifacts were not appropriately managed. To obtain a view of what versions and changes were pending, or currently installed, in the various environments required manual aggregation of the data held in centralised online forms. Thus there was a critical dependency on team members filling these forms out accurately.
All these process and technology challenges had resulted in teams who had a limited ability to take ownership of the changes they made, a high level of dependencies on other teams and a lengthy governance processes, and an overall risk-averse culture.
Cevo was engaged to work with the central technology teams to select the best Continuous Integration (CI) toolset, build a CI pipeline to reduce risk from manual processes involved in releases, and roll this out to the R&C team.
Cevo worked within the key R&C team and central DevOps/operations team to help establish the toolchain. From this they were able to migrate existing scripts/artifacts to these tools and build the initial CI Pipeline. The pipeline included functionality for creating deployment packages, with automated governance checks based on the team’s requirements.
A large part of this project was a cultural shift towards a more modern approach to developing and releasing changes. Moving from manual processes to using source code management and automatic packaging was a significant difference in how the team would perform day-to-day work.
The new process for developing and releasing changes was trialled initially with a few select team members. Cevo assisted with training, as well as creating transparency and visibility to the rest of the team. Once the initial team members had been onboarded, the process was rolled out across other teams and projects, allowing for adjustments and fast feedback into the pipeline and the processes.
Once a greater portion of team members began using the tools and seeing the benefits (including reduced bottlenecks, more power to own their changes, reduced manual processing, and greater access to the source code), they were advocates of the solution, helping to bring across some in the broader team who were initially sceptical and slower to adopt new ways of working.
An additional part of the cultural change was to impart on the team was how useful automation can be for not only simple, repeatable tasks, but also more complex tasks, and those particularly prone to human error. With this change in mindset the team implemented a number of scripts to help with automation of tasks identified as-needed.
By adopting the CI toolset and changing their processes the following benefits were recognised:
The governance role played by R&C changed significantly, completely removing the largest bottleneck for project teams. The R&C team were able to shift their focus to more value-adding work over manual management of the scripts and vendor patches.
More accountability and responsibility shifted to the project teams by being able to access the source control and artifact storage repositories, and trigger package builds themselves. This empowers the teams to own the changes they are making.
The foundation for future changes was established in the Continuous Integration space, including automated testing and on-demand immutable environments, which would have been virtually impossible before implementation of the new toolset and processes.