Waiting on our Cycle time
- Firstly, I can’t test any of the code without a deployment, even if it’s a single function that doesn’t touch any hosted services.
- The deployment pipeline is slow. I’m deploying a tiny change to a small lambda, and it shouldn’t take ten minutes to do that.
- I’m running the entire data set through the deployed change, only to throw away 99.99% of the data. Clearly, that’s inefficient.
- Finally, I’m testing things manually. It might be “only” ten minutes to look at the data, but there’s already 60 odd commits in this repository, which means that I’ve probably spent a good ten hours testing this manually, not to mention waiting around for fifteen minutes every time a deployment went out. A good set of automated tests with a much smaller dataset could have cut this to a few minutes, or even seconds.
Unit Testing the important parts
Cutting Deployment times
The deployment pipeline is a bit more tricky, as there’s a tremendous variety of reasons for slow pipelines. In this case, it turned out that there were a couple of docker images being built in the same repository. These are being built from scratch for every run, but 95% of the custom pieces are shared with every other docker image deployed by our organisation. The common docker image base was split off into its own repository, and we can now build just the docker layer we need. This change cuts the deployment from ten to three minutes.
This kind of problem is common in CI/CD pipelines. If you’re not sure where to start, then Cevo is your go to cloud partner for solving DevOps issues.
Removing Manual Testing
Finally, there’s the manual testing. We can cut down the ten minute wait by setting up each branch to deploy it’s own cloudformation stack, and running tests against a much smaller dataset. This reduces the run time to less than a minute.
The unit tests earlier mean that we no longer need to test every possible case, just the entry and exit handling of the lambda, so we’re down to the happy data processing path, and error path – so that’s just two manual test cases, which is about two minutes all up.
My thirty five minute cycle of…
- Ten minutes to build and deploy.
- Fifteen minutes to run.
- Ten minutes to test (manually).
… is now a total of six minutes:
- Three minutes to build and deploy.
- One minute to run.
- Two minutes to test (manually).
In this process, I’ve radically reduced the time to get feedback on my changes in two different ways. One was by making the process more efficient, and the other was by removing the need to go through the full cycle for every change. These two strategies are the heart of fast feedback.
If your team or organisation is suffering under the weight of a slow process and needs fast feedback, give Cevo a call. We’re your DevOps and Cloud Partners, and fast feedback is our jam.