CFN-LINT AND BUILDKITE BASICS
cfn-lint is an open source linting tool which can be used for testing CloudFormation templates before they are deployed to AWS. Linting is a process of analysing the source code to identify potential errors & bugs. It has the potential to save a lot of time and money by avoiding creating a CloudFormation stack in AWS to then realise it has failed due to errors that could have been corrected beforehand.
For an introduction to cfn-lint please refer to this article by my colleague https://cevo.com.au/post/using-cfn-lint-to-validate-iam-resources-in-aws/.
Buildkite is a CI/CD tool which is used for automating manual jobs. Refer to the official Buildkite-Documentation for setup and other details.
WHAT ARE WE TRYING TO SOLVE?
Creating a CloudFormation (CFN) stack may take anywhere from one minute to more than 30 minutes based on the number and type of resources that are being created.
The challenge for one of our clients was to identify a tool for testing/scanning a CFN template before it was deployed to AWS. This would allow them to realise and correct any errors and issues with the CFN template, instead of waiting for the stack to fail after creation. As mentioned above, avoiding the deployment of faulty stacks on to the AWS console will save a lot of time, effort and cost.
WHAT DID WE DO?
We took this requirement to the next level by integrating cfn-lint with the Buildkite pipeline. This means every time a Pull Request is submitted to the master branch, the Buildkite pipeline automatically runs and scans all the CFN yaml files in the github repository with cfn-lint. If there are no issues with the CFN, the pipeline completes successfully and displays a success message in Github on the ‘Approve Pull Request’ page.
If cfn-lint check fails, it will also be reported at the Approve Pull Request stage.
The person assigned to the Pull Request can either choose to approve or reject the request based on the Buildkite pipeline output. cfn-lint also gives the line number of the error and suggestions on how to resolve it.
HOW DID WE ACHIEVE THIS?
Prerequisites: Basic understanding of Linux, Git, Buildkite, Shell scripting and Docker
1 Create a new Github repository or create a folder structure as below in your existing Github repository.
2 Create a file pipeline.yml inside .buildkite folder with the below code:
plugins:
- seek-oss/github-merged-pr#v1.0.1:
mode: trigger
steps:
- label: "container"
command: "container.sh"
wait
- label: "tests"
command: "test.sh"
When a Buildkite pipeline is run, it first checks to see if there is a .buildkite folder in the root of the branch and then looks for pipeline.yml inside the folder. This yaml file defines the steps of a pipeline as code.
seek-oss/github-merged-pr is a plugin that helps in triggering the buildkite pipeline as soon as the Pull Request is submitted from any child branch to the master branch. More info about the plugin can be found at https://github.com/seek-oss/github-merged-pr-buildkite-plugin
We will be installing cfn-lint in a docker container in the container.sh and testing the CFN templates in the script test.sh
3 Create Dockerfile at the root of the branch with the below code:
FROM amazonlinux
WORKDIR /home/ec2-user
RUN yum install python-pip -y
RUN pip install cfn-lint
COPY cloudformation cloudformation/.
As a part of this dockerfile we are using a base amazonlinux image and installing cfn-lint on top of it. Cloudformation is the folder at the root of the github branch which contains all the CFN templates (in yaml format) that need to be scanned.
4 Create a shell script container.sh at the root of the branch with the below code
#!/bin/bash
sudo yum update -y
sudo yum install docker -y
sudo usermod -a -G docker ec2-user
sudo service docker start
docker build -f dockerfile -t testcfn .
This script installs docker and builds a docker container using dockerfile where we will be testing our CFN templates
5 Create a shell script test.sh at the root of the branch with the below code
#!/bin/bash
rm -f Output.txt
find ./cloudformation -type f > CFN_list.txt
cat CFN_list.txt | while read Filename; do
echo "The file $Filename is being tested " >> Output.txt
sudo docker run -t testcfn cfn-lint -iW -iE3012 $Filename
done
This script checks for all the files present under the CloudFormation directory and scans them one by one with cfn-lint inside the docker container. The option W is used for ignoring warnings and E for ignoring specific error numbers of cfn-lint. Specific error numbers can be ignored in case the errors are already known and need to be ignored for testing purposes.
6 Once all the scripts are ready, create a Buildkite pipeline as per the Buildkite documentation.
In the settings section, configure it to use the pipeline.yml file. Specify the Github repository URL where we have our scripts and CFN templates in the Github section. Ensure the below options are selected for the Pull Request plugin to work correctly.
Once all the above configurations are made, raise a Pull Request from any child branch to the master branch and that should trigger the Buildkite Pipeline which will scan all the CloudFormation templates
Example of a failed build:
SUMMARY
We’ve seen that validating CloudFormation templates with cfn-lint can be very useful in terms of saving time, money and effort. Integrating with Buildkite adds an extra edge by automating all the manual tasks so that the CFN templates can be tested before they are deployed to AWS.
Contact Cevo if you need any help with Cloud or DevOps solutions.
