Using Kitchen and InSpec enables you to develop infrastructure faster, more reliably and with greater confidence.
As the cycle times for providing new servers and services decrease, the demands on infrastructure developers is increasing. To cope with this increased demand, we need new ways to think about building and validating servers.
A powerful combination is to couple Kitchen with InSpec, to provide a modern development approach to both the provisioning of servers and validating their configuration is correct.
For a long time infrastructure was the sort of thing you pulled out of a box, plugged in and then set about configuring and testing. The cycle between needing new equipment and having it ready was measured in weeks, if not months.
Modern cloud computing and infrastructure as code approaches are becoming the norm, with this cycle times on delivering new infrastructure is now measured in seconds. We can no longer afford lengthy validation and compliance testing cycles. We need a rapid, repeatable and scalable approach to testing the new infrastructure configuration.
While we can launch new hardware rapidly, there is still a lot of work to be done to complete the installation and configuration of server software and applications. The great news is that there are a number of different tools already in wide adoption to make this task much easier.
We have seen an evolution of organisations through three phases when adopting an infrastructure as code approach.
Phase 1 – pre-automation – This is the phase most, if not all, businesses start in. Even those starting cloud native will most probably launch hardware, and then login to the machine and manually configure the software. You can actually get pretty far in this phase, you can still even use a stateless infrastructure approach by manually taking server images and scaling out your solution.
But this soon becomes tedious, error prone and a constraint on innovation of your solution. You’ll bite the bullet (hopefully sooner rather than later) and invest in adopting a server provisioning tool.
Phase 2 – server automation – Once you have a non-trivial number of servers, manually creating, patching and operating the systems becomes a complex and expensive task, by this stage you will be looking for a server automation tool – Chef, Ansible, Puppet, SaltStack, etc.
By the time you’ve automated your first half dozen server images, you’ll be looking for a way to ensure that your servers not only work the way you want them, but that they stay working as intended.
While it might seem obvious, we now need a way to test what has been developed.
Phase 3 – validated server configuration – It’s at this phase that you start to accelerate the confidence of your system. Not only now do you have a solid approach to provisioning servers, but you have extended your automated quality process to cover your infrastructure provisioning process.
BUT HOW DO WE DO IT?
It’s all good to say that we want to create repeatable, reliable and consistent server images, but how do we do it?
Enter Kitchen as a test harness to validate the results of your infrastructure code on one or more platforms in isolation.
By introducing Kitchen to your tool chain you get benefits in two ways:
- ensure the validity of any server image and
- speed up the development of the infrastructure code – this will be expanded in a future article
Kitchen itself does not actually perform any tests, it is the test harness which ensures the tests have access to the server, are able to be run and reports back on the status of the test cases.
A powerful combination is to couple Kitchen with InSpec, while both of these tools find their genesis from the Chef provisioning tool chain, there is nothing restricting the use alongside other tools.
One awesome feature of InSpec is the ability to create re-usable test suites (called Profiles) which allow you to adopt the DRY principal by re-using existing test profiles across your different server images.
A great starting point for your InSpec Profiles is to look at the existing dev-sec profiles available on github. The dev-sec team has put together a number of generic profiles that looks for common server hardening and patching issues.
A WORKING KITCHEN AND INSPEC EXAMPLE
STEPS
- Create a
Gemfile(content below) - Execute
bundle install - Create the
.kitchen.ymlfile (content below) - Install the dependencies with
bundle install - Setup your environment
KITCHEN_SSH_KEY_NAMEshould point to a valid EC2 SSH keyKITCHEN_SSH_KEY_PEMshould be the path to your local SSH private key for the above- (optional)
SERVER_IMAGE_IDto specify the EC2 AMI to test- Otherwise will launch a Red Hat Enterprise Linux 7.4 server
- Run a full test with
bundle exec kitchen test
source "https://rubygems.org" gem "rake" gem "serverspec" gem "test-kitchen" gem "kitchen-ec2" gem "kitchen-inspec".kitchen.yml
---
driver:
name: ec2
aws_ssh_key_id: <%= ENV['KITCHEN_SSH_KEY_NAME'] || "development" %>
region: ap-southeast-2
instance_type: m3.medium
associate_public_ip: true
transport:
ssh_key: <%= ENV['KITCHEN_SSH_KEY_PEM'] || "~/.ssh/development.pem" %>
connection_timeout: 10
connection_retries: 5
username: ec2-user
# Busser runs the tests, and assumes that the ChefDK is installed by default.
# Because we are not using Chef, we need to tell it to use the system Ruby.
# (see: https://github.com/test-kitchen/test-kitchen/issues/347)
busser:
ruby_bindir: /usr/bin
provisioner:
name: shell
platforms:
- name: rhel-7
driver:
image_id: <%= ENV['SERVER_IMAGE_ID'] || "ami-c92b3eaa" %>
suites:
- name: default
verifier:
inspec_tests:
- name: dev-sec/linux-baseline
- name: dev-sec/ssh-baseline
After running bundle exec kitchen test you should see kitchen launch your image and run the tests. If all of the tests pass the program will exit with success, otherwise you might see something like
Profile Summary: 32 successful, 88 failures, 1 skipped Test Summary: 112 successful, 110 failures, 1 skipped >>>>>> ------Exception------- >>>>>> Class: Kitchen::ActionFailed >>>>>> Message: 1 actions failed. >>>>>> Verify failed on instance <default-rhel-7>. Please see .kitchen/logs/default-rhel-7.log for more details >>>>>> ---------------------- >>>>>> Please see .kitchen/logs/kitchen.log for more details >>>>>> Also try running `kitchen diagnose --all` for configurationWhat this is telling you is that your image has failed tests, and you can now work to resolve them.
KITCHEN WORKFLOW
When you run the test mode of Kitchen, it actually runs multiple steps.
- create
- converge
- setup
- verify
- destroy
These multiple steps help you with the development phase of your server provisioning and validation code.
A simple workflow is to run:
bundle exec kitchen setup– to create your EC2 instances and have it ready to run your tests- the machine will continue to run, allowing you to develop your tests
- when ready you can run
bundle exec kitchen verifyto execute the tests against the running server - you can continue this loop as often as you need before finally
- executing
bundle exec kitchen destroywill shutdown and cleanup any EC2 resources you were using.
This article is focused on validating server images, but the approach works just as well to help you develop your infrastructure provisioning code, be it puppet, chef, ansible, etc.
USING IN A CONTINUOUS INTEGRATION PROCESS
While it’s great that tools like Kitchen make development and manual validation tasks easier to do, you don’t get the full benefit of these tools until they are integrated with your Continuous Integration server.
TEST VISIBILITY
Another benefit of integrating with CI for your infrastructure testing allows a central place people can review to see what tests are passing or failing. Again, Kitchen in it’s default configuration will output the status of the tests to the console, meaning users need to scroll through console logs to find the test status.
Again, you can configure the verifiers to output the test results in JUnit XML format, and then process these results into your CI server.
verifier:
name: inspec
format: junit
output: ./results/%{platform}_%{suite}_inspec.xml
TESTING A FRESHLY BUILT IMAGE
Now that you have the ability to execute as set of InSpec tests on a stock AWS image, you need to update your configuration so that it tests your candidate AMI’s.
Kitchen supports injection of environment variables into the yaml files, this allows you to create a common kitchen file, but alter the AMI that is under test.
By adding a platform as follows, you can export the AMI id to a SERVER_IMAGE_ID environment variable, and execute your kitchen test command without the need to alter the test scripts.
platforms:
- name: custom-image
driver:
image_id: <%= ENV['SERVER_IMAGE_ID'] || "ami-abcd1234" %>
DON’T FORGET TO CLEAN UP
Kitchen in its default configuration will leave servers running when tests fail. This may be a useful feature for you when you are running the tool yourself, as you have a running server that you can poke around on to see what happened. But this is a really bad pattern when run by CI. Every time a test failure occurred from your CI server, you will have test servers sitting around, costing money with no process to clean them up.
Kitchen supports altering from the default behaviours by appending --destroy=always to your test run. In this mode Kitchen will destroy all instances it creates on every test run.
CONCLUSION
As demands the on teams to deliver increase, and systems grow in size and complexity, ensuring we have a way to scale out our approach to quality and validation is critical.
Through introducing Kitchen and InSpec to your toolkit you will be well positioned to have confidence that the systems you build work to specification, and continue to align with these requirements as time goes by.
A new post discuses extending this approach to replace packer entirely
