Python as a Cloud Glue Language

One promise of the cloud is that you have less moving parts to manage. No longer do you need to worry about drives or servers, or even networks. Just package up your application and deploy it to the cloud.

For those of you working in the cloud, you’ll know it’s not that simple. While a lot of heavy lifting has been done for you, nothing is quite as easy as it seems.

When using a cloud provider like AWS, you get access to a large range of compute and storage services that you can combine to meet your unique business requirements. Think of these services as the large building blocks that your infrastructure and applications are built upon.

But blocks alone are not enough to build a stable foundation; while these services hide lots of implementation details, they regularly need external support from you and your teams to glue them together.

One option is to glue the components together by hand. For example, you can use the AWS console to take the output of one element and pass it into another.

The more reliable and dependable solution however is to automate this connectivity with a little bit of scripting.

WHY PYTHON?

Python is a perfect language to implement these glue scripts as it is:

• Cross platform – the same scripts can work on Windows, MacOS and Linux
• Easy to pick up and use for all levels of programming experience
• Well supported by the AWS community – lots of examples and prior art
• Has a number of existing libraries to make your job easier – boto, requests etc
• A fully featured programming language, with all the power and capability that brings

Taking the time to learn python and adopting it as your utility glue language will pay great returns in the long run.

You might be inclined to pick up a simple scripting language like bash, and while this works well for simple scripts, once your glue gets more complicated your bash scripts start to become a couple jungle of jq, sed, awk and curl commands.

Python has native support for structured collections like arrays, lists and dictionaries. Through the use of modules such as boto, json and requests, you can build complex conditional scripts with a few lines that are not only easy to write, but still easy to understand in 6 months time when you have to come back and change them.

The pattern for most glue utilities falls into the following pattern:

1. call an AWS API with a set of parameters
2. inspect the response information to extract some value or make a decision
3. call another AWS API to update some state or initiate a process

Let’s look at some examples of sample glue utilities.

REPLICATE AN AMI BETWEEN REGIONS

In this example we need to copy AMIs between regions, while ensuring that we have traceability between both the source and destination image.

import boto3
import sys


def lookup_ami_tag(image, key):

    if image.tags is not None:
        for tag in image.tags:
            if key == tag["Key"]:
                return tag["Value"]

    return None


ami_name = sys.argv[1]
dest_region = sys.argv[2]
src_region = "ap-southeast-2"
dest_region_tag = dest_region+".origin.id"

ec2_resource = boto3.resource('ec2', region_name=src_region)
ec2_destination_client = boto3.client('ec2', region_name=dest_region)

ami_id = amis[ami_name]
image = ec2_resource.Image(ami_id)

dump_ami_details(image)

origin_ami_id = lookup_ami_tag(image, dest_region_tag)
if origin_ami_id is None:
    new_name = image.name+"-from-"+ami_id

    result = ec2_destination_client.copy_image(Name=new_name,
                                               SourceRegion=src_region,
                                               SourceImageId=ami_id)

    tag = image.create_tags(
        Tags=[ { 'Key': dest_region_tag, 'Value': result['ImageId'] } ]
    )
else:
    print("AMI has already been shared as '{}'".format(origin_ami_id))

SHOW DETAILS ON DEPLOYED AMI

Here we are wiring together some CloudFormation calls to lookup the AMI ids that are deployed and exporting the tag details so we can see information about the assets.

We first look up the specified CloudFormation stack, then extract the parameters for the DB, APP and WEB AMIs. Then using EC2 AMI calls, we look up details on the AMIs and their tags. This activity is possible to achieve through the console, but takes a lots of clicks and is error prone.

With a script like this running as part of a Jenkins deployment, you’ll get traceabilty on what is currently deployed, what has just been deployed, or even both.

import boto3
import sys

AMI_PARAMETER_VALUES = { "DB_AMI", "APP_AMI", "WEB_AMI" }

cfn_client = boto3.client('cloudformation')
ec2 = boto3.resource('ec2')


def lookup_ami_details(ami_id):
    image = ec2.Image(ami_id)
    return image


def dump_ami_tags(image):
    for tag in image.tags:
        print("    > {} - {}".format(tag["Key"], tag["Value"]))

    return image


stacks = cfn_client.describe_stacks(
    StackName=sys.argv[1]
)


for stack in stacks["Stacks"]:

    for param in stack["Parameters"]:
        param_key = param["ParameterKey"]
        param_value = param["ParameterValue"]

        if param_key in AMI_PARAMETER_VALUES:

            ami_obj = lookup_ami_details(param_value)
            print("[{}] - {} ({}) {}".format(param_key, ami_obj.name, ami_obj.image_id, ami_obj.creation_date))
            dump_ami_tags(ami_obj)

SUMMARY

You can see from these small examples that they are very specific to an individual platform need; the subtlety of the names of the tags and the structure of the content requires these tools to be bespoke. And you can also see that these outcomes can be achieved in minimal Python code.

So what are you waiting for? Take a look at your current cloud platform and you’ll see a number of opportunities to glue things better together with Python.