Imagine for a second that your company has compute resources and workloads on Google’s Cloud Platform (GCP), and that you also need to manage resources within an AWS account. This multi-cloud scenario is becoming more common nowadays, as organisations experiment with different service offerings – but it does present some challenges.
Not least of those challenges is how to manage credentials when systems in one cloud provider need to manage resources in the other. The easy way isn’t always the best.
SETTING THE SCENE
For the sake of clarity, let’s say that you already have an AWS account, and a GCP account. Your company has decided to deploy CI/CD systems in the GCP account, but wants to use those pipelines to deploy and manage resources in the AWS account (this isn’t an entirely fabricated example, by the way, it’s based on a true story).
In this example, the job running inside a GCP VM needs to interact with an SQS Queue to publish a message in an AWS account; an equivalent command-line might be:
aws sqs send-message --queue-url https://sqs.ap-southeast-2.amazonaws.com/123456789012/example-queue --message-body '{"message":"hello, world"}'
Pretty simple, and not uncommon. In order for this command to succeed, the CI/CD pipeline user (the identity that the build agent runs as) must have AWS credentials available to it, and those credentials must be able to make the relevant API calls.
THE SIMPLEST (AND MOST DANGEROUS) WAY: IAM USERS
Creating an IAM User in your AWS account allows you to generate an Access Key and corresponding Secret Key, export them, and inject them somehow into your build pipeline; whether via the capabilities of the CI/CD tool (eg storing them as a “secret”), putting them on the filesystem of the build agents, injecting them as environment variables, or something else.
The risk here is that these credentials never expire, they have to be transferred somehow from the AWS environment to the GCP environment, and in most cases, people want them to be stored somewhere so that they can be used to re-create the CI/CD environment later if required.
The number 1 route for AWS account compromise (and subsequent exploitation for, say, mining cryptocurrencies) is via IAM User credentials; unwittingly publishing them to a GitHub repository, deliberately putting them into a shared documentation page, storing them on a USB stick and accidentally leaving it in a taxi … the list is endless.
There are multiple clever solutions for storing IAM User credentials securely (I should know, I wrote one of them: http://credulous.io ), but all of them work around the point: if you never create these credentials, you never have to manage them.
THE MOST SECURE WAY: IAM ROLES AND IDENTITY PROVIDERS
By this point, someone reading this blog post will have piped up: “just use IAM Roles!” and they’d be right. IAM Roles offer the same capability to perform AWS actions as IAM Users, plus the credentials are only valid for a limited period of time (between 15 minutes and 12 hours, depending on the settings on the role itself). This nicely avoids much of the risk associated with long-lived credentials leaking out into the wild.
Of course, the challenge is: how do you get these time-limited credentials into a VM running inside GCP? You certainly don’t want to have to copy them in manually each time.
Let’s bring AWS IAM Identity Providers to the rescue. IAM supports creating a trusted identity provider resource, based on some external attributes, which will then be trusted by the IAM Role and can be relied upon to generate time-limited credentials via the Security Token Service.
That suddenly went fast, didn’t it? It’s ok, I’ll break it down.
IDENTITY PROVIDERS
In IAM-speak, an Identity Provider (IdP) is some external service which can be trusted to authenticate an entity. IAM takes some kind of cryptographically-signed assertion from that trusted provider, and generates time-limited credentials which are then issued to the entity. Each time the entity needs new credentials, it goes via the IdP, gets a fresh assertion, presents it, and gets fresh credentials.
IAM trusts two types of external IdP natively: ones which use a protocol called Security Assertion Markup Language (SAML), and ones which use OpenId Connect (OIDC).
GOOGLE SERVICE ACCOUNTS
When you create a VM inside GCP, you can associate a Service Account with that VM. The VM can then use automatically-generated, time-limited credentials of the service account to make GCP API calls. You can think of this as being pretty much identical to the way that IAM Roles and IAM Instance Profiles work inside AWS with EC2 instances.
It turns out that the Service Account credentials are OIDC Tokens, which can be presented to an OIDC IdP inside AWS via the sts:AssumeRoleWithWebIdentity
API call to return … IAM Role credentials!
LOVELY! HOW?
CREATE SERVICE ACCOUNT IN GCP
Before you create the VM in GCP, you should create the Service Account that’ll be used. You could just use the default Service Account that GCP creates for all Compute VMs, but then any VM in your fleet could acquire IAM credentials at any time, and you probably don’t want that.
- In the GCP console, go to the IAM & Admin menu, then choose Service Accounts.
- Click “Create Service Account”
- Fill in the details of the service account name and its description and click Create
- In the Permissions screen, add the “Service Account Token Creator” Role and click Continue
- You do not need to grant users or groups access to this service account, and you do not need to create keys. Just click Done.
Once the Service Account is created, click on it to see its details
- Click “View Domain-Wide Delegation Client ID”
- Copy the numeric Client ID – you’ll need it for creating the IAM Identity Provider in AWS step (below)
CREATE VM IN GCP WITH ASSOCIATED SERVICE ACCOUNT
Now we’re going to create a simple VM and associate the service account that we’ve just created with it. Processes running inside that VM will then be able to act as that service account.
- In the GCP console, go to Compute Engine menu and then choose the VM Instances submenu
- Click Create to start building a new VM
- Name the VM, and choose an appropriate region for it; I used
australia-southeast1
because that’s the closest to where I am, but it really doesn’t matter. I also chose ann1-standard-1
type, which is a small VM that doesn’t cost very much. I chose Ubuntu as the operating system. - In the Identity and API Access section, click on the pulldown which (by default) says “Compute Engine default service account”; you should see the name of the service account that we created above as an option, and you should choose that service account.
- Click Create and wait for the VM to be active
CREATE IAM IDENTITY PROVIDER IN AWS
Using the code at https://github.com/cevoaustralia/gcp-sa-to-aws-iam-role
Using the numeric Client ID you found above, create the CloudFormation stack (you’ll need to be using credentials which allow you to create and manage IAM Identity Providers, Lambda Functions, and IAM Roles):
$ CLIENT_ID= make
This creates an OIDC IdP which trusts the Google Accounts signing key, which will be used to authenticate the credentials from the service account
Once the stack has created, copy the
DeploymentUserRoleArn
from the Outputs and save them for the next step.
USING THE GCP SERVICE ACCOUNT
Here’s the payoff.
- Log in to the GCP VM which you created above (you can use the browser window, or the gcloud command-line, or … really, whatever you like)
- Download the GitHub repo code, which contains the python code which will fetch the AWS credentials (you can use any AWS SDK, Python is just what I wrote this example in):
wget https://github.com/cevoaustralia/gcp-sa-to-aws-iam-role/archive/master.zip
unzip the downloaded archive:
unzip master.zip
change to the directory containing the example credential acquisition code:
cd gcp-sa-to-aws-iam-role-master/gcp
Install the dependencies:
pip3 install -r requirements.txt
Before running the code, it’s worth explaining what it’s doing: it requests an OIDC token from the local VM’s metadata endpoint, then calls the AWS AssumeRoleWithWebIdentity API using that token. The resulting AWS credentials are written into a standard location, at $HOME/.aws/credentials
after which they can be used automatically by the AWS CLI
To acquire those AWS credentials, simply run the script with the ARN of the IAM Role that you copied from the stack outputs, above:
./get_aws_creds.py arn:aws:iam::123456789012:role/DeploymentRole
Congratulations! You now have a GCP VM with time-limited credentials for the IAM Role available to it. You can validate this via the AWS CLI tool:
aws sts get-caller-identity
{
"UserId": "AROAYS4LBXD7LMQMP42FD:example",
"Arn": "arn:aws:sts::123456789012:assumed-role/DeploymentRole/example",
"Account": "123456789012"
}
THE LAST BITS
These credentials are only valid for a limited time period, of course – by default, 3600 seconds (1 hour). If you want to keep these credentials fresh, you could use a cron
job, or use the above script as a pre-execution task before running your actual awscli
commands … the possibilities are endless.
CONCLUSION
I hope this has been helpful to you – demonstrating how you can acquire and use AWS time-limited, role-based credentials on a Virtual Machine in Google Cloud.
If you’d like to know more about identity management; architecting, deploying, migrating or managing workloads in AWS; or multi-cloud and hybrid cloud environments, please get in contact with us; we’re the Cloud Enablement Experts!