Introduction
In the first part of this series, I explained what the EC2 Instance Metadata Service (IMDS) was and showcased how users could configure their instances to either enable, disable or restore IMDS to v1 or v2 using bash scripts.
In the final part of this series, I will demonstrate how to configure IMDS through the use of AWS Systems Manager (SSM) Run Commands using a tag based approach to target EC2 instances to run our IMDS commands against. In addition, I will look at automating this process through the construction of scheduled events that will invoke our SSM Run Commands on a defined schedule (e.g. every five minutes, every hour, or daily).
Overview
Before I demonstrate how to create and set up the tagging system within the SSM Run Command, let’s quickly outline what we are trying to achieve. When creating an SSM Run Command, we can define a targeting system that allows us to specify EC2 instances based on resource tags (e.g. tag IMDSVersion = V1, IMDSVersion = V2, IMDSVersion = Disable). Using this system, we would simply need to add the key name pair to our EC2 instances and SSM will include that instance as a target for our Run Command. We can create SSM Documents that define the actions we need to perform to manage IMDS on the EC2 instance and configure our Run Command to use those SSM Documents.
Let’s start by defining the SSM Documents.
Creating the SSM Documents
To create the SSM Documents, we can use two approaches; Console or CloudFormation. In this post, I will be using CloudFormation. We begin by creating a CloudFormation template that will contain the three SSM document resources which will have the commands needed to enable, disable and restore IMDS.
---
AWSTemplateFormatVersion: "2010-09-09"
Description: >
This is a cfn template that will be used to create our SSM Documents
Resources:
EnableIMDSV2RunCommand:
Type: AWS::SSM::Document
Properties:
Name: EnableIMDSv2RunDocument
Content:
schemaVersion: "2.2"
description: 'Enable IMDS'
mainSteps:
- action: aws:runShellScript
name: runCommands
inputs:
runCommand:
- "aws ec2 modify-instance-metadata-options --instance-id $AWS_SSM_INSTANCE_ID --http-tokens required --http-endpoint enabled --region $AWS_SSM_REGION_NAME"
DocumentType: Command
DisableIMDSRunCommand:
Type: AWS::SSM::Document
Properties:
Name: DisableIMDSDocument
Content:
schemaVersion: "2.2"
description: 'Remove IMDS'
mainSteps:
- action: aws:runShellScript
name: runCommands
inputs:
runCommand:
- "aws ec2 modify-instance-metadata-options --instance-id $AWS_SSM_INSTANCE_ID --http-endpoint disabled --region $AWS_SSM_REGION_NAME"
DocumentType: Command
RestoreIMDSV1RunCommand:
Type: AWS::SSM::Document
Properties:
Name: RestoreIMDSv1Document
Content:
schemaVersion: "2.2"
description: 'Restore IMDS'
mainSteps:
- action: aws:runShellScript
name: runCommands
inputs:
runCommand:
- "aws ec2 modify-instance-metadata-options --instance-id $AWS_SSM_INSTANCE_ID --http-tokens optional --http-endpoint enabled --region $AWS_SSM_REGION_NAME"
DocumentType: Command
- Note: See the SOLVING THE CATCH-22 PROBLEM section below for details on where those AWS_SSM_* environment variables come from.
- Once you have created the template, create the stack either via the console or through the CLI. Here, I will be using the console method, but the CLI command can be found in the following document: https://docs.aws.amazon.com/cli/latest/reference/cloudformation/create-stack.html
- First, navigate to CloudFormation in the console by typing CloudFormation in the search bar.
- Select the “Create Stack” button.
- Select the following configuration and choose the recently created template:
- After uploading the template, you will need to input a stack name and review the template before creating the stack.
- After creating the stack, you will be able to view the current progress of the stack in the CloudFormation console.
Giving SSM Permission to Run Commands
We need to give SSM permission to perform actions on our EC2 instances to set the Instance Metadata Service to our desired outcome.
To do this, we need to create an IAM Role and attach the AmazonSSMManagedInstanceCore managed policy and create a new policy that grants the EC2 ModifyInstanceMetadataOptions permission.
To enable these permissions users will need to:
- Navigate to the IAM console, by typing in IAM within the search bar.
- Once here, users will need to select the role option underneath the access management tab and select the “Create role” option.
- Once here, users will need to select the “EC2” use case and select the next button.
- After selecting the use case, users will need to search for the following policies:
- AmazonSSMManagedInstanceCore.
- CloudWatchLogsFullAccess (which will help view the output of the run commands for our instances).
Ideally, we do not want to create IAM roles that have policies that contain full access, however, in this scenario, we will for the ease of the demonstration.
- CloudWatchLogsFullAccess (which will help view the output of the run commands for our instances).
- Once the policies have been attached, users will be asked whether they would like to create tags (for this post, it is not needed), so just proceed to the review stage.
- Once at the review stage, users will need to provide a role name (eg EC2SSMManagerInstanceRole) and select “Create role”.
- After creating the role, users will need to create an inline policy, which will enable users to modify the instance’s metadata. To create this inline policy:
- Search for the newly created policy within the roles section of the IAM console.
- Once here select the “add inline policy”, where users will need to search and add the following policy: EC2ModifyInstanceMetadataOptions.
- Now that we have added the policies, we can now set up the instance tagging.
Setting up the EC2 Tags
To configure the tags on your instances (which were created in the first post):
- Navigate to EC2 in the console by typing EC2 within the search bar.
- Once here, users can view their currently run instances as shown below:
- To create the tagging, simply select the instance (e.g. Enable) and select the “Tag” tab.
- Once selected add the following tags to the instances as shown in the images below:
Create the CloudWatch Log Group
Enabling CloudWatch Logs in the SSM Run Command will log the output of the commands and enable us to confirm whether the EC2 instance’s metadata has been changed to the desired state.
To create a CloudWatch Log group:
- Navigate to CloudWatch in the console by typing CloudWatch in the search bar.
- Once here, select the Log groups tab and then select the Create Log Group option, where users will need to pass in a name.
- After providing a name, select create log group and add the following configuration:
Configuring the Run Commands
Now that we have created and attached the policies to an IAM role as well as configure tagging for our instances, we can now look at configuring our Run Commands.
Before we configure the commands, let’s check that they were created correctly:
- Navigate to Systems Manager, by typing Systems Manager in the search bar.
- Once here, select the Documents option underneath the Shared Resources tab.
- Once selected, users should see their newly created SSM Documents as shown below:
- Note: Within the search bar, make sure to select the “Owned by me” tab and filter by the search term “IMDS”. That way users can view the newly created SSM Documents for IMDS.
Now that we have checked that they have been created, let’s configure the Run Commands:
- Select the Run Command option underneath the Node Management tab.
- Once here select the Run command button, which will require users to search for the SSM document they wish to run.
- Given that there are a few pages of documents, simply type IMDS to filter the search to only include documents with IMDS in the name as shown below:
- After locating your documents, select the document that you would like to configure.
Note: that steps 4 – 8 of the process will be identical for the other documents, apart from the value of tags.
- Once here users will be given the option of choosing the targets from either For our example, users will need to select the “specify instance tags”:
- After selecting this option users will need to pass in the specified tag corresponding to the SSM Document to run.
- Enter IMDSVersion:V2 for EnableIMDSv2RunDocument.
- Enter IMDSVersion:Disable for DisableIMDSDocument.
- Enter IMDSVersion:V1 for RestoreIMDSv1RunDocument.
- Once users have configured the instance tags, within the output options section select Enable CloudWatch Logs and enter the name of the Log group we created above (SSMRunCommandLogGroup).
- Once selected, press the Run button.
- Once pressed, users should see the some details of the command as shown in the image below:
- Once the command has run, navigate over to the newly created CloudWatch log group and select the most recent entry, where ….“drum roll”…, users should see the output of the command run on the instance.
{
"InstanceId": "i-0248f42437330efc4",
"InstanceMetadataOptions": {
"State": "pending",
"HttpEndpoint": "enabled",
"HttpTokens": "required",
"HttpPutResponseHopLimit": 1
}
}
- Repeat the above steps for each of the IMDS SSM Documents we created above.
Solving the Catch-22 Problem
It is also worth noting that I originally looked at using the IMDS endpoint of the host instance to extract the instance ID to use to modify the IMDS. However, this approach would only work for the enabling and disabling commands, since if a user wants to restore IMDS to V1, we can not extract the instance ID using IMDS as it is now a disabled endpoint and thus can not be retrieved.
When experimenting with SSM Run Commands I created a Run Command to run a shell script that simply outputted all environment variables. I was able to discover that Systems Manager sets a number of environment variables which includes both the instance ID and the AWS Region (both needed to run the AWS CLI command).
- AWS_SSM_INSTANCE_ID
- AWS_SSM_REGION_NAME
After finding this out, I modified my SSM Documents to use these environment variables to run the command without having to retrieve the ID from the IMDS endpoint.
Creating the Scheduled Event
Now that we have tested the Run Commands work as expected, let’s automate this process by creating a scheduled event to run each SSM Document against all EC2 instances that contains our specified tag (e.g. IMDSVersion: V2, IMDSVersion: Disable, and IMDSVersion: V1).
To set up the scheduled event:
- Navigate to EventBridge by simply typing EventBridge within the search bar.
- Once here, select “Create Rule” and add the following configuration to create an Event Rule that will run every 5 minutes. Users might want to run this daily but I am using 5 minutes for demonstration purposes.
- Configure SSM Run Command as the target for our event specifying the SSM Document and corresponding resource tags as we did when manually creating the Run Commands.
- Select “Create a new role for this specific resource” the first time through to allow EventBridge to create an IAM role for us.
- Select “Use existing role” for the other two Event Rules.
- Repeat the steps above for the other two documents as follows:
| Setting | Value |
| Document | DisableIMDSRunDocument |
| Target key | tag:IMDSVersion |
| Target value | Disable |
| Use existing role | Selected |
| Setting | Value |
| Document | RestoreIMDSv1Document |
| Target key | tag:IMDSVersion |
| Target value | V1 |
| Use existing role | Selected |
Note: Ideally, we would like to also configure our scheduled Run Command to log output to our CloudWatch log group as we did when we created it manually. However, as of the time of writing, there seems to be no way of setting this via an EventBridge Target.
Despite not being able to configure CloudWatch logging, we can see within the SSM Run Command history that our commands are executing based on our scheduled pattern (of five minutes):
CONCLUSION
Well, this concludes our two-part series on managing the EC2 Instance Metadata Service and how users can use AWS Systems Manager Run Commands to execute shell scripts against selected EC2 Instances using resource tags. We have also seen how we can automate this process using EventBridge scheduled events. I hope that the information that was presented was informative and useful in managing IMDS on your EC2 instances.
