AWS Config is a powerful service that helps organisations monitor and assess their cloud environments. However, many users experience unexpectedly high AWS Config bills. In a previous blog, we discussed how to optimise AWS Config costs: AWS Config Cost Optimization.
In this blog, we will dive deeper into understanding why AWS Config charges can be high, how to detect cost-driving factors, and effective strategies to optimise costs.
1. Understanding AWS Config billing
To effectively manage AWS Config costs, we need to first understand how AWS Config pricing works. AWS Config has both direct and indirect costs:
- Direct Costs: These are the main factors that contribute to AWS Config charges:
- Number of configuration items recorded
- Number of active AWS Config rule evaluations
- Number of conformance pack evaluations
- Indirect Costs: These are additional costs that arise from using AWS Config:
- S3 storage (AWS Config stores configuration snapshots in an S3 bucket).
- SNS notifications (if enabled, AWS Config sends changes through SNS, incurring messaging costs).
For more details, visit the AWS pricing page: AWS Config Pricing.
The more changes occurring within an AWS account, the higher the AWS Config charges. Environments with frequent infrastructure modifications, such as development environments using Infrastructure as Code (IaC) tools like Terraform or dynamic infrastructures with ephemeral components (e.g., storage or networking in ECS) can lead to increased AWS Config costs.
2. Detecting high AWS Config Costs using Amazon Athena
At Cevo, AWS Config ranks among the top five most costly AWS services in some accounts. To analyse why AWS Config costs are high, we can use Amazon Athena to investigate configuration changes in a development account.
Why use Amazon Athena?
Amazon Athena is a serverless service that lets you analyse data in an S3 bucket using SQL, eliminating the need to set up or manage infrastructure.
AWS Config stores configuration history files in an S3 bucket, as shown below:
For more details on viewing AWS Config configuration history, refer to this guide: AWS Config Resource Management.
By using Amazon Athena, we can read and analyse these configuration history files to identify the resources with the highest number of changes.
Analyse AWS Config costs
To analyse AWS Config costs, we will create two scripts:
- A script to create an Athena table to store the results.
- A script to query and group configuration changes by resource type and ID.
Step 1: Creating an Athena table to store AWS Config records
Since AWS Config records configuration history in an S3 bucket, we first need to create a table in Athena to query these records and store it in a formatted way. Below is the script to create the table:
CREATE EXTERNAL TABLE awsconfig (
fileversion string,
configSnapshotId string,
configurationitems ARRAY < STRUCT <
configurationItemVersion : STRING,
configurationItemCaptureTime : STRING,
configurationStateId : BIGINT,
awsAccountId : STRING,
configurationItemStatus : STRING,
resourceType : STRING,
resourceId : STRING,
resourceName : STRING,
ARN : STRING,
awsRegion : STRING,
availabilityZone : STRING,
configurationStateMd5Hash : STRING,
resourceCreationTime : STRING > >
)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
LOCATION 's3://AWS_CONFIG_BUCKET_NAME/AWSLogs/ACCOUNT_ID/config/REGION';
Script 1 – Create Table in Amazon Athena to store configuration changes
Note: Update the LOCATION parameter to match the S3 bucket storing AWS Config history files.
The default path for AWS Config configuration files follows this format:
config-bucket-XXXXXXXX/AWSLogs/<AccountID>/Config/<Region>/YYYY/MM/DD
This script creates a table named awsconfig in the Athena database and stores configuration changes. We will use this table in the next query to identify the most frequently changing resources.
Step 2: Identifying the most frequently changing resources
The second query identifies the resources with the highest number of configuration changes. It retrieves data from the previous query and groups it by resource type and ID to determine which resources undergo the most modifications.
SELECT
configurationItem.resourceType,
configurationItem.resourceId,
COUNT(configurationItem.resourceId) AS NumberOfChanges
FROM awsconfig
CROSS JOIN UNNEST(configurationitems) AS t(configurationItem)
WHERE "$path" LIKE '%ConfigHistory%'
GROUP BY configurationItem.resourceType, configurationItem.resourceId
ORDER BY NumberOfChanges DESC
Script 2 – Query to show the number of changes for each resource
Running this query generates a result with three columns:
- Resource type
- Resource ID
- Number of recorded changes
We found that the most frequently changing resources were of type ResourceCompliance. This means AWS Config is frequently recording compliance status changes, which contribute significantly to costs, often more than recording actual resource configuration changes.
If you have AWS Control Tower enabled with a Conformance Pack, AWS Config costs may be even higher. Control Tower continuously records ResourceCompliance changes and evaluates all Conformance Pack rules, leading to frequent configuration updates – each of which incurs a charge.
Investigating specific resources
Let us take a closer look at the first resource in our query results:
AWS::Lambda::Function/aws-controltower-NotificationForwarder
To inspect what changes occurred, navigate to Resource Inventory in AWS Config, search for the resource, and click Resource Timeline.
The timeline reveals that AWS::Config::ResourceCompliance changes occur daily. These changes stem from compliance status updates for the aws-controltower-NotificationForwarder Lambda function.
Filtering the event type by Compliance Events for the Lambda function confirms that its compliance status updates every day when the Conformance Pack runs.
We encourage you to investigate the Athena query results in your own AWS environment to identify the resources generating the highest AWS Config costs.
In our Cevo Dev account, the change in compliance status for resources contributes the most into the cost of AWS Config. The top 10 most resources have configuration changes are type AWS::Config::ResourceCompliance.
3. Cost optimisation strategy
Several strategies can effectively reduce AWS Config costs. In our previous blog, we outlined some approaches: AWS Config Cost Optimization.
In addition to those strategies, here are other ways to optimise AWS Config costs:
1. Disable ResourceCompliance recording
A major cost driver is ResourceCompliance recording. If your organisation does not require a detailed compliance change history (or if you use AWS Security Hub for security monitoring), you can disable ResourceCompliance recording to cut costs.
To exclude this resource type, update the AWS Config recorder settings by overriding the default configuration.
For more details, refer to this post: Chris Farris – Security Hub Optimization.
Disabling ResourceCompliance prevents unnecessary configuration recordings while still maintaining security monitoring through Security Hub.
In our Dev account, we successfully reduced the average daily AWS Config cost from $2.00 to $1.30, a 35% savings. This reduction is significant, especially for businesses managing hundreds of accounts and spending thousands of dollars on AWS Config.
The chart below displays the daily average cost for AWS Config. ResourceCompliance was disabled on February 6th.
2. Disable AWS Config in Development Accounts
Development environments are primarily used for testing new changes and exploring features. As a result, developers frequently create, destroy, and reconfigure resources, leading to constant changes. This high volume of activity can cause AWS Config costs in Development accounts to be higher than in production. Additionally, since Development accounts are often not part of an enterprise’s auditing process, configuration change tracking may not be necessary. Disabling AWS Config in these accounts can be a practical way to reduce costs.
3. Check the anomaly change numbers in AWS Config
As seen in our Cevo Dev account, some resources experience high-frequency changes. Monitoring these changes can help detect unexpected cost spikes.
This issue has affected many AWS users before:
Setting up alerts for unusual spikes in configuration changes can help prevent unexpected AWS Config charges.
Another effective way to detect cost spikes is by enabling AWS Cost Anomaly Detection. This allows you to create customised monitors and receive alerts whenever anomalous spending is detected.
Summary
In this blog, we explored how AWS Config billing works and the factors that drive high costs. Additionally, we learned how to use Amazon Athena to analyse AWS Config costs and identify frequently changing resources. Moreover, we provided few cost-saving strategies; by applying these strategies, you can reduce AWS Config costs while still maintaining necessary compliance and security monitoring.
