Overview
CloudWatch Lambda Insights is a monitoring and troubleshooting solution for serverless applications running on AWS Lambda. It collects, aggregates, and summarises system-level metrics including CPU time, memory, disk, and network. It also collects, aggregates, and summarises diagnostic information such as cold starts and Lambda worker shutdowns to help you troubleshoot health and performance issues with your Lambda functions.
Lambda Insights uses a new CloudWatch Lambda extension, which is provided as a Lambda layer. Lambda extensions allow us to extend the functionality of our Lambda functions without touching function code. When you install this extension on a Lambda function, it collects system-level metrics and emits a single performance log event for every invocation of that Lambda function.
See the AWS Docs for more information on CloudWatch Lambda Insights features and pricing.
In this post, we’ll take a sample serverless application and walk-though the required steps to enable CloudWatch Lambda Insights for your Lambda functions.
Getting Started
Clone the following GitHub repository containing an AWS SAM based sample serverless application.
$ git clone https://github.com/ScottScovell/well-architected-serverless.git In the README you will find a walk-through of the sample serverless application and details on how to run locally before deploying to AWS.
You can either implement the changes below as you walk through this post, or checkout the completed feature branch part-1/enable-lambda-insights and review the code before deploying into your AWS account.
ADDING THE LAMBDA INSIGHTS LAYER
Lambda Insights uses a new CloudWatch Lambda extension, which is provided as a Lambda layer. To add the layer to all functions, we can use the Globals section of the SAM template and specify which version (ARN) of the Lambda extension to use.
Note: Available versions of the Lambda Insights Extension can be found here
Add the Layer to the Globals section of our template.yaml file as follows
Globals:
Function:
Timeout: 30
Layers:
- !Sub "arn:aws:lambda:${AWS::Region}:580247275435:layer:LambdaInsightsExtension:14"Now add the CloudWatchLambdaInsightsExecutionRolePolicy IAM policy to each function (global policies are currently not supported). This grants our Lambda function(s) permission to access Lambda Insights.
ListEntitiesFunction:
Type: AWS::Serverless::Function
Properties:
...
Policies:
- CloudWatchLambdaInsightsExecutionRolePolicy
- DynamoDBCrudPolicy:
TableName: !Ref EntitiesTableUse the SAM CLI to validate the changes to the SAM template.
$ sam validateYou should see template.yaml is a valid SAM Template
Now use the SAM CLI to build and deploy the application to AWS.
$ sam build --use-container
$ sam deploy --guidedUse the following settings during the guided deployment
Setting default arguments for 'sam deploy'
=========================================
Stack Name [sam-app]: well-architected-serverless
AWS Region [us-east-1]:
Parameter REGION [us-east-1]:
Confirm changes before deploy [y/N]: y
Allow SAM CLI IAM role creation [Y/n]: Y
ListEntitiesFunction may not have authorization defined, Is this okay? [y/N]: y
GetEntityFunction may not have authorization defined, Is this okay? [y/N]: y
CreateEntityFunction may not have authorization defined, Is this okay? [y/N]: y
DeleteEntityFunction may not have authorization defined, Is this okay? [y/N]: y
Save arguments to configuration file [Y/n]: Y
SAM configuration file [samconfig.toml]:
SAM configuration environment [default]: VIEWING LAMBDA INSIGHTS IN THE CLOUDWATCH CONSOLE
Generate some traffic using the API Gateway console, Postman, or curl to execute requests against our application. For example, the following bash command simulates 100 invocations of the ListEntitiesFunction via the deployed API Gateway endpoint.
$ for run in {1..100}; do curl https://3zk9g9go06.execute-api.us-east-1.amazonaws.com/Prod/entities/; sleep 5; doneIn the CloudWatch Console, navigate to Lambda Insights and select Multi-Function to view aggregated metrics for our Lambda functions.
Switch to Single-function and select a specific Lambda function to view performance metrics, performance logs, and application logs for that function.
ENABLE X-RAY TRACING
AWS X-Ray is a diagnostics service to help visualise the components of your application, identify performance bottlenecks, and troubleshoot errors in your serverless applications. Lambda functions send trace data to X-Ray where it is used to generate a service map and searchable trace summaries.
X-Ray tracing can be enabled in services that invoke your function and Lambda will automatically send traces to X-Ray. The upstream service samples incoming requests and adds a tracing header that tells Lambda to send traces or not. You can also trace requests that don’t have a tracing header set by the caller. In these cases you can enable active tracing in your function’s configuration.
For the scope of this blog post, we’ll assume the API Gateway does not have X-Ray tracing enabled and configure tracing in our Lambda functions. To do this, enable tracing in the Globals section of your SAM template.
Globals:
Function:
Timeout: 30
Tracing: Active
Layers:
- !Sub "arn:aws:lambda:${AWS::Region}:580247275435:layer:LambdaInsightsExtension:14"Redeploy your SAM stack and generate some traffic once again so we can see trace activity in the console. (see above)
VIEWING THE X-RAY SERVICE MAP, TRACES, AND ANALYTICS
In the X-Ray console, navigate to the Service map. After a few mins you should see your Lambda functions appearing in the map showing health and performance metrics. Icons can be sized by health or volume of traffic.
Selecting a function also shows details of the service with links through to trace logs and analytics.
Selecting one of the trace entries shows health and performance details of that invocation.
CONCLUSION
In this post, we have taken a sample serverless application and walked though the steps required to enable Amazon CloudWatch Lambda Insights. Once deployed we explored the Lambda Insights console and saw how we can quickly gather health and performance insights of our serverless applications in AWS and drill down into trace logs automatically sent to AWS X-Ray for runtime troubleshooting and diagnostics.
