Monitoring Solution for AWS ECS Fargate Applications with New Relic Integration

April 26, 2023

Ravi Doddahalli

Summary

When it comes to running applications on AWS ECS Fargate, monitoring is an essential component for ensuring availability, performance, and reliability. Without proper monitoring, it can be challenging to identify and address issues that may impact your application’s performance or cause downtime. This is where New Relic comes in, providing a comprehensive monitoring solution for AWS ECS Fargate applications that allows you to gain real-time insights into your application’s performance and take proactive measures to address issues.

In this blog post, we’ll explore how you can leverage the monitoring solution for AWS ECS Fargate applications with New Relic integration to gain visibility into your applications, in addition to the benefits of this solution, how it works, and how to get started with it.

New Relic is a complete monitoring solution that provides insights into the performance of AWS applications. This platform helps businesses optimise the performance of their applications to help resolve any issues that may arise.

Why choose New Relic over AWS CloudWatch?

The below table compares AWS CloudWatch and New Relic.

Particulars	AWS CloudWatch	New Relic
Usage costing	Based on the amount of data ingested and the number of custom metrics and alarms	Based the number of hosts, containers or VMs monitored and the level of features and services selected
Features	Basic monitoring features – primarily infrastructure monitoring	Comprehensive monitoring features – application performance management (APM), infrastructure, monitoring, customisable dashboards, mobile monitoring, and synthetic
Scalability	Highly scalable	Highly scalable
System	Native	Integrated
Alerting integration	Advanced alerting capabilities are missing that ability to set up custom alert policies and notification channels are not readily available. CloudWatch Alerts required to integrate with AWS Lambda and AWS SNS to build custom policies and notifications.	Integrates with a range of tools and services, including ServiceNow, Slack, OpsGene and many more
Mobile monitoring	Not available – Required to build own app	Readily available
Synthetic monitoring	Monitoring capabilities like real user monitoring(RUM) is available. That is, insights into the actual experience of users interacting with web applications and APIs in real time is possible.	Enables businesses to simulate user interactions with their applications to test their performance.

Types of Monitoring

The table below shows the types of New Relic monitoring that are integrated with AWS applications.

Monitoring type	Description
Infrastructure Monitoring	System monitoring captures AWS cloud services metrics. Auto ingestion to New Relic from AWS cloud services. Example – for a service AWS ECS – its CPU utilisation, memory utilisation etc.
Application Performance Monitoring	Application monitoring captures application metrics based on application type.
Custom Events Monitoring	Custom events can represent an activity that is triggered by business logic. For example, a customer purchasing a large quantity of a product, or a sales person reaching the target sales

Infrastructure Monitoring

New Relic infrastructure Monitoring is a powerful solution that can help you gain deep insights into the performance and health of your AWS ECS Fargate infrastructure. One way to leverage the full capabilities of New Relic Infrastructure Monitoring is by using the New Relic sidecar. The New Relic sidecar is a lightweight container that can be deployed alongside your ECS Fargate tasks to collect real-time performance data, including CPU usage, memory usage, disk space, and network traffic. Here, we will explore how to set up New Relic Infrastructure Monitoring for AWS ECS Fargate with the New Relic sidecar.

Prerequisites: Obtain a New Relic licence key by signing up for a New Relic account.

Install New Relic Sidecar using Terraform

Save New Relic licence credentials in Secrets Manager

//ssm.tf

resource “aws_secretsmanager_secret” “nr_license_secret” {
name = “sample-nr-license”
description = “The New Relic License Key”
}

resource “aws_secretsmanager_secret_version” “nr_license_secret_value” {
secret_id = aws_secretsmanager_secret.nr_license_secret.id
secret_string = “${var.NR_LICENSE_KEY}”
// pipeline environment variable/ Secret variables which are encrypted variable
}

Create an IAM policy to access the NR licence key

//iam.tf

resource “aws_iam_policy” “ECSTaskPolicy” {
name = “sample-ECSTaskPolicy”

policy = <<EOF
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: “ssm:GetParameter*”,
“Resource”: “arn:aws:ssm:${aws_region}:${aws_account_id}:parameter/sample/*”
}
]
}
EOF
}

resource “aws_iam_policy” “ECSExecutionPolicy” {
name = “sample-ECSExecutionPolicy”

policy = <<EOF
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: “ssm:GetParameter*”,
“Resource”: “arn:aws:ssm:${aws_region}:${aws_account_id}:parameter/sample/*”
},
{
“Effect”: “Allow”,
“Action”: “secretsmanager:GetSecretValue”,
“Resource”: “${aws_secretsmanager_secret.nr_license_secret.arn}”
}
]
}
EOF
}

Create an IAM role to be used as the task execution role

//iam.tf

resource “aws_iam_role” “ECSTaskRole” {
name = “sample-ECSTaskRole”
assume_role_policy = <<EOF
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Principal”: {
“Service”: [
“ecs-tasks.amazonaws.com”
]
},
“Action”: “sts:AssumeRole”
}
]
}
EOF
}

resource “aws_iam_role” “ECSExecutionRole” {
name = “sample-ECSExecutionRole”
assume_role_policy = <<EOF
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Principal”: {
“Service”: [
“ecs-tasks.amazonaws.com”,
“ecs.amazonaws.com”
]
},
“Action”: “sts:AssumeRole”
}
]
}
EOF
}

Attach the policy and role created in step 1 and 2

//iam.tf

resource “aws_iam_policy_attachment” “ECSExecutionPolicyAttach” {
name = “sample-ECSExecutionPolicyAttach”
roles = [aws_iam_role.ECSExecutionRole.name]
policy_arn = aws_iam_policy.ECSExecutionPolicy.arn
}

resource “aws_iam_policy_attachment” “ECSTaskPolicyAttach” {
name = “sample-ECSTaskPolicyAttach”
roles = [aws_iam_role.ECSTaskRole.name]
policy_arn = aws_iam_policy.ECSTaskPolicy.arn
}

Deploy Sample ECS task definition with the sidecar container for AWS Fargate launch type.

// service.tf

resource “aws_ecs_task_definition” “ecs_task_definition” {
family = “sample-with-nr-infra-sidecar-ecs-task-definition”
network_mode = “awsvpc”
execution_role_arn = aws_iam_role.ECSExecutionRole.arn
task_role_arn = aws_iam_role.ECSTaskRole.arn
requires_compatibilities = [“FARGATE”]
cpu = “1024”
memory = “4096”
container_definitions = <<EOF
[
{
“name”: sample,
“image”: sample:latest,
“essential”: true,
“portMappings”: [
{
“containerPort”: 8080,
“hostPort”: 8080
}
],
“environment”:[
{
“name”: “NEW_RELIC_LICENSE_KEY”,
“value”: “${var.NR_LICENSE_KEY}”
// pipeline environment variable/ Secret variables which are encrypted variable

},
{
“name”: “NEW_RELIC_APP_NAME”,
“value”: “sample”
},
{
“name”: “NEW_RELIC_NO_CONFIG_FILE”,
“value”: “true”
},
]
},
{
“environment”: [
{
“name”: “NRIA_OVERRIDE_HOST_ROOT”,
“value”: “”
// by default the agent uses the hostname of the system it is running on as the root of the hostname for agent communication.
},
{
“name”: “NRIA_IS_FORWARD_ONLY”,
“value”: “true”
},
{
“name”: “FARGATE”,
“value”: “true”
},
{
“name”: “NRIA_PASSTHROUGH_ENVIRONMENT”,
“value”: “ECS_CONTAINER_METADATA_URI,ENABLE_NRI_ECS,FARGATE”
},
{
“name”: “NRIA_CUSTOM_ATTRIBUTES”,
“value”: “{\\”nrDeployMethod\\“:\\”downloadPage\\“}”
}
],
“secrets”: [
{
“valueFrom”: “${aws_secretsmanager_secret.nr_license_secret.arn}”,
“name”: “NRIA_LICENSE_KEY”
}
],
“cpu”: 256,
“memoryReservation”: 512,
“image”: “newrelic/nri-ecs:2.8.40”,
“name”: “newrelic-infra”
}
]
EOF
}

Application Monitoring

New Relic Application Monitoring is an essential tool for gaining visibility into the performance and health of your applications running on AWS ECS Fargate. By leveraging New Relic’s Node.js agent, you can monitor your Node.js applications and gain deep insights into their performance and health metrics.

In order to have application logs or custom events monitoring, one should configure the agent.

To meet your application’s needs, you have the option to modify the Node.js agent by adjusting either the newrelic.js configuration file or an environmental variable.

Configure agent via the newrelic.js file

Main required setup are license_key setting and app_name

exports.config = {
/**
* Array of application names.
*/
app_name: [process.env.SERVICE],
/**
* Your New Relic license key.
*/
license_key: process.env.NEW_RELIC_LICENSE_KEY,

};

Environmental variables

NEW_RELIC_APP_NAME
NEW_RELIC_LICENSE_KEY

After configuring the New Relic agent, one should use the recordLogEvent method to log an event.

//index.js
import newrelic from ‘newrelic’;

.
.
.
const error = new SystemError(‘invalid state’);
const timestamp = Date.now();

newrelic.recordLogEvent({message: ‘an error happened’, level: ‘error‘, timestamp, error});

Custom Events Monitoring

In addition to infrastructure and application monitoring, New Relic also enables custom events monitoring. Custom events can represent an activity that is triggered by business logic that sends data about an event so that it can be analysed in New Relic.

After configuring the New Relic agent, one should use recordCustomEvent to record an event as shown below code sample.

//index.js
import newrelic from ‘newrelic’;

.
.
.

//pass the value to newrelic record
newRelicRecord[‘sales’] = 10000;

//Create a record as custom event
newrelic.recordCustomEvent(`application_name_sales_prod`, newRelicRecord);

How is the New Relic Monitoring solution aligned with the AWS Well-Architectured Framework?

The monitoring solution for AWS ECS Fargate applications with New Relic Integration aligns well with several pillars of the AWS Well-Architected Framework, including Operational Excellence, Reliability, Performance Efficiency, and Cost Optimisation.

Operational Excellence: The solution enables real-time monitoring and alerting of ECS Fargate applications, allowing for proactive issue resolution and minimising the time-to-recovery.
Reliability: By providing detailed insights into the health and performance of ECS Fargate applications, the solution helps ensure that applications are available and performant.
Performance Efficiency: The solution provides deep insights into application performance, including transaction traces, database queries, and external API calls. This allows you to quickly identify bottlenecks and optimise your application’s performance.
Cost Optimisation: By providing detailed usage metrics for your ECS Fargate applications, allowing you to identify opportunities for cost savings. Additionally, New Relic’s performance monitoring capabilities can help you identify areas where you can optimise your application’s performance, reducing the need for additional compute resources.

Conclusion

The blog post provides a comprehensive overview of how New Relic can be used to monitor AWS ECS-Fargate applications. It compares New Relic to AWS CloudWatch, highlighting the limitations of CloudWatch and how New Relic can complement it with its advanced monitoring and alerting capabilities.

The article covers the different types of monitoring and alerting that can be integrated with AWS applications using New Relic. It includes container monitoring, distributed tracing, and synthetic monitoring. The post also provides practical guidance on implementing the New Relic monitoring solution on AWS ECS-Fargate. It includes code samples for installing the New Relic sidecar using Terraform and configuring the New Relic Node.js agent.

The post concludes by explaining how the New Relic monitoring solution aligns with the AWS Well-Architected Framework. By using New Relic, AWS customers can improve their application performance, reduce downtime, and optimise their costs while ensuring that their applications are built and operated according to the Well-Architected Framework.

Overall, the blog post provides a comprehensive overview of the benefits of using New Relic for monitoring AWS ECS-Fargate applications and how it can help AWS customers to achieve their operational and business goals. And to learn more about how New Relic can help you monitor application performance, visit New Relic site.

Enjoyed this blog?

Share it with your network!