Posted by Benjamin Wootton on 06/12/16 10:01

The eye of the whale: how to monitor Docker containers on AWS with CloudWatch

Blog_CloudWatch_AWS_containers.png

The AWS EC2 Container Service (ECS) is a handy platform if you want to deploy Docker containers with the speed and convenience of the public cloud.

With ECS, you get a (mostly) turn-key solution for running and orchestrating Docker containers without having to set up your own infrastructure. You also get easy scalability, since your ability to scale up is not constrained by the size of your infrastructure, as it would be if you ran containers on-premises.

Container deployment, however, is only half the battle. If you want to create an efficient container environment for production purposes, you also need to make sure you are monitoring your containers effectively.

But because containers are so agile and portable, they are built up, torn down and shuffled about very frequently, which massively increases the difficulty of monitoring them.

That’s where AWS CloudWatch comes in. CloudWatch provides an easy, centralised interface for collecting and analysing data generated by your ECS environment.

This article explains why CloudWatch is useful for container monitoring, and how to get started with it.

What is CloudWatch?

CloudWatch is Amazon’s main monitoring service for apps running on AWS infrastructure. It is  not specifically designed to work only with containers running on ECS, you can use CloudWatch for almost any type of environment that is hosted on AWS.

CloudWatch supports a wide array of metrics. You can use it to monitor application performance data, like CPU use and data transfer speeds. You can also keep track of disk usage and collect generic logs generated by almost any type of application.

But CloudWatch does more than just collect information and logs. It also offers data visualisations and reports to help you interpret that data. It may not be a replacement for a full-blown data analytics solution, but for many organisations that host apps or services on AWS, CloudWatch is more than sufficient for identifying the major trends and potential problems that admins need to monitor in order to maintain performance and guarantee high availability.

CloudWatch and Docker containers

Amazon introduced support for sending Docker logs to CloudWatch in 2015. This makes it possible to store container logs in a central location. It also saves space because the logs do not have to be stored inside the containers.

In addition, CloudWatch can monitor statistics related to the health of your ECS cluster, such as CPU and memory usage.

Getting started with CloudWatch for ECS

To use CloudWatch to monitor your containers, you need to perform the following steps:

  • Create an IAM policy for container monitoring and attach it to your ecsInstanceRole.

  • Install the CloudWatch agent on your container instances. On Red Hat/CentOS-like systems, you can do this with yum install awslogs. On other systems, Amazon provides an installation script.

  • Configure and start the agent.

For full details on these procedures, including specific information related to configuring the IAM policy and the CloudWatch agent, check out the AWS documentation. 

Using CloudWatch

Once you have CloudWatch configured and installed for your ECS cluster, you can open the CloudWatch console (if you’re already logged in, you can do that by clicking this link), select the log group for your container cluster, and start monitoring the data. The interface will look something like this:

ContainersOnAWS.png

The interface is a convenient way to keep track of the current status of your cluster in real time (or almost real time - there is a slight delay between data collection and reporting). If there is a sudden spike in CPU or memory usage, for example, it will be evident from your dashboard, so that you can take appropriate action. 

In addition to summarising key metrics related to the current status of an ECS cluster, CloudWatch also reports information over time. This is useful for measuring long-term trends and performance. For example, if you rebuild a container image in order to make an app more efficient by lowering memory consumption or CPU usage, CloudWatch graphs can help you determine whether your changes were effective by comparing a metrics summary from before the change with one after the change.

Further reading

There is much more you can do with CloudWatch. For an interesting example using CloudWatch in conjunction with rsyslog and a dedicated logging container to aggregate all of your container logs, check out this AWS blog post. You may also find it useful to set up CloudWatch alarms or to configure Auto Scaling for your ECS clusters so that they scale automatically on the basis of CloudWatch metrics.  

Benjamin Wootton

Benjamin Wootton - Benjamin Wootton is the Co-Founder and CTO, EMEA of Contino. He has worked with tens of enterprise organisations on DevOps transformation and is a hands on DevOps engineer with expertise in cloud and containers.

Topics: AWS, Containers

Meet Contino At An Online or Offline Event

Comments