Live Elasticsearch Online Training starting on Oct 10! See all classes

Understand Key Docker Metrics: Monitoring and Logging for Docker Datacenter

3 Understand Key Docker Metrics

3.1 Operating System Metrics for each Node

3.1.1 Host CPU

Understanding the CPU utilization of hosts and containers helps one optimize the resource usage of Docker UCP nodes. The container CPU usage can be throttled in order to avoid a single busy container slowing down other containers by taking away all available CPU resources. Throttling the CPU time is a good way to ensure a minimum of processing power for essential services – it’s like the good old nice levels in Unix/Linux.

When the resource usage is optimized, a high CPU utilization might actually be expected and even desired, and alerts might make sense only for when CPU utilisation drops (service outages) or increases for a longer period over some max limit (e.g. 85%).

3.1.2 Host Memory

The total memory used in each Docker UCP node is important to know for the current operations and for capacity planning. Dynamic cluster managers like Docker Swarm use the total memory available on the node and the requested memory for containers to decide on which host a new container should ideally be launched. Deployments might fail if a cluster manager is unable to find a host with sufficient resources for the container. That’s why it is important to know the host memory usage and the memory limits of containers. Adjusting the capacity of new cluster nodes according to the footprint of Docker applications could help optimize the resource usage.  

3.1.3 Host Disk Space

Docker images and containers consume additional disk space. For example, an application image might include a Linux operating system and might have a size of 150-700 MB depending on the size of the base image and installed tools in the container. Persistent Docker volumes consume disk space on the host as well. In our experience watching the disk space and using cleanup tools is essential for continuous operations of Docker hosts.

Disk Space used on two nodes

Disk Space used on two nodes

Because disk space is very critical it makes sense to define alerts for disk space utilization to serve as early warnings and provide enough time to clean up disks or add additional volumes. For example, SPM automatically sets alert rules for disk space usage for you, so you don’t have to remember to do it.

A good practice is to run tasks to clean up the disk by removing unused containers and images frequently.

3.2 Total Number of Running Containers

The current and historical number of containers is an interesting metric for many reasons.  For example, it is very handy during deployments and updates to check that everything is running like before.

When Docker UCP automatically schedules containers to run on different hosts using different scheduling policies, the number of containers running on each host can help one verify the activated scheduling policies. A stacked bar chart displaying the number of containers on each host and the total number of containers provides a quick visualization of how the cluster manager distributed the containers across the available hosts.

Container counts per Docker host over time

Container counts per Docker host over time

This metric can have different “patterns” depending on the use case. For example, batch jobs running in containers vs. long running services commonly result in different container count patterns. A batch job typically starts a container on demand, or starts it periodically, and the container with that job terminates after a relatively short time. In such a scenario one might see a big variation in the number of containers running resulting in a “spiky” container count metric. On the other hand, long running services such as web servers or databases typically run until they get re-deployed during software updates. Although scaling mechanisms might increase or decrease the number of containers depending on load, traffic, and other factors, the container count metric will typically be relatively steady because in such cases containers are often added and removed more gradually. Because of that, there is no general pattern we could use for a default Docker alert rule on the number of running containers.

Nevertheless, alerts based on anomaly detection, which detect sudden changes in the number of the containers in total (or for specific hosts) in a short time window, can be very handy for most of the use cases. The simple threshold-based alerts make sense only when the maximum or minimum number of running containers is known, and in dynamic environments that scale up and down based on external factors, this is often not the case.

3.3 Container Metrics

Container metrics are basically the same metrics available for every Linux process, but include limits set via cgroups by Docker, such as limits for CPU or memory usage. Please note that sophisticated monitoring solutions like SPM for Docker
are able to aggregate Container Metrics on different levels like Docker Hosts/Cluster Nodes, Image Name or ID and Container Name or ID. Having the ability to do that makes it easy to track resources usage by hosts, application types (image names) or specific containers. In the following examples we might use aggregations on various levels.

3.3.1 Container CPU – Throttled CPU Time

One of the most basic bits of information is information about how much CPU is being consumed by all containers, images, or by specific containers. A great advantage of using Docker is the capability to limit CPU utilisation by containers. Of course, you can’t tune and optimize something if you don’t measure it, so monitoring such limits is the prerequisite. Observing the total time that a container’s CPU usage was throttled provides the information one needs to adjust the setting for CPU shares in Docker. Please note that CPU time is throttled only when the host CPU usage is maxed out. As long as the host has spare CPU cycles available for Docker it will not throttle containers’ CPU usage. Therefore, the throttled CPU is typically zero and a spike of this metric is a typically a good indication of one or more containers needing more CPU power than the host can provide.

Container CPU usage and throttled CPU time

Container CPU usage and throttled CPU time

The following screenshot shows containers with 5% CPU quota using the command docker run –cpu-quota=5000 nginx, we see clearly how the throttled CPU grows until it reaches around 5%, enforced by the Docker engine.

Container CPU usage and throttled CPU time with CPU quota of 5%

Container CPU usage and throttled CPU time with CPU quota of 5%

3.3.2 Container Memory – Fail Counters

It is a good practice to set memory limits for containers. Doing that helps avoid a memory-hungry container taking all available memory and starving all other containers on the same server. Runtime constraints on resources can be defined in the Docker run command.
For example, “-m 300M” sets the memory limit for the container to 300 MB. Docker exposes a metric called container memory fail counters. This counter is increased each time memory allocation fails — that is, each time the pre-set memory limit is hit. Thus, spikes in this metric indicate one or more containers needing more memory than was allocated. If the process in the container terminates because of this error, we might also see out of memory events from Docker.

A spike in memory fail counters is a critical event and putting alerts on the memory fail counter is very helpful to detect wrong settings for the memory limits or to discover containers that try to consume more memory than expected.

Container Memory, Memory Fail counters and Swap activity

Container Memory, Memory Fail counters and Swap activity

3.3.3 Container Memory Usage

Different applications have different memory footprints. Knowing the memory footprint of the application containers is important for having a stable environment. Container memory limits ensure that applications perform well, without using too much memory, which could affect other containers on the same host. The best practice is to tune memory setting in a few iterations:

  1. Monitor memory usage of the application container
  2. Set memory limits according to the observations
  3. Continue monitoring of memory, memory fail counters, and Out-Of-Memory events. If OOM events happen, the container memory limits may need to be increased, or debugging is required to find the reason for the high memory consumptions.

Container memory usage

Container memory usage

3.3.4 Container Swap

Like the memory of any other process, a container’s memory could be swapped to disk. For applications like Elasticsearch or Solr one often finds instructions to deactivate swap on the Linux host – but if you run such applications on Docker it might be sufficient just to set “–memory-swap=-1” in the Docker run command!

Container swap, memory pages, and swap rate

Container swap, memory pages, and swap rate

3.3.5 Container Disk I/O

In Docker multiple applications use the same resources concurrently.  Thus, watching the disk I/O helps one define limits for specific applications and give higher throughput to critical applications like data stores or web servers, while throttling disk I/O for batch operations. For example, the command docker run -it –device-write-bps /dev/sda:1mb mybatchjob would limit the container disk writes to a maximum of 1 MB/s.

Container I/O throughput

3.3.6 Container Network Metrics

Networking for containers can be very challenging. By default all containers share a network, or containers might be linked together to share a separated network on the same host. However, when it comes to networking between containers running on different hosts an overlay network is required, or containers could share the host network. Having many options for network configurations means there are many possible causes of network errors.

Moreover, not only errors or dropped packets are important to watch out for. Today, most of the applications are deeply dependent on network communication. Throughput of virtual networks could be a bottleneck especially for containers like load balancers. In addition, the network traffic might be a good indicator how much applications are used by clients and sometimes you might see high spikes, which could indicate denial of service attacks, load tests, or a failure in client apps. So watch the network traffic – it is a useful metric in many cases.

Network traffic and transmission rates

Network traffic and transmission rates

For more information visit Sematext Docker Agent page.