octi-docker-datacenter

Monitoring Docker Datacenter Logs & Metrics

Docker Datacenter (DDC) simplifies container orchestration and increases the flexibility and scalability of application deployments.  However, the high level of automation create new challenges for monitoring and log management. Organizations that introduce Docker Datacenter manage container deployments in various scenarios e.g., on bare metal, virtual machines, or hybrid clouds. That’s why at Sematext we are seeing a shift from traditional server monitoring to container-centric monitoring. This post is an excerpt from the newly published “Reference Architecture: Monitoring and Logging for Docker Datacenter” and shows how Docker Datacenter could be extended with Logging and Monitoring services.

Download Reference Architecture Logging & Monitoring for Docker Datacenter

The Docker Universal Control Plane (UCP) management functionalities include real-time monitoring of the cluster state, real-time metrics and logs for each container. However, operating larger infrastructures requires a longer retention time for logs and metrics and the capability to correlate metrics, logs and events on several levels (cluster, nodes, applications and containers).  A comprehensive monitoring and logging solution ought to provide the following operational insights:

  • Auditing of Docker events
  • Resource usage for capacity planning and tuning
  • Detailed metrics for cluster nodes and containers
  • Centralized log management with full-text search, filtering, and analytics across all containers
  • Anomaly detection and alerting for all logs and metrics
  • Long retention time for logs, metrics and events

The Docker Datacenter architecture is open for extensions, such as Monitoring and Logging. In this blog post, you will learn how to set up monitoring and logging for a Docker Datacenter.

ddc-archDocker Datacenter Architecture (source: www.docker.com)

We will use Sematext Docker Agent, a monitoring agent, which connects to the Docker API for metrics, events and log collection for all nodes and containers in Docker Datacenter.

Set up Monitoring and Logging

  1. Configuration of the agent
    1. Connection to the Docker API
    2. Connection to SPM/Logsene
  2. Deployment of the agent to all cluster nodes

Docker-UCP-Deployment

Meet Sematext Docker Agent

Sematext Docker Agent is a modern, Docker-aware metrics and log collection agent.  It runs as a tiny container on every Docker host collecting logs, metrics and events for all cluster hosts and all containers from the Docker Remote API. It discovers all containers including the containers for Docker UCP services. Sematext Docker Agent streams metrics, events, and logs via TLS (https) to SPM and Logsene.  After the deployment of the agent all logs and metrics are immediately available in SPM and Logsene.

Connect Docker-Compose to Docker UCP Cluster

After installing UCP, you can run the docker command, against UCP cluster nodes. Since all traffic between users and UCP nodes is secure and authenticated, when using the Docker CLI client, you’ll need to provide client certificates. Download the bundle that contains the client certificates for a user from the UCP web app:

    1. If you haven’t already done so, log into UCP.
    2. Navigate to your profile.
    3. As an example, if you’re logged in as the Admin user, on the right-hand menu, navigate to Admin > Profile.
    4. Click the Create Client Bundle button.
    5. The browser downloads the ucp-bundle-admin.zip file.
    6. Copy the client bundle in your working directory and extract the files.
      $ cp ~/Downloads/ucp-bundle-admin.zip .
      $ unzip ucp-bundle-admin.zip
    7. Run the included shell script to set the environment variables to access UCP with docker CLI client.
       
      $ source env.sh
      
    8. Check the settings by running “docker info” to see if docker CLI is connected to UCP
      $ docker info

Configure Sematext Docker Agent for Docker UCP

Sematext Docker Agent is configured via environment variables. The Sematext Docker Agent Github page lists all options (e.g. filter for specific images and containers), but we’ll keep it simple here:

    1. Get a free account at apps.sematext.com, if you don’t have one already.
    2. Create an SPM App of type “Docker” to obtain the SPM Application Token. SPM App will hold your Docker UCP performance metrics.
    3. Create a Logsene App to obtain the Logsene Application Token. Logsene App will hold your Docker UCP logs.
    4. Edit the values of LOGSENE_TOKEN and SPM_TOKEN in the docker-compose.yml file as shown below: 
      sematext-agent-docker:
      image: 'sematext/sematext-agent-docker:latest'
      environment:
       - LOGSENE_TOKEN="REPLACE THIS WITH YOUR LOGSENE TOKEN"
       - SPM_TOKEN="REPLACE THIS WITH YOUR SPM TOKEN"
       - affinity:container!=*sematext-agent-docker*
      restart: always
      volumes:
       - /var/run/docker.sock:/var/run/docker.sock

      In case you have to use a proxy add HTTPS_PROXY=https://your-proxy-server:port to the environment section in the docker-compose.yml.

    5. To deploy Sematext Docker Agent to all nodes, we have to figure out how many Docker UCP nodes are running. The following command prints the number of nodes: 
      $ docker info | grep Nodes | awk '{ print $2 }'
      3
      

      Finally run the combined command to scale Sematext Docker Agent to all nodes, with the docker-compose project name “monitoring”:

      $ docker-compose -p monitoring scale sematext-agent=$(docker info | grep Nodes | awk '{ print $2 }')
      Creating and starting 1 ... done
      Creating and starting 2 ... done
      Creating and starting 3 ... done

      Check the number of running agent with 

      • docker-compose -p monitoring ps
      • or in the Application section in Docker UCP:  Docker-UCP-screenshot

After one minute you should see the performance metrics of all Docker UCP nodes and deployed containers in your SPM Application, and logs are stored in Logsene.

docker-monitoring-overview

The Server view should show all Docker UCP nodes:

server-overview-docker

SPM is now able to show the logs from the right Logsene Application (remember, there might be multiple for different applications) in same screen as Docker Metrics and Events are shown.

spm-screenshot-docker-events

To take full advantage of Sematext Docker Agent, Logsene and SPM you might be interested in the complete  “Reference Architecture: Monitoring and Logging for Docker Datacenter” (PDF version), which explains all design considerations, Docker Key Metrics, and all available options for monitoring and log management.

If you’d like to learn even more about Docker Monitoring and Logging stay tuned on sematext.com/blog or follow @sematext.  To monitor your Docker environment (including Docker Swarm and Docker Cloud), any of Sematext’s other pre-built integrations, your app’s custom metrics, or your Docker logs, sign up for a free 30-day trial account below.

SIGN UP – FREE TRIAL

docker-partner-logo

Leave a Reply