Monitoring & Logging Deployment

Sematext Docker Agent runs as a tiny Docker container on each Docker node.  From there it calls Docker API to get logs, metrics, and events for all containers running on the same node, as well as the node itself.  It then streams those data to Sematext SPM and Logsene over an encrypted connection.

Sematext docker agent

7.1 Docker Remote API Integration Options

There are several options to connect Sematext Docker Agent to the Docker Remote API, depending on the configuration of the Docker daemon:

  • Connect to Docker host via UNIX domain socket
  • Connect to Docker host via TLS socket
  • Connect to Docker host via TCP socket
  • Connect to the Docker UCP Remote API endpoint (proxy to Docker Swarm) via TLS

Sematext Docker Agent supports all of these methods, please refer to Appendix A – Configuration options for Sematext Docker Agent. For TLS connections, the agent requires access to the TLS certificates (e.g. mounted to a volume). The UNIX socket connection requires access permission to Docker’s UNIX socket.

Keeping in mind that Docker UCP cluster looks like a single Docker host from the Docker Remote API point of view, it should be very easy to monitor Docker UCP / Swarm with existing Docker monitoring tools! Connecting a monitoring agent to the Swarm Master API endpoint is one potential option. The Sematext Docker Agent would collect all container metrics, events and all logs from the Swarm Master. The following considerations lead to the requirement to have the monitoring and logging agent running on each Docker UCP node:

  • If a single monitoring agent were to connect only to the master node, it would miss host metrics for all other nodes because the Docker API doesn’t provide any host metrics. We could also not see how much memory, disk space, or CPU the Docker UCP / Swarm node itself consumes. Solution: deploy the monitoring agents to each node for collecting the host metrics locally.
  • In a larger cluster with a high volume of logs, events and metrics to collect, a single monitoring agent connected to the master node would need to handle all operational data of the cluster. This would work for a small cluster but such an architecture would obviously be destined for failure on larger clusters. It’s much better to have an agent running on each node and spread the monitoring and logging work over all nodes. Another positive side-effect of this is that there is no need to change the deployment strategy later, when the cluster scales out.
  • If the monitoring agent were to lose the connection to the master node, the monitoring of all containers would fail. To avoid a single point of failure it makes sense to monitor each node individually.

7.2 Summary

Monitoring and logging agent should run on every node for the following reasons:

  • Collection of performance metrics on each Docker UCP node provides complete information about the node and the containers running on each node
  • Load sharing of monitoring and logging workloads
  • No single point of failure
  • The connection from the agent to the Docker Remote API should be established to the local Docker daemon via UNIX socket and not to the Docker UCP endpoint with TLS

Docker monitoring agent deployment

Target setup

To deploy Sematext Docker Agent to all Docker UCP nodes we will use docker-compose scale command, connected to the Docker API endpoint for the Docker UCP cluster. See chapter 8 for details.

For more information visit Sematext Docker Agent page.