Until recently, automating the deployment of Performance Monitoring agents in Docker Swarm clusters was challenging because monitoring agents had to be deployed to each cluster node and the previous Docker releases (<Docker engine v1.12 / Docker Swarm 1.2.4) had no global service scheduler (Github issue #601). Scheduling services with via docker-compose and scheduling constraints required manual updates when the number of nodes changed in the swarm cluster – definitely not convenient for dynamic scaling of clusters! In Docker Swarm Monitoring and Logging we shared some Linux shell acrobatics as workaround for this issue.
The good news: All this has changed with Docker Engine v1.12 and new Swarm Mode. The latest release of Docker v1.12 provides many new features for orchestration and the new Swarm mode made it much easier to deploy Swarm clusters.
With Docker v1.12 services can be scheduled globally – similar to Kubernetes DaemonSet, RancherOS global services or CoreOS global fleet services
Tweet
We can finally completely automate the setup of Docker Swarm monitoring! So, how do you deploy monitoring agents like Sematext Docker Agent or any other service with similar requirements like cAdvisor, logspout etc. to all Swarm nodes with Docker v1.12?
We started with docker service --help and the most relevant new command for this task is docker service create, which now has --mode global option. As most monitoring agents need access to the Docker UNIX socket we have to bind the socket to the Sematext Docker agent container with the service instruction –mount, which is different from the well known Docker syntax for volume mounts, e.g. -v /var/run/docker.sock:/var/run/docker.sock”. We could not find details in the documentation but this readme is ahead and explains all details of the docker create service –mount options. Mounting the Docker UNIX socket is done like this: --mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock. Initially, we used type=volume, which failed 🙁 – the the correct option is “type=bind”. Why? We have to use “bind” when you share files from the filesystem and “volume” for existing data containers/named volumes. If you have your own monitoring and logging, you can skip this, but we use SPM for Docker metrics and Logsene for our Docker logs, so the last bits we need are SPM and Logsene App tokens (generated in SPM/Logsene UI) passed as environment variables to the service. Here is the full command to deploy the agent to all Swarm nodes as global service:
docker service create --mode global --name sematext-agent-docker \
--mount type=bind,src=/var/run/docker.sock,dst=/var/run/docker.sock \
-e SPM_TOKEN=YOUR_SPM_TOKEN -e LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN sematext/sematext-agent-docker
The command outputs the Docker Swarm service ID, which could be used with the docker service command. Great!
If you check the status of the service with docker service ps SERVICE_ID you’ll see sematext-docker-agent getting scheduled for deployment on each node shortly after running the above command.
Now it gets interesting. What happens when we add a new node to the Swarm cluster? Will Swarm deploy the agent there? Let’s add new Swarm nodes:
docker swarm join --token SWMTKN-1-34crsjviacp2bay11xv6yeimdgvrf43olvj6xgy2xztu5bl7m7-9hmczsc5mhlm8ps9u6fkkkkk 188.166.160.246:2377
and check again the service status with docker service ps SERVICE_ID.
Voilà! We see all nodes nodes with the monitoring service – Swarm did indeed deploy Sematext Docker Agent to the new nodes automatically:
As soon as the agent starts on a Docker Swarm node it ships metrics, logs and events to SPM/Logsene and data shows up in real-time charts and reports.
Server overview showing all swarm nodes
Docker Swarm Monitoring Reports (summary for the whole Swarm cluster)
We are excited that a single command can deploy Sematext Docker Agent for the collection of metrics, events and logs from all containers in the Swarm cluster! We really appreciate the work done by Docker for the latest release – tailored to customers needs.
Learning Docker v1.12 features takes some time, but it pays off quickly!
Please check the options for Docker service creation, to define more details such as update and restart policies or reserving resources for your deployed services. We hope this post gets you started faster with the new orchestration features and ease your life when dealing with Docker Swarm monitoring and log management.
Any question or feedback? Get in touch with us via Twitter or e-mail.