Over the last several months we’ve made all kinds of improvements to Sematext Docker Agent (SDA). If you’re not familiar with SDA yet, here it is in a nutshell:
Sematext Docker Agent is a modern, open-source, Docker-native monitoring and log collection agent. It runs as a tiny container on each Docker host and provides automatic collection and processing of Docker Metrics, Events and Logs for all cluster nodes and all auto-discovered containers. It works with Kubernetes, Mesos, Docker Swarm, Docker Datacenter, Docker Cloud, as well as Amazon EC2, RancherOS, and CoreOS.
By default SDA collects all logs from all containers, but we’ve recently added LOGSENE_ENABLED_DEFAULT, a new flag that lets you control if log collection should be enabled by default or not. When set to false, SDA will collect logs only from containers labelled with a LOGSENE_TOKEN=<Logsene app token> that, in addition to enabling log collection, also specifies where logs should be shipped. This is really useful because it:
- Gives each team in an organization full control over log collection for their containers and the routing to their Logsene Apps or Elasticsearch indices. As you can imagine, this is very handy for larger organizations where e.g. one Swarm or Kubernetes cluster is shared by several teams, and each team needs to enable/disable log collection however they see fit and ship logs to different destinations.
- Lets one exclude collection of logs from infrastructure containers whose logs may not contain enough value to be worth collecting.
The figure below illustrates this. There are 3 containers running Nginx, Rancher agent, and MySQL. We want to have explicit control over which container logs are collected and where they are shipped. We want to collect only Nginx logs, and we want to ship it to a specific Logsene app, not the default one set in SDA. We can accomplish that by doing the following
- Set LOGSENE_ENABLED_DEFAULT=false flag in SDA config
- Set Docker label LOGSENE_ENABLED=true for the Nginx container to enable its log collection
- Set Docker label LOGSENE_TOKEN=…. for the Nginx container to specify which Logsene app we want to ship logs
Sematext Docker Agent log routing via container labels – disable log collection by default and enable it only for the Nginx container
We can also inverse the setup and enable log collection for all containers by default, as illustrated in the following figure.
Sematext Docker Agent log routing via container labels (default settings) – collect and ship all logs, either to default Logsene app or to container specific apps.
We’ve also introduced TAGGING_LABELS. This flag lets you enrich your container logs with data extracted from your existing Docker environment variables or labels. Just specify patterns for values to extract, e.g. TAGGING_LABELS=”com.docker.,com.myorg.,role*”. This will add extracted values to your log events as additional fields (you can also think of this as meta-data) and let you easily slice and dice your logs by using values from labels or environment variables. You can, of course, also build custom reports using this data, thus extracting more value and operational insight from your existing data in Sematext Cloud.
Enriched logs with Docker labels
Need a tool that collects your containers Metrics + Events + Logs? Try Sematext Docker Agent, it’s all open-source and can send your data to Sematext Cloud so you don’t have to manage or build out the backend for storing all the monitoring data, alerting, etc. For feature requests, bugs, or PRs, see sematext/sematext-agent-docker.
Check out our log management guide for more useful logging content.