Innovative Docker Log Management

[ Note: Click here for the Docker Monitoring webinar video recording and slides. And click here for the Docker Logging webinar video recording and slides. ] ——- In the dynamic world of “Microservices” the traditional method of static setups for log routing and parsing doesn’t work very well; in fact, it creates additional complexity and resource usage.  This, in turn, reduces the number of microservices that could run on a single server.  Sematext has come up with a better method. The integrated log management functions in Sematext Docker Agent support the microservice approach by reducing setup complexity, startup time and minimizing the utilized resources. Sematext Agent for Docker collects metrics, events and logs and runs in a container on every Docker Host. In addition to standard log collection functionality, we recently integrated the automatic log format detection and field extraction for Container Log Messages. The processing is hooked into the stream from the Docker API where logs are collected to the log indexing service of our centralized logging tool, Logsene.  This means that — and if you’ve dealt with logs before you’ll know this is huuuge — there’s no set-up of syslog with Docker log drivers, there is no routing to a heavy Logstash process for parsing, and there is no maintenance of Elasticsearch to keep the logs, or even a need to run your own Kibana! SPM for Docker and Logsene provides everything out of the box! There are many ways to collect logs from Docker (you can learn about that in our Docker Logging Webinar); so what is the advantage of using Logsene for Docker Logs? Let me show you… For starters, let’s look into the details of the new Log Management functionality now available to Docker users:

1. Setup – Log specific options for SPM for Docker
2. Automatic Format Detection and Parsing for Container Logs
3. Correlation with Metrics
4. Alerting and Anomaly Detection for Container Logs
5. Visualization with Logsene and Kibana
6. Search indexed logs from the UI or the command line and process it with UNIX tools

1. Setup – Log specific options for SPM for Docker

First of all, we made it super simple to collect logs along with Docker events and Docker metrics using SPM for Docker. There is not much to the installation, simply add “-e LOGSENE_TOKEN=your-logsene-token” to the Docker run command of Sematext Docker Agent and you get all logs from Docker containers into Logsene.  Just like that! Not all logs might be of interest, which is why we provide the capability to whitelist or blacklist log outputs by image or container names. The relevant parameters are:

• -e MATCH_BY_NAME – A regular expression to white list container names
• -e MATCH_BY_IMAGE – A regular expression to white list image names
• -e SKIP_BY_NAME – A regular expression to black list container names
• -e SKIP_BY_IMAGE – A regular expression to black list image names
• -v PATH_TO_YOUR_FILE:/etc/logagent/patterns.yml – optional, custom log parser definitions for your applications

A full example to collect Metrics, Events, and Logs:

docker run --name sematext-agent 
-e HOSTNAME=$HOSTNAME 
-e SPM_TOKEN=YOUR_SPM_TOKEN 
-e LOGSENE_TOKEN=YOUR_LOGSENE_TOKEN 
-e MATCH_BY_IMAGE=”nginx|mysql|mongodb|myapp” 
-v /var/run/docker.sock:/var/run/docker.sock 
sematext/sematext-agent-docker

2. Automatic Parser for Container Logs

Docker logs are console output streams from containers. This data might be a mix of plain text messages from start scripts and structured logs from applications.  The problem is obvious – you can’t just take a stream of log events all mixed up and treat them like a blob.  You need to be able to tell which log event belongs to what container, what app, parse it correctly, etc. SPM for Docker analyzes the line format and converts it to JSON to make all fields available and extracts named fields from plain text logs. Traditionally it was necessary to use log shippers like Logstash, Fluentd or rsyslog to parse log messages — but these setups are typically set up to be very static for each input source. That would not work well in the dynamic microservice world! We have seen people juggling the syslog driver, configurations for parsers, log routing and more! That’s why we’ve integrated automatic format detection into SPM for Docker using logagent-js to take away this pain – and the waste of resources – both computing and human time that goes into dealing with such things! This integration has a low footprint, doesn’t need retransmissions of logs to external services, and it detects log types for the most popular applications and generic JSON and line-oriented log formats out of the box!

Example: Apache Access Log fields generated by SPM Docker Agent

For example, SPM Docker Agent can parse logs from official images like:

• MongoDB, MySQL, Redis, …
• NGINX and Apache logs, …
• Any JSON output with special support for Logstash or bunyan
• Plain text messages with or without timestamps in various formats
• Any format specified in a custom patterns definitions file by the mount option: “-v /mypatterns/patterns.yml:/etc/logagent/patterns.yml”

The component for detecting and parsing log messages — logagent-js — is open source and contributions for even more log formats are welcome.

3. Correlation with Metrics

Having logs stored in Logsene provides the unique capability to correlate performance metrics like CPU usage, Memory usage, Network and Disk I/O with log messages – with a single click you can discover log messages around any spike in the performance metrics via SPM!  If you troubleshoot performance and other issues in your organization, you know how much time quick access to metrics AND logs AND events can save you!

Correlation of logs and metrics for NGINX

4. Alerts on Docker Logs

Logsene provides the capability to define alerts and use anomaly detection to notify you about logs you most likely want to know about (like errors or warnings, security issues etc.) Please check the related blog posts on this topic.

5. Visualization using Kibana

We automatically provide 1-click access to Kibana for all Logsene applications. So if your logs include numeric data — or if you like to generate statistics about log frequency, message types or top errors — feel free to create your Kibana dashboards in Logsene.

6. Search indexed logs from command line and process it with UNIX tools

If you live in the console and love your command line, please check Logsene command line interpreter.  There is no need to switch to the web UI for search, and, even better, no need to use JSON queries. Logsene + awk + grep + sort + uniq + … whatever you need for your task.

Get started in 5 Minutes

You’ve just plowed through a lot information — congratulations!  If you like what you just read you can sign up here, run SPM for Docker and see for yourself which of the functionalities are most interesting for you. Small startups, startups with no or very little outside investment money, non-profit and educational institutions special pricing – just get in touch with us.  If you’d like to help us make SPM and Logsene even better, we are hiring! Check out our Log Management Guide for more useful Log Management content.