Docker Logging: A Complete Guide
Get Started with Docker Logging from Scratch
When it comes to developing applications, logging is probably one of the most important parts to get right from a DevOps point of view. Log management helps the DevOps team tackle troubleshooting faster and easier, identify patterns, spot opportunities, and ensure compliance and security.
Check our full Log Management Guide to find out more about how logging works, why it’s important and how you can set it up for yourself.
Docker Logging: Why are Logs Important when using Docker?
The importance of logging applies to Dockerized applications as well, maybe even more so, as you have to deal with three types of logs – container logs, daemon logs, and host logs. All of them are vital in troubleshooting errors and issues when using Docker.
Docker logging is the process of handling log events generated by Docker and the applications that live inside the containers.
In this article, we will refer to Docker logging in terms of container logging as these are specific to Docker and are generated by the containers. Later on, we’ll touch upon Docker daemon logs as well – logs that refer to the Docker service generated by Docker itself.
To recap: What is a Docker Container?
A container is a standard unit of software that packages up an application to deploy it from host to host. It keeps it isolated which in return enables it to work the same regardless of the differences in infrastructure. A Docker container is ultimately a runtime instance of an image that’s like a template for creating the environment you want. An image is an executable package that includes everything that the app needs to run uniformly: the code, libraries, configuration files, and environment variables.
Containers also allow to break down applications into microservices – multiple small parts of the app that can interact with each other via functional APIs. Each microservice is responsible for a single feature so development teams can work on different parts of the app at the same time. That makes building an app easier and faster.
Articles related to Docker Containers:
- Docker Container Monitoring and Management Challenges
- Docker Container Performance Metrics to Monitor
- Docker Container Monitoring Open Source Tools
- Docker Container Monitoring with Sematext
How is Docker Logging Different?
Most conventional log analysis methods don’t work on containerized logging – troubleshooting becomes more complex compared to traditional hardware-centric apps that run on a single node and need less troubleshooting. You need more data to work with so you must extend your search to get to the root of the problem.
Containers are Ephemeral
Docker application logs are stored inside the container. However, Docker containers do not store data persistently, so when you shut down a container all the data that’s inside is wiped out by default – unless you move it elsewhere.
That’s where logging comes in: you can collect the data with a log aggregator and store them in a place where they’ll be available after the container shuts down.
Containers are Multi-Tiered
This is one of the biggest challenges to Docker logging. However basic your Docker installation is, you will have to work with two levels of aggregation. One refers to the logs from the Dockerized application inside the container. The other involves the logs from the host servers, which consist of the system logs, as well as the Docker Daemon logs which are usually located in /var/log or a subdirectory within this directory.
A simple log aggregator that has access to the host can’t just pull application log files as if they were host log files. Instead, it must be able to access the file system inside the container to collect the logs. Furthermore, your infrastructure will, inevitably, extend to more containers and you’ll need to find a way to correlate log events to processes rather than their respective containers.
- Top 10 Docker Logging Gotchas
- Docker Logging Best Practices (coming soon)
Get Started with Docker Container Logs
When you’re using Docker, you work with two different types of logs: daemon logs and container logs.
What are Docker Container Logs?
Docker container logs are generated by the Docker containers, thus needing to be collected directly from said containers. Any messages that a container sends to STDOUT or STDERR is logged then passed on to a logging driver that forwards them to a remote destination of your choosing.
What is a Logging Driver?
Logging drivers are Docker’s mechanisms for gathering data from running containers and services to make it available for analysis. Whenever a new container is created, Docker automatically provides a logging driver if another log driver option is not specified. At the same time, it allows you to implement and use logging driver plugins if you would like to integrate another logging tool.
The logging driver enables you to choose how and where to ship your data. The default logging driver is a JSON-structured file located on local disk: /var/lib/docker/containers/[container-id]/[container-id]-json.log.
However, you have other log driver options as follows:
- logagent: A general purpose log shipper. The Logagent Docker image is pre-configured for log collection on container platforms.
- syslog: Ships log data to a syslog server. This is a popular option for logging applications.
- journald: Sends container logs to the systemd journal.
- fluentd: Sends log messages to the Fluentd collector as structured data.
- gelf: Writes container logs to a Graylog Extended Log Format (GELF) endpoint such as Graylog or Logstash.
- awslogs: Sends log messages to AWS CloudWatch Logs.
- splunk: Writes log messages to Splunk using HTTP Event Collector (HEC).
- gcplogs: Ships log data to Google Cloud Platform (GCP) Logging.
- logentries: Writes container logs to Rapid7 Logentries.
- etwlogs: Writes log messages as Event Tracing for Windows (ETW) events, thus only available on Windows platforms.
How to Configure the Docker Logging Driver?
When it comes to configuring the logging driver, you have two options:
- setup a default logging driver for all containers
- specify a logging driver for each container
In the first case, the default logging driver is a JSON file, but, as mentioned above, you have many other options such as logagent, syslog, fluentd, journald, splunk, etc. You can switch to another logging driver by editing the Docker configuration file and changing the log-driver parameter or using your preferred log shipper.
Alternatively, you can choose to configure a logging driver on a per-container basis. As Docker provides a default logging driver when you start a new container, you need to specify the new driver from the very beginning by using the –log-driver and –log-opt parameters.
docker pull sematext/logagent docker run -d --restart=always --name st-logagent -e LOGS_TOKEN=YOUR_LOGS_TOKEN -e LOGS_RECEIVER_URL="https://logsene-receiver.sematext.com" -v /var/run/docker.sock:/var/run/docker.sock sematext/logagent
This will start sending all container logs to Sematext.
How to Work with Docker Container Logs?
Docker has a dedicated command which lists container logs. The docker logs command. The flow will usually involve you checking your running containers with docker ps, then check the logs by using a container’s ID.
docker logs <container_id>
This command will list all logs for the specified container. You can add a timestamp flag and list logs for particular dates.
docker logs <container_id> --timestamps
docker logs <container_id> --since (or --until) YYYY-MM-DD
What you’ll end up doing will be tailing these logs, either to check the last N number of lines or tailing the logs in real time.
The –tail flag will show the last N lines of logs:
docker logs <container_id> --tail N
Using the –follow flag will tail -f (follow) the Docker container logs:
docker logs <container_id> --follow
But what if you only want to see specific logs? Luckily, grep works with docker logs as well.
docker logs <container_id> | grep pattern
This command will only show errors:
docker logs <container_id> | grep -i error
Once an application starts growing, you tend to start using Docker Compose. Don’t worry, it has a logs command as well.
This will display the logs from all services in the application defined in the Docker Compose configuration file.
Get started with Docker with our Docker Commands Cheat Sheet!
What about Docker Daemon Logs?
Docker daemon logs are generated by the Docker platform and located on the host. Depending on the host operating system, daemon logs are written to the system’s logging service or to a log file.
If you were to collect only container logs you’d get insight into the state of your services. However, by traditional logging methods, you also need to be aware of the state of your Docker platform, which is what Docker daemon logs are for. They paint a clear picture of your overall microservices architecture.
On that note, the Docker daemon logs two types of events:
- Events generated by the Docker service itself
- Commands sent to the daemon through Docker’s Remote API
Where are Docker Daemon Logs Located?
Depending on your Operating System, the Docker daemon log file is stored in different locations. Here are a few examples:
- Ubuntu (old, using upstart) – /var/log/upstart/docker.log
- Ubuntu (new, using systemd) – sudo journalctl -fu docker.service
- Boot2Docker – /var/log/docker.log
- Debian GNU/Linux – /var/log/daemon.log
- CentOS – /var/log/daemon.log | grep docker
- CoreOS – journalctl -u docker.service
- SUSE – journalctl -u docker.service
- Fedora – journalctl -u docker.service
- Red Hat Enterprise Linux Server – /var/log/messages | grep docker
- Amazon Linux AMI – /var/log/docker
- OpenSuSE – journalctl -u docker.service
- OSX – ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/log/docker.log
- Windows – Get-EventLog -LogName Application -Source Docker -After (Get-Date).AddMinutes(-5) | Sort-Object Time.
If you start the Docker daemon in the terminal using the docker daemon command, Docker writes the log file directly to the terminal.
Docker Log Management: Logging Tools & Software
Logging is a key part of gathering insight into the state of your infrastructure, but only if it’s analyzed. However, log data comes in huge volumes so doing it manually would be like looking for a needle in a haystack. Which is why you need a log data analysis platform. You can opt for open-source solutions or commercial software to get the most out of your Docker logs.
Open-Source Log Analysis Solutions
With open source solutions, you need an expert team at the ready to handle everything stack-related, from setup to configuration, providing the infrastructure, maintenance, and management.
The most popular open source log analysis software is Elastic Stack (formerly known as ELK Stack). It’s a robust platform comprising three different tools – Elasticsearch to store log data, Logstash to process it, and Kibana to visualize log data.
Read more on Elasticsearch on Docker:
- Docker Logs to Logstash/ELK (coming soon)
For more information on Elasticsearch, check out our Elasticsearch Complete Guide.
Commercial Log Analysis Tools: Logging as a Service
If you don’t have the resources to deal with Docker log data on your own, you can reach out to vendors who provide “logging as a service” as part of a full log management solution. You only need to point out the Docker logs and they’ll take over managing your log data from collection to storage, analysis, monitoring, and presentation.
Sematext as a Cloud Log Management Solution for Docker Logs
Sematext Cloud is an all-in-one solution that provides hassle-free log management and analytics of your stack. By handling your Docker logs, events and metrics Sematext helps you spot and troubleshoot issues easier, gather better insights, and spot opportunities for both your platform and Dockerized applications.
Read more on Sematext Cloud here.
Also read: Docker Container Monitoring with Sematext