Nowadays, users expect 24/7 availability and very short MTTR (Mean Time To Repair). Logs contain valuable information about the health of your applications, can point you directly to application errors or uncover security problems in your infrastructure. Therefore, Log Management is one of the most important tasks for DevOps Engineers and IT Administrators.
However, with logs being stored in different formats and locations across your infrastructure, dealing with logs manually would be quite the Sisyphean task. That’s where log management solutions come into play. They give you a real-time visual picture of how your applications and systems are being used by your users and provide deeper insights and opportunities you can leverage to improve the code quality, increase efficiency, mitigate risks and offer better customer experience.
In this post, we’re going to explore how log management solutions work, how to choose one and share with you a list of the best open source tools to help you handle your logs efficiently.
How do Log Management Solutions Work?
The procedure of transforming logs into operational intelligence requires a few steps. Applications generate logs, which are collected, parsed and shipped by log shippers to central log storage. The central log storage often includes a search index for fast search and a component that manages log retention. Search and visualization tools provide a user interface for DevOps Engineers while Alerting tools make sure the responsible team members are proactively notified about any critical or suspicious log messages.
Without further ado, this is how log management solutions work, step-by-step:
Integration of various log sources
Applications must provide useful log messages. Typically a logging library is used by programmers to generate logs in an unstructured (text) or structured format (key/value log format or JSON), writing to console, log files, Syslog or Windows Event stream. Linux Server processes write logs to /var/log via syslog or store logs via journald in the system journal. On Windows, logs are stored as Windows Events. Routers, switches an IoT devices support typically log forwarding to a syslog server. Containers write logs to the console and log drivers handle the log storage or forwarding.
Log Parser. Unstructured log messages must be parsed to structure logs at least into timestamp, severity and message field. Depending on the log content, it is very useful to extract more fields for the analysis or visualization.
Secure log shipping to a central log storage.
Learn about the differences between logging directly from the application or using a dedicated log shipper, from our post on Logging Libraries vs. Log Shippers, where we discuss the pros and cons of each approach.
The log storage must be scalable to deal with high event frequency during peak hours and always growing log volume. Data is typically kept for a limited time. A challenge is here to manage different retention times for different log sources. In many cases, log storages keep an index for fast log search.
The idea of centralized log management is to have the capability to search for logs in one central place. A good example is journald as logging server and journalctl as a command line tool to search logs. Other systems provide web UI’s or windows applications to view and search log events.
Tools like Kibana or Grafana help to visualize log statistics, e.g. to show a timeline of error events or the total amount of logs over time. Charts could be created for any information provided in structured logs – e.g. visualization of web server logs can tell you a lot about user behavior.
Nobody has time to check logs frequently for suspicious messages. Therefore alerting to ChatOps tools like Slack, PagerDuty are important to get notified in real-time when the log management system detects any suspicious log behavior.
The Best Free Log Management Tools & Software: Pros & Cons Comparison
In this part, we’re going to share the advantages and cons of different open source log management solutions and how each works.
Here’s our list of the best open source log management tools:
1. System Journal
Did you know that most Linux systems have a complete log management solution on board? Distributions based on systemd contain journald and journalctl.
systemd-journald – All Linux system processes write logs to the system journal, which is managed by journald. The system journal is local log storage.
journalctl is the command line client to display logs with various filter options like time, system unit or any other field stored in the log event. For advanced searches, it is possible to pipe the output to grep, which makes it easy to apply complex search expressions to journalctl output.
The journalctl client is not only useful for log search, but it also provides various other functions such as management of the system journal storage.
Journal-upload is a service to forward log events to a remote journald instance. Configuring journal-upload on all your Linux machines forwarding log events to a central log server is the best way to centralize logs. Then you can use journalctl on the central log server for log search. Another interesting approach is shipping journald logs via log shippers into the Elastic Stack to benefit from Elastic Stack features.
- Available on most Linux systems
- Configurable log storage
- High-performance log collection
- Option to centralize logs
- No UI, only Linux console
2. Elastic Stack (formerly known as ELK Stack)
The Elastic Stack contains most of the tools for a log management solution:
- Log shippers such as Logstash and Filebeat
- Elasticsearch as a scalable search engine
- Kibana as UI to search for logs or visualizations
- Alerting for logs (only available under the commercial Elastic license)
- Security (partially free to use, however under Elastic license)
Starting with ELK is easy. However, you should consider operational complexity at a larger scale and the potential license catch. To avoid the license catch, you might look into X-Pack alternatives, which shows open-source and commercial alternatives or you start with Amazon’s Open-Distro for Elasticsearch, which is completely Apache 2 licensed and contains important features like security, SQL, and monitoring.
Due to the popularity of the Elastic Stack, we can find many specialized flavors of Elastic Stack, e.g.
- ElastiFlow is a set of Docker containers to monitor networks (Netflow, SFlow) by providing mainly very complex Logstash configurations and Kibana dashboards
- Wazuh integrating log sources like OSSec and Suricata and Kibana plugins
- Graylog 2 having a management UI and many log source integrations for switches and routers. The management UI comes to some overhead of running one or more MongoDB instances in parallel to the Elasticsearch cluster.
Due to the complexity of the Elastic Stack, we can also find various companies providing Elasticsearch consulting or SaaS offerings for hosted Elastic Stack such as Elastic Cloud, Amazon Elasticsearch Service or specialized to the log management and monitoring use case Sematext Cloud.
- Scalable search engine as log storage
- Mature log shippers
- Web UI and visualizations in Kibana
- Complex to manage at a larger scale
- Limited security and alerting features using the Apache 2 licensed Elastic Stack
3. Grafana Loki
What is Grafana?
Grafana is a monitoring tool to visualize time series data. Initially, the project focused on the visualization of metrics. It started as an early fork of Kibana. Unlike Kibana, Grafana did not stick to Elasticsearch as the only data source. There was for some time no support for Elasticsearch. The history of the project might be the reason that Grafana is still missing some log search features. Grafana supports several time-series data stores such as InfluxDB or Prometheus, and many data sources are available as plugins. The web UI is secured with name and password and it has no limits on the user management. Grafana server provides API’s to manage users, dashboards and data sources. Because the tool can connect to many data sources, includes multi-user support and beautiful visualizations many DevOps engineers. Grafana is not even missing simple alerting features and could be considered as a complete open-source solution together with good log storage. So let’s have a look at Grafana Loki, which complements Grafana with a log shipper and log storage.
What is Grafana Loki?
The Grafana Loki stack contains 3 components:
- Grafana user interface
- Promtail – The Loki log shipper promtail collects logs from Kubernetes pods and attaches the same labels as Prometheus does for monitoring. The tool is very much limited to Kubernetes container as log source. Having the same labels attached to logs and metrics makes it easier to correlate logs and metrics – for sure a convenient solution for Prometheus users.
- Loki Server provides scalable log storage based on CNCF Cortex engine – a multi-tenant Prometheus time-series storage.
Here are some interesting difference compared to other log management tools:
- promtail does not parse logs
- Loki does not index logs for full-text search
These differences might be a limiting factor for log search and visualization. The Loki plugin for the fluentd log shipper is available and is an option to collect data from other log sources via fluentd.
- Fast and scalable log storage
- Integrates well with CNCF Kubernetes and Prometheus projects
- Makes Grafana a tool for metrics and log correlation
- Limited full-text search and log analytics
- No alerting for logs, work in progress
- Very young project and potentially not mature
Originally a syslog daemon, rsyslog (rocket-fast system log for processing) has evolved into a free general-purpose logging tool that can read data from multiple sources, parse or enrich it, buffer it, and finally ship it to various destinations. It implements basic syslog protocol and extends it with content-based filtering, flexible configuration options, advanced filtering capabilities and adds new features such as using TCP, SSL, and RELP for transport. It offers high-performance, high security and modular design.
If you want to learn more, download our free e-book on how to use ryslog to collect and parse data and/or go through the following related articles:
- Recipe: rsyslog + Redis + Logstash
- Recipe: rsyslog + Elasticsearch + Kibana
- Recipe: How to integrate rsyslog with Kafka and Logstash
- Recipe: Apache Logs + rsyslog (parsing) + Elasticsearch
- Monitoring rsyslog’s Performance with impstats and Elasticsearch
- Monitoring rsyslog with Kibana and SPM
- Rsyslog 8.1 Elasticsearch Output Performance
- Structured Logging with Rsyslog and Elasticsearch
Logstash is a log collection and processing engine that comes with a wide variety of plugins that enable you to easily ingests data from various sources, transform and forward it to a defined destination. It’s part of the Elastic Stack along with Elasticsearch and Kibana, which is why it’s most often used to ship data to Elasticsearch.
If you want to understand better how Logstash works, check out our Logstash Tutorial, as well as other related posts:
- 5 Logstash Alternatives
- Elasticsearch Ingest Node vs Logstash Performance
- Handling Multiline Stack Traces with Logstash
- Recipe: Reindexing Elasticsearch Documents with Logstash
- Replaying Elasticsearch Slowlogs with Logstash and JMeter
- Logstash Performance Monitoring
Fluentd is a high-performance open-source data collector that enables you to implement a unified logging layer. More specifically, it structures data as JSON to allow you to bring together all aspects of log data processing: collecting, buffering, and outputting data across various sources and destinations.
A good Logstash alternative, Fluentd is a favorite among DevOps, especially for Kubernetes deployments, as it has a rich plugin library, uses little system resources, provides meaningful metadata, and is very reliable against data loss.
Filebeat is a lightweight log shipper installed on hosts for forwarding various types of data to the Elastic Stack for further analysis. It can send them either to Logstash or directly to Elasticsearch for advanced processing. Filebeat is not a replacement for Logstash, as it can’t transform data into a meaningful and structured set of fields. Instead, it was designed to make up for Logstash’s significant memory usage by having a small footprint, handling large volumes of data, and supporting SSL and TLS encryption. Filebeat can and should, in most cases, be used together with Logstash.
- Using Filebeat to Send Elasticsearch Logs to Sematext Logs
- How to ship Kibana Server Logs to Elasticsearch
Logagent is a modern, lightweight, and open-source log shipper featuring extensible log parsing, on-disk buffering, secure transport and bulk indexing to Elasticsearch or Sematext Cloud.
As it uses few system resources, it’s suitable for deploying on edge notes and devices, while its ability to parse and structure logs make it a great Logstash alternative. Logagent is designed to be very easy to use even for those who haven’t used a log shipper before.
Learn more about Logagent from our posts on:
- Better Observability with New Container Agents
- Logagent Meets Apache Kafka
- The New Version of Logagent Enriches Container Logs with Metadata and GeoIP
- Shipping data to AWS Elasticsearch with Logagent
What tools will you use?
There are some great open-source log management solutions out there that you can choose from, but it depends on your particular specifications and even personal preferences on which one suits your use-case best.
If you need help deciding or any support regarding logging, we at Sematext offer Logging Consulting, so feel free to reach us out.
Similarly, if you’re looking for a solution to avoid the hassle of running the Elastic Stack (formerly known as ELK Stack) on your servers, check out Sematext Logs, our Managed ELK Solution (in the cloud or on-premise), which works with all the shippers we covered above.