Logging vs Monitoring: How are They Different & Why You Need Both

By Chris Tozzi

Logging or monitoring? If you deploy and manage an application, these are the two key techniques available to you for helping to ensure that the application meets availability and performance expectations. One of them is Application Performance Management, or APM, though you can also find it referred to as ‘Application Performance Monitoring’ or simply ‘monitoring’. The other is log analytics and management or just ‘logging’.

It may be tempting to assume that you need only one or the other in order to keep your application running healthfully. But that is not the case. To understand why, let’s explore the differences and similarities between logging and monitoring, and why you need both to gain the greatest level of visibility.

What Is Logging and Why It Is Important?

Logging is the practice of managing all of the log data produced by your applications and infrastructure.

Application logging can be broken down into several smaller processes, including:

• Log aggregation or log shipping, which refers to the process of collecting logs from disparate sources and moving them to a central location.
• Log storage and archiving, which involves establishing and implementing the right strategy for storing log files over the long term, as well as deleting them after a retention period has ended.
• Controlling the quality of log data by addressing missing entries, out-of-sync timestamps, redundant logs, and so on.
• Ensuring the security and privacy of logs, which sometimes contain highly sensitive information or data that is subject to regulations such as the GDPR. Read our GDPR logging best practices to learn more about this topic.
• Log enrichment, which entails adding more information to logs that helps to contextualize data, such as inserting geographic data based on IP addresses.
• Log analysis, or the process of making sense of log data using log analyzers. Some folks might consider log analysis to be part of monitoring more than logging — we’ll explain why in a moment.

The extent of your logging operation will vary depending on factors such as how many logs you have to manage, how widely distributed your infrastructure is (highly distributed infrastructures make log aggregation more challenging, as do hybrid infrastructures where log data has to move between on-premises and cloud-based environments in a secure way) and which types of log data are available to you. Some types of services generate fewer log data than others; for example, cloud-based serverless computing environments create minimal log data, while an on-premises data center typically churns out quite a bit.

To go further into logging basics, resources, and tips & tricks, have a look at our log management guide our download our eBook on Log Management & Analytics. If you’re already convinced logging is for you, then all you need to do is learn what are the logging best practices you should swear by to get the most out of your logs and find the right solution for your use case. If you’re into that, read our comparisons of the best cloud logging services, log management tools, or log analysis software available today.

What Is Monitoring and Why It Is Important?

Put simply, monitoring is the art and science of ensuring that an application both remains available and responds to user requests within an acceptable amount of time. More broadly speaking, monitoring can also involve goals such as optimizing code or reducing infrastructure costs, but we’ll stick to the simpler definition for now.

Monitoring tools help to achieve these goals by monitoring metrics such as whether an application or service is responding at all, how quickly it is responding, and how much memory, network bandwidth, or CPU time it is consuming.

Monitoring encompasses a variety of different techniques and approaches. Depending on which types of monitoring tools you use, and which features you leverage within them, your monitoring strategy could include processes such as:

• Real user monitoring, or RUM, which uses actual user data to assess the health of an application. Real user monitoring is a type of monitoring technology for digital businesses that analyzes customers’ digital experiences by looking at how online visitors are interacting with a website or application, analyzing everything from page load events to AJAX requests to Apdex Score and frontend application crashes.  Real user monitoring is also known as End-user experience monitoring (EUEM) or Digital experience monitoring (DEM) and as such, it’s a part of application performance monitoring. We discussed the differences between them in our blog post about APM vs. RUM. If you’re already a RUM pro and looking for the right solution, check out this pros and cons comparison of the best real user monitoring tools.
• Synthetic monitoring, which relies on synthetic interactions to monitor an application. If you’re new to this, learn the basics from our synthetic monitoring guide and check out this review of the best synthetic monitoring tools available today.
• Network monitoring, which analyzes network traffic as a way of monitoring application availability and performance.
• Distributed tracing, which is helpful for monitoring the execution of an application at the level of the code itself, rather than the infrastructure that hosts it. Learn more in OpenTracing: Distributed Tracing Emerging Industry Standard.

Because there are so many different ways to approach monitoring, it’s best to think of it as an overall strategy for managing application performance and availability, rather than a specific method.

Logging vs. Monitoring

What’s the difference between logging and monitoring, then?

The simple answer is that they serve two quite distinct purposes. Monitoring helps you manage application performance, while logging is all about managing the data inside logs. This ensures that, when it comes time to troubleshoot or optimize your applications, you have all the tools and data you need to have a complete and accurate understanding of your infrastructure and applications’ availability and performance.

Thus, logging and monitoring are closely related because log data is one of the critical data sources available to you for performing application monitoring. While logs are not the only source of valuable application metrics for your monitoring tools (you can also leverage data like stack traces, as noted above), logs created by applications, servers, network infrastructure, and more offer detailed windows into the ways in which an application is performing. You can use separate tools, but having logging and monitoring in a single platform makes troubleshooting critical issues much faster and much simpler. One such tool is Sematext Cloud. It supports logging and monitoring, providing you the holistic set of functionality required to perform both tasks efficiently.

Put another way: Without effective logging, you can’t do monitoring efficiently.

This does not mean, however, that the sole purpose of logging is to support performance monitoring.

Logging has other uses besides troubleshooting and performance monitoring. Properly structured logs help you to:

• Respond Better to Data Breaches and Other Security Incidents 
• Ensure Compliance with Security Policies, Regulations & Audits 
• Understand Online User Behavior

For security and audit requirements you may want to create an organization or department-wide logging and monitoring policy for each of these. Having such policies makes it easier and more efficient for your team to decide what to log, where logs are stored and for how long, whether they need to be archived for audit purposes, encrypted, and so on.

Just remember to log mindfully. Don’t get carried away with logging everything and creating so much noise that you can’t spot data that is actually the most valuable to you. Read our article about common best practices to reduce log volume with Logagent to learn more about that.

Conclusion: Why You need Both Logging And Monitoring

Without properly managed logs that make data from across an environment available to application monitoring tools, you’ll lack a critical source of data for monitoring. You’ll also likely run into other problems that extend beyond the realm of application performance monitoring, such as the risk of non-compliance for failure to manage log data effectively.) And without monitoring, you will not be able to make sense of log data in order to understand how your application is performing; nor will you be able to troubleshoot problems and prevent problems from recurring.

If you’re looking for a tool to help with both monitoring and logging, take a look at Sematext Cloud, our cloud monitoring tool that monitors logs and correlates them with metrics to help ensure peak application performance and reduce downtimes.

Bio

Chris Tozzi

Chris Tozzi has worked as a journalist and Linux systems administrator. He has particular interests in open source, agile infrastructure, and networking. He is Senior Editor of content and a DevOps Analyst at Fixate IO. His latest book, For Fun and Profit: A History of the Free and Open Source Software Revolution, was published in 2017.