As a Site Reliability Engineer (SRE) or DevOps professional, you are well aware of the importance of observability in ensuring the smooth functioning and performance of your applications. Observing and monitoring your applications can help you identify and resolve issues in real-time, resulting in increased reliability and improved user experience. Logs play a crucial role in this process as they provide detailed information about the activity and behavior of your applications.
One way to improve application observability is by creating log-based metrics. Log-based metrics derive metric data from the content of log entries. They are often used to identify performance issues, diagnose problems, and improve system reliability.
Here are a few examples of log-based metrics:
- User activity: the number of log entries that record user actions or interactions within a system or application
- Request rate: the number of requests, as recorded in the log entries
- Latency: the time taken to respond to a request, as recorded in the log entries
- Shopping cart checkout value over time – average, median, or whatever percentile
In this post, we will show you how to use Sematext Quick Charts to create log-based metrics and improve the observability of your applications. Note, however — and you can see from this last example above — even though logs are typically in the domain of SREs and DevOps and our example below is from the observability realm, logs can be a great source of highly valuable product or business insights and metrics.
Integrating Your Application Logs with Sematext Logs
Shipping your application logs to Sematext Logs is the first step in creating log-based metrics. Sematext Logs is a centralized log management solution that allows you to collect, store, and analyze all your log data in one location. By integrating your logs, you can access all your log data in one place, making it easier to process and extract the metrics you need.
To ship your logs, you’ll need to install the Sematext Agent and then just perform a few clicks in the UI to configure the ship logs it discovered for you via the Logs Discovery.
Once you do this your logs will show up in Sematext. Next, let’s move to the extraction of metrics and show you how to create charts and alerts to monitor your applications.
Creating metrics with Sematext Quick Charts
Quick Charts is a simple, but powerful tool you will use to create charts from individual log event fields in just a few clicks.
Numerous use cases are supported, including:
- Charting of values from a simple non-numeric field
- Charting of values from a numeric field
- Charting of values from a non-numeric field using Logs Pipelines
For the first use case, let’s take an example of an HTTP Server where we created an Apache web server logs App to explore the logs. Let’s assume that we want to see the top clients connected, the top HTTP methods used, and the top paths that are hit.
To create our first Quick Chart you would:
- Go to the Explore view of the Apache Logs App
- Expand one log entry in the Logs table
- Click on the log field you want to visualize, in our case client_ip
Here is what that looks like:
You can choose from a variety of chart types, such as time series charts, line charts, bar charts, and pie charts, to best represent your data. There are also a variety of customization options, such as the ability to change the color and style of your chart, and the rollup method. In our case, we will select a Pie Chart visualization and we will save the chart in our current view, the Explore report.
We can now see the end result within the same report:
The same way we can create the top HTTP methods used and the top paths. For the first we can use the method field:
And we can create a Bar Chart:
Similarly, for the top paths we can use the path field and create a Bar Chart as well:
If we choose to save all the new charts in the Explore report the end result will look like the one below:
As we saw we can easily create and customize as many charts as we like from non-numerical log fields and save them in any report we wish. Using the exact same approach we can chart values from numeric fields, like latency information or other useful data.
Another interesting use case is charting values using Sematext Logs Pipelines. Let’s say we want to extract the HTTP response status code from our log events and let’s use the same Apaches Logs App as we had in our previous example. The response status code can be found in the message field as shown below:
To extract it to a separate field, we can use Log Pipelines:
And create a new Field Extractor processor:
We will use Field Extractor and write a simple Grok pattern to extract the HTTP status code and name it as http_status_code:
The new field will be now visible in our log events, so we can now easily use the same approach that we followed in our previous example, to create a quick chart:
In this case, it will be a Bar Chart:
The end result with our new HTTP Status Code chart is shown below:
Creating Custom Dashboards
Creating custom dashboards in Sematext Cloud allows you to get a complete overview of your applications in one place. This feature enables you to combine your charts, logs, and alerting rules into a single view, making it easy to monitor and analyze the performance and behavior of your applications.
Creating a custom dashboard can be done during Quick Chart creation, simply click on the “Chart Target Location” and select “Add New Dashboard”.
You can also choose to display your dashboard in full screen, making it easier to view on a larger screen or monitor. Last, using Sematext Scheduled Reports you can receive emails with the dashboards embedded in them at a specified frequency.
Summary
In summary, Sematext Logs provides a solution for improving the observability of your applications through log-based metrics. Sematext Quick Charts can be used for visualizing both numeric and non-numeric values, while Sematext Logs Pipelines enable you to customize your log events according to your specific requirements.