In the world of modern business, data drives decision-making. Every interaction, every transaction, and every click generates a series of data in the form of logs. These logs, often seen as plain text records, have the potential to unlock valuable business insights when analyzed correctly.
In How to Create Log-Based Metrics to Improve Application Observability, we described the process of creating log-based metrics to improve application observability using Sematext Cloud. In this article, we will explore the power of extracting business insights from logs and how it can transform raw data into actionable intelligence. We’ll use an e-commerce company for our examples.
The E-commerce Landscape
Consider an e-commerce company that operates an online shop. This company has several components:
- Employees: Sales representatives, customer support agents, and other personnel for serving customers
- Products: A large range of products that customers can purchase
- Customers: Customers place orders, make payments, and engage in various transactions
The Tech Stack
The online shop is built on a proven infrastructure to handle the demands of e-commerce operations efficiently. In short, the website is powered by a CMS, it’s hosted on a dedicated Apache web server, and integrates with a secure payment gateway.
The backend relies on the MySQL database, which is responsible for storing critical data, including product details, customer information, and transaction records.
Monitoring and Logging Tools
To monitor the infrastructure and the application services, the company is using Sematext Cloud with several available monitoring and logging integrations. To collect and analyze logs, the company decided to use the Sematext MySQL Logs integration. This integration has been enabled by simply creating a MySQL Logs App and installing the Sematext Agent. Everything is a few clicks away in the UI to configure the ship logs it discovered for you via the Logs Discovery.
Extracting Business Data from Logs
In our example here, the MySQL logs contain data that is valuable for business. Here are a few examples of MySQL logs from which we will be extracting useful business data.
231009 14:24:19 13 Query INSERT INTO employees (employeeNumber, lastName, firstName, extension, email, officeCode, jobTitle) VALUES ( 1100, 'Smith', 'Jane', 'x123', 'jane.smith@example.com','1', 'Sales Representative');
231009 10:22:14 13 Query INSERT INTO customers (customerNumber, customerName, contactLastName, contactFirstName, phone, addressLine1, addressLine2, city, state, postalCode, country, salesRepEmployeeNumber, creditLimit) VALUES (520, 'CR ABC Company', 'Ivan', 'Maric', '123-456-7890', '123 Main St', NULL, 'Split', NULL, '90001', 'Croatia', 1100, 75000.00)
231009 15:11:08 7 Query INSERT INTO orders (orderNumber, orderDate, requiredDate, status, comments, customerNumber) VALUES (10101, '2023-10-06', '2023-10-10', 'In Process', 'Urgent order', 101);
231009 15:12:35 7 Query INSERT INTO orderdetails (orderNumber, productCode, quantityOrdered, priceEach, orderLineNumber) VALUES (10101, 'S10_1678', 10, 81.35, 1), (10101, 'S10_1949', 5, 204.16, 2);
In your case it may be your own application that logs valuable business data, not MySQL. In either case, we want to extract the pieces of data that we will then use to create business dashboards, alerts, etc. To perform this data extraction we will use Logs Pipelines. In general, Logs Pipelines are really handy as they can help you structure your logs, drop noisy logs, and a lot more.
Once logs are ingested, Logs Pipelines allows you to define parsing rules using the Pipeline Builder. You can add one or more Processors and use regular expressions or built-in parsers to extract structured data from raw log messages. The Field Extractor Processor is used to extract fields using a grok pattern. For example, if your logs contain information about customer transactions, you can define a grok pattern to extract fields like “country”, “shipping status” and “price”.
As we can see below, in our imaginary online shop’s log monitoring setup there are multiple Processors used to extract a number of valuable fields from the logs, like “order_number”, “product_code”, “price”, “customer_name”, “country”, “employee_number”, “amount” and so on.
These fields are now part of our structured log events and, as you can see below, both numeric and non-numeric values can be extracted:
Charting Extracted Business Data
We have now extracted a couple of interesting bits of information from the original “message” field:
customer_name: the name of the customercountry: the country of the customeremployee_number: the Account Manager for the customeramount: the available credit from the customershipping_status: the status of the shipping: shipped, in progress or canceledcustomer_id: a unique identifier for the customerpayment_date: the date of paymentpayment_amount: checkout amount of the orderorder_number: the number of the orderproduct_code: a unique identifier for each productprice: price of the product
Now what? We may want to create visualizations that show things like:
- Checkout amount over time
- Top 10 biggest checkout amounts
- Top selling products by country
- How much is being sold in each country, optionally showing trends over time
- Top 10 and bottom 10 countries in terms of the sales amounts
- …see more examples below
In other words, the newly created fields can now be used as building blocks for creating new Dashboards and visualizing business insights.
To create charts with business insights, we will use the Quick Chart functionality. Simply clicking on the field value will open a context menu where you can click to add a chart for it.
More information about this can be found in Quick Charts docs.
Extracting Business Insights
Now, let’s explore how log analysis can help this e-commerce company gain valuable business insights.
Product Popularity
Logs can reveal which products are the most popular among customers. By tracking the frequency of product purchases, you can determine which items are driving the most revenue. This information can be used to optimize product display and placement, cross-sell, marketing strategies, etc.
Customer Behavior
Understanding customer behavior is crucial in e-commerce. Logs can help track customer interactions, including interest in related products, available credit, ordering preferences, and completed purchases. This data can be used to improve user experience and personalize recommendations.
Payment Analysis
Logs can provide insights into payment trends. By analyzing payment transactions, you can identify common payment methods and optimize payment processing for efficiency.
Inventory Management
E-commerce businesses rely on efficient inventory management. Logs can be used to track product stock levels, predict when reordering is necessary, and prevent out-of-stock scenarios that can lead to lost sales.
Building a Comprehensive Dashboard
To make these insights accessible and actionable, it’s essential to create a comprehensive dashboard for the e-commerce team. Here are some components of such a dashboard, created using the Chart Builder:
In the dashboard above we can find the following insights:
- Order status updates to gain insights into the efficiency of the order processing pipeline
- Customers by region together with the available credit by country to guide decisions regarding market expansion and where to focus the marketing efforts
- Payments made by customers to track revenue trends
- Top-selling products in general and top-selling products by price
Once you’ve built a dashboard similar to this one, but catered to your specific data and needs, you may want to enable auto-refresh and have it displayed somewhere permanently – perhaps on your second monitor or on a big screen in the office for everyone to see.
However, if you don’t want to be going to the dashboard and glancing at it periodically, you could have Sematext “get in touch” with you instead though:
- Alerts that will notify you when certain thresholds are crossed or when there are anomalies, or when data in logs matches certain patterns
- Scheduled Reports that will send you the dashboard periodically, so you can passively receive these data via email
Business Alerts
Incorporating the dashboard already set up on office monitors gives you access to a powerful tool for real-time business insights. This dashboard not only provides you with an easy-to-understand representation of critical business data but also detects anomalies, alerts you and notifies you on time.
Sematext Alerts act as a safeguard against potential glitches in operations, unexpected drops in sales, or spikes in the monitored insights, allowing you to proactively address all discovered issues.
Scheduled Reports
For those who may not want to log into Sematext regularly and prefer alternative ways to stay informed, you can use Scheduled Reports. Scheduled Reports enable you to receive daily, weekly, or monthly reports via email to keep track of all your business data without the need for frequent manual checks.
Summary
Logs, once considered solely technical artifacts, are also invaluable sources of business intelligence. By utilizing logs smartly, organizations can optimize sales performance, enhance product offerings, improve customer experiences, and a lot more. It’s really on your team to be imaginative and to think of logs as not only a debugging and troubleshooting tool, but also a carrier of valuable business data that can be turned into insights if used as we’ve illustrated here.