In the world of modern business, data drives decision-making. Every interaction, every transaction, and every click generates a series of data in the form of logs. These logs, often seen as plain text records, have the potential to unlock valuable business insights when analyzed correctly.

In How to Create Log-Based Metrics to Improve Application Observability, we described the process of creating log-based metrics to improve application observability using Sematext Cloud. In this article, we will explore the power of extracting business insights from logs and how it can transform raw data into actionable intelligence. We’ll use an e-commerce company for our examples.

The E-commerce Landscape

Consider an e-commerce company that operates an online shop. This company has several components:

• Employees: Sales representatives, customer support agents, and other personnel for serving customers
• Products: A large range of products that customers can purchase
• Customers: Customers place orders, make payments, and engage in various transactions

The Tech Stack

The online shop is built on a proven infrastructure to handle the demands of e-commerce operations efficiently. In short, the website is powered by a CMS, it’s hosted on a dedicated Apache web server, and integrates with a secure payment gateway.

The backend relies on the MySQL database, which is responsible for storing critical data, including product details, customer information, and transaction records.

Monitoring and Logging Tools

To monitor the infrastructure and the application services, the company is using Sematext Cloud with several available monitoring and logging integrations. To collect and analyze logs, the company decided to use the Sematext MySQL Logs integration. This integration has been enabled by simply creating a MySQL Logs App and installing the Sematext Agent. Everything is a few clicks away in the UI to configure the ship logs it discovered for you via the Logs Discovery.

Extracting Business Data from Logs

In our example here, the MySQL logs contain data that is valuable for business. Here are a few examples of MySQL logs from which we will be extracting useful business data.

231009 14:24:19        13 Query    INSERT INTO employees (employeeNumber, lastName, firstName, extension, email, officeCode, jobTitle) VALUES ( 1100, 'Smith', 'Jane',  'x123',  'jane.smith@example.com','1',  'Sales Representative');

231009 10:22:14        13 Query    INSERT INTO customers (customerNumber, customerName, contactLastName, contactFirstName, phone, addressLine1, addressLine2, city, state, postalCode, country, salesRepEmployeeNumber, creditLimit) VALUES (520, 'CR ABC Company', 'Ivan', 'Maric', '123-456-7890', '123 Main St', NULL, 'Split', NULL, '90001', 'Croatia', 1100, 75000.00)

231009 15:11:08        7 Query     INSERT INTO orders (orderNumber, orderDate, requiredDate, status, comments, customerNumber)
VALUES (10101, '2023-10-06', '2023-10-10', 'In Process', 'Urgent order', 101);

231009 15:12:35        7 Query     INSERT INTO orderdetails (orderNumber, productCode, quantityOrdered, priceEach, orderLineNumber) VALUES  (10101, 'S10_1678', 10, 81.35, 1),  (10101, 'S10_1949', 5, 204.16, 2);

In your case it may be your own application that logs valuable business data, not MySQL. In either case, we want to extract the pieces of data that we will then use to create business dashboards, alerts, etc. To perform this data extraction we will use Logs Pipelines. In general, Logs Pipelines are really handy as they can help you structure your logs, drop noisy logs, and a lot more.

Once logs are ingested, Logs Pipelines allows you to define parsing rules using the Pipeline Builder. You can add one or more Processors and use regular expressions or built-in parsers to extract structured data from raw log messages. The Field Extractor Processor is used to extract fields using a grok pattern. For example, if your logs contain information about customer transactions, you can define a grok pattern to extract fields like “country”, “shipping status” and “price”.

As we can see below, in our imaginary online shop’s log monitoring setup there are multiple Processors used to extract a number of valuable fields from the logs, like “order_number”, “product_code”, “price”, “customer_name”, “country”, “employee_number”, “amount” and so on.

These fields are now part of our structured log events and, as you can see below, both numeric and non-numeric values can be extracted:

Charting Extracted Business Data

We have now extracted a couple of interesting bits of information from the original “message” field:

• customer_name: the name of the customer
• country: the country of the customer
• employee_number: the Account Manager for the customer
• amount: the available credit from the customer
• shipping_status: the status of the shipping: shipped, in progress or canceled
• customer_id: a unique identifier for the customer
• payment_date: the date of payment
• payment_amount: checkout amount of the order
• order_number: the number of the order
• product_code: a unique identifier for each product
• price: price of the product

Now what? We may want to create visualizations that show things like:

• Checkout amount over time
• Top 10 biggest checkout amounts
• Top selling products by country
• How much is being sold in each country, optionally showing trends over time
• Top 10 and bottom 10 countries in terms of the sales amounts
• …see more examples below

In other words, the newly created fields can now be used as building blocks for creating new Dashboards and visualizing business insights.

To create charts with business insights, we will use the Quick Chart functionality. Simply clicking on the field value will open a context menu where you can click to add a chart for it.

More information about this can be found in Quick Charts docs.

Extracting Business Insights

Now, let’s explore how log analysis can help this e-commerce company gain valuable business insights.

Product Popularity

Logs can reveal which products are the most popular among customers. By tracking the frequency of product purchases, you can determine which items are driving the most revenue. This information can be used to optimize product display and placement, cross-sell, marketing strategies, etc.

Customer Behavior

Understanding customer behavior is crucial in e-commerce. Logs can help track customer interactions, including interest in related products, available credit, ordering preferences, and completed purchases. This data can be used to improve user experience and personalize recommendations.

Payment Analysis

Logs can provide insights into payment trends. By analyzing payment transactions, you can identify common payment methods and optimize payment processing for efficiency.

Inventory Management

E-commerce businesses rely on efficient inventory management. Logs can be used to track product stock levels, predict when reordering is necessary, and prevent out-of-stock scenarios that can lead to lost sales.

Building a Comprehensive Dashboard

To make these insights accessible and actionable, it’s essential to create a comprehensive dashboard for the e-commerce team. Here are some components of such a dashboard, created using the Chart Builder:

In the dashboard above we can find the following insights:

1. Order status updates to gain insights into the efficiency of the order processing pipeline
2. Customers by region together with the available credit by country to guide decisions regarding market expansion and where to focus the marketing efforts
3. Payments made by customers to track revenue trends
4. Top-selling products in general and top-selling products by price

Once you’ve built a dashboard similar to this one, but catered to your specific data and needs, you may want to enable auto-refresh and have it displayed somewhere permanently – perhaps on your second monitor or on a big screen in the office for everyone to see.

However, if you don’t want to be going to the dashboard and glancing at it periodically, you could have Sematext “get in touch” with you instead though:

• Alerts that will notify you when certain thresholds are crossed or when there are anomalies, or when data in logs matches certain patterns
• Scheduled Reports that will send you the dashboard periodically, so you can passively receive these data via email

Business Alerts

Incorporating the dashboard already set up on office monitors gives you access to a powerful tool for real-time business insights. This dashboard not only provides you with an easy-to-understand representation of critical business data but also detects anomalies, alerts you and notifies you on time.

Sematext Alerts act as a safeguard against potential glitches in operations, unexpected drops in sales, or spikes in the monitored insights, allowing you to proactively address all discovered issues.

Scheduled Reports

For those who may not want to log into Sematext regularly and prefer alternative ways to stay informed, you can use Scheduled Reports. Scheduled Reports enable you to receive daily, weekly, or monthly reports via email to keep track of all your business data without the need for frequent manual checks.

Summary

Logs, once considered solely technical artifacts, are also invaluable sources of business intelligence. By utilizing logs smartly, organizations can optimize sales performance, enhance product offerings, improve customer experiences, and a lot more. It’s really on your team to be imaginative and to think of logs as not only a debugging and troubleshooting tool, but also a carrier of valuable business data that can be turned into insights if used as we’ve illustrated here.

Start Free Trial