We finally have a worthy partner for the long-standing (but recently improved!) Elasticsearch Monitoring Integration. If you see a spike in query latency or a node count drop in metrics, you’d naturally want to check the logs.
You may already be using a Generic Logs App to centralize your Elasticsearch logs, but there’s a better way now: the new Elasticsearch Logs Integration gives you out of the box dashboards for all the important information:
- Errors breakdown
- Master and clustering-related logs
- Start and stop
- Slowlog query breakdown by time, frequency, index, node, etc
Setup takes literally two clicks once you have Sematext Agent installed, thanks to Logs Discovery. Sematext Agent is already installed if you’re using our Elasticsearch Monitoring. Otherwise you’ll need two copy-paste commands to install it.
We’ve just added Google Chat to our growing list of available notification channels! Google Chat is seamlessly integrated with Google Workspace. If you are a Google Workspace customer and Google Chat user this removes the need to use a different notification channel from the one you already have.
You can, of course, choose which people or channels you want to receive notifications, as well as personalize their content.
To get started with Google Chat notifications, check out our docs on how to configure Google Chat.
If you run applications on top of the JVM you know how critical garbage collection is for application performance. Some of the information about JVM performance can be collected from JMX, which is what our JVM monitoring integration exposes in a set of out of the box dashboards. But the JVM also logs a bunch of useful information. Unfortunately, these logs are not designed for humans. They are not easy to interpret quickly. That’s why we are happy to announce a new integration – the JVM Logs Integration. It gleans insights about the JVM garbage collection performance from the JVM logs and presents them in a set of easy to understand dashboards. Using the Split Screen you can also correlate other JVM performance metrics with JVM GC data extracted from logs.
Here are some insights this new integration provides:
- Details about your JVM garbage collector runs including new and tenured generations statistics
- Region information, such as survivor region size after garbage collection and new generation region size after garbage collection.
- Time your application threads were stopped because of garbage collection, and much more.
The easiest way to get started is using the Logs Discovery that, if you are running a recent version of Sematext Agent, will let you select the JVM logs you want to ship to Sematext, so you can set up log shipping in about a minute just through the Discovery screen (EU). Learn more about this JVM Garbage Collection logs integration. Enjoy!
Our Elasticsearch monitoring integration just got a whole lot better! You get new metrics, better dashboards and more default alerts out of the box. Stemmed from many years of consulting and production support experience, our new Elasticsearch monitoring makes troubleshooting clusters a lot faster and simpler. Let me explain why.
First off, the dashboards: the new Overview screen allows you to spot health issues (e.g. unassigned shards, thread pool rejections) as well as the #1 performance killer: load imbalance. To detect the source of said load, there’s an index breakdown on both reads and writes:
The old Overview dashboard was super-useful and we kept it, renamed as Essential Metrics. There’s another new dashboard called Daily Patterns, which allows you to see the day of the week and hours with more traffic. Here’s an example, which shows that e.g. on Monday morning queries are slower in this cluster.
Notice that TIP markdown on top? We’ve added those whenever the charts aren’t self-explanatory. You now know what to look for – and if there are issues, you have hints on what to do. Check the tip on refresh time below. Also, notice how we have breakdowns by index/node on the right, allowing you to look closer at the source of load. Like with tips, we’ve added breakdowns like this in many places.
Other new dashboards, such as Ingest or Scripting, are made of new metrics. There are 52 new metrics in total. You can now identify the most expensive Ingest pipelines, how many times circuit breakers tripped, script compilations and more. Existing metrics classes are enhanced, too: cache hit ratios, disk IOPS, merge and recovery throttling, etc.
There’s a new dimension to filter and group by: node role. Dedicated masters and data nodes have different load patterns, so you’d often select the data nodes to see their aggregate load (maybe group by host later). In a large cluster, selecting all data nodes used to be tricky, but now it’s just one click!
Last but not least: default alerts. There were 4 such alerts before on some clear red flags: heartbeats, almost 100% heap and disk usage, and anomalous number of nodes. We added 8 more, for example on anomalous unassigned shards or if load is much higher than the number of processors. This way you’ll know better when the cluster has performance issues or when it’s behaving abnormally.
As you may tell, we’re quite excited about those improvements! We hope you’ll find them useful, too. Either way, feel free to reach out for any feedback or questions.
We are thrilled to announce Logs Pipelines, an exciting new functionality you can now use in Sematext Cloud!
Raw log events are not always perfectly structured. They may include extra fields and be overly verbose, or they may lack structure that would enable you to derive more insights from your logs. For various reasons it might be hard to set up a log shipper to transform or filter log events at the source. An alternative approach is to configure Logs Pipelines inside Sematext.
Each Pipeline consists of one or more Processors that are executed in the order in which they are defined in a Pipeline. They can change, drop, or even produce additional events or fields. They are chained to form a Pipeline. The output of one Processor is the input for the next Processor.
The following Processors are currently available:
- Rename Fields
- Remove Fields
- Field Extractor
- Field Masking
- Script Field
Each of your Logs Apps can have a different Pipeline, with a different set or order or Processors. Pipelines and Processors can be modified at any time and changes take effect immediately upon saving. Building a Pipeline is super simple and intuitive with a Pipeline Builder that includes a before/after preview so you can examine a Pipeline’s output before saving it.
Some use cases:
- Extraction of numerical values or metrics out of text fields, so you can create charts and alerts on them
- Standardization of field names
- Masking or dropping of sensitive data, such as PII
- Geo-enrichment based on IPs or latitude and longitude information
- Reduction in log volume
Let us know what other Processors you would like to see in Sematext.
Importantly, we’ve adjusted the pricing in a small way. This will affect you, most likely positively, only if you start using Pipelines. We’re now separately tracking the volume of logs received and stored. We’ve reduced the cost of stored logs slightly and made it very cheap to ship logs to Sematext. You can see the adjusted pricing in both Sematext Cloud and on the Sematext pricing page. Learn more in Logs Pipelines docs.
If you’ve missed our dark theme announcement, check it out. It’s “just” a visual thing, but we all like to look at prettier things given the choice. Along the same lines we’ve recently exposed the ability to style charts using the gradient style.
Here are a few examples:
Combined with the dark theme, these charts look HOT! What do you think?
To use the gradient style for your charts simply edit any existing component using Chart Builder, select the Dataseries you want to style, and then switch to the Style tab where you will find the newly added “Gradient” option.
Are there any other visualization improvements you would like to see in Sematext? Let us know via our live chat or via email.
This is a brand new integration for NGINX logs. Besides monitoring NGINX or NGINX Plus web server performance metrics you can now ship NGINX logs to Sematext and enjoy a number of useful out of the box dashboards and alert rules. Of course, You can customize, add, or delete any dashboards or alert rules. The simplest way to get started is to head over to Discovery (EU) and look for NGINX logs there. If you don’t see any of your NGINX servers there that means you haven’t installed Sematext Agent (EU) on them. Once you do that you will see NGINX in both Services and Logs sections of Discovery and will be able to set up both NGINX performance monitoring and log monitoring through the UI.
The default overview report looks like this:
As you can see on the left side of this screenshot, there are other pre-built reports there that will save you time and give your additional insight into how your NGINX web server is doing, such as:
- Errors: Logs specifically about errors. You can see which status codes are returned and which errors are logged by your NGINX server.
- HTTP: Logs about NGINX access logs. Use this report to view any and all data about HTTP requests and responses.
- Sources: Logs about NGINX source files. You can view where the logs are collected from and from where in the world users are accessing your NGINX server.
If you are using Sematext for monitoring both NGINX performance metrics and NGINX logs, then consider connecting the Apps. Doing that will help you jump from one to the other and back faster, correlate them side by side easier, and see dashboards from both/all Connected Apps in your alert notifications sent via email, as well as on Alert Landing Pages. All this will make it easier and faster to troubleshoot issues.Learn more about NGINX logs integration or set up NGINX logs monitoring (EU).
This is a brand new integration for Apache logs. Besides monitoring Apache web server performance metrics you can now ship Apache logs to Sematext and enjoy a number of useful out of the box dashboards and alert rules. Of course, You can customize, add, or delete any dashboards or alert rules. The simplest way to get started is to head over to Discovery (EU) and look for Apache logs there. If you don’t see any of your Apache servers there that means you haven’t installed Sematext Agent (EU) on them. Once you do that you will see Apache in both Services and Logs sections of Discovery and will be able to set up both Apache performance monitoring and log monitoring through the UI.
The default overview report looks like this:
As you can see on the left side of this screenshot, there are other pre-built reports there that will save you time and give your additional insight into how your Apache web server is doing, such as:
- Errors: Logs specifically about errors. You can see which status codes are returned and which errors are logged by your Apache server.
- HTTP: Logs about Apache access logs. Use this report to view any and all data about HTTP requests and responses.
- Sources: Logs about Apache source files. You can view where the logs are collected from and from where in the world users are accessing your Apache server.
If you are using Sematext for monitoring both Apache performance metrics and Apache logs, then consider connecting the Apps. Doing that will help you jump from one to the other and back faster, correlate them side by side easier, and see dashboards from both/all Connected Apps in your alert notifications sent via email, as well as on Alert Landing Pages. All this will make it easier and faster to troubleshoot issues.Learn more about Apache logs integration or set up Apache logs monitoring (EU).
Regardless of which notification hook(s) you are using for your alerts in Sematext, the notification always contains a link to a view in Sematext that we internally call Alert Landing Page. This page provides a number of details about a specific alert incident and serves as the jump off point for any further troubleshooting you may need to perform. A while back we’ve added the ability to add Runbooks and Descriptions for alert rules and we’ve recently tweaked the Alert Landing Page to expose the Runbook more or, if you have not defined the Runbook for a given alert rule, make it easy for you to add it, as shown in the screenshot below.
Runbooks are “recipes” that consist of actions one may want to perform when something breaks in a previously observed and documented fashion. You can style Runbooks using Markdown, so you can structure them however you wish to make them easier for your teammates to follow even when they are half asleep.
You can view all your alert rules here (EU) and edit them to add Runbooks. We strongly recommend you also add a description to each alert rule. They will help both you and your teammates understand why some alert rule was created long after the person who created it is no longer around to ask.
Learn more about Runbooks in Wikipedia.
If you are using Sematext for monitoring Nginx, Apache, or MongoDB, you’ll be pleased to hear we’ve just made some improvements to the Nginx, Apache, and MongoDB performance monitoring integrations.
The changes are based on the new, unified Sematext Agent that provides numerous integrations (EU) out-of-the-box and enables exciting features like infrastructure monitoring, auto-discovery and many more. The old Node.js-based agent that was used for just these three integrations will continue to be supported for a while longer, but we suggest you upgrade to the new Sematext Agent as it brings additional benefits. You can do that by:
- removing the existing Node.js-based agent – see how (n.b. you can also remove the Node.js runtime if you don’t use it for something else since the new agent doesn’t need it)
- following the installation instruction for your Nginx, Apache, or MongoDB Monitoring Apps (EU) in Sematext
One notable difference between the new and the old MongoDB integration is in the metrics they provide. A few of the old reports will be removed in the near future, but we are adding several new ones for WiredTiger, TCMalloc, Shards, Latency, Cursor and reworked Locks.
We are excited to announce you can now choose between light and the new dark theme!
The dark theme was designed to reduce strain on your eyes, especially while working in a low-light environment.
And of course, it looks way cooler than the light theme 😉
To change your theme just go to Settings in the bottom-left of Sematext Cloud screen and then choose Preferences, as shown below.
If you use a shared Team Account, which we recommend you use if you are using Sematext with others at your organization, the theme preference you’ve set in your account will be applied when you switch into the Team Account, or any other account you have access to.
We hope you’ll like it.
As always, our team will be delighted to hear your feedback, whether it’s through the live chat or via email firstname.lastname@example.org.
We’ve just incorporated the Journald into our arsenal of collectible log sources. Journald Discovery brings all your systemd service logs under one roof where you can granularly define log shipping rules by including/excluding specific services.
Needless to say, we’ve made sure it works seamlessly in all environments. You’re deploying your workloads in Kubernetes but still want to grab Journald logs from worker nodes? No problem!
With Kubernetes Logs Discovery, setting up cluster-wide logs aggregation and parsing becomes a no-brainer!
Kubernetes Logs Discovery surfaces all containerized services and classifies them into corresponding groups. You can then instantly start shipping logs or define more advanced filtering criteria by pod labels.
Podman is a drop-in replacement for the Docker container engine bringing some interesting features like the ability to switch between root/rootless modes for stringent security isolation.
To start monitoring your podman containers, head to the agent installation or update instructions in the Sematext Cloud US or EU region. Sematext will dig out valuable podman metrics including memory and CPU utilization, disk I/O stats, and network metrics.
How do you use Sematext to monitor the web sites in your intranet? How do you monitor APIs behind your firewall? We’re proud to announce the availability of private agents for Synthetics Apps on Standard and Pro plans. You can now run Synthetics Monitors from your own private locations. This allows you to monitor services that are behind firewalls, or otherwise not reachable from our servers.
To get started take a look at Private Locations.
From today you can select your personal preferences on the account level. These settings include customizing chart presets for charts and time formats across your entire account.
The values you set in the user preferences will be applied across all charts, but you can still edit these chart settings on the component-level.
We’ll be adding more user preferences soon! Stay tuned for updates.
With Synthetics you can now create public status pages where your users can check the status of your service. When there are issues with your service, you can publish information about the incident directly to the status page. Your users can check the status page manually or subscribe to the incident RSS feed. Here is the key functionality:
- Percent uptime
- Markdown support, so you can embed custom and nicely formatted content
- Adding, updating, and resolving incidents
- Incidents for scheduled maintenance can be created ahead of time
- One can subscribe to an RSS feed to be notified of new incidents
- Automatic 60-second refresh
- Support for multiple status pages
- Custom domain names, so you can have status.yourcompany.com for example
Any services that are monitored with Synthetics can be automatically added to the status page for uptime monitoring.
We now use Synthetics for our own official status page at status.sematext.com. This is also a good example of cross-datacenter monitoring. Sematext Cloud US monitors Sematext Cloud EU, and vice versa.
The Sematext Infra App integration and its functionality is now available to all Sematext Cloud users!
Infra Apps act as a storage for metrics, processes and events related to your bare-metal, virtual machine and container infrastructure. Your whole core infrastructure will be monitored through Infra Apps.
Infra Apps will also absorb all container-related functionality and reports that were previously part of the Docker integration which is getting retired.
As part of that all Docker integration users will have their Docker Apps automatically converted to Infra Apps, while keeping their names intact. Their plans will be kept as well. If your Docker App was on a Standard plan, it will stay on the Standard plan after this conversion. The price remains the same and all container usage will from now on be charged through Infra Apps.
Infra Apps that were created while they were still in Beta will now be assigned a free plan to avoid any unwanted billing. To gain access to paid features, like no host or container limits, longer data retention, no alert rule limits and similar, you can upgrade any of your Infra Apps to a paid plan at any point.
There are no changes to any other integrations.
As always, due to the flexible nature of the Sematext Cloud pricing model, you can still choose a different plan for each of the Apps you own depending on your needs.
Check the documentation to learn more about Infra Apps.
After containerd joined the family of monitoring integrations, we’re following the trend and expanding our container monitoring capabilities with cri-o container runtime support. cri-o aims to deliver a native container runtime for Kubernetes workloads.
To start monitoring your cri-o containers, install the latest version of Sematext Agent. You can dig out valuable cri-o metrics including memory and CPU utilization, disk I/O stats, and network metrics.
Node.js is more than twice as commonly seen in containerized environments than in non-containerized environments. We listened to our customers and decided to enhance the ability of our Node.js agent to gather container-related metadata and tags.
You’ll now be able to filter and group metrics by a set of container-specific tags:
These tags are collected no matter which containerized environment you are using. Everything is supported, anything from Docker, Docker Swarm, and Kubernetes.
When you install the latest version of Sematext Agent it will provide you with information about all log files and containers discovered on the host where it was installed. Its comprehensive log patterns catalog structures the raw log events into events with meaningful fields out of the box making them suitable for dashboarding and the creation of precise alert rules.
The new Discovery (or this Discovery if you use Sematext Cloud EU) gives you a unified control plane that enables you to instantly see all of your log files and services that are producing them. You can configure automatic log shipping with just a few clicks directly from the UI.
To get started with Logs Discovery, install the latest version of Sematext Agent. If you want to learn more about Logs Discovery, head to our docs. Once your log sources are discovered head over to Discovery (or EU Discovery if you use Sematext EU).
Happy log digging!