Monitoring Elasticsearch with Sematext

As shown in Elasticsearch Key Metrics, the setup, tuning, and operations of Elasticsearch require deep insights into the performance metrics such as index rate, query rate, query latency, merge times, and many more. Sematext provides an excellent alternative to other Elasticsearch monitoring tools.

Open-source tools to monitor Elasticsearch are free, but your time is not. Relatively speaking, they’re rather expensive. Thus, Sematext Cloud aims to save you time, effort, and your hair! We even offer training and additional support for Elasticsearch. Check out the short video below to get more information on how we can help with that:

How Sematext Helps you Save Time

Here are a few things you will NOT have to do when using Sematext for Elasticsearch monitoring:

• figure out which metrics to collect and which ones to ignore
• give metrics meaningful labels
• hunt for metric descriptions in the docs so that you know what each one actually shows
• build charts to group metrics that you really want on the same charts, not several separate charts
• figure out which aggregation to use for each set of metrics (min? max? avg? something else?)
• build dashboards to combine charts with metrics you typically want to see together
• set up basic alert rules

All of the above is not even a complete story. Do you want to collect Elasticsearch logs? How about structuring them? Sematext does all this automatically for you!

Sematext’s service auto-discovery feature lets you automatically start monitoring your Elasticsearch instances directly through the user interface.

In this post, we will look at how Sematext provides more comprehensive – and easy to set up – monitoring for Elasticsearch and other technologies in your infrastructure. By combining events, logs, and metrics together in one integrated full stack observability platform and using the Sematext open-source monitoring agent and its integrations, which are also open-source, you can monitor your whole infrastructure and apps, not just your Elasticsearch cluster. You can see how it compares to similar solutions in our article about the best IT monitoring tools.

You can also get deeper visibility into your entire software stack by collecting, processing, and analyzing your logs.

Elasticsearch Monitoring

Collecting Elasticsearch Metrics

Sematext Elasticsearch integration collects over 100 different Elasticsearch metrics for JVM, index performance, cluster health, query performance and more. Sematext maintains and supports official Elasticsearch monitoring integration. Moreover, the Sematext Elasticsearch integration is customizable and open source.

Bottom line: you don’t need to deal with configuring the agent for metrics collection, which is the first huge time saver!

Installing Monitoring Agent

Setting up the monitoring agent takes less than 5 minutes:

1. Create an Elasticsearch App in the Integrations / Overview (or Sematext Cloud Europe). This will let you install the agent and control access to your monitoring and logs data. The short What is an App in Sematext Cloud video has more details.
2. Name your Elasticsearch monitoring App and, if you want to collect Elasticsearch logs as well, create a Logs App along the way.
3. Install the Sematext Agent according to the setup instructions displayed in the UI.

App creation and setup instructions in Sematext Cloud

For example, on Ubuntu, add Sematext Linux packages with the following command:

echo "deb http://pub-repo.sematext.com/ubuntu sematext main" | sudo tee /etc/apt/sources.list.d/sematext.list > /dev/null
wget -O - https://pub-repo.sematext.com/ubuntu/sematext.gpg.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install spm-client

Then setup Elasticsearch monitoring by providing Elasticsearch server connection details:

sudo bash /opt/spm/bin/setup-sematext
--monitoring-token
--app-type elasticsearch
--agent-type standalone
--SPM_MONITOR_ES_NODE_HOSTPORT 'localhost:9200'
--infra-token

In case you have Elasticsearch secured with HTTPS and basic authentication, you can add the following parameters to the command:

--SPM_MONITOR_ES_NODE_BASICAUTH_USERNAME userName
--SPM_MONITOR_ES_NODE_BASICAUTH_PASSWORD passWord

In addition, you need to specify the HTTPS as protocol SPM_MONITOR_ES_NODE_HOSTPORT, as shown in the complete setup command:

sudo bash /opt/spm/bin/setup-sematext
--monitoring-token <your-token-goes-here>
--app-type elasticsearch
--agent-type standalone
--SPM_MONITOR_ES_NODE_HOSTPORT 'https://localhost:9200'
--infra-token <your-token-goes-here>
--SPM_MONITOR_ES_NODE_BASICAUTH_USERNAME userName
--SPM_MONITOR_ES_NODE_BASICAUTH_PASSWORD passWord

Go grab a drink, but hurry! Elasticsearch metrics will start appearing in your charts in less than a minute.

Elasticsearch Monitoring Dashboard

When you open the Elasticsearch App you find a predefined set of dashboards that organize more than 100 Elasticsearch metrics and general server monitoring in predefined charts grouped into an intuitively organized set of monitoring dashboards:

• Overview with charts for all key Elasticsearch metrics
• Operating System metrics such as CPU, memory, network, disk usage, etc.
• JVM metrics for Garbage collection, JVM Memory, JVM Threads, and JVM open files
• Elasticsearch metrics
  • Cluster Health:  The number of Elasticsearch nodes and shard status (active, relocating, initializing, ..)
  • Shard Stats: The number of shards, shard status per index
  • Index Stats: The number of indexed documents, size on disk, indexing rate, merging rate, merged documents
  • Search: Request rate, query and fetch latency, realtime-get latency
  • Thread Pools: Number of threads per pool, thread pool size
  • Circuit Breakers: Field data stats, request size stats
  • Connections: Connected sockets, Node-Node-Transport stats, TCP socket and traffic stats

Elasticsearch key metrics in Sematext Cloud

Setup Alerts for Elasticsearch Metrics

To save you time Sematext automatically creates a set of default alert rules such as alerts for low disk space. You can create additional alerts on any metric. Watch Alerts in Sematext Cloud for more details.

Alerting on Elasticsearch Metrics

There are 3 types of alerts in Sematext:

• Heartbeat alerts, which notify you when a Elasticsearch DB server is down
• Classic threshold-based alerts that notify you when a metric value crosses a predefined threshold
• Alerts based on statistical anomaly detection that notify you when metric values suddenly change and deviate from the baseline

Let’s see how to actually create some alert rules for Elasticsearch metrics in the animation below. The request query count chart shows a spike. We normally have up to 100 requests, but we see it can jump to over 600 requests. To create an alert rule on a metric we’d go to the pulldown in the top right corner of a chart and choose “Create alert”. The alert rule applies the filters from the current view and you can choose various notification options such as email or configured notification hooks (PagerDuty, Slack, VictorOps, BigPanda, OpsGenie, Pusher, generic webhooks etc.). Alerts are triggered either by anomaly detection, watching metric changes in a given time window or through the use of classic threshold-based alerts.

Alert creation for Elasticsearch request query count metric

Elasticsearch Logs

Shipping Elasticsearch Logs

Since having logs and metrics in one platform makes troubleshooting simpler and faster let’s ship Elasticsearch logs too. You can use many log shippers, but we’ll use Logagent because it’s lightweight, easy to set up, and because it can parse and structure logs out of the box. The log parser extracts timestamp, severity, and messages. For query traces, the log parser also extracts the unique query ID to group logs related to query execution.

1. Create a Logs App to obtain an App token
2. Install Logagent npm package

   sudo npm i -g @sematext/logagent

   you don’t have Node.js, you can install it easily. E.g. On Debian/Ubuntu:

   curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
   sudo apt-get install -y nodejs

3. Install the Logagent service by specifying the logs token and the path to Elasticsearch log files.  You can use -g ‘var/log/**/elasticsearch*.log` to ship only logs from Elasticsearch server. If you run other services, on the same server consider shipping all logs using -g `/var/log/**/*.log` The default settings ship all logs from /var/log/**/*.log when the -g parameter is not specified. Logagent detects the init system and installs Systemd or Upstart service scripts. On Mac OS X it creates a launchd service. Simply run:

   sudo logagent-setup -i YOUR_LOGS_TOKEN -g `var/log/**/elasticsearch*.log`
   #for EU region:
   #sudo logagent-setup -i LOGS_TOKEN
   #-u logsene-receiver.eu.sematext.com
   #-g `var/log/**/elasticsearch*.log`

   The setup script generates the configuration file in /etc/sematext/logagent.conf and starts Logagent as system service.

Note, if you run Elasticsearch in containers, setup Logagent for container logs.

Log Search and Dashboards

Once you have logs in Sematext you can search through them when troubleshooting, save queries you run frequently or create your individual logs dashboard.

Search for Elasticsearch Logs

Log Search Syntax

If you know how to search with Google, you’ll know how to search your logs in Sematext Cloud.

• Use AND, OR, NOT operators – e.g. (error OR warn) NOT exception
• Group your AND, OR, NOT clauses – e.g. message:(exception OR error OR timeout) AND severity:(error OR warn)
• Don’t like Booleans? Use + and – to include and exclude – e.g. +message:error -message:timeout -host:db1.example.com
• Use field references explicitly – e.g. message:timeout
• Need a phrase search? Use quotation marks – e.g. message:”fatal error”

When digging through logs you might find yourself running the same searches again and again. To solve this annoyance, Sematext lets you save queries so you can re-execute them quickly without having to retype them. Please watch how using logs for troubleshooting simplifies your work.

Alerting on Elasticsearch Logs

To create an alert on logs we start by running a query that matches exactly those log events that we want to be alerted about. To create an alert just click to the floppy disk icon.

Similar to the setup of metric alert rules, we can define threshold-based or anomaly detection alerts based on the number of matching log events the alert query returns.

Please watch Alerts in Sematext Cloud for more details.

Elasticsearch Metrics and Log Correlation

A typical troubleshooting workflow starts from detecting a spike in the metrics, then digging into logs to find the root cause of the problem. Sematext makes this really simple and fast. Your metrics and logs live under the same roof. Logs are centralized, the search is fast, and the powerful log search syntax is simple to use.  Correlation of metrics and logs is literally one click away.

Elasticsearch logs and metrics in a single view

Monitor Elasticsearch with Sematext

Comprehensive monitoring for Elasticsearch involves identifying key metrics for Elasticsearch, collecting metrics and logs, and then connecting everything in a meaningful way. In this post, we’ve shown you how to monitor Elasticsearch metrics and logs in one place. We used OOTB and customized dashboards, metrics correlation, log correlation, anomaly detection, and alerts. And with other open-source integrations, like Apache Kafka, you can easily start monitoring Elasticsearch alongside metrics, logs, and distributed request traces from all of the other technologies in your infrastructure. Get deeper visibility into Elasticsearch today with a free Sematext trial.

Or, if you need some additional help with the Elasticsearch ecosystem, Sematext can help with that, too.