Live Elasticsearch Online Training starting on Oct 10! See all classes

Sematext Cloud News & Updates 2018


ClickHouse Monitoring released

Yandex ClickHouse is an open source column-oriented database management system capable of real-time generation of analytical data reports using SQL queries.

ClickHouse allows you to not only lower your TCO (Total Cost of Ownership) but also allows you to benefit from unmatched performance and scalability.

With the release of the new Sematext Monitoring Agent, you can now monitor ClickHouse DB clusters.

The setup, tuning and operations of ClickHouse require deep insights into the performance metrics such as locks, replication status, merge operations, cache usage and many more…

The new ClickHouse integration supports over 70 different ClickHouse metrics for system, queries, merge tree, replication, replicas, mark cache, R/W buffers, dictionaries, locks, distributed engine and Zookeeper errors & wait times.

Screen Shot 2018 10 15 at 12.45.38

ClickHouse key metrics in Sematext Cloud

How to use it

To monitor ClickHouse create a ClickHouse App in Sematext Cloud and install the Sematext Agent according to the setup instructions. Like magic, in less than a minute you will receive all ClickHouse metrics in your ClickHouse monitoring App.


Because ClickHouse heavily relies on ZooKeeper we recommend setting up ZooKeeper monitoring and log collection for related logs. By doing this you can benefit from combined dashboards thus having your favorite ZooKeeper and ClickHouse metrics together with related logs for faster troubleshooting. Save time, gain visibility: the benefits are endless.

New Sematext Monitoring Agent released

You asked, we listened. The new Sematext Monitoring Agent includes many features that will make your infrastructure and application monitoring a whole lot easier.

The recently opened-sourced Sematext Monitoring Agent is fully pluggable, making it possible for you to collect metrics from a number of additional sources. The new agent supports the latest Java versions and has updates for many integrations such as Apache Kafka, introduces support for ClickHouse monitoring, etc. In addition, we changed how the agent talks to Sematext. Instead of using a custom protocol, we’ve adopted the popular Influx Line Protocol for sending metrics and all their metadata via HTTPS. The old proprietary format will continue to work for a while to ensure backward compatibility with older agent versions.

You can find all details about the release in Open-sourcing Sematext Monitoring Agent.

If you are an existing Sematext Cloud user, we encourage you to log in and update your agent to start benefiting from the lower agent footprint and flexible configuration options.

The agent update is super simple: just use your package manager to update the spm-client package.

New Container View in infrastructure monitoring


The new Containers view provides you with enhanced visibility into all your containers and their hosts, drawing your attention to the ones that need it the most.

The main view is very much like “top for containers” – applied across all of your containers and their hosts. You can sort containers by memory or CPU usage to find the hottest containers, or simply display the top N containers. The detail view of each container displays container metrics in real-time.

In containers table, the user can see basic information and metrics about each server and by clicking on table row user can expand it to get more details about the container. Thus, for each container user can view metrics, review information about containers grouped by tags like host, image name, pod name …

By using the heatmap, user can get an overview of a large number of containers at once and see if selected metric has a high value on any of the monitored containers or host.

Want to find out more about the new container view? Read our blog post. Can’t wait to try the new containers view? Here is the direct link for Sematext Cloud USA or Sematext Cloud Europe

Sematext Logs and Elasticsearch API Security Features

The Sematext API for log ingestion and search has always been compatible with the Elasticsearch API and provides additional level of security, namely:

  • HTTPS encryption on all Sematext data receivers to help ensure the authenticity, integrity, and privacy of data in transit.
  • The API uses App’s token as index name and generates multiple read-only or write-only tokens in the App Settings. Log shipper requires the write-only token only and does not need to know the read-capable token, needed only if you want to query logs via the API.

Log searches via the API now require not only the App token, but also the API key as an additional security measure, yet no changes to any of your log shipping techniques are needed. This change affects access only via read APIs and not write APIs. Get your API key here and you can regenerate it at any time should it ever get compromised.

The following example shows a search request with curl using the new authentication header:

curl '<APP TOKEN HERE>/_search?size=0&pretty' -H 'Authorization: apiKey <API KEY HERE>'

The API also supports HTTP Basic Authentication which allows third-party tools using client libraries with Basic Authentication for Elasticsearch to do the same. The username is then the “apiKey” literal, and the password is the actual API key.

Here is another example using curl with basic authentication:

curl -v -u apiKey:<API KEY HERE> '<APP TOKEN HERE>/_search?size=0&pretty'

Better Servers View

Better Servers View

Servers view is giving overview over host servers used by your apps. User can use GroupBy and FilterBy
filters to decide which servers or group of servers he wants to monitor. There two main components for
monitoring: table-view and heat-map.

In servers table user can see basic informations and metrics about each server and by clicking on table
row user can expands it to get more details about server. Thus, for each server user can view metrics,
review informations about docker containers on that host and review metrics and informations for apps on
that host.

By using heat-map user can get overview over large number of servers at once and see if selected metric
has high value on any of monitored servers.

Automatic Metrics Correlation

Automatic Metrics Correlation

If you work with monitoring or monitoring tools much, you’ve probably seen the phrase “correlating”
here and there. Correlation is what can help you find metrics with similar pattern and see dependencies
between metrics even if you don’t suspect they are in dependency relation. When troubleshooting,
automatic metrics correlation is the easiest way to find root cause.

Automatic metrics correlation in Sematext Cloud work pretty straight forward: you select a metrics of
interested, select time range and Correlation mechanism will find all metrics which have similar behaviour
in the same time range. Similar spikes, similar trends, similar waves.

Added Logtrail to Kibana

Added Logtrail to Kibana

Dealing with log files or extracting data from various data sources is a daily task in the IT
administration. Adding logtrail to Kibana gives users the “terminal experience”,
that experience of tailing and filtering log files like in the good old terminal that tons of
people are used to and that many of us still love – typically black background, new logs appearing
at the bottom of the screen instead of the top screen, simply way to filter logs and “jump”
to various parts of the log stream.

To summarise this plugin enables you and your team to view, analyze and search log events from a
centralised clean & developer friendly interface.

Log in to see for yourself and check it out….. or read our blog post to find out about a few more improvements we’ve made.