Skip to main content
Monitoring

Service Autodiscovery & Automatic Monitoring with Sematext

Bojan Šmid Bojan Šmid on

If you are anything like us here at Sematext, you are likely always trying to automate any tedious, repetitive tasks. Repetitio est mater… boringdorum. Setting up monitoring falls in that category. You either do it manually every time you provision a new piece of infrastructure or service, or you automate it. Note that by “service” I mean either an instance of your own application or something like Nginx or Elasticsearch or MySQL or …

This automation is especially important in the modern, containerized world where you want to automate anything that is slow, repetitive or error prone. Because what value is having a nice and shiny Kubernetes or Swarm cluster running 100s of services if monitoring them — and you’ll definitely want to monitor them — is complicated and doesn’t happen automatically? Even if you don’t use containers but make use of auto-scaling, say via AWS Elastic Beanstalk or anything similar, you’ll need monitoring that works with such dynamic setup. So how do you make this happen? The solution to this conundrum consists of two parts:

  1. Autodiscovery
  2. Automatic Monitoring

Let’s dig into each of them.

What exactly is Service Autodiscovery?

In dynamic environments services are started, stopped or just moved around between nodes. Having real-time, detailed, up to date information about all your services, such as their health, performance metrics, logs, traces, process information, etc. provides much needed insight into your cluster. Autodiscovery is what finds out about your newly started services and, like a good watchdog, keeps tabs on them, knows when they disappear, die, move to another host, etc. So what do you think Sematext does once it discovers a new service? Yeah, it lets you auto-monitor it. But what in the world is auto-monitoring you ask? Read on!

Automatic Monitoring FTW

Automatic monitoring builds on top of Autodiscovery. It lets you choose which types of services to monitor and where to route their performance metrics. For example, you may want to monitor your Elasticsearch nodes, but not your Nginx servers. Or you may have Elasticsearch in both production and dev environments, but want to monitor only the production environment. Or you may have Elasticsearch clusters in multiple datacenters or regions and you want to monitor them separately, shipping each of their metrics to a separate App.

To unlock this functionality, install the Sematext Agent. Once you install the Sematext Agent everything is done via the Sematext user interface — no config editing, restarting, or deploying is needed. So that’s what Automatic Monitoring means. We’ve been using it ourselves, we love it, and hope you will love it, too!

Getting Started with Service Autodiscovery

So, how does one get to use Sematext Autodiscovery?

First, you’ll want to make sure you have a recent version of Sematext Agent. If you don’t have a recent-enough version you will see that in your Discovery screen (EU). We recommend you just grab the latest Sematext Agent. You’ll want to install it on any hosts where you run services that you think you might want to monitor, regardless of whether these hosts run containers with applications inside them or you run them directly on the host. The hosts can be bare metal machines, VMs, cloud instances, it doesn’t matter.

Installing the Sematext Agent

Let’s assume you have your applications/services running inside containers. You could be using Kubernetes or Docker Swarm or Docker Enterprise or Hashicorp Nomad or…

When installing the Sematext Agent in a container environment, you will be asked to create a Docker App if you don’t already have one. Docker App is a place in Sematext Cloud where container metrics live – things like counts of containers, container I/O, container memory usage, and similar. These container metrics are useful on their own, but Autodiscovery is about monitoring services running inside of your containers, so read on…

If you are not using containers the steps are similar. You just won’t need to create a Docker App, that’s the only difference.

Next, you will be presented with instructions to install the Sematext Agent, as shown below.

Once the Sematext Agent is up and running, the screen will guide you to the Discovery screen (EU). Remember: you’ll want to install Sematext Agent on as many of your hosts as possible to get the full benefit of discovering your services.

If some services are already running on a machine/cluster where you installed Sematext Agent, they will instantly appear in Discovery. In the example below, Sematext Agent discovered one ClickHouse instance and one MySQL instance, each running in its own container.

As services are created or stopped, Sematext Agent will instantly report them and Discovery will reflect the current state. For example, as shown in the screenshot below, an Elasticsearch node instance in a newly started container will be visible within seconds. Clicking on the Elasticsearch row in the Discovery screen would enable you to automatically start monitoring not just this particular Elasticsearch instance, but also any other Elasticsearch instance discovered on any host where the Sematext Agent is running.

A few clicks and seconds later the service status will change from Unmonitored that you see above…

…to Monitored as you can see below:

Finally, clicking on “Go To Report” takes you to metrics that you will immediately start seeing, as shown below:

I hope you can see how simple this is. Besides installing the Sematext Agent, only a few clicks were required to start monitoring. The Elasticsearch instance was just an example I used to illustrate service discovery and auto-monitoring capabilities.

Monitoring via JMX

What if you have some JVM based service, like ZooKeeper, Solr, or Cassandra, or even some custom Java or Kotlin application? Would Automatic Monitoring still be as simple? In short, yes!

Applications running on the JVM often expose their metrics via JMX, as does the JVM itself. Anything that exposes metrics via JMX typically requires that the JMX connector is exposed while starting up. However, Sematext Agent is clever. It doesn’t require you to enable the JMX connector or restart your service. It knows how to attach the monitoring agent automatically. This means JVM monitoring and monitoring of all JVM based services that use JMX for metrics can be monitored just as easily as in the example above.

Database monitoring

How about discovering and monitoring MySQL, PostgreSQL, ClickHouse and other database instances? Monitoring databases typically requires agents to have a database username and password to connect to it and collect metrics. The Discovery UI shows where and how to provide the necessary credentials. Note that for MySQL monitoring you may need to grab the MySQL driver.

Which services can be discovered and monitored automatically?

The list of services Sematext can discover and start monitoring automatically is listed here. If you don’t see your favorite service there give us a shout!

What about bare-metal and virtual-machines?

All of the above works equally well in non-container environments. The difference will be just in how credentials, when required, are passed to Sematext Agent. Autodiscovery, Automatic Monitoring… all work the same way as in containerized environments. Learn more.

What can’t be automated?

Obviously, credentials needed to access some services still have to be defined in some way. In pure Docker setup they can be defined as environment variables in your service containers, which will require a restart of your container. Kubernetes is simpler as credentials would be defined as secrets, so there is no need to restart the pods. But this still requires some minimal manual work on your end. Other than this, everything else happens automatically. Learn more about using secrets with Kubernetes & Sematext.

Can monitoring be set up without the UI?

Yes! If you love your terminal more than the GUI, the ability to set up monitoring in a fully manual way, via the console, is still present. The Discovery screen shows all discovered services and their statuses regardless of how their monitoring agent was set up. Learn more.

What’s next for Sematext Autodiscovery?

The software infrastructure we’ve built for Autodiscovery and Automatic Monitoring paves the way to other exciting capabilities. It enables us to provide you with the ability to profile your applications and services on demand, or collect their heap dumps for analysis. Another exciting capability that we are already working on is the ability to let you collect logs in a similar automated fashion. Finally, we are looking to expand the list of services recognized by our Autodiscovery engine. Got applications and services that we don’t autodetect yet? Please tell us! The live chat bubble you see on this page is an easy way to get in touch.