Skip to main content
Monitoring, Sematext Cloud

Now Open Source: Sematext Monitoring Agent

sematext sematext on

You can now create monitoring integrations for your technology stack easier than ever before. We’ve recently opened-sourced Sematext’s Monitoring Agent, reworked it, and made it fully pluggable, making it possible for you to collect metrics from a number of additional sources.

Security, Transparency, Collaboration

Infrastructure and application monitoring agents are a sensitive piece of software. Deployed to as many servers and container hosts as possible, they collect critical data digital businesses need to operate efficiently and remain competitive. Every modern organization needs to know what exactly is running on their infrastructure for a number of reasons: maximizing resource utilization, cost savings, threat and vulnerability detection, applying security policies, compliance rules, etc. Open-source agents come with many more advantages such as:

Transparency

Anyone can inspect an open source project for bugs, malware, etc. Transparency is tremendously important to governments, regulated industries like banking or healthcare, and security software, but it is also critically important to all other organizations.

Collaboration

Most technologies are changed and released several times a year. As they change, their metrics evolve with them.  Often these changes are backward compatible, but sometimes they are not.  Thus, monitoring agents need occasional adjustments to keep up with new releases of technology they monitor. In the past, that is what we have done behind the closed doors, but that meant you depended on Sematext releasing new agent versions.  Open-sourcing our agent means we are no longer this bottleneck.  You are now in control and can adjust the agent for your own needs, optionally contributing your changes for the benefit of the community.

Adoption and Remixing

Open source projects can be used by anyone for nearly any purpose. For open source monitoring agents, this means that it is possible to integrate data collectors by anyone who has the need to monitor a specific technology or use the metrics for analysis, control application auto-scaling or tune data collectors for an optimized resource usage.

Out of the Box Integrations

The monitoring agent comes with a number of out of the box integrations for monitoring Kafka, Cassandra, Elasticsearch, Solr, ClickHouse, etc.  This means that you don’t  have to do anything to get your dashboards – as soon as you install it the agent will start collecting metrics, visualize them in charts, group them logically, and organize them for you in the UI to make it easy for you to gain insights from all the data.  Each metric has a clear label and description.  Very importantly, you can change all this should you prefer something different simply by editing a few simple configuration files!  Metrics collected out of the box are listed in integration docs.

Pluggable Integrations

The agent comes with a number of out of the box integrations, but you can add your own, and all it takes are a few configs.  No coding needed. You can expect a detailed “How to” post very soon, the work on the documentation is still in progress.  

Metric Sources

Metrics can be collected from a few supported sources:

  • JMX
  • REST API
  • SQL Database

Each of these metric-sources is defined in configuration files.  To fetch metrics from JMX you only need to specify MBeans you want to collect. The JMX data source simplifies the monitoring stack by eliminating the need for JMX/HTTP bridges like Jolokia.  For REST you just specify the HTTP/HTTPS connection info and JSON elements to extract.  To get metrics from an SQL database you provide the connection string along with queries that return database statistics data.  Additional sources will be added.  Pull requests welcome, of course!

Influx Line Protocol Support

The Influx Line Protocol is a very popular protocol for sending metrics to InfluxDB.  It is widely supported by a number of open source monitoring agents. To make it simple for anyone to send metrics to Sematext we’ve done several things:

This makes the following possible:

      • You can now use Sematext Monitoring Agent to send metrics to your own InfluxDB, too, not just to Sematext. The connection settings can be configured in the Sematext agent properties, e.g.:
        server_base_url=https://your-influxdb-server:8086
        metrics_endpoint=/write?db=metrics&u=UserName&p=Password
      • You can use a completely open source stack with Sematext Agent for metric collection, InfluxDB for metric storage, Grafana for dashboards, etc.
      • The open source release of Sematext Monitoring Agent complements other open-source agents. It fills the niche for the monitoring of application layer metrics via out of the box integrations like JVM, databases, web servers, message queues, big data tools, etc. or any of the pluggable configurations from other metric sources like SQL, JMX, REST (see above).
      • You can use other agents to collect and ship metrics to Sematext via their Influx Line Protocol outputs. By doing so you benefit from all other features Sematext platform provides – anomaly detection, ChatOps integrations, correlation with logs, distributed transaction tracing, etc. on top of data from other open-source agents.

    Sematext Java Agent 4

    Sematext Agent: Input and output options

    JVM Profiling

    Although this agent is written in Java, it is not limited to monitoring Java applications.  For example, you can use it to monitor MySQL or HAProxy, Redis, and so on, none of which are written in Java.  However, because it is written in Java this agent can not only collect metrics but also perform on-demand JVM profiling.  By “on-demand” we mean that you can turn the JVM profiler on and off in Sematext Cloud with a single button click.  The JVM profiler is very light-weight and collects CPU Time, Garbage Collection Time with class and function names to detect e.g. slowest function calls in the JVM.  You can choose which nodes and which specific JVMs to run it on and for how long.

    Screen Shot 2018 10 05 at 12.26.33

    JVM profiling data in Sematext Cloud

    Distributed Transaction Tracing

    This agent uses Java Bytecode Instrumentation (BCI) to instrument a set of known APIs and trace distributed transactions spanning multiple services and nodes. Enabling tracing in the agent will also enable database tracing for SQL (JDBC), Solr, and Elasticsearch clients, providing you insight into all slow transactions down to database-level detail (e.g. slow SQL, slow search queries, etc.) or transactions that resulted in an error.  To instrument your own JVM-based application all you need to do is define the APIs to trace via custom pointcuts. The nice thing about BCI-based instrumentation is that it does not require you to change your source code.  An alternative, should you prefer the annotation approach, is to use the trace annotations.

    Apache Software License

    The Sematext Java Agent and all its application integrations are released under the Apache Software License v2 license, just like our previously open-sourced monitoring agents:

  • Update your Monitoring Agent

    If you are an existing Sematext Cloud user, we encourage you to log in and update your agent to start benefiting from the lower footprint and flexible configuration options. If you are new to Sematext, have a look at the live demo to poke around and explore not just Sematext Monitoring, but also Sematext Logs.