Our platform provides ready to use monitoring agents and log shippers.
It exposes the Elasticsearch API and syslog receivers, so you use your favorite
open-source tools to get data in or out of Sematext.
Ready Made Dashboards
Create a monitoring app. Install the agent. Invite teammates. It takes only 5 minutes.
You won’t need to figure out what to monitor or how to collect key metrics. We’ve done that for you.
Want custom Dashboards? No problem!
Default Set of Alerts
If you’re monitoring a JVM app, you’ll find alert rules for JVM heap. If you’re monitoring Elasticsearch you’ll see anomaly detection rule for the number of nodes in your cluster. Don’t like your default roles? Delete or disable them and add your own!
Solr is highly reliable, scalable and fault tolerant, providing distributed indexing, replication and load-balanced querying, automated failover and recovery, centralized configuration and more. Solr powers the search and navigation features of many of the world’s largest internet sites.
Monitor search latency by core, indexing rate, caches, etc. plus JVM and OS metrics
Apache Solr Traces
Elasticsearch is a powerful open source search and analytics engine that makes data easy to explore based on Lucene. It provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents. Elasticsearch is developed in Java and is released as open source under the terms of the Apache License.
Cluster status, nodes, index, shard, search, cache, thread pool, circuit breaker and more
Worker count, event loop latency, process memory and GC metrics, request and error count and rate, etc.
Apache Spark™ is a fast and general engine for large-scale data processing. In contrast to Hadoop’s two-stage disk-based MapReduce paradigm, Spark’s in-memory primitives provide performance up to 100 times faster for certain applications. By allowing user programs to load data into a cluster’s memory and query it repeatedly, Spark is well suited to machine learning algorithms. For persistence, Spark can use either Hadoop Distributed File System (HDFS) or Cassandra.
Master, workers, drivers, executors, straming, apps, jobs, stages, storage, etc.
Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact.
Collect all key producer, consumer, and broker metrics
Docker is an open platform for developers and sysadmins to build, ship, and run distributed applications. Consisting of Docker Engine, a portable, lightweight runtime and packaging tool, and Docker Hub, a cloud service for sharing applications and automating workflows, Docker enables apps to be quickly assembled from components and eliminates the friction between development, QA, and production environments.
Monitor all container and host metrics and collect container events
CoreOS is an open-source lightweight operating system based on the Linux kernel and designed for providing infrastructure to clustered deployments, while focusing on automation, ease of applications deployment, security, reliability and scalability. As an operating system, CoreOS provides only the minimal functionality required for deploying applications inside software containers, together with built-in mechanisms for service discovery and configuration sharing.
The Apache Cassandra database is the right choice when you need scalability and high availability without compromising performance. Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data. Cassandra’s support for replicating across multiple datacenters is best-in-class, providing lower latency for your users and the peace of mind of knowing that you can survive regional outages.
Cluster, read/write latency, pending reads/writes/cluster ops, caches, SSTable and more
Apache HBase™ is the Hadoop database, a distributed, scalable, big data store. Use Apache HBase when you need random, realtime read/write access to your Big Data. This project’s goal is the hosting of very large tables — billions of rows X millions of columns — atop clusters of commodity hardware. Apache HBase is an open-source, distributed, versioned, non-relational database modeled after Google’s Bigtable. Just as Bigtable leverages the distributed data storage provided by the Google File System, Apache HBase provides Bigtable-like capabilities on top of Hadoop and HDFS.
Major and minor compactions, flush queue, region servers, MOB, WAL, replication, and other metrics
Apache Hadoop – YARN, HDFS, MapReduce
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.
Monitor MapReduce, HDFS NameNode and DataNode, YARN ResourceManager and more
MongoDB is a cross-platform document-oriented database. Classified as a NoSQL database, MongoDB eschews the traditional table-based relational database structure in favor of JSON-like documents with dynamic schemas (MongoDB calls the format BSON), making the integration of data in certain types of applications easier and faster. Released under a combination of the GNU Affero General Public License and the Apache License, MongoDB is free and open-source software.
Storage, memory, locks, documents, operations, journal, flushes, OS metrics, and more
Apache Storm is a free and open source distributed realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Storm is simple, can be used with any programming language, and is a lot of fun to use!
Cluster, supervisor, topology, bolt input/output, and spout metrics
Amazon Web Services – AWS
Amazon Web Services (AWS) is a collection of remote computing services, also called web services that together make up a cloud computing platform by Amazon.com since 2006. The most central and well-known of these services are Amazon EC2 and Amazon S3. The service is advertised as providing a large computing capacity (potentially many servers) much faster and cheaper than building a physical server farm.
Monitor ELB, EBS, and EC2 metrics
Amazon Relational Database Service (Amazon RDS) makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and resizable capacity while managing time-consuming database administration tasks, freeing you up to focus on your applications and business. Amazon RDS provides you six familiar database engines to choose from, including Amazon Aurora, Oracle, Microsoft SQL Server, PostgreSQL, MySQL and MariaDB.
Amazon Aurora is a MySQL-compatible, relational database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. Amazon Aurora provides up to five times better performance than MySQL at a price point one tenth that of a commercial database while delivering similar performance and availability.
Apache Tomcat™ is an open source software implementation of the Java Servlet and JavaServer Pages technologies. The Java Servlet and JavaServer Pages specifications are developed under the Java Community Process. Apache Tomcat powers numerous large-scale, mission-critical web applications across a diverse range of industries and organizations.
Monitor cache size and utlization, request counts, traffic in/out, threads, tasks, data sources, JVM, etc.
Apache HTTP Server
The Apache HTTP Server, colloquially called Apache, is the world’s most widely-used Web server software. Originally based on the NCSA HTTPd server, development of Apache began in early 1995 after work on the NCSA code stalled. Apache played a key role in the initial growth of the World Wide Web, quickly overtaking NCSA HTTPd as the dominant HTTP server, and has remained the most popular HTTP server since April 1996. In 2009, it became the first Web server software to serve more than 100 million Web sites.
Workers, connections, and multiple scoreboard metrics, with PHP FPM Pool support
Nginx & Nginx Plus
Nginx (pronounced engine-x) is a high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server. Nginx is known for its high performance, stability, rich feature set, simple configuration, and low resource consumption. Unlike traditional servers, Nginx doesn’t rely on threads to handle requests. Instead it uses a much more scalable event-driven (asynchronous) architecture. This architecture uses small, but more importantly, predictable amounts of memory under load.
Request count and rate, read/write connections, PHP FPM Pool requests and processes, and OS metrics.
Request count and rate, read/write connections, status zone traffic requests, responses and more.
Redis is an open source, BSD licensed, advanced key-value cache and store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs. Redis maps keys to many types of values. A key difference between Redis and other structured storage systems is that Redis supports not only strings, but also abstract data types. The type of a value determines what operations (called commands) are available for the value itself. Redis supports high-level, atomic, server-side operations like intersection, union, and difference between sets and sorting of lists, sets and sorted sets.
Used memory, fragmentation ratio, evictions, commands, expirations, keyspace, connected clients and more
MySQL is a popular choice of database for use in web applications, and is a central component of the widely used LAMP open source web application software stack (and other ‘AMP’ stacks). LAMP is an acronym for “Linux, Apache, MySQL, Perl/PHP/Python.” Free-software-open source projects that require a full-featured database management system often use MySQL.
All key MySQL and OS metrics, compatible with AWS RDS and MariaDB
MariaDB is a community-developed fork of the MySQL relational database management system intended to remain free under the GNU GPL. Being a fork of a leading open source software system, it is notable for being led by the original developers of MySQL, who forked it due to concerns over its acquisition by Oracle. Contributors are required to share their copyright with the MariaDB Foundation.
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them ,which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.
Connection and node stats, request latency, outstanding requests, etc.
Free & open source, high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load. Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering.
Size, get, set, touch, and flush rates, system stats, and cache network traffic
HAProxy is a free, open source high availability solution, providing load balancing and proxying for TCP and HTTP-based applications by spreading requests across multiple servers. It is written in C and has a reputation for being fast and efficient (in terms of processor and memory usage). HAProxy is used by a number of high-profile websites including GitHub, Stack Overflow, Reddit, Tumblr, and Twitter and is used in the OpsWorks product from Amazon Web Services.
Session count and rate, traffic in/out, failures, server status, role, OS metrics, etc.
PHP is a general-purpose scripting language that is especially suited to server-side web development, in which case PHP generally runs on a web server. Any PHP code in a requested file is executed by the PHP runtime, usually to create dynamic web page content or dynamic images used on websites or elsewhere.
Java is a general-purpose computer programming language that is concurrent, class-based, object-oriented, and specifically designed to have as few implementation dependencies as possible. It is intended to let application developers “write once, run anywhere” (WORA), meaning that code that runs on one platform does not need to be recompiled to run on another. Java applications are typically compiled to bytecode that can run on any Java virtual machine (JVM) regardless of computer architecture. Java is, as of 2014, one of the most popular programming languages in use, particularly for client-server web applications, with a reported 9 million developers.
Slack is a team communication tool that provides Mac, iOS, and Android apps, as well as Web versions. It offers one-on-one messaging, private groups, persistent chat rooms, direct messaging as well as group chats organized by topic. All content inside Slack is searchable from one search box. Slack integrates with a number of third-party services, including Google Docs, Dropbox, Heroku, Crashlytics, GitHub and Zendesk
PagerDuty provides alerting, on-call scheduling, escalation policies and incident tracking to increase uptime of your apps, servers, websites and databases. PagerDuty is an operations performance platform delivering visibility and actionable intelligence across the entire incident lifecycle.
Nagios is the industry-standard in IT infrastructure monitoring. Nagios provides enterprise-class Open Source monitoring of hosts, services, applications, and networks. Learn why organizations around the world trust Nagios to handle their IT infrastructure monitoring and business process continuity.
A webhook (or WebHook), in web development, is a method of augmenting or altering the behavior of a web page, or web application, with custom callbacks. These callbacks may be maintained, modified, and managed by third-party users and developers who may not necessarily be affiliated with the originating website or application.
Icinga is an open source network and computer system monitoring application. It was originally created as a fork of the Nagios system monitoring application in 2009. Icinga is attempting to get past perceived short-comings in Nagios’ development process, as well as adding new features such as a modern Web 2.0 style user interface, additional database connectors (for MySQL, Oracle, and PostgreSQL), and a REST API that lets administrators integrate numerous extensions without complicated modification of the Icinga core.
Shinken is a monitoring framework. It’s a Python Nagios® Core total rewrite enhancing flexibility and large environment management. Features include ready to run monitoring packs, endless scalability and RAID-like availability, focus on critical business impacts only, full virtualization integration, and ability to keep your Nagios configuration and plugins.
SenseiDB is a distributed data system that was built to support many product initiatives at LinkedIn.com, e.g. LinkedIn Signal and the LinkedIn Homepage. It is foundation to the LinkedIn’s search and data infrastructure. Sensei is both a search engine and a database. Sensei is designed to query and navigate through documents with parts that contain text and are unstructured, as well as parts containing meta information that have well-formed structures.
Brokers, nodes, partitions, gateways, and more
Kamon is Open Source tool for monitoring applications running on the JVM. Its core components provide a clean and simple API for recording metrics and trace information for any application running on the JVM. Enhance your onboarding experience with bytecode instrumentation modules that automatically measure and trace your application with modules for Scala, Akka, Spray and Play!
Akka is an open-source toolkit and runtime simplifying the construction of concurrent and distributed applications on the JVM. Akka supports multiple programming models for concurrency, but it emphasizes actor-based concurrency, with inspiration drawn from Erlang. Language bindings exist for both Java and Scala. Akka is written in Scala, and as of Scala 2.10, Akka’s actor implementation is included as part of the Scala standard library.
Actors time in mailbox, processing time, error counts, dispatchers, routers, JVM and OS metrics
Red Hat is the world’s leading provider of open source solutions, using a community-powered approach to provide reliable and high-performing cloud, virtualization, storage, Linux, and middleware technologies. Red Hat also offers award-winning support, training, and consulting services. Red Hat is an S&P 500 company with more than 80 offices spanning the globe, empowering its customers’ businesses.
CentOS (abbreviated from Community Enterprise Operating System) is a Linux distribution that attempts to provide a free, enterprise-class, community-supported computing platform which aims to be functionally compatible with its upstream source, Red Hat Enterprise Linux (RHEL). In January 2014, CentOS announced the official joining with Red Hat while staying independent from RHEL, under a new CentOS governing board.
Debian is the name for several Gnu/Linux operating systems, composed primarily of free and open-source software, most of which is under the GNU General Public License, and developed by a group of individuals known as the Debian project. At each point in time the Debian project offers three Linux distributions, named “stable”, “testing”, “unstable” officially. The Debian Stable distribution is one of the most popular for personal computers and network servers, and has been used as a base for several other Linux distributions.
Deis is an open source Platform as a service (PaaS) that leverages Docker, CoreOS and Heroku Buildpacks to provide a private application platform that is lightweight and flexible. It is an open source PaaS that makes it easy to deploy and manage applications on your own servers. Deis builds upon Docker and CoreOS to provide a lightweight PaaS with a Heroku-inspired workflow.
Scala is a programming language for general software applications. Scala has full support for functional programming and a very strong static type system. This allows programs written in Scala to be very concise and thus smaller in size than other general-purpose programming languages. Many of Scala’s design decisions were inspired by criticism of the shortcomings of Java.
Groovy is a powerful, optionally typed and dynamic language, with features similar to those of Python, Ruby, Perl, and Smalltalk, and static-typing and static compilation capabilities, for the Java platform aimed at multiplying developers’ productivity thanks to a concise, familiar and easy to learn syntax.
SolrCloud is the name of a set of new distributed capabilities in Solr. Passing parameters to enable these capabilities will enable you to set up a highly available, fault tolerant cluster of Solr servers. Use SolrCloud when you want high scale, fault tolerant, distributed indexing and search capabilities.
Monitor search latency by collection, indexing rate, caches, JVM and OS metrics
BigPanda offers the world’s first Autonomous Digital Operations Platform that intelligently automates IT incidents. Powered by their unique Open Box Machine Learning technology, BigPanda helps your IT Ops team handle more incidents, respond more quickly, and dramatically improve on key metrics such as MTTD, MTTA and MTTR.
Sematext Logsene exposes the Elasticsearch API that thrives in the center of the rich ecosystem of data shipping, analysis and visualization tools and services.
- Use any data shipper with Elasticsearch output to send your data to Sematext Cloud in a jiffy
- Use any visualization or analysis tool with Elasticsearch support to get your data out of Elasticsearch – Kibana, Grafana, Apache Zeppelin, Zoomdata, etc.
- Use the Logsene Elasticsearch API for indexing, searching, aggregations, scan and scroll with curl or any Elasticsearch client library
- Run an in-house Elastic/ELK Stack, tired of managing it, and want to give Sematext a go? Keep using your existing data shippers like Logstash, Filebeat, or Fluentd and just point them to Sematext HTTP API using your app token as your index.