Skip to main content

2020

Logs in Chart Builder

We’re excited to introduce logs in Chart Builder and hope you’ll enjoy using it as much as we’ve enjoyed developing it!

This gives you new possibilities to draw charts using data from Sematext Logs. We’ve made Logs charting more powerful and easier to use. You can select counts or any numeric fields to chart data. You can also use mathematical operations and functions to apply transformations. As always you can filter, group, and roll up metrics by various criteria, as well as style them to your heart’s content.

With variables, you can now dynamically filter and group charts by tags. This makes it even easier to drill down to issues when troubleshooting!

How do you start using Chart Builder for Logs? Two ways:

1. By editing any chart in your Logs Apps
2. By adding a new component to any dashboard or any report, as shown below

Charts you create with Chart Builder will be added to your dashboards and reports as dashboard components.

Stay tuned for more updates! We’re currently developing new types of components for Logs and Metrics in Chart Builder.

Introducing Mobile Application Log Analysis for Android and iOS

With the new mobile application logs integration, you get out of the box insights into what your mobile apps are doing. By using the open-source iOS or Android mobile app SDKs you can collect your mobile apps' logs and use them for troubleshooting, analytics, and more.

This integration includes a number of out of the box dashboards. There is a dashboard specifically for iOS and another one for Android. There is a dashboard for errors across both Android and iOS with several reports to quickly let you get the sense of error patterns, trends, and numbers. The overview report, which you can see above, is designed to be easily scannable and digestible and give you a birds-eye view of the health of your mobile applications. All data can be filtered and grouped by a number of tags, so you can easily narrow things down to a particular device type, version, geographical region, and more. Note that in Sematext you can create multiple Apps, so if you have more than one mobile app you could ship all their logs to a single App or multiple Apps in Sematext. It’s up to you how you like to see and analyze your data!

Once you have your mobile app logs in Sematext you can create alerts on and be notified about unexpected errors and other types of problematic conditions that could otherwise be overlooked and lead to higher customer churn.

The Geo dashboard shows you where on the planet your mobile users are located. The location data is collected automatically if you enable that via Sematext mobile app SDKs and if users have their location services turned on, of course. The SDKs send latitude and longitude data and Sematext does the rest to show you users by country or continent…

Discovery is here: Automatically monitor discovered services!

Have you ever wished you could monitor your software stacks with zero configuration changes and no service disruptions?

autodiscovery service  

With Discovery, we automate the discovery and monitoring of your services regardless of their runtime target. It works across bare metal and VM environments, Docker containers, including Docker Swarm, and Kubernetes workloads.

We give you a simple way to monitor services with the click of a single button. Even in dynamic environments and with auto-scaling. Our monitoring rules ensure replicas are automatically monitored when you scale them up. Monitoring also stops as soon as those replicas are removed as a side effect of scaling down.

To see what we’ve discovered for you, make sure you have a recent Sematext Agent (or Sematext Europe) and head over to Discovery in Sematext (or Sematext Europe) or learn about Discovery in the docs.

New Vercel Integration with error tracking for serverless functions and Jamstack websites!

With our new Vercel Logs Integration you get insight into your whole Vercel account. View logs from all your projects, serverless functions, static and Jamstack websites, including build logs and errors!

We’ve given you a quick way of integrating Vercel with Sematext Logs by adding our Sematext Log Drain Integration to your Vercel account and hooking it up to forward logs to Sematext.

We wanted to make your life easier by giving you out-of-the-box dashboards that show your build, static, and function logs for a complete overview of your Vercel deployments, projects, and apps. Now you can analyze all data about your Vercel projects and apps in one place.

The cherry on top is that you can set alerts for errors, timeouts, and cold starts, keeping a close eye on your serverless functions. Even better, you can search for specific logs with a full-text search and filter logs based on sources, status codes, errors, and much more!

Vercel enables you to host Jamstack websites and web services that deploy instantly, scale automatically, and require no supervision, with no configuration. Sematext enables you to make sure your Jamstack websites and web services are operating at peak performance.

 

Introducing Sematext Synthetics, an API & Website Monitoring Solution

We are happy to announce the release of Sematext Synthetics. With Synthetics, you can monitor availability (uptime) of APIs and websites. You can also monitor your website performance and user journeys.

Sematext Synthetics monitors your APIs/websites 24×7 from multiple locations around the globe and alerts you when they fail or perform poorly. Sematext Synthetics also provides detailed reports on their availability and performance.

With the addition of Synthetics to Sematext Cloud along with infrastructure monitoring, log management, and real user monitoring, you get end-to-end visibility to your application performance.

With Sematext Synthetics, you can:
  • Monitor the availability of your APIs and track their performance
  • Monitor SSL certificate expiration time
  • Monitor your websites and track various web performance metrics like page load time, paint metrics, etc.
  • Monitor individual resource performance in a website by domain, type and response code and identify the slowest resources
  • Monitor third-party resources performance
  • Investigate slow pages & identify bottlenecks using a waterfall chart
  • Script user journeys with custom assertions based on contents of the webpage
  • Collect custom metrics from your sites and track them
  • Benchmark against competitors
  • Create custom dashboards and alerts based on Synthetic performance metrics
  • Debug downtimes with end-to-end visibility of your application performance
  • Get alerted when things fail via multiple alert channels like emails, Slack, Pager Duty, WebHooks, etc.

Check out this short video for more info.

Read more about Sematext Synthetics, here.

Flexible, Tag-aware Heartbeat Alerts

When we first released Heartbeat Alerts years ago, we made it possible to get alerted when a monitored service running on a host stopped sending metrics to Sematext. This works well in the static world of virtual machines and bare-metal servers. However, it isn’t suitable for the dynamic world of containers.

We have now added the ability to create Heartbeat Alerts based on any combination of tags. Meaning, Heartbeat Alerts can be tied to specific services regardless of where they are running. You can specify tags to include or exclude, as well as tags to group by, then create Heartbeat Alerts for services matching your criteria. This way, even if a containerized service is moved from one host to another, it will not trigger false alerts.

To use tag-based Heartbeat Alerts you’ll want to update the Sematext Agent to the latest version. It includes tags from the Sematext Common Schema that tag-based alerts use:

  • os.host
  • container.id
  • container.image.name
  • container.image.tag
  • container.name

Using these tags you can do things like:

  • Filtering by and grouping by container.image.name to get alerted when all containers with the specified image name are gone.
  • Grouping by container.id to get alerted when that particular container dies. Yes, this could lead to a lot of alert noise, so use this sparingly and wisely.
  • Grouping by os.host and container.image.name to get alerted when a host stops running all containers with some image name (container may have moved to another host)

We have also improved the Heartbeat Alerts preview, which now shows all “heartbeats” – one for each combination of selected tags – in a tooltip when you hover, as shown in the screenshot below.

Read more about creating Heartbeat alerts here.

Brand new GitHub Webhook Events Integration with pre-built reports!

Store all data about triggered GitHub Events in one place. When an event is triggered, GitHub sends an HTTP POST payload to Sematext.

Gain insight into the entire GitHub workflow in your Organization, including your Organization’s team pulse, patterns, metrics, development velocity, progress, trends, helping you to spot opportunities for improvement in your team’s daily workflow.

Keep your team happy by keeping them productive and fulfilled.

The GitHub Events we support are:

  1. Commit comments
  2. Branch or tag creation
  3. Branch or tag deletion
  4. Issues
  5. Issue comments
  6. Pull requests
  7. Pull request reviews
  8. Pull request review comments
  9. Pushes
  10. Releases

The path to increasing developer happiness and productivity starts here!

New in Experience – Long Tasks

Please note, to get this data for your website you will need to update the JavaScript snippet. Go to App Settings -> Installation to copy the new JavaScript snippet.

Long tasks are tasks that block the main thread for 50 milliseconds or more.

Long tasks can cause issues such as:

  • Clunky animations and scrolling affecting user experience
  • High input latency
  • A long delay when triggering event handlers
  • A long delay between your application’s first-time-to-paint and when it becomes interactive

Understanding what causes these long tasks in your browser is crucial for improving performance. Since browser JavaScript is single-threaded, tasks that take longer than 50 milliseconds to finish can interfere with the user interacting with the website. This might cause render blocking behavior and flickering when loading your website.

We now display Long Tasks information in Experience. Read more about Long Tasks in Experience here.

Say Hello to Monitoring ARM64 with Sematext!

With the growth in popularity of ARM-based processors, we’ve added support for all ARM v8 architectures to our Sematext Agent and Logagent. This includes all ARM-based EC2 instances – both Graviton and Graviton 2, as well as embedded devices, Raspberry Pi, etc.

Our agents can run natively, but also as Docker containers, on ARM-based machines. You’ll get the same experience with collecting metrics and logs as with our current agents.

With our Infrastructure dashboards that contain metrics about servers, containers, processes, and much more, you can monitor your ARM instances alongside your existing machines.

New and Improved Alerts for Experience

When we first released Experience, we also made it possible to create alerts for different Apdex scores. That worked well enough but now we have improved alerts support in Experience and added additional types of alerts that you can create for your Experience Apps.

This gives you additional flexibility in deciding what is important for your website or webapp.

In addition to Apdex scores, you can now create alerts for:

  • First contentful paint (FCP) time
  • First paint (FP) time
  • HTTP requests count
  • HTTP requests load time
  • Long tasks count
  • Long tasks duration
  • Page load backend load time
  • Page load frontend load time
  • Page load time
  • HTTP resource load count
  • HTTP resource load time
  • HTTP resource transfer size
  • User sessions count
  • Users count

We have also improved the interface for creating alerts so you can see a preview of the alert.

Learn more about creating Experience alerts.

Brand new Kubernetes Audit Logs Integration with pre-built reports!

Kubernetes audit logs are detailed descriptions of each call made to the Kubernetes API Server. They provide a chronological sequence of activities that lead to the state of the system at a specific moment.

Configured with either a Dynamic Backend or a Log Backend, get access to Kubernetes audit logs without using any agents.

Gain insight into Authentication and Authorization requests hitting your Kubernetes API server, alongside errors and timeouts that help you pinpoint users, hosts and IP addresses that should not be accessing your resources.

Track error codes, verbs, and resources to pinpoint what is getting accessed with what kubectl command.

As you can see, Kubernetes audit logs are extremely useful for security and compliance purposes, telling you exactly who did what, when, and how.

Introducing a new type of chart – the Heatmap!

The Heatmap is a two-dimensional representation of data, where values are represented using color saturation. The vertical axis shows weekdays, while the horizontal axis represents the hours in a day.

With our latest release of a brand new Alerts Overview page, you gave us invaluable feedback about the Heatmap chart. With Heatmap you can visually represent historical data to gain actionable insight into how to improve performance.

You can select one or more metrics and use mathematical operations and functions to apply transformations to them. You can also filter, group, and roll up metrics by various criteria.

You can create Heatmap components in two places in Sematext Cloud:

  1. In any report in your Monitoring Apps
  2. In any dashboard or any report

Introducing a brand new Alerts Overview with improved UI!

Yes, it’s April 1st. No, we are not joking! We’ve been working hard to make our Alerts even more awesome! This is what we’ve come up with. A brand new Alert Landing Page. From now on when you receive an alert notification it takes you to a new page dedicated to giving you actionable insight into the issue that happened.

The Alert Landing Page contains all necessary details about the cause of a problem, when it happened, what broke, and any related information about other alerts that happened around the same time, as well as Alert Rule details and related charts.

We also improved the way you navigate through Alert Rules and Alert Notifications. We’ve introduced an Alert Page that contains a stream of Alert Events belonging to an Alert Rule and a heatmap that helps you find what point had the highest density of Alert Events.

Lastly, we also have a new Alert Rules table that contains links to both Alert Rule details and info about the most recent Alert Events belonging to an Alert Rule.

Read more in our docs about how to configure Alerts.

Updated Support for Kubernetes Logs

Kubernetes Logs

We have updated our Kubernetes logs integrations! You can now ship a wide variety of Kubernetes logs to Sematext by configuring our Logagent. We now also support the Containerd and CRI-O container runtimes as well as Docker, which is currently the most popular runtime.

After creating a Logs App, choose a Kubernetes integration in the Containers sections and follow the instructions to get started.

The instructions allow you to choose which setup you’re using. We support everything from Helm 2 and Helm 3 to Kubectl, Docker Stack, Swarm and Compose.

If you’re familiar with our Logagent, this is all possible because we’ve developed new Kubernetes filters.

Read more in our docs about how to configure log shipping.

We have a new Integration for Kubernetes Audit Logs!

Kubernetes Audit Logs

We have a new Kubernetes Audit logs integration! You can now ship Kubernetes Audit logs directly to Sematext, without any overhead, by configuring either a Dynamic Backend or Log Backend. You don’t need to install any agents!

After creating a Logs App, choose the Kubernetes Audit integration and follow the instructions to get started.

If you are already using Logagent, we’ve developed a new Kubernetes Audit plugin there as well. Configuring it is as simple as always.

Read more in our docs about how to configure log shipping.

Inventory addition: Container Images Report

Do you know which images your containers are based on? Do you know how many different versions of each image you have in production? Does your container image digest match the one in the registry?

These are all important questions you need answers to. Guess what? We’re giving you the answers today!

Container images inventory

You can find the Container Images report in the Inventory section of the Infrastructure menu.

For each container image you can see its version, the host where it’s downloaded, which registry it came from, the digest, time of creation, and size. You can also group and filter by relevant tags.

We have a new Logs Integration for Journald!

Journald Logs

We have a new Journald logs integration! You can now ship Journald logs directly to Sematext, without any overhead, by configuring a single URL. You don’t need to install any agents!

After creating a Logs App, choose the Journald integration and follow the instructions to get started.

If you are already using Logagent, we’ve developed a new Journald plugin there as well. Configuring it is as simple as always.

Read more in our docs about how to configure log shipping.

Introducing the new Process Overview Infrastructure report with 4 brand new charts!

We’ve been hard at work to bring you a new Process Overview report to our Infrastructure Process Monitoring. First update your instance of Sematext Agent, or run the latest version of the container if you’re using Docker or Kubernetes.

You’ll get insight into crucial process metrics in your infrastructure, giving you an easy way to aggregate, group and filter process-related metrics. Head over to our docs to set it up if you haven’t already.

Want to see CPU usage per PID? Check! Want to see the memory usage for each type of process you have? We have it!

This is a helpful feature if you have several different runtimes, such as Python, Node.js, or Java running across your infrastructure. You can quickly group the processes by their name and then do further aggregations and filtering to visualize data.

You can also group by the process type, which can be crucial in runtimes like Node.js where you have master and child processes.

With the addition of custom tagging, giving each process a name for easy filtering is something you need to try! Check out this blog post to learn more.

Latest performance tweaks for the Sematext Agent cut the CPU & Memory usage almost in half!

50% memory decrease

The latest release of the Sematext Agent cuts down the CPU and memory footprint by almost a half! If you haven’t updated your Sematext Agent instances in a while, now’s the time to do it.

We’ve completely reworked the event pipeline for intercommunication within the agent. It’s now replaced with an even more lightweight version. This, alongside a significant number of low-level tweaks, cut the already tiny CPU usage by a third, while cutting the memory usage in half!

Meanwhile, we took the liberty to add even more data collection features for you, such as even more detailed process metrics. But, you’ll have to wait and see what we have in store for you there. You’ll have access to it in Sematext very soon.

Now you want to update Sematext Agent, don’t you? It’s simple, check it out.

Introducing Metrics Chart Builder

We’re excited to give you the new Chart Builder and hope you’ll enjoy using it as much as we’ve enjoyed developing it!

Besides its complete visual overhaul, we’ve made it more powerful and easier to use. You can select one or more metrics data series to chart and use mathematical operations and functions to apply transformations to them. You can also filter, group, and roll up metrics by various criteria, as well as style them to your heart’s content. If you are a Sematext power user you might like the new support for variables, which are used to dynamically filter and group charts by tags.

How do you start using Chart Builder? Two ways:

1. By editing any chart in your Monitoring Apps 2. By adding a new component to any dashboard or any report, as shown below

Charts you create with Chart Builder will be added to your dashboards and reports as dashboard components.

You will soon be able to use Chart Builder with your logs in Sematext as well. Stay tuned!

Experience: Load Time Breakdown

Load Time Breakdown

We now show a more detailed load time breakdown whenever we have the data available. Hard page loads, same-origin HTTP requests and resources have detailed load time breakdown.

This breakdown helps you understand which part of the process affects your load times the most. Read more in the documentation to understand what each part means.

Group Performance Data in Experience

The new Group By selector allows you to slice the data and find performance differences between browsers, countries, device types, etc. Click on the Group By selector next to the filters and select the field to get started.

Group by also works well in combination with filters.

If you have defined any custom tags for your users, you can group by those as well.

Resources Insights added to Experience

Resources Insights

Resources pages will give you detailed information about each resource that was fetched while loading your website.

This includes images, fonts, stylesheets, and other media. To better understand the metrics shown in the table, read more in the documentation.

2019

Sematext Experience Anomaly Detection and Alerting

Monitoring the user experience for your website is not a one-time task. While you iterate versions of your website, constantly deploying new changes, you need to monitor the user experience throughout the process. Any new change could lead to a drop in the Apdex Score.

You can now configure alert rules for the Apdex Score in Sematext Experience! Creating an alert rule is easy, click/hover on the Apdex Score emoji, then click Create Alert.

Alert sources can be:
  • Page Load Apdex
  • HTTP Requests Apdex
  • Transaction Apdex

All alerts support filters such as Country, OS, Browser, User, URL, Device Type, or Connection Type.

You can set the Apdex value for a threshold-based alert or the time window for anomaly detection. The alert notification is sent by email, or any chat ops integration such as PagerDuty, Slack, Nagios, BigPanda, Victor Ops or arbitrary web-hooks.

Not monitoring your website or web app yet? Try if free for 30 days in either Sematext Cloud US or Sematext Cloud EU.

Data Sampling in Sematext Experience

If you have a high traffic website or web app, we’ve got good news for you! With high-traffic websites or web apps you often don’t need to track 100% of your website traffic. Taking a reasonable sample of the website traffic is often good enough to gather reasonably precise, valuable, and actionable insights. For those using Sematext Experience for monitoring high traffic websites or web apps, sampling may provide a good tradeoff between the loss of precision and cost reduction.

Configure Sampling in Sematext Experience

If your Sematext Experience App is on the PRO plan you can now configure the sampling rate from 10 to 100% (default). Here’s how:

Not monitoring your website or web app yet? Try if free for 30 days in either Sematext Cloud US or Sematext Cloud EU.

Reactive Grid Layouts for Dashboards

We love dashboards! Our goal is to help you troubleshoot faster and glean valuable insights from charts, counters, log tables, and more on a single screen, without losing track of your data. When you work with dashboards regularly you often find things to improve, often by resizing existing widgets, changing their placement, removing them, or adding new widgets. We had a look into various dashboard layout frameworks, each with a different approach to managing dashboard layouts. Here are the 3 most common ones:

    • Free placement of widgets.You can place any widget anywhere. When you start adding multiple widgets you start needing to move or resize other widgets. Because the placement and layout is completely up to you this process can be difficult and time-consuming.
    • Grid layout with rows and columns. You can place widgets by snapping them to grid cells. The layout might be limited to a defined number of rows and columns. Widgets always have the size of one or more grid cells in a row. Due to the fixed size of widgets, such a layout might waste a lot of precious space.
    • Reactive grid layout. Best of both worlds. You can easily place and resize widgets, while other widgets are adjusted automatically. A preview and automatic size adjustment help you drag and drop widgets quickly and precisely where you want them. The flexible widget sizing helps to create layouts with a higher density of information, using the screen optimally.
This reactive grid layout is exactly what you will now see your dashboards using! You can still save your dashboard layout, or revert any changes you made but don’t want to keep. Have a look at the video and give it a try!

Track User Satisfaction Trends with Sematext Experience

The Apdex score is a great metric for tracking user satisfaction for your website. In the initial release of Sematext Experience, our real user monitoring solution, we displayed a smiling, neutral or sad emoji.

Screen Shot 2019 10 24 at 11.44.47

Assuming you took the sad emoji seriously you have probably tuned your website to get more smileys.

Unfortunately, improving web performance is not that easy – many factors influence user experience. Therefore, we’ve added a new chart so you can see how the Apdex score changes over time. This Apdex score chart gives you much better insights into user satisfaction for your website.

Using the new chart for Apdex score over time, together with the page loads chart, you can discover a clear relationship between spikes in load times and decreased user satisfaction. You can also correlate changes in user satisfaction with your web app deployments or website changes.

Screen Shot 2019 10 24 at 11.49.46 Apdex score chart and page load count in Sematext Experience

Compute, Network, and Storage Reports for Infrastructure Monitoring

Infrastructure monitoring gives you a complete overview of the storage, network, and compute metrics of your resources.

You can find network, storage and compute reports in the Infrastructure section. All new reports include event histograms and a sorted grid view displaying servers with the highest resource usage. Another addition is the ability to group and filter metrics by relevant tags.

Following metrics are available in the new reports:

  • Compute: CPU, memory, system load
  • Network: send and receive transmission rates
  • Storage: IO read, IO write, disk space used

Autodiscovery for Kubernetes and Docker Enterprise

Do you run Elasticsearch, Solr, Zookeeper, Apache Kafka, or any other App in Kubernetes, Docker Enterprise or Swarm?

Then you know that it could be challenging to set up monitoring agents to scale with your dynamic applications. Some challenges are configuring and discovering dynamic IP addresses, port numbers, DNS resolutions, JMX access options or database credentials.

If you’ve ever wondered how to use Sematext for monitoring services running in containers, we’ve just made that incredibly simple. We’ve completely revamped the agent instructions to make them easier to follow with copy-paste-execute commands for all popular Sematext monitoring integrations. Instructions for running services in plain containers, Kubernetes, or Docker Swarm are all available.

The new autodiscovery feature in the Sematext Agent automates the deployment of application monitoring agents completely.

The first time you set up monitoring for an App, you add labels or annotations to your App containers with the monitoring token and any other optional properties for configuring the Agent.

The Sematext UI displays the copy/paste instructions for your App containers and Sematext Agent covering:

  • Docker
  • Docker Enterprise / Swarm
  • Kubernetes / OpenShift
  • Helm

Real User Monitoring with Sematext Experience

You can now use Sematext Cloud for user experience monitoring as well. New Experience Apps collect data directly from your website or webapp visitors and give you a complete picture of their experience and satisfaction. Monitoring your infrastructure and processes that run on your side gives you only half the story about how users perceive the performance of your website or single page app when they access it through their browser or mobile device.

Do you use your website to sell products or services? Website visitors are more likely to leave if your site is performing poorly.

Do you have a SaaS product? Your web app performance will directly impact your customers’ satisfaction, and therefore the adoption of your service.

Maybe your customers from North America are happy with the website performance, but customers from EU or Asia might have a completely different experience.

Browsers, network connectivity, operating systems, geographical location all influence your website performance and by getting the measurements directly from your visitors you can be sure that you are not missing any data you need to make your website perform better.

Experience Apps have support for both websites and single-page apps built with React, Angular, Ember.js, Vue.js and other frameworks. We track page loads, HTTP requests, user profiles, and sessions, and very soon crash reports as well. You can read the announcement or the documentation for more information, as well as start using Experience in Sematext Cloud US or in Sematext Cloud EU.

For more information, take a look at the Sematext Experience demo and dive into some of the features we offer in our new real user monitoring.

Visualize Any Metric with New Custom Charts

Monitoring integrations come with a predefined set of charts and dashboards. Great if you want to get started quickly and don’t really know which metrics you should be monitoring or don’t want to spend time building charts and dashboards from scratch. Now, in addition to these out of the box charts and dashboards you can also craft completely custom charts for monitoring your applications using all available metrics. These charts can be added to dashboards to visualize your infrastructure and application behavior.

 

Custom charts give you full control over how you want to visualize your data. Some data series are easier to analyze when presented as bar charts, others are better rendered as line charts. You can also customize colors, line styles, stacking, and more. It’s all to make your monitoring easier. You decide which presentation method gives you the most power. Ship metrics to Sematext with input format compatible with Influx and build your own dashboards with any metrics by its name.

  Imagine you want to correlate CPU and memory usage. The easiest way to see the correlation is to draw CPU and memory usage on the same chart/ CPU usage is expressed in percentages while memory is measured in bytes. No worries – just configure two Y-axes units and assign CPU usage to one and memory usage to the other Y-axis, as shown in this short video:

Do you want to group data series by hostname and sum them up to get total memory usage in your cluster? Just pick the tag to group by and the aggregation function. Do you want to see memory usage of each host separately? That’s just as easy: pick a grouping tag and no aggregation function (‘all separately’).  

Custom charts are a beta feature for now. Over the coming weeks you’ll be able to:
  • Perform math operations on your data. For example, you’ll be able to create a data series derived from two or more metrics.
  • Use filter variables, so all charts on the dashboard can be controlled interactively with a single click without the need to change each chart configuration separately.
  • Use the query editor if you prefer typing over clicking to create your charts.
  • Building charts that combine logs counts and metrics on a single chart.

Introducing Core Infrastructure Monitoring

Core infrastructure monitoring with Sematext just got better. How? The new agent collects container metrics and events just like the old agent, but it also collects Kubernetes metrics and events. As a result, you can now see a number of Kubernetes dashboards out of the box.

 

You can now identify processes that use the most resources, whether CPU or memory, across your entire infrastructure. Forget about ssh-ing to machines and running top. In Sematext you can see the hungriest processes right now, but also their behavior over time. Processes running in containers are also monitored, giving you visibility inside containers as well.

 

The new inventory monitoring captures information about installed packages, their versions, operating system information, Linux distributions, and many other useful bits. Package install and uninstall events details are also captured, so you can always trace who changes what, where, and when. Using Sematext you now have a place where you can identify outdated packages, packages with vulnerabilities, groups of servers with identical sets of packages as well as those machines that you thought have the same setup but in fact don’t, and more.

 

Network monitoring was rewritten to utilize eBPF whenever possible instead of relying on network packets. This new implementation has almost no overhead.

 

The agent was rewritten in Go, shedding the need for the JVM, resulting in smaller CPU and memory footprint.

 

All of the above-mentioned reports are a part of the new type of Monitoring App we are calling Infra App. You can have any number of Infra Apps in your Sematext account, though most people will have one Infra App per monitored environment. For example, you might create one Infra App for monitoring all of your production applications and infrastructure and another for monitoring your staging environment. Learn more about infrastructure monitoring.

Introducing Sematext Operator for Kubernetes Monitoring

If installing the Sematext Agent on Kubernetes manually felt too old fashioned and if Sematext Agent Helm Chart was not good enough, you’ll be pleased to hear that Sematext Agent is now available as a Kubernetes Operator. Sematext Operator is available on Github at https://github.com/sematext/sematext-operator. You can also find it certified in the RedHat’s OpenShift Container Platform via OperatorHub.io. Learn more in Kubernetes agent docs.

Inventory Monitoring

Do you know how many different Linux distros you are running in production? Do you know which ones you are running and their versions? How can you tell which version of openssh is installed on your servers? Or maybe you are running several different versions and you can’t really easily check that? Do you know if you are running any software with known vulnerabilities?

 

With Sematext’s new Inventory Monitoring you can now answer such questions with ease. The new infrastructure Inventory gives you information about your operating systems, Linux distributions, hardware specs, installed packages, log files, and more. The agent also detects package installation and removal events. Your event stream shows which packages were installed, removed, or updated, by who, where, and when. All inventory data is collected by the Sematext Agent and is part of core infrastructure monitoring. To start seeing your inventory information in Sematext be sure you are running a recent version of Sematext Agent. For more info check out inventory monitoring documentation and the official announcement.

Real-time Process Monitoring

You can now use Sematext for real-time process monitoring. With process monitoring you gain real-time visibility into which processes use the most CPU or memory, whether they are running inside containers, in cloud instances, your own VMs, or bare-metal servers. To start collecting information about processes make sure you are using a recent Sematext Agent. For more information see process monitoring documentation and the official announcement.

Alert Scheduling

You can now control during which times of the day and which days of the week alert rules are active or inactive. By default an alert rule is active all day, every day. The new alert scheduling lets you pick, for each day of the week, one or more time slots when the alert rule is active. Each alert rule can have its own schedule. Each day of the week can have different times when an alert rule is active. You can have different schedule Monday to Friday vs. Weekend, for example. The alert schedule can be modified at any time and it comes into effect immediately. Have a look!

2018

Custom Reports for Monitoring & Logs Apps

Sematext Cloud comes with predefined dashboards for every Monitoring App. A huge time saver, because you don’t need to spend hours creating your monitoring or logging dashboards. However, the predefined reports might not be optimal for your particular observability needs. Especially for Logs Apps, you might find that Sematext’s native reports are very handy as you won’t need to switch to Kibana to have a quick look at your report. All reports can be scheduled and delivered via e-mail (click to email icon in the report actions bar). We recently introduced several features that let you customize and extend Sematext App reports:
  • Modify the existing App reports e.g. by moving charts via drag & drop or remove a chart via the top-left menu in the chart.
  • Add or remove widgets in existing reports
  • Add or change the displayed filters
  • Add a new report page with your favorite metrics charts and filters
  • Clone a report
  • Organize reports in “folders” by adding groups for reports
  • Change the order of groups and reports in the left sidebar using the handle on the left side of each group or report in the left sidebar

Screen Shot 2018 11 02 at 17.39.52

 
Note: All changes to App reports are visible to all App users with access to that App after hitting the “save” button. Don’t be afraid of playing around – you can always reset the App to the original state using the “Reset all reports” menu item, next to the App name in the left navigation bar.
 

Custom Reports

All widgets, supported in Dashboards, are available in every App-specific report too. To get started with Dashboard functionality check out the Creating Dashboards in Sematext Cloud video.

ClickHouse Monitoring released

Yandex ClickHouse is an open source column-oriented database management system capable of real-time generation of analytical data reports using SQL queries. ClickHouse allows you to not only lower your TCO (Total Cost of Ownership) but also allows you to benefit from unmatched performance and scalability. With the release of the new Sematext Monitoring Agent, you can now monitor ClickHouse DB clusters. The setup, tuning and operations of ClickHouse require deep insights into the performance metrics such as locks, replication status, merge operations, cache usage and many more… The new ClickHouse integration supports over 70 different ClickHouse metrics for system, queries, merge tree, replication, replicas, mark cache, R/W buffers, dictionaries, locks, distributed engine and Zookeeper errors & wait times. Screen Shot 2018 10 15 at 12.45.38

ClickHouse key metrics in Sematext Cloud

How to use it
To monitor ClickHouse create a ClickHouse App in Sematext Cloud and install the Sematext Agent according to the setup instructions. Like magic, in less than a minute you will receive all ClickHouse metrics in your ClickHouse monitoring App.
  Tips  
Because ClickHouse heavily relies on ZooKeeper we recommend setting up ZooKeeper monitoring and log collection for related logs. By doing this you can benefit from combined dashboards thus having your favorite ZooKeeper and ClickHouse metrics together with related logs for faster troubleshooting. Save time, gain visibility: the benefits are endless.

New Sematext Monitoring Agent released

You asked, we listened. The new Sematext Monitoring Agent includes many features that will make your infrastructure and application monitoring a whole lot easier.

The recently opened-sourced Sematext Monitoring Agent is fully pluggable, making it possible for you to collect metrics from a number of additional sources. The new agent supports the latest Java versions and has updates for many integrations such as Apache Kafka, introduces support for ClickHouse monitoring, etc. In addition, we changed how the agent talks to Sematext. Instead of using a custom protocol, we’ve adopted the popular Influx Line Protocol for sending metrics and all their metadata via HTTPS. The old proprietary format will continue to work for a while to ensure backward compatibility with older agent versions.

You can find all details about the release in Open-sourcing Sematext Monitoring Agent.

If you are an existing Sematext Cloud user, we encourage you to log in and update your agent to start benefiting from the lower agent footprint and flexible configuration options.

The agent update is super simple: just use your package manager to update the spm-client package.

Upgrade to Elasticsearch and Kibana 6.4

We’re excited to announce that Sematext Logs has upgraded to Elasticsearch 6.4 and along with it to the shiny, new Kibana 6.4! All your visualizations and dashboards have been migrated to the latest Kibana 6.4 version for you, so you don’t have to do anything.Kibana 6.x comes with a number of improvements including:
  • New colors to improve accessibility
  • Vega interactive visualizations
  • Ability to apply Vega filters to dashboard context
  • Visualization of other and missing terms buckets
  • Lab visualizations
  • Improved workflow for inspecting data
  • Dashboard input controls
  • And more!
You can find the Kibana button on the menu […] next to the App Switcher in the logs view.
Start Kibana 6.4 in Sematext Cloud and explore new features and visualizations. Enjoy!

New Container View in infrastructure monitoring

The new Containers view provides you with enhanced visibility into all your containers and their hosts, drawing your attention to the ones that need it the most.

The main view is very much like “top for containers” – applied across all of your containers and their hosts. You can sort containers by memory or CPU usage to find the hottest containers, or simply display the top N containers. The detail view of each container displays container metrics in real-time.

In containers table, the user can see basic information and metrics about each server and by clicking on table row user can expand it to get more details about the container. Thus, for each container user can view metrics, review information about containers grouped by tags like host, image name, pod name …

By using the heatmap, user can get an overview of a large number of containers at once and see if selected metric has a high value on any of the monitored containers or host.

Want to find out more about the new container view? Read our blog post. Can’t wait to try the new containers view? Here is the direct link for Sematext Cloud USA or Sematext Cloud Europe.


Sematext Logs Elasticsearch API Extra Security

Sematext Logs Elasticsearch API Extra Security | July 2018

The Sematext API for log ingestion log search has always been compatible with the Elasticsearch API. On top of that, the Sematext API has always had additional security in place, for example:

  • HTTPS encryption on all Sematext data receivers
  • The API used the App tokens as index name, with the ability to generate multiple read-only or write-only tokens in the App settings. For example, somebody who needs to set up a log shipper really requires only the write-only token and doesn’t need to know the read-capable token, which is needed only if you want to query logs via the API.

To further secure access to our customers’ data we now require not just the App token, but also the API key for log searches via API. Importantly, note that you do not have to change any of your log shipping.

This change affects only access via read APIs, not write APIs. You can get your API key from https://apps.sematext.com/ui/account/api and, as you can see there, it can be regenerated at any time should it ever get compromised.

The following example shows a search request with curl using the new authentication header:

curl 'https://logsene-search.sematext.com:443/<APP TOKEN HERE>/_search?size=0&pretty' -H 'Authorization: apiKey <API KEY HERE>'

The API also supports HTTP Basic Authentication. Therefore third-party tools using Elasticsearch client libraries that use Basic Authentication with Elasticsearch can authenticate as well. The username is then the “apiKey” literal, and the password is the actual API key. Here is another example using curl with basic authentication:

curl -v -u apiKey:<API KEY HERE> 'https://logsene-search.sematext.com/<APP TOKEN HERE>/_search?size=0&pretty'

Better Servers View

Better Servers View

Servers view is giving overview over host servers used by your apps. User can use GroupBy and FilterBy filters to decide which servers or group of servers he wants to monitor. There two main components for monitoring: table-view and heat-map.

In servers table user can see basic informations and metrics about each server and by clicking on table row user can expands it to get more details about server. Thus, for each server user can view metrics, review informations about docker containers on that host and review metrics and informations for apps on that host.

By using heat-map user can get overview over large number of servers at once and see if selected metric has high value on any of monitored servers.


Automatic Metrics Correlation

Automatic Metrics Correlation

If you work with monitoring or monitoring tools much, you’ve probably seen the phrase “correlating” here and there. Correlation is what can help you find metrics with similar pattern and see dependencies between metrics even if you don’t suspect they are in dependency relation. When troubleshooting, automatic metrics correlation is the easiest way to find root cause.

Automatic metrics correlation in Sematext Cloud work pretty straight forward: you select a metrics of interested, select time range and Correlation mechanism will find all metrics which have similar behaviour in the same time range. Similar spikes, similar trends, similar waves.


Added Logtrail to Kibana

Added Logtrail to Kibana

Dealing with log files or extracting data from various data sources is a daily task in the IT administration. Adding logtrail to Kibana gives users the “terminal experience”, that experience of tailing and filtering log files like in the good old terminal that tons of people are used to and that many of us still love – typically black background, new logs appearing at the bottom of the screen instead of the top screen, simply way to filter logs and “jump” to various parts of the log stream.

To summarise this plugin enables you and your team to view, analyze and search log events from a centralised clean & developer friendly interface.

Log in to see for yourself and check it out….. or read our blog post to find out about a few more improvements we’ve made.


2017

Tighter Metrics & Logs Integration

Tighter metrics and logs integration sematext

Sematext has been offering monitoring and log management for a while now, but in the most recent release we’ve further unified it, making it really simple for people to go through the well-established DevOps path: Alert Notification ⇒ Metrics Charts ⇒ Log Inspection for details

Moreover, we’ve made it dead-simple to see metrics and logs in the single view and visually correlate between the two.  End result? Much faster troubleshooting ⇒ much more efficient root cause analysis ⇒ much shorter MTTR ⇒ happier DevOps & happier users!

For more details about this and the benefits this functionality enables for your team check out our blog post or read more about monitoring and logging with Sematext Cloud.

Improved Navigation, Discoverability, and Accessibility

Improved navigation, discoverability, and accessibility sematext

We’ve restructured the UI. Everything you need can be quickly accessed via tabs on the left.  Account section is exposed, and Team-related functionality (invitations to apps or to account, user roles, etc.) is in its own Teams section. Dashboards are in top-left. Metrics, Logs, and Events are right there next to each other. AppMap, NetMap, and a top-like view of all your Servers is under Infrastructure. Jumping to a specific Metrics or Logs App or a Dashboard can be done quickly from the searchable pulldown in top-right. If you have access to multiple accounts you can switch between them much like you can switch between multiple Google accounts.

Log in to see for yourself and check it out… or read our blog post to find out about a few more improvements we’ve made.

React.js & Redux, Pretty Charts, Better Dashboards

Reactjs redux pretty charts better dashboards sematext

Sematext Cloud UI looks and behaves differently. Indeed, we rewrote the whoooooole front-end using modern front-end technologies like React.js and Redux. We’ve switched to Chart.JS for charts, and contributed a few PRs to it along the way. We’ve made it possible to show/hide chart legends, change their positioning, to easily add/remove individual timeseries from metrics charts, and so on. Dashboards have been completely rewritten and are much, much faster and lighter than before. You can easily create a Dashboard that contains charts from multiple different Metrics or Logs apps. You can also add logs themselves to a Dashboard, or event charts, or components with custom Markdown in them.

For more details about this check out our blog post.

Log Event Context

Log event context sematext

When you’re troubleshooting and inspecting logs what do you do? You search and filter logs to narrow things down, to find log messages that will help you understand what, how, why, or when something happened. The key part here is that you search, and when you search what you typically see are only logs that matched what you searched for (think grep). Log Context in Sematext Cloud is much like that. Once you’ve found a log event of interest a single click will expose its context. It’s like grep -C NUM, but applicable across all your infrastructure, not just one log file at a time.

For more details on log event context check out our blog post or see it for yourself.

S3 Archiving & Longer Retention for Logs

S3 archiving and longer retention for logs sematext

Need to have your recent logs searchable for your DevOps team(s) to troubleshoot issues, but also need to archive old logs for compliance reasons? Many organizations need that and Sematext Cloud makes that simple. You can certainly simply pick a longer data retention option along with your plan and daily log volume. Doing that makes all your logs searchable for a long time. If, however, you simply need to store your older logs, so that you can retrieve them later on, but do not need to search them, you can use S3 Archiving option provided by Sematext Cloud. All you need to provide are Access Key ID, your Secret Access Key, and the name of the S3 bucket where you want your logs to be archived.

For more details on log event context check out our blog post.

Alert Rules & Notification Hooks Sharing

Alert rules and notification hooks sharing sematext

Alert Rules and related Notification Hooks (aka Integrations) sharing option was updated. Now each Alert Rule and each Notification Hook is tied to account under which it was created. This way any member of your team will still see all Alert Rules created under the common shared account. Each team member will be able to use and create new Alert Rules and Notification Hooks, which will also be visible to other members of the team (but editable only by the shared account owner, or users with admin role). Such Alert Rules and Hooks will not be visible to App guests though, so your team’s work in the common shared account will not be affected by them.

For more details on this new sharing option and its benefits over plain App sharing check out our blog post or see it for yourself.

Alert Rule & Notification Hook Management

Alert Rules and Notification hook management sematext

We’ve changed how Alert Rules & Notification Hooks are handled in Sematext Cloud. They now live at Account level, not App level, which makes their management much simpler, more intuitive, much more suitable for modern teams, as described in Alert Rules & Hook Notification Sharing. The new “Account Default” switch makes it easy to quickly change where alert notifications are sent for all Alert Rules that are set up to use default account notifications.  Very handy for agile teams who may change their alerting channel preferences over time, have lots of Alert Rules, and would go crazy if they had to modify each of them.

See it for yourself.

Expanded ChatOps Integrations

Expanded chatops integrations sematext

You can now hook up Sematext Cloud Alerts to a number of different Notification Hooks: email, any custom WebHook, Slack, PagerDuty, HipChat, VictorOps, OpsGenie, BigPanda, Pushover – for Android, iPhone, iPad, and Desktop (Pebble, Android Wear, and Apple Watch, too!), Nagios – yes, people still use it, Zappier – and from there you can connect Sematext Alerts to a number of other services. Setting any of these notification hooks takes only seconds.  You only need to configure them once and then, because notification hooks and alerts are set at account-level, you can reference them when creating alert rules.

Elasticsearch 5.x and Kibana 5.x

Elasticsearch 5.x and kibana 5.x sematext

We’ve refreshed our backend, or one of them at least. We’ve updated Elasticsearch to 5.x version. Along with it, we’ve updated the integrated version of Kibana to which you can easily switch from the “Native UI” if you are a Kibana aficionado. From there you can create all kinds of charts and dashboards. We’ve also made sure Kibana URLs are shareable, so you can share them with your teammates. If they have access to the Logsene App whose Kibana URL you are sharing they, too, will be able to access the shared charts, dashboards, etc.

For more details on this check out our blog post or see it for yourself by logging in and going to one of your Logsene apps.

Events surfaced all the way to the top

events log metrics correlation sematext

Unlike logs, which tend to be high volume, events are a little more infrequent, a little more special. Events are things like alerts, or deploys, or builds, or… Ultimately though, it is up to you and your team to choose what to push into Sematext as an event. In this release we’ve surfaced Events all the way to the top, so they now stand shoulder to shoulder to metrics and logs. You can view logs in their own event stream-like view and you can search them of course.  Importantly, however, you can also correlate events with metrics, as shown below, much like you can correlate logs and metrics. Why is correlating events with metrics valuable? Think about being able to correlate a bad build or not quite fully tested release deployment with a change in metric values. When it’s sudden and things break right away you don’t necessarily need any tools to point out the correlation – you’ve got fire you need to put out and the correlation is obvious.

For such things, it is handy to be able to look at metrics and events and see the correlation. Sounds good? Log in to see for yourself and check it out!

Introducing Instant Templates & Mappings

Logsene, our hosted ELK stack now part of Sematext Cloud, exposes the Elasticsearch API. If you have experience with Elasticsearch though, if you’re aware of its rich ecosystem of tools, you know how valuable this is. When you send your logs to a Logsene app their structure (their mapping in Elasticsearch parlance) is automatically detected. Log event fields are created and their types are deduced from the received data. The mapping and the index template are also created. Under the hood, Logsene maintains a sequence of indices for each Logsene app. Every so often a new index is added and its mapping is based on the index template for that Logsene app.

For more details on this check out our blog post or see it for yourself.

Payment Logic Changes

Up until now, Sematext Cloud users had to first provide their credit card details under their account and then manually assign that credit card to each monitoring or logging App they wanted to use with a paid plan. It turns out this is a bit of a pain when you have a lot of Apps, so we’ve done away with that.  From now on, the payment method defined at the account level is automatically applied to all Apps with a paid plan. This simplifies management, especially in cases when users have many Apps (think about credit card expiring and having to assign a new card to each of the 57 Apps you have!).

For more details on this check out our blog post.

Android SDK for Log Shipping & Analytics

We’ve built and open-sourced an Android SDK for shipping logs from mobile apps to Sematext Cloud. This library not only lets you send your Android app logs for troubleshooting purposes, but it also lets you collect app usage for analytics purposes! In other words, shipping logs and usage events from an Android app will give you the ability to both troubleshoot issues with the app, as well as let you gain insights about your app’s usage, adoption, etc. The library sends data in batches to preserve battery (every 60s), or if there are more than 10 events queued up.

For more info, check out our blog post and see Sematext Logsene Android project on Github.

iOS SDK for Log Shipping & Analytics

Want to ship logs and other events from your iOS apps? We’ve open-sourced the iOS SDK for shipping logs from iOS apps to Sematext Cloud. This library not only lets you send your iOS app logs for troubleshooting purposes, but it also lets you collect app usage for analytics purposes!  In other words, shipping logs and usage events from an iOS app will give you the ability to both troubleshoot issues with the app, as well as let you gain insights about  your app’s usage, adoption, etc. If you don’t see the events in the dashboard immediately, note that this library sends data in batches to preserve the battery (every 60s), or if there are more than 10 messages queued up.

For more info, check out our blog post and see Sematext Logsene iOS project on Github.

 
Loading...

Stay up to date

Get tips, how-tos, and news about Elastic / ELK Stack, Observability, Solr, and Sematext Cloud news and updates.

Sematext Newsletter