Logging

Imagine you're a detective trying to solve a crime, but all the evidence is invisible.  Sounds impossible, right?  That's exactly what it's like trying to debug a Java application without

Imagine waking up to a critical system failure that has brought your business operations to a standstill.  Panic sets in as you scramble to understand what went wrong.  You sift

Ever found yourself drowning in a sea of log data, struggling to make sense of the overwhelming noise? Or perhaps faced a major system breakdown, only to find that your

Logs act as silent sentinels, recording every whisper of your application's activity. They are invaluable chronicles illuminating system behavior, diagnosing issues, and providing crucial insights into your application's health.  However, the

As a Site Reliability Engineer (SRE) or DevOps professional, you are well aware of the importance of observability in ensuring the smooth functioning and performance of your applications. Observing and

In the world of modern business, data drives decision-making. Every interaction, every transaction, and every click generates a series of data in the form of logs. These logs, often seen

OK, so you've decided to move from Elasticsearch to OpenSearch. Maybe our comparison helped you decide and maybe you've checked our guide on how to perform the migration. But how

Any software application or a system can have bugs and issues in testing or production environments. Therefore, logging is essential to help troubleshoot issues easily and introduce fixes on time.

Logging is the process of keeping records of activities and data of a software program. It is an important aspect of developing, debugging, and running software solutions as it helps

Apache Tomcat is the Java web server that implements many Java features like web site APIs, Java server pages, Java Servlets, etc. It's an open-source software widely used in the

Ubuntu provides extensive logging capabilities, so most of the activities happening in the system are tracked via logs. Ubuntu logs are valuable sources of information about the state of your

HAProxy is generally the frontend layer of your application, which means it plays a critical role since all traffic first lands on this layer. Because of this, you need to

Logs are a critical aspect of any production workload, as they give you insight into what is happening in your system and tell you which components may be having issues.

For this Black Friday, Sematext wishes you: more products sold more traffic and exposure more logs :) Now seriously, applications tend to generate a lot more logs on Black Friday,

PostgreSQL is an open-source relational database management system that's been utilized in continuous development and production for 30 years now. Nearly all the big tech companies use PostgreSQL, as it

RabbitMQ is one of the most popular open-source message brokers available. Its ability to be deployed in various configurations and on various platforms makes it a widely used tool; it

NGINX is one of the most widely used reverse proxy servers, web servers, and load balancers. It has capabilities like TLS offloading, can do health checks for backends, and offers

Every developer's worst nightmare is digging through a huge log file, trying to pinpoint problems. The troubleshooting most likely won't stop there. They'll have to follow the trail to multiple

All the applications that you're trying to execute require memory. It doesn't matter if the application was developed using assembly language. Or if you used a low-level programming language like

On December 9, 2021, a vulnerability was reported that could allow a system running Apache Log4j 2 version 2.14.1 or below to be compromised and allow an attacker to execute

Apache - the technology that powers the web. I'm not sure if that is correct, but I think that we wouldn't see the world wide web in its current shape

As developers, we would like our users to interact with applications that run smoothly and without issues. We want the libraries that we create to be widely adopted and successful.

When working with Java or any other JVM-based programming language we get certain functionalities for free. One of those functionalities is clearing the memory. If you’ve ever used languages like

Logging is a crucial part of the observability of your Java applications. Java logs combined with JVM metrics and traces give full observability into the application behavior and are invaluable

Troubleshooting issues in your applications can be a complicated task requiring visibility into various components. In the worst-case scenario, to understand what is happening and why it is happening you

When it comes to troubleshooting Java application performance, JVM metrics are no longer enough. To fully understand the environment you also need Java logs and traces. Today, we're going to

When it comes to troubleshooting application performance, the more information you have the better. Java logs combined with JVM metrics and traces give you full visibility into your Java applications.

Getting visibility into your application is crucial when running your code in production. What do we mean by visibility? Primarily things like application performance via metrics, application health, and availability,

Don't you hate it when you see an uncaughtException error pop up and crash your Node.js app? Yeah... I feel you. Can anything be worse? Oh yeah, sorry, unhandledRejection I

Let's say you get an alert that one or more queries is slow. Or that your users complain, whichever comes first 🙂  We've all been there… How do you find

Sematext Logs is a Log Management-as-a-service. Think of it as your own central location for logs in the cloud. If you prefer or need to keep logs in your own

Building SaaS products is hard. Making customers happy is even harder. I should know, I've built a start-up that failed! But, not everything is that bad. I learned a lot.

Making sense out of logs is not an easy task. Log management solutions gather and accept data from multiple sources. Those sources can have different log structures, providing a different

Logs are one of the most valuable assets when it comes to IT system management and monitoring. As they record every action that took place on your network, logs provide

Having visibility into your Java application is crucial for understanding how it works right now, how it worked some time in the past and increasing your understanding of how it

TL;DR note: if you want the bzip2 -9 version of this post, scroll down to the very last section for some quick pointers. If you want to learn a bit

I'm sure you bumped into journald: it's what most distros use by default for system logging in Linux. Most applications running as a service will also log to the journal.

By Chris Tozzi Logging or monitoring? If you deploy and manage an application, these are the two key techniques available to you for helping to ensure that the application meets

Kubernetes is gaining popularity every day. Using an Ingress controller is the preferred method of allowing external access to the services in a cluster. This makes ingress logs incredibly important

There’s no need to look outside the ELK Stack for apps to ensure data protection.  Basic Elasticsearch Security features are free and include a lot of functionality to help you

Looking to learn about Logstash as quickly as possible? This Logstash Tutorial is for you: we’ll install Logstash and push some Apache logs to Elasticsearch in less than 5 minutes.

When managing cloud-native applications, it’s essential to have end-to-end visibility into what’s happening at any given time. This is especially true because of the distributed and dynamic nature of cloud-native

The Linux Audit framework is a kernel feature (paired with userspace tools) that can log system calls. For example, opening a file, killing a process or creating a network connection.

As shown in Elasticsearch Key Metrics, the setup, tuning, and operations of Elasticsearch require deep insights into the performance metrics such as index rate, query rate, query latency, merge times, and

The key to gaining serverless observability is sending all AWS Lambda Logs to a central location where you can later group, filter and make sense of them. Sematext is a full-stack

Elastic Stack Features (formerly X-Pack) is an Elastic Stack extension that bundles security, alerting, monitoring, reporting, and graph capabilities. One could use either all or specific components. Elastic Stack Features as

By Chris Riley What is the Elastic Stack? Elasticsearch, Logstash, and Kibana — the trio better known as Elastic Stack (or ELK, if you prefer a term that is now

We're excited to announce the release of the Field Stats API plugin for Elasticsearch. The Field Stats API used to be present from Elasticsearch 1.6 to 5.6, to provide efficient

Unless you are using a very old version of Elasticsearch you’re able to define pipelines within Elasticsearch itself and have those pipelines process your data in the same way you’d

This recipe is similar to the previous rsyslog + Redis + Logstash one, except that we’ll use Kafka as a central buffer and connecting point instead of Redis. You’ll have

Here at Sematext we use Java and rely on Elasticsearch for our hosted ELK logging SaaS, a lot. We like them so much that we regularly share our logging experience

Logtrail is a popular Kibana plugin that makes Kibana look like Papertrail. Sematext’s Logtrail view in Kibana gives you the terminal-like experience of exploring logs without the pain of needing

"Why does the 'docker logs' command fail?", is one of our frequently asked questions. The answer is simple and mentioned in the Docker documentation: "The docker logs command is not available for drivers

The rather broad definition of personal data in the GDPR requires paying special attention to log data. GDPR and personal data in web server logs is a popular topic in

What can you do with a couple of IoT devices, Node.js, Elasticsearch, and MQTT?  You can put together your own Internet of Things setup for measuring air pollution, like I

Centralizing logs to Elasticsearch? Of course, the first log shipper that comes to mind is Logstash. When you get into it, you realize centralizing logs often implies a bunch of

Many of our customers use AWS EC2. In the context of Elasticsearch consulting or support, one question we often get is: should we use AWS Elasticsearch Service instead of deploying Elasticsearch ourselves? The

Docker changed not only how applications are deployed, it also changed the workflow for log management. Instead of writing logs to files, containers write logs to the console (stdout/stderr) and

In the world of DevOps, metric collection, log centralization and analysis Apache Kafka is the most commonly used middleware. More specifically, it is used as a fast, persistent queue between

Kubernetes is a really hot topic at the moment. All major cloud providers adopted it as a solution for deploying cloud native apps. Just a few weeks ago, AWS at

When dealing with log centralization in your organization you have to start with something. Often times people start by collecting logs for the most crucial pieces of software, and frequently

This is a guest post from Filippo Balicchia. Filippo contributed Logagent plugins for Apache Kafka the details of which he is sharing in this post.  Filippo is a software engineer

Elasticsearch is already quite popular and its popularity just keeps growing. Looking at Google Trends chart for the last 5 years shows this nicely: There are a number of reasons

TL;DR: The main question here is: How Does Java 9 Work with Elasticsearch 6? It works well, but don't expect miracles. Unless you're using G1, then there are some miracles. With

Do you know what portion of your traffic comes from bots? Do you know which bots are good and which ones are bad? Do you know what the bad bots

Logsene, our hosted ELK stack now part of Sematext Cloud, exposes the Elasticsearch API. As a Logsene user you may or may not be aware of this.  If you have

Every modern organization depends on software, and where there is software there are metrics and logs and, one would hope, monitoring of these two critically important types of data. Operations

Note: This is a guest post by Jochen Kressin, the CTO of floragunn GmbH, the makers of Search Guard, an open-source X-Pack Security alternative. Elasticsearch is a great piece of software.

Note: This is a guest post by Jochen Kressin, the CTO of floragunn GmbH, the makers of Search Guard, an open-source X-Pack Security alternative. In this article, we show you how

Over the last several months we’ve made all kinds of improvements to Sematext Docker Agent (SDA).  If you’re not familiar with SDA yet, here it is in a nutshell: Sematext

Need to have your recent logs searchable for your DevOps team(s) to troubleshoot issues, but also need to archive old logs for compliance reasons?   Many organizations need that and

Want to ship logs and other events from your iOS apps?   We’ve open-sourced the iOS SDK for shipping logs from iOS apps to Sematext Cloud.  This library not only

A few months back we’ve silently built and open-sourced an Android SDK for shipping logs from mobile apps to Sematext Cloud.   This library not only lets you send your

This post shows how to use Docker Secrets in Docker Swarm mode. For our example we’ll show how to use Docker Secrets to encrypt and safely store tokens used by

Intrusion detection systems generate highly valuable logs with network usage details and alerts. They collect vast amounts of data and typically store them in structures with a large number of

Pretty much everyone's heard about syslog: with its roots in the 80s, it's still used for a lot of the log management done today. Mostly because of its long history, syslog

This is a guest post by Nedim Šabić, developer of Fibratus, a tool for exploration and tracing of the Windows kernel.  Unlike Linux / UNIX environments which provide a plethora of open

We are known worldwide for our Elasticsearch, ELK stack and Solr consulting services, and we are always happy to help others improve their skills in these technologies, not only through

Not everyone uses Splunk or ELK stack for logs. A few weeks ago, at the Lucene/Solr Revolution conference in Boston, we gave a talk about using Solr for logging, along

In earlier posts, we explained how one can reindex data from one Elasticsearch cluster to another, or within the same Elasticsearch cluster, via tools like Logstash and rsyslog. The same

Use-case: you're using AWS VPC and want visibility over the connection to your VPC. Which IPs are allowed or denied connections to certain ports, how much traffic goes through each

Update: We made it much easier to ship Heroku logs. Please read 5 Minute Recipe: Heroku Log Drain Setup for more details.  Heroku is a cloud platform based on a managed

UPDATE: There is a much easier way to ship your Windows events to Sematext now. See the New Windows logs integration update. There are a lot of sources of logs

This recipe shows how to send CloudTrail logs (which are .gz logs that AWS puts in a certain S3 bucket) to a Logsene application, but should apply to any kinds

Grafana is an open-source alternative to Kibana. Grafana is best known as a visualization / dashboarding tool focused on graphing metrics from various data sources, such as InfluxDB. Even though

In this presentation from Lucene/Solr Revolution 2015, Sematext engineers -- and Solr and centralized logging experts -- Radu Gheorghe and Rafal Kuć talk about searching and analyzing time-based data at

More than two years ago we posted a recipe on how to centralize syslog in Elasticsearch in order to search and analyze them with Kibana, all by using only rsyslog.

OK, so you want to hook up rsyslog with Logstash. If you don't remember why you want that, let me give you a few hints: Logstash can do lots of

[Note: this post was written by Sematext engineer Marko Bonaći] In vino veritas, right?  During a recent team gathering in Kraków, Poland, and after several yummy bottles of țuică, vișinată,