Better Observability with New Container Agents

Why a New Docker Agent?

If you liked Sematext Docker Agent you’ll love our new agent for Docker monitoring that provides you with even more insight into your Docker, Kubernetes, and Swarm clusters.  Because of its power, small footprint, and ease of installation the old Sematext Docker Agent enjoyed high adoption by the Docker DevOps community.

An all-in-one Docker monitoring tool, certified by Docker since 2015, it could monitor all key Docker metrics, container events, as well as collect and parse logs.  However, container technology is developing rapidly – Docker Enterprise and Kubernetes gained in popularity, as did cloud container platforms like Google GKE.  The Docker engine is not the only reliable container runtime anymore. There are several alternatives available such as containerd. A popular example is IBM Cloud, where we find Kubernetes and containerd.  It was time for an update.  Except, we didn’t just update it.  We rewrote it, made it even smaller and more modular, and much more powerful.  The new Sematext Agent can monitor not only Docker containers but also see inside them.  It has first-class Kubernetes monitoring support and kernel tracing capabilities, all with super low CPU and memory footprint.

To better serve the need for advanced monitoring and advanced logging functionality we’ve split the agent in two.  This enables faster release cycles and even easier deployment for each specific use case – embracing containerized architectures and orchestration tools. There are two separate images but – importantly – you still benefit from having a single deployment via Helm on Kubernetes.

The following new images replace the old Sematext Docker Agent:

• sematext/agent – container monitoring, infrastructure monitoring, cluster monitoring and events from container engines and orchestration tools
• sematext/logagent – log collection, log parsing, log enrichment, and log shipping for containers
• Both images are Docker certified

Together with the new monitoring agent, we introduced new Dashboards to display the collected data, such as Container Infrastructure monitoring and Kubernetes cluster metrics.

Container monitoring with heatmap

Kubernetes Dashboard – tracking deployment status and Pod restarts over time

So let’s introduce you to sematext/logagent & sematext/agent.

Container Logs Processing with Logagent

Logagent is a general-purpose open-source log shipper. The Logagent Docker image is pre-configured for the log collection on container platforms. It runs as a tiny container on every Docker host and collects logs for all cluster nodes and their containers. All container logs are enriched with Kubernetes, Docker Enterprise, and Docker Swarm metadata.

The deployment of Logagent is very similar to the deployment of Sematext Docker Agent and is fully compatible with all its configuration options for logs. The format for log parser patterns also remains the same. Logagent, like its predecessor, recognizes log formats from various applications / official images out of the box.

The following little example shows how easy it is to deploy Logagent, run a web server, and get structured web server logs for web analytics in Sematext.

# Start Logagent
docker run -d --restart=always -e LOGS_TOKEN=YourLogsToken
-v /var/run/docker.sock:/var/run/docker.sock
sematext/logagent
# Start Nginx web server
docker run -d -p 8081:80 nginx
# Access the web server
curl http://127.0.0.1:8081

A few seconds later, we see the result in Sematext, beautiful, structured web server logs including container metadata.

Structured web server logs with container metadata

With a few clicks, we can add widgets to create a web server logs dashboard, showing Top IP addresses and Top URLs or containers.

Sematext UI with Top N widgets for various log fields

That was easy for logs, so let’s have a look at the new Docker monitoring agent.

Monitoring Containers with Sematext Agent

Sematext Agent collects metrics about hosts (CPU, memory, disk, network, processes), containers and orchestrator platforms and ships that to Sematext Cloud. To gain deep insight into the Linux kernel, Sematext Agent relies on eBPF to implant instrumentation points (attach eBPF programs to kprobes) on kernel functions. Using Linux kernel instrumentation allows Sematext Agent a very efficient and powerful system exploration approach. It has the ability to auto-discover services deployed on physical machines, virtual hosts, and containers, as well as a mechanism for collecting infrastructure inventory info. It also collects events from different sources such as OOM notifications, container or Kubernetes events.

The plethora of information collected to provide you with full stack observability of your applications, services, and infrastructure is neatly organized in dashboards for infrastructure monitoring, container monitoring and Kubernetes cluster monitoring.

Kernel Tracing with eBPF

Many traditional monitoring agents are based on checks running periodically. Such checks run scripts and may even use commands like ‘ps -efa’ to discover running processes. There are several disadvantages to periodical checks. For example, periodical checks could miss short-running processes. Depending on the frequency of the checks they have their own overhead. Linux kernel observability using eBPF, on the other hand, can trace any kernel function call in the user space. Using eBPF makes it possible to automatically discover new processes without periodical checks. There is a lot more eBPF can do.  For instance, it can also discover any file system changes or network activity of all processes, including containers.

eBPF architecture – Source: https://github.com/cilium/cilium/

The new Sematext Agent makes heavy use of eBPF for auto-discovery of processes and their activity. To do that, Sematext Agent attaches bytecode at various hook points in the kernel to detect:
– Process creation and termination

• Socket listen/accept

• Signals

• Out of memory errors

• File system activity

Because eBPF is not available in older Linux kernels the agent also has fallback mechanisms like polling /procfs. If you are curious whether eBPF is available on your machines, have a look at the new Inventory Monitoring in Sematext, as it displays all Linux kernel versions used across your infrastructure.

Low CPU & Memory Footprint

Saving money on cloud resources is a hot topic for every company deploying applications to the cloud.  Keeping costs down is a must for any company that wants to be competitive in today’s markets.  We are very keenly aware of that, being a fully bootstrapped and cost-conscious organization ourselves.  Sematext Agent is a native binary.  As such, it doesn’t have the overhead of a runtime environment such as JVM, Ruby, Python, etc.  Moreover, we have put a lot of effort into profiling and minimizing the Sematext Agent memory and CPU footprint to make it nearly invisible when it’s running on your infrastructure.

Container and Kubernetes Monitoring

What are the advantages of the new agent for container monitoring?

First of all the Docker Remote API is limited to Docker environments, while Kubernetes emerged to the most popular orchestration tool. In addition more and more alternative container runtimes are available on the market. Therefore the new Sematext Agent takes a container runtime agnostic approach for container monitoring.

• Container runtime agnostic discovery and monitoring
  • Automatic container discovery
  • Support for Docker, Rkt and containerd container engines
• Container metrics
  • CPU usage
  • Disk space usage and IO stats
  • Memory usage, memory limits, and memory fail counters
  • Network IO stats
• Host inventory information
  • Host kernel version and other system information, like distro, architecture, number of CPUs, etc.
  • Information about installed software packages
• Container metadata
  • Container name
  • Image name
  • Container networks
  • Container volumes
  • Container environment
  • Container labels including relevant information about orchestration
    • Kubernetes metadata such as Pod name, UUID, namespace
    • Docker Swarm metadata such as service name, swarm task, etc.
• Collection of container events
  • Docker events such as start/stop/die/volume mount, etc.
  • Kubernetes events such as Pod status changes deployed, destroyed, etc.
• Tracking deployment status and Pod restarts over time
• Process metrics such as CPU usage, memory usage and disk IO

Let’s see how Sematext Agent is deployed.

Getting Started with Sematext Agent

To run Sematext Agent you will need a Infra App token. If you don’t have any Infra Apps yet, you can create one now.
The Sematext UI displays copy and paste instructions for various ways of deployments for Docker, Docker Enterprise/Swarm, Kubernetes DaemonSets or Helm charts.

The Sematext Agent Documentation contains all configuration options. After a short time you will see container information in the infrastructure monitoring, Docker and Kubernetes reports.

Migration

When migrating to the new agents you can do a simple “nearly in-place replacement” by first removing the old agent and then quickly setting up the new ones.  This may result in a bit of gap in your metrics and logs between the time you remove the old Sematext Docker Agent and set up the new ones, but the switch to new agents should take only a few minutes.  If this short gap in data is not acceptable, but a bit of data duplication is, then you can switch the order of operations – set up new agents first and then remove the old one.

Please read Monitoring Docker With Sematext, it shows all details, useful options for the agent deployments, log search tips, alert rule definition and more.

What else is Planned in the Near Future?

The new Sematext Agent has the ability to auto-discover running inside containers.  Once the discovered services are exposed in Sematext the Sematext Agent will enable you to seamlessly start monitoring applications you have running inside your containers. The automatic deployment will work on bare metal and VM servers, Docker Enterprise and Kubernetes.

Logagent will grow its open-source repository of supported log formats and plugins for hands-free log collection and parsing.

The new agent includes process monitoring and package inventory collection capabilities, which will soon start showing up in the Sematext UI, so if you don’t have the new agent now is a good time to upgrade your Sematext Agent!