Skip to content
share

Sending Log Events to Sematext

Logs Discovery

The simplest - and the recommended way - to ship logs to Sematext is via the Discovery screen after the initial setup. Setting up log shipping from there will be the easiest and will yield the best results, resulting in a the most suitable type of Logs App being created with the most useful pre-built dasboards and the most useful alert rules.

Follow these step-by-step instructions for setting up the Sematext Agent through the Logs App.

Create Logs App

A Logs App is the space in Sematext that holds your logs. You can have any number of Logs Apps in your account and each of them can have a different plan, retention, users who can access it, etc.

Integration-specific Logs Apps

It is important to note that most Logs Apps are "typed". That is, most Logs Apps are built for a specific integration, for logs with some specific, known log format. For example, you can see an Nginx Logs App in a screenshot below. If you want to ship Nginx logs that is the Logs App you want to create. Not only will that Logs App have a bunch of ready to use Nginx-specific reports and charts out of the box, but it will also come with a set of default alert rules that are applicable to monitoring Nginx logs. Make use of this and save time by creating an appropriate Logs App. You can find other integrations in integrations docs.

Nginx Logs App

Generic Logs App

If you need to ship logs whose "type" you don't see among the list of offered Logs Apps types, create a Generic Logs App. This type of Logs App has log search, alerting, and all other capabilities, but has general reports that work for all types of logs regardless of their format. You can later create additional reports, charts, etc. to customize things to your liking.

Select Your Environment Type

After creating the Logs App, the environment selection screen for agent installation will be displayed.

Logs App Select an Environment

Choose the environment in which you intend to install the Sematext Agent. The installation instructions specific to that environment will be displayed. Follow the provided instructions to install the agent. Once installed, the Logs App will identify the host and display the discovered log sources within it.

View Discovered Logs

Discovered Logs shows a list of log sources discovered within the host grouped by service type.

Logs App Discovered Logs

Set Up Automatic Log Shipping

Click on the Set Up button next to the services from which you want to ship logs. This enables automatic log shipping for the selected services. For example, you can configure the system to send all log sources from Linux daemon services to Sematext Cloud as soon as they are discovered.

Logs App Set Up Log Shipping

Congratulations! You have successfully configured automatic log shipping for logs discovered under Linux daemon services without any additional configuration.

Shipping to the Same App from Additional Sources

If you want to ship logs from additional hosts and containers to the same Logs App, you can always navigate to the Ship Logs screen from the left Menu panel and repeat the steps to install the agent to another host.

Logs App Ship Logs

Furthermore, at any time, you can navigate to the Discovery screen to view the log sources and services identified across your infrastructure, from all the containers and hosts where the Sematext Agent is installed. From this screen, you have the option to configure automatic log shipping or metrics without the need for any additional installations.

Sematext Discovery

Custom Integration Options

Alternatively, you can explore various custom integration options for shipping log events to Sematext. These include support for Elasticsearch API, syslog-ng, various programming languages, and popular log shippers such as rsyslog, Logstash, Fluentd, and many many more. You can discover all supported custom integrations by accessing any Logs App, clicking 'Ship Logs' on the left menu panel, and navigating to the Custom Integrations tab. We recommend you use this only if, for some reason, setting up log shipping from Discovery screen in Sematext Cloud UI doesn't work and our support is unable to help you.

Logs App Custom Integrations

For further insights, refer to Logging Libraries vs Log Shippers.

Log Shippers

  • Logagent - cross platform, Smart and lightweight Log Parser and Log Shipper written in Node.js
  • rsyslog - easy to get started, very fast and very light on resources, docs are harder to navigate for beginners though.
  • Logstash OSS - cross platform, very simple to set up, well documented, but a little heavy on resource usage
  • Filebeat OSS - cross platform, much lighter on resource usage, requires a Logstash instance to aggregate logs
  • syslog-ng - very fast and very light on resources, good docs, available as both free and paid version
  • syslogd - quite old, light on resources, not very feature rich
  • Fluentd - cross platform, easy to get started, horizontally scalable, available as both free and paid version
  • Fluent Bit - FluentBit is an open source specialized data collector. It provides built-in metrics and general purpose output interfaces for centralized collectors such as Fluentd.
  • NXLog - cross platform but mostly used on Windows, easy to get started, available as both free and paid version

Programming Languages

Operating Systems

Containers

Cloud IaaS / PaaS

iOS

For iOS apps use Sematext Logs for iOS library.

Android

For Android apps use Sematext Logs for Android library.

AWS EC2

If you're an EC2 user, you can log Sematext from your instances by creating a US region Logs App or EU region Logs App and setting up a log shipper like you would from any other physical or virtual machine.

AWS ECS on AWS Fargate With FireLens

We recommend you use the AWS ECS Logs Integration to get more detailed out-of-the-box reports.

AWS ECS on AWS Fargate With AWS Logs

We recommend you use the AWS ECS Logs Integration to get more detailed out-of-the-box reports.

AWS ECS on AWS EC2

We recommend you use the AWS ECS Logs Integration to get more detailed out-of-the-box reports.