Skip to content

Overview

Logs Discovery

The simplest - and the recommended way - to ship logs to Sematext is via the Discovery screen after the initial setup. Setting up log shipping from there will be the easiest and will yield the best results, resulting in a the most suitable type of Logs App being created with the most useful pre-built dasboards and the most useful alert rules.

Follow these step-by-step instructions for setting up the Sematext Agent through the Logs App.

Create Logs App

A Logs App is the space in Sematext that holds your logs. You can have any number of Logs Apps in your account and each of them can have a different plan, retention, users who can access it, etc.

Integration-specific Logs Apps

It is important to note that most Logs Apps are "typed". That is, most Logs Apps are built for a specific integration, for logs with some specific, known log format. For example, you can see an Nginx Logs App in a screenshot below. If you want to ship Nginx logs that is the Logs App you want to create. Not only will that Logs App have a bunch of ready to use Nginx-specific reports and charts out of the box, but it will also come with a set of default alert rules that are applicable to monitoring Nginx logs. Make use of this and save time by creating an appropriate Logs App. You can find other integrations in integrations docs.

Nginx Logs App

If you select a specific integration, you will be asked to create an Infra App. The Infra App works with the Sematext Agent to automatically discover log sources on your host for the selected service.

Next, choose the environment where you want to install the Sematext Agent.

Logs App Environments

Once the agent is installed, the log sources are discovered automatically, and logs from your selected service will start shipping to Sematext Cloud without any additional setup.

Logs App Finish Setup

Generic Logs App

If you need to ship logs whose "type" you don't see among the list of offered Logs Apps types, create a Generic Logs App. This type of Logs App has log search, alerting, and all other capabilities, but has general reports that work for all types of logs regardless of their format. You can later create additional reports, charts, etc. to customize things to your liking.

From there you will be provided with 2 options to ship logs from.

Use Sematext Agent to ship custom logs

Install the Sematext Agent, which can automatically discover log sources on your hosts. When you select this option, you will be asked to choose an Infra App. If you don’t have one, we will create it automatically for you. An Infra App works with the Sematext Agent to discover available log sources on the host where the agent is installed.

After creating the Infra App, select the environment where you want to install the Sematext Agent. Once the agent is installed, you can choose which logs to ship from the list of discovered log sources.

Generic Logs Discovery

Click on the Set Up button next to the services from which you want to ship logs and enable automatic log shipping for the selected service.

Generic Logs Discovery Setup

If you are shipping custom logs to Sematext Cloud, we recommend reviewing the Logs Pipelines guide to structure, enrich, or transform your log documents.

Use Custom Integrations

If you don’t see your environment in the Install Sematext Agent step, you can use your own log shipper, such as Logstash, Fluentd, Syslog, cloud libraries, or send log events directly from your application using any Elasticsearch or OpenSearch compatible library to set up log shipping manually. The Custom Integrations screen explains how to manually send data to your new Logs App

Shipping to the Same App from Additional Sources

If you want to ship logs from additional hosts and containers to the same Logs App, you can always navigate to the Ship Logs --> Install Agent screen from the left Menu panel and repeat the steps to install the agent to another host.

Furthermore, at any time, you can navigate to the Discovery screen to view the log sources and services identified across your infrastructure, from all the containers and hosts where the Sematext Agent is installed. From this screen, you have the option to configure automatic log shipping or metrics without the need for any additional installations.

Sematext Discovery

Custom Integration Options

Alternatively, you can explore various custom integration options for shipping log events to Sematext. These include support for Sematext API which is compatible with Elasticsearch and OpenSearch, syslog-ng, various programming languages, and popular log shippers such as rsyslog, Logstash, Fluentd, and many many more. You can discover all supported custom integrations by accessing any Logs App, clicking 'Ship Logs' on the left menu panel, and picking Custom Integrations option. We recommend you use this only if, for some reason, setting up log shipping from Discovery screen in Sematext Cloud UI doesn't work and our support is unable to help you.

Custom Integrations Selected

Custom Integrations

For further insights, refer to Logging Libraries vs Log Shippers.

Log Shippers

  • Logagent - cross platform, Smart and lightweight Log Parser and Log Shipper written in Node.js
  • rsyslog - easy to get started, very fast and very light on resources, docs are harder to navigate for beginners though.
  • Logstash OSS - cross platform, very simple to set up, well documented, but a little heavy on resource usage
  • Filebeat OSS - cross platform, much lighter on resource usage, requires a Logstash instance to aggregate logs
  • syslog-ng - very fast and very light on resources, good docs
  • syslogd - quite old, light on resources, not very feature rich
  • Fluentd - cross platform, easy to get started, horizontally scalable
  • Fluent Bit - FluentBit is an open source specialized data collector. It provides built-in metrics and general purpose output interfaces for centralized collectors such as Fluentd.
  • NXLog - cross platform but mostly used on Windows, easy to get started

Programming Languages

Operating Systems

Containers

Cloud IaaS / PaaS

iOS

For iOS apps use Sematext Logs for iOS library.

Android

For Android apps use Sematext Logs for Android library.

AWS EC2

If you're an EC2 user, you can log Sematext from your instances by creating a US region Logs App or EU region Logs App and setting up a log shipper like you would from any other physical or virtual machine.

AWS ECS on AWS Fargate With FireLens

We recommend you use the AWS ECS Logs Integration to get more detailed out-of-the-box reports.

AWS ECS on AWS Fargate With AWS Logs

We recommend you use the AWS ECS Logs Integration to get more detailed out-of-the-box reports.

AWS ECS on AWS EC2

We recommend you use the AWS ECS Logs Integration to get more detailed out-of-the-box reports.