Skip to content
share

Generic Logs Integration

Stored data is received through the Elasticsearch API and also through a variety of Syslog protocols.

The Elasticsearch API lets you:

  • send log events directly from your application, using any Elasticsearch library
  • send log events using a "log shipper" application such as Logstash, rsyslog, Apache Flume, Fluentd, or anything that can output to Elasticsearch
  • search for logs from your own application, or by configuring/adapting existing Elasticsearch UIs, such as Kibana
  • optionally define custom mappings for your log types, so you can tweak the way your logs are indexed

Syslog Protocols

We accept Syslog messages using any log shipper and any Syslog library, as long as they either contain a valid token or the source IP is authorized.

Journald

We accept Journald logs using the systemd-journal-remote package. Everything you need to do is point the systemd-journal-remote to send Journald logs to Sematext Logs.

Log Shippers

  • Logagent - cross platform, Smart and lightweight Log Parser and Log Shipper written in Node.js
  • rsyslog - easy to get started, very fast and very light on resources, docs are harder to navigate for beginners though.
  • Logstash - cross platform, very simple to set up, well documented, but a little heavy on resource usage
  • Filebeat - cross platform, much lighter on resource usage, requires a Logstash instance to aggregate logs
  • syslog-ng - very fast and very light on resources, good docs, available as both free and paid version
  • syslogd - quite old, light on resources, not very feature rich
  • Fluentd - cross platform, easy to get started, horizontally scalable, available as both free and paid version
  • Fluent Bit - FluentBit is an open source specialized data collector. It provides built-in metrics and general purpose output interfaces for centralized collectors such as Fluentd.
  • NXLog - cross platform but mostly used on Windows, easy to get started, available as both free and paid version

Programming Languages

Operating Systems

Containers

Cloud IaaS / PaaS

iOS

For iOS apps use Sematext Logs for iOS library.

Android

For Android apps use Sematext Logs for Android library.

AWS EC2

If you're an EC2 user, you can log Sematext from your instances by setting up a log shipper like you would from any other physical or virtual machine.

AWS S3 (CloudTrail, Flow logs, ELB access logs, etc.)

If you have logs stored in S3, you can ship them to Sematext via this AWS Lambda function. This method also works for when you periodically upload logs to S3 buckets, like Amazon CloudTrail does.

AWS CloudWatch Logs

If you want to ship CloudWatch logs, you can use another AWS Lambda function. If logs are VPC flowlogs, the Lambda function will also parse them and add geoIP information on the source IP addresses.

Centralized Logging for AWS Lambda

If you want to automatically subscribe to AWS Lambda log streams you can use this CloudFormation stack.

It'll let you run a single command and set up log group subscriptions, funnel all CloudWatch logs to Kinesis, and use a dedicated Lambda function to ship these logs to Sematext.

Read the full tutorial on our blog!