Generic Logs Integration
Stored data is received through the Elasticsearch API and also through a variety of Syslog protocols.
The Elasticsearch API lets you:
- send log events directly from your application, using any Elasticsearch library
- send log events using a "log shipper" application such as Logstash, rsyslog, Apache Flume, Fluentd, or anything that can output to Elasticsearch
- search for logs from your own application, or by configuring/adapting existing Elasticsearch UIs, such as Kibana
- optionally define custom mappings for your log types, so you can tweak the way your logs are indexed
We accept Syslog messages using any log shipper and any Syslog library, as long as they either contain a valid token or the source IP is authorized.
We accept Journald logs using the
systemd-journal-remote package. Everything you need to do is point the
systemd-journal-remote to send Journald logs to Sematext Logs.
- Logagent - cross platform, Smart and lightweight Log Parser and Log Shipper written in Node.js
- rsyslog - easy to get started, very fast and very light on resources, docs are harder to navigate for beginners though.
- Logstash - cross platform, very simple to set up, well documented, but a little heavy on resource usage
- Filebeat - cross platform, much lighter on resource usage, requires a Logstash instance to aggregate logs
- syslog-ng - very fast and very light on resources, good docs, available as both free and paid version
- syslogd - quite old, light on resources, not very feature rich
- Fluentd - cross platform, easy to get started, horizontally scalable, available as both free and paid version
- Fluent Bit - FluentBit is an open source specialized data collector. It provides built-in metrics and general purpose output interfaces for centralized collectors such as Fluentd.
- NXLog - cross platform but mostly used on Windows, easy to get started, available as both free and paid version
Cloud IaaS / PaaS¶
- AWS S3
- AWS CloudTrail
- AWS CloudWatch
- AWS VPC Flow Logs
- Cloud Foundry
- Google App Engine
- GitHub Webhook Events
If you're an EC2 user, you can log Sematext from your instances by setting up a log shipper like you would from any other physical or virtual machine.
AWS S3 (CloudTrail, Flow logs, ELB access logs, etc.)
If you have logs stored in S3, you can ship them to Sematext via this AWS Lambda function. This method also works for when you periodically upload logs to S3 buckets, like Amazon CloudTrail does.
AWS CloudWatch Logs¶
If you want to ship CloudWatch logs, you can use another AWS Lambda function. If logs are VPC flowlogs, the Lambda function will also parse them and add geoIP information on the source IP addresses.
Centralized Logging for AWS Lambda¶
If you want to automatically subscribe to AWS Lambda log streams you can use this CloudFormation stack.
It'll let you run a single command and set up log group subscriptions, funnel all CloudWatch logs to Kinesis, and use a dedicated Lambda function to ship these logs to Sematext.
Read the full tutorial on our blog!