Sematext Logagent Log Shipper
What is Logagent?¶
Logagent is a modern, open-source, lightweight log shipper written entirely in Node.js with a low memory footprint and low CPU overhead!
It comes with out of the box and extensible log parsing, on-disk buffering, secure transport, and log shipping with bulk indexing to any Elasticsearch endpoint, including Sematext Logs.
If you're eager to get started, here's how you start shipping logs.
# Make sure you have Node.js installed curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash - sudo apt-get install -y nodejs # Install Logagent and run it as a system service sudo npm i -g @sematext/logagent sudo logagent-setup -i <LOGS_TOKEN or ES_INDEX>
To read more jump to Installation right away!
- Install Logagent with:
- Linux Systemd
- Linux Upstart
- Windows service
- Mac OS X Launchd service
- Log shipping with a disk buffer
- A simple YAML configuration file
- Built-in data parser with configurable patterns
- Command-line tool
- Inputs (files, streams, sockets, databases)
- Input filters (grep/grok filters)
- Outputs (Elasticsearch, Sematext Cloud, Kafka, etc.)
- Output filters (SQL aggregation of parsed data, enrichment of data)
- Node.js API
- Install as a system service
- Run as a Docker Container
- Deploy to Heroku as Heroku Log drain
- Deployment to Cloud Foundry as Cloud Foundry Log drain (thus usable with Pivotal, IBM Bluemix, etc.)
Logagent Log Shipping with Disk Buffer¶
Logagent doesn't lose data. It stores parsed logs to a disk buffer if the network connection to the Elasticsearch API fails. Logagent retries shipping logs later, when the network or Elasticsearch is available again.
Logagent YAML Configuration File¶
After installing Logagent you have a CLI tool and can run
logagent-setup to create a system service and start shipping logs right away. It'll also create a simple
YAML configuration file for you in
# /etc/sematext/logagent.conf # Global options options: # print stats every 60 seconds printStats: 60 # don't write parsed logs to stdout suppress: true # Enable/disable GeoIP lookups # Startup of logagent might be slower, when downloading the GeoIP database geoipEnabled: false # Directory to store Logagent status and temporary files # this is equals to LOGS_TMP_DIR env variable diskBufferDir: /tmp/sematext-logagent input: # a list of glob patterns to watch files to tail files: - '/var/log/**/*.log' output: # index logs in Elasticsearch or Sematext Logs sematext: # output a name, Eg. elasticsearch, sematext, etc. module: elasticsearch url: https://logsene-receiver.sematext.com # default Elasticsearch index or Sematext Logs token to use index: <LOGS_TOKEN or ES_INDEX> # indices for shipping logs to multiple locations indices: <LOGS_TOKEN_1 or ES_INDEX_1>: # list of log sources or filenames - syslog\.log - access\.log - auth\.log <LOGS_TOKEN_2 or ES_INDEX_2>: # list of RegEx matching a log source or filename - .*wifi.* - .*bluetooth.*
Logagent Command-line Tool¶
Logagent can also be used as a command-line tool without running
logagent-setup and using the default configuration file.
- Works with Unix pipes, stdin, and stdout
Log parser and format converter
- text to JSON
line delimited JSON or YAML
cat access.log | logagent --yaml
Import files into Elasticsearch
cat access.log | logagent -n nginx -e http://localhost:9200 -i logs
Watch multiple log files in the terminal
logagent -yaml -g '/var/log/**/*.log'
Store logs in Elasticsearch
logagent -e http://localhost:9200 -i logs
Built-in Log Parser for Logagent¶
You can configure custom data patterns for parsing logs.
- Log format detection and intelligent pattern matching
- Pattern library included covering a set of common databases, web servers, message queues, etc.
- Easy to extend with custom patterns and JS transform functions
- Hot reload of changed pattern definitions without service restart
- Auto-detection of date and numeric fields
- Masking of sensitive data with configurable hashing algorithms (SHA-1, SHA-256, SHA-512, …)
- GeoIP lookup with automatic GeoIP DB updates (Maxmind GeoIP-Lite files)
A comprehensive collection of plugins for data input, processing, and output are available. See the complete list of Logagent Plugins.