Elasticsearch
Elasticsearch Output Plugin¶
The elasticsearch output plugin forwards parsed logs to Sematext Logs using the Sematext API, which supports the Elasticsearch output format.
Features¶
- log routing by log source to multiple Elasticsearch-compatible endpoints
- log routing by log source to multiple OpenSearch/Elasticsearch indices (or Sematext Logs Apps
- SSL/TLS by default, when using Sematext
- Two-way SSL Authentication, also known as Mutual Authentication as part of PKI, secure client authentication with SSL client certificates
- bulk indexing with timeout (1000 docs or 10 second timeout by default)
- disk buffer an re-transmit when connection to Elasticsearch fails
- renaming of invalid field names
- limit field size (240k by default)
Simple config¶
The following example configuration ships all log files in /var/log (including sub-directories) to one /Elasticsearch index.
input:
files:
- '/var/log/**/*.log'
output:
my-logs-app:
module: elasticsearch
url: https://logsene-receiver.sematext.com
index: bb308f80-0453-485e-894c-f80c054a0f10
Log routing to multiple targets¶
In some situations, it is required to ship data from different sources to different OpenSearch/Elasticsearch servers or clusters. The output section in the Logagent configuration file accepts multiple definitions for the Elasticsearch output module.
Each Elasticsearch output might have a list of indices followed by a list of regular expressions matching the log source (e.g. file name of the log file).
The following example ships logs from wireless devices and authentication log to a local OpenSearch/Elasticsearch server and other server logs to multiple Sematext Logs Apps.
input:
files:
- '/var/log/**/*.log'
output:
# index logs in Elasticsearch or Sematext
local-elasticsearch:
module: elasticsearch
url: http://localhost:9200
# default index to use, for all logs that don't match any other configuration
index: other_logs
# specific indices to use per logSource field of parsed logs
indices:
wireless_logs: # use regex to match log source e.g. /var/log/wifi.log
- wifi|bluetooth
security_logs:
- auth\.log
sematext:
module: elasticsearch
url: https://logsene-receiver.sematext.com
indices:
bb308f80-0453-485e-894c-f80c054a0f10:
- [nginx|httpd]\.log
a0ca5032-62da-467d-b6d5-e465a7ce45bb
- mysql|postgres|oracle
969020b4-f11c-41dd-86e4-24e67759cdb3
- mongo.*\.log
- myapp1\/app.log
- myapp2\/app.log
HTTP, HTTPS and authentication options¶
The Elasticsearch output module accepts http(s) options. Client side certificates and keys are specified with a file name. If you use self-signed certificates, set rejectUnauthorized to false.