Ingest data in Elasticsearch
Elasticsearch Output Plugin¶
The Elasticsearch output plugin forwards parsed logs to Elasticsearch or Sematext Logs.
Features¶
- log routing by log source to multiple Elasticsearch servers
- log routing by log source to multiple Elasticearch indices (or Sematext Logs Apps)
- SSL/TLS by default, when using Sematext
- Two-way SSL Authentication, also known as Mutual Authentication as part of PKI, secure client authentication with SSL client certificates
- bulk indexing with timeout (1000 docs or 10 second timeout by default)
- disk buffer an re-transmit when connection to Elasticsearch fails
- renaming of invalid field names
- limit field size (240k by default)
Simple config¶
The following example configuration ships all log files in /var/log (including sub-directories) to one Elasticsearch index.
input:
files:
- '/var/log/**/*.log'
output:
my-logs-app:
module: elasticsearch
url: https://logsene-receiver.sematext.com
index: bb308f80-0453-485e-894c-f80c054a0f10
Log routing to multiple targets¶
In some situations, it is required to ship data from different sources to different Elasticsearch servers or clusters. The output section in the Logagent configuration file accepts multiple definitions for the Elasticsearch output module.
Each Elasticsearch output might have a list of indices followed by a list of regular expressions matching the log source (e.g. file name of the log file).
The following example ships logs from wireless devices and authentication log to a local Elasticsearch server and other server logs to multiple Sematext Logs Apps.
input:
files:
- '/var/log/**/*.log'
output:
# index logs in Elasticsearch or Sematext
local-elasticsearch:
module: elasticsearch
url: http://localhost:9200
# default index to use, for all logs that don't match any other configuration
index: other_logs
# specific indices to use per logSource field of parsed logs
indices:
wireless_logs: # use regex to match log source e.g. /var/log/wifi.log
- wifi|bluetooth
security_logs:
- auth\.log
sematext:
module: elasticsearch
url: https://logsene-receiver.sematext.com
indices:
bb308f80-0453-485e-894c-f80c054a0f10:
- [nginx|httpd]\.log
a0ca5032-62da-467d-b6d5-e465a7ce45bb
- mysql|postgres|oracle
969020b4-f11c-41dd-86e4-24e67759cdb3
- mongo.*\.log
- myapp1\/app.log
- myapp2\/app.log
HTTP, HTTPS and authentication options¶
The Elasticsearch output module accepts http(s) options. Client side certificates and keys are specified with a file name. If you use self-signed certificates, set rejectUnauthorized to false.
output:
secure-elasticsearch:
module: elasticsearch
url: "https://user:password@localhost" # password characters will have to be urlencoded
index: logs
httpOptions:
key: /ssl-keys/client.key
cert: /ssl-keys/client.crt
ca: /ssl-keys/ca.pem
rejectUnauthorized: true