share share share 

Use Grok patterns to filter raw logs

Input Filter: Grok

Input plugin for @sematext/logagent. Uses Grok patterns to filter data from input plugins before data are parsed.

Installation

Install @sematext/logagent and logagent-input-filter-grok npm package:

npm i -g @sematext/logagent
npm i -g logagent-input-filter-grok

Configuration

Add the following section to the Logagent configuration file. Please note you could use the plugin with multiple configurations. The output of the first filter is passed into the next one ...:

input: 
  files:
    - '/var/log/**/*.log'

inputFilter:
  - module: grok
    config:
      # Logagent uses node-grok. It loads all patterns from the given file. Using 'matchSource' parameter it is possiible to define a custom pattern.
      # See https://github.com/Beh01der/node-grok/tree/master/lib/patterns for patterns loaded at start
      matchSource: '%{IP:client} \[%{TIMESTAMP_ISO8601:timestamp}\] "%{WORD:method} %{URIHOST:site}%{URIPATHPARAM:url}" %{INT:code} %{INT:request} %{INT:response} - %{NUMBER:took} \[%{DATA:cache}\] "%{DATA:mtag}" "%{DATA:agent}"'
      filePath: /tmp/grok-patterns
      idpattern: USER

output:
  elasticsearch:
    module: elasticsearch
    url: http://localhost:9200
    index: mylogs

Run Logagent:

logagent --config myconfig.yml