Remove fields from logs

Output filter: remove-fields¶

This plugin removes fields before sending them to the output destination.

In the context of data protection regulations like GDPR you might need to mask data fields, especially when you hand over log data to third parties. This plugin can replace all occurrences of the original field values with maskValuesString. To mask the field values any other field, simply add a list of field names in maskValuesInFields (see example below).

Configuration¶

Add the following 'outputFilter' section to the Logagent configuration file. Note that you can use the plugin with multiple configurations for different event sources.

# tail web server logs
input: 
  files:
    - '/var/log/*/access_log'

# log agent parses web server logs out of the box ...
# output filter to remove client_ip and user field in web server logs
outputFilter:
  remove-fields:
    module: remove-fields
    # JS regular expression to match log source name
    matchSource: !!js/regexp .*
    # List of fields, where the values from removed field should be
    # replaced with maskValuesString 
    maskValuesInFields:
      - message
      - path
    # String to replace masked values from removed fields
    maskValuesString: "ANONYMIZED-DATA"
    fields:
      - user
      - client_ip

Run Logagent with your config:

logagent --config logagent-example-config.yml -n httpd --yaml

The output does not contain the fields client_ip and user. Optionally the user is replaced with "ANONYMIZED-DATA" in the message and path field:

logSource:    httpd
_type:        access_common
remote_id:    -
method:       GET
path:         /user/ANONYMIZED-DATA
message:      GET /user/ANONYMIZED-DATA
http_version: HTTP/1.1
status_code:  304
size:         0
@timestamp:   Thu Apr 26 2018 22:02:26 GMT+0200 (CEST)