Processors
Processors are units of processing in Pipelines. They can change, drop, or even produce additional events. They are chained to form a Pipeline. The output of one Processor is the input for the next Processor.
Processors available in Logs Pipelines are:
- XML Parser: Parse XML formatted fields into separate fields.
- Size processor: Estimates size of log event and extracts the information into a new field.
- Drop processor: Drop all log events that match a specified filter.
- Flattener processor: Used to flatten a nested structure into separate fields.
- User-Agent processor: Extract OS, browser, device and other data from a User-Agent attribute using an external library.
- URL Parser: Extracts query parameters and other important parameters from a URL.
- Rename Fields Processor: Change the names of log event fields.
- Remove Fields Processor: Remove fields that contain sensitive information or redundant data to save from costs. See Reducing Log Monitoring Costs for more information.
- Field Extractor Processor: Extract fields using a grok pattern.
- Field Masking Processor: Mask fields using a regex pattern and hide sensitive data. See Handle Sensitive Data for more information.
- Script Field Processor: Add or change a field using a script.
- Sampling Processor: Keep only a certain percentage of your data using random sampling.
- Geo Processor: Enrich log events with reverse geocoding location or IP address.
Configuration section will be empty in most cases when you initially open Pipeline Builder.
In such cases, instead of a selected Processor configuration, there will be a list of available Processors. Clicking on any Processor card will add a Processor with default values. Processors can also be added by clicking the "Add Processor" button.
When clicking this button, a modal with a searchable list of Processors will appear. You can then select which Processor to add. You can also add Processors by duplicating existing Processors.
The duplicate action is below the standard action button in the list of Processors. Note that the order of Processors matters since the output of one Processor is the input for the next Processor.
You can reorder Processors using the drag handle to the left of each Processor's name.
The selected Processor configuration will be displayed on the right side. Every Processor has at least a name and a description field, but all Processors will also have additional, Processor-specific configurations.
Errors¶
If a Processor is not configured correctly, the number of errors will be displayed next to the Processor's name and the Save Changes button will be disabled.
Filters¶
Almost all Processors have optional filters. With filters, you can select only a subset of events that the Processor should be applied to. Events that do not match filters will be ignored.
Processors use the same filtering you are already used to in Sematext.
The only difference is that you can also enter fields that are not keywords or even defined yet.
Wildcards¶
Another important difference is that you can use wildcards for values. There is support for standard wildcard characters: * matches any number of characters, and ? matches a single character.
This way you can, for example, drop events whose message field contains *raws* as in example below
