We are thrilled to announce Logs Pipelines, an exciting new functionality you can now use in Sematext Cloud!
Raw log events are not always perfectly structured. They may include extra fields and be overly verbose, or they may lack structure that would enable you to derive more insights from your logs. For various reasons, it might be hard to set up a log shipper to transform or filter log events at the source. An alternative approach is to configure Logs Pipelines inside Sematext.
Each Pipeline consists of one or more Processors that are executed in the order in which they are defined in a Pipeline. They can change, drop, or even produce additional events or fields. They are chained to form a Pipeline. The output of one Processor is the input for the next Processor.
The following Processors are currently available:
- Drop
- Rename Fields
- Remove Fields
- Field Extractor
- Field Masking
- Script Field
- Sampling
- Geo
Each of your Logs Apps can have a different Pipeline, with a different set or order or Processors. Pipelines and Processors can be modified at any time and changes take effect immediately upon saving. Building a Pipeline is super simple and intuitive with a Pipeline Builder that includes a before/after preview so you can examine a Pipeline’s output before saving it.
Some use cases:
- Extraction of numerical values or metrics out of text fields, so you can create charts and alerts on them
- Standardization of field names
- Masking or dropping of sensitive data, such as PII
- Geo-enrichment based on IPs or latitude and longitude information
- Reduction in log volume
- …
Let us know what other Processors you would like to see in Sematext.
Importantly, we’ve adjusted the pricing in a small way. This will affect you, most likely positively, only if you start using Pipelines. We’re now separately tracking the volume of logs received and stored. We’ve reduced the cost of stored logs slightly and made it very cheap to ship logs to Sematext. You can see the adjusted pricing in both Sematext Cloud and on the Sematext pricing page. Learn more in Logs Pipelines docs.