The new Sematext Agent version 3.6.0 introduces custom log shipping. This feature provides you with greater flexibility and control over your logs, augmenting Sematext Discovery when log files are not automatically discovered or when you need to specify custom log parsing rules.
Key benefits
- Ease of Use: You can specify log file locations and custom parsing rules through a YAML file. This is particularly useful for log files that are not automatically discovered or require custom parsing rules
- Unlimited Log Sources, Parsing Rules, and Destinations: Configure multiple log files, parsing rules, and destination Logs Apps within a single configuration file
- Flexible and Adaptable: Ideal for various environments, supporting both predefined and custom log formats
How it works
- Configuration File: The Sematext Agent periodically checks the
custom-logs.ymlfile for valid log shipping rules - Log Shipping: Upon finding valid rules, the agent starts collecting, parsing, and shipping logs to the specified Sematext Cloud Logs App using the provided App token
- Easy Updates: To stop log shipping, simply remove the rule(s) from the
custom-logs.ymlfile
Getting started
- Create a Logs App: Create a new Logs App in Sematext Cloud. For custom log files, create a Generic Logs App. For supported integrations, choose the relevant type.
- Retrieve App Token: After installing the Sematext Agent, ignore the Discovery screen and retrieve the App token from the Ship Logs screen under Custom Integrations
- Configure Custom Logs: Edit the `custom-logs.yml` file with your log rules
Example configuration
Below is an example of configuring custom log shipping for a custom application:
myapp-logs:
files:
- "/var/log/myapp/myapp.log"
- "/var/log/myapp/errors.log"
token: "your-sematext-logs-app-token"
regexPatterns:
- '^(?P\d{4}/\d{2}/\d{2}\s[\d|:]+)\s\[(?P.+?)]\s(?P\d+)#(?P\d+):\s(?P.*)'
patternTypes:
timestamp: "string"
level: "string"
pid: "int"
tid: "int"
message: "string"Upgrade to Sematext Agent 3.6.0 with custom log shipping, giving you more control over handling your logs.
For more information, please refer to Custom Logs Shipping docs.