More than two years ago we posted a recipe on how to centralize syslog in Elasticsearch in order to search and analyze them with Kibana, all by using only rsyslog. This works well, because rsyslog is fast and light, as […]
Recipe: rsyslog + Redis + Logstash
OK, so you want to hook up rsyslog with Logstash. If you don’t remember why you want that, let me give you a few hints: Logstash can do lots of things, it’s easy to set up but tends to be […]
Replaying Elasticsearch Slowlogs with Logstash and JMeter
Sometimes we just need to replay production queries – whether it’s because we want a realistic load test for the new version of a product or because we want to reproduce, in a test environment, a bug that only occurs […]
Recipe: Reindexing Elasticsearch Documents with Logstash
If you’re working with Elasticsearch, it’s very likely that you’ll need to reindex data at some point. The most popular reason is because you need a mapping change that is incompatible with your current mapping. New fields can be added […]
Monitoring rsyslog’s Performance with impstats and Elasticsearch
If you’re using rsyslog for processing lots of logs (and, as we’ve shown before, rsyslog is good at processing lots of logs), you’re probably interested in monitoring it. To do that, you can use impstats, which comes from input module […]
Using Elasticsearch Mapping Types to Handle Different JSON Logs
By default, Elasticsearch does a good job of figuring the type of data in each field of your logs. But if you like your logs structured like we do, you probably want more control over how they’re indexed: is time_elapsed […]
Parsing and Centralizing Elasticsearch Logs with Logstash
NOTE: this configuration was tested with Logstash 2.2 on logs generated by Elasticsearch 2.2 No, it’s not an endless loop waiting to happen, the plan here is to use Logstash to parse Elasticsearch logs and send them to another Elasticsearch […]
Encrypting Logs on Their Way to Elasticsearch Part 2: TLS Syslog
In part 1 of the “encrypted logs” series we discussed sending logs to Elasticsearch over HTTPS. This second part is about TLS syslog. If you wonder what this has to do with Elasticsearch, the point is that TLS syslog is […]
Encrypting Logs on Their Way to Elasticsearch
Let’s assume you want to send your logs to Elasticsearch, so you can search or analyze them in realtime. If your Elasticsearch cluster is in a remote location (EC2?) or is our log analytics service, Logsene (which exposes the Elasticsearch […]