Last updated: 26 January, 2018 Many of our clients use AWS EC2. In the context of Elasticsearch consulting or support, one question we often get is: should we use AWS Elasticsearch Service instead of deploying Elasticsearch ourselves? The question is valid whether “self […]
Elastic Stack Import-Export with Logstash & Logsene
In earlier posts, we explained how one can reindex data from one Elasticsearch cluster to another, or within the same Elasticsearch cluster, via tools like Logstash and rsyslog. The same recipes apply to Logsene, as it exposes the Elasticsearch API. […]
Scalable and Flexible Elasticsearch Reindexing via rsyslog
Earlier on, we posted a recipe on reindexing data from within an Elasticsearch 2.3+ cluster. But this doesn’t work if you want to reindex in a different cluster or if your Elasticsearch is older than 2.3. Or both, when you’re […]
Monitoring rsyslog with Kibana and SPM
A while ago we published this post where we explained how you can get stats about rsyslog, such as the number of messages enqueued, the number of output errors and so on. The point was to send them to Elasticsearch […]
Sending AWS CloudWatch/VPC Logs to Logsene
Use-case: you’re using AWS VPC and want visibility over the connection to your VPC. Which IPs are allowed or denied connections to certain ports, how much traffic goes through each connection and so on. Solution: send AWS VPC logs […]
How to forward CloudTrail (or other logs from AWS S3) to Logsene
This recipe shows how to send CloudTrail logs (which are .gz logs that AWS puts in a certain S3 bucket) to a Logsene application, but should apply to any kinds of logs that you put into S3. We’ll use AWS […]
Recipe: Apache Logs + rsyslog (parsing) + Elasticsearch
More than two years ago we posted a recipe on how to centralize syslog in Elasticsearch in order to search and analyze them with Kibana, all by using only rsyslog. This works well, because rsyslog is fast and light, as […]
Recipe: rsyslog + Redis + Logstash
OK, so you want to hook up rsyslog with Logstash. If you don’t remember why you want that, let me give you a few hints: Logstash can do lots of things, it’s easy to set up but tends to be […]
Replaying Elasticsearch Slowlogs with Logstash and JMeter
Sometimes we just need to replay production queries – whether it’s because we want a realistic load test for the new version of a product or because we want to reproduce, in a test environment, a bug that only occurs […]