Using Filebeat to Send Elasticsearch Logs to Logsene

One of the nice things about our log management and analytics solution Logsene is that you can talk to it using various log shippers.  You can use Logstash, or you can use syslog protocol capable tools like rsyslog, or you can just push your logs using the Elasticsearch API just like you would send data to a local Elasticsearch cluster. And like any good DevOps team, we like to play with all the tools ourselves.  So we thought the timing was right to make Logsene work as a final destination for data sent using Filebeat.

With that in mind, let’s see how to use Filebeat to send log files to Logsene.  In this post, we’ll ship Elasticsearch logs, but Filebeat can tail and ship logs from any log file, of course.

Installing Filebeat

The first step is the easiest — you just need to go to the Filebeat download page and get the package for your operating system. For the purposes of this article, we’ve used Filebeat 7.5.2 though the older version will be good as well. You just need to remember that Sematext Cloud will only work with the Apache 2.0 licensed Filebeat.

After you download the package you need to unpack it into a directory of your choice.

Configuring Filebeat

If you already have your Logsene application created — great!  If not, please go here to get set up. You will need your token, which you can find in the App Settings section of the menu:

Once you have the Logsene app token you are ready to configure Filebeat. To do that you first need to create a new configuration file called logsene.yml and put in it a configuration snippet similar to the one below:

filebeat:
  inputs:
    - type: log
      paths:
        - /opt/elasticsearch/logs/*.log

processors:
- rename:
    fields:
     - from: host
       to: host_object

setup.template:
  enabled: false
  overwrite: false
output.elasticsearch:
  hosts: ["https://logsene-receiver.sematext.com:443"]
  protocol: https
  index: "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"

logging:
  to_files: true
  files:
    path: /var/log/filebeat/
    name: filebeat_eslogs.log
    rotateeverybytes: 10485760
  level: info

A brief comment on the above configuration: the first section describes which log files should be read and sent to Logsene.  In this example we are shipping logs from files ending with the .log in the /opt/elasticsearch/logs/ directory.

In the processors section, we are renaming a field called host and we give it a new name – host_object. This is done to match the needs of Sematext Cloud templates.

Next we disable sending the defaults Filebeat template and finally we configure the output in the section called output. It tells Filebeat to send data to Elasticsearch. Yes — to Elasticsearch — because Logsene provides the Elasticsearch API. You’ll need to provide three properties here:

1. The first one, called hosts, needs to point to https://logsene-receiver.sematext.com on port 443. You’ll want to use SSL, but you could also use HTTP and send data to port 80 if you don’t want to use SSL.
2. The second option specifies the protocol, which in our case is the https.
3. The third option is the index, and you’ll need to specify your Logsene app token here. In the above example, you can see a token of aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee, but you should use your own.

The last section specifies that you would like to know what Filebeat is doing (and you’ll want to save that information to file).

Running Filebeat with your configuration

You can now run Filebeat and use your configuration.  To do that, run the following command:

$./filebeat -c logsene.yml

This tells Filebeat to use the configuration file you’ve created and send Elasticsearch log files to Logsene.

That’s all

You can now go to your Logsene application and look at the logs you’ve sent:

Isn’t that fast and easy? 😉

Once your logs are in Logsene you can build all kinds of reports with Kibana, which is integrated into Logsene, you can get alerts based on data in your logs, you can invite your teammates, so you can all have access to all your logs in one place, and so on.

If you’re not using Logsene yet you can have a 30-day trial up and running in minutes — just sign up for a free account!  There’s no commitment and no credit card required.  And drop us an email or hit us on Twitter with suggestions, questions or comments about this post.