Custom Elasticsearch Index Templates in Logsene

One of the great things about Logsene, our log management tool, is that you don’t need to care about the back-end – you know, where you store your logs. You just pick a log shipper (here are Top 5 Log Shippers), point it to Logsene (here’s How to Send Logs to Logsene) and you are done. Logsene takes care of everything for you – your logs stop filling up your disk, you don’t have to worry about log compression and rotation, your logs get indexed so when you need to troubleshoot issues you have one place where you get see and search all your logs from all your applications, servers, and environments. This is all nice and dandy, but what if your logs are special and you want them analyzed in a specific way, and not the way Logsene’s predefined index templates and analysis work?  To handle such use cases we’ve recently made it possible for Logsene users to define how their logs are analyzed. Let’s look at an example.

Registering Log Index Template in Logsene

Logsene is built on top of Elasticsearch and exposes a subset of its API to the users. Because of that, all the great tools available for Elasticsearch work with Logsene.  For example, you can use Logstash to ship logs to Logsene and you can use Kibana to search and graph logs stored in Logsene. In fact, Kibana is the alternative UI available out of the box for Logsene users. Logsene users can now use Elasticsearch Index Templates functionality to define new templates for their indices in Logsene. Let’s say that we want to have a new type of logs that contain a new type, let’s call it messages, with one analyzed text field – message, and two non-analyzed text fields – tag and nick. Our index template for that might look as follows:

curl -XPUT 'logsene-receiver.sematext.com/_template/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee_MyTemplate' -d '{
 "template" : "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee*",
 "order" : 21,
 "settings" : {
  "index.analysis.analyzer.my_own_lowercase.type" : "custom",
  "index.analysis.analyzer.my_own_lowercase.tokenizer" : "keyword",
  "index.analysis.analyzer.my_own_lowercase.filter.0" : "lowercase",
 },
 "mappings" : {
  "message" : {
   "properties" : {
    "message" : { "type" : "string" },
    "tags" : { "type" : "string", "analyzer" : "my_own_lowercase" },
    "nick" : { "type" : "string", "analyzer" : "my_own_lowercase" }
   }
  }
 }
}'

That “aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee” is a fake Logsene app token we are using in this example.  You should, of course, use your own Logsene app token instead.

There are a few things one should remember when registering an index template in Logsene:

  1. To register a template one should use HTTP PUT method and send it to logsene-receiver.sematext.com/_template/TOKEN_NAME. The TOKEN part is your Logsene app’s token and the NAME part is the name of the template, which should be unique for your Logsene app.
  2. The template property inside the JSON request should be set to TOKEN* (yes, with the trailing asterisk), otherwise Logsene will reject the template.
  3. The order property must to be higher than 10 and should be unique for your templates.
  4. Only mappings and settings sections of the templates are allowed, with the limitation that settings section can only contain analysis definition.
  5. You can register multiple index templates by just using a different NAME.
  6. Very importantly, keep in mind that registered index templates do not come into effect immediately — they become active within the next 24 hours — specifically, at 00:00 UTC.

The above command should result in a response similar to the following one:

{"acknowledged":true}

This means that the template was successfully registered. If an error occurs the response from Logsene will be different, for example:

{"error":"Error occured during template verification","errorId":"2739358978185","status":"400"}

Reading Defined Templates

Of course, once your templates are in you can also read them. Doing that is very simple, you just need your Logsene app token and a request that looks as follows:

curl -XGET 'logsene-receiver.sematext.com/_template/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee*'

The response will contain all the templates defined for the application with the specified token and will look as follows:

{"aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee_MyTemplate":{"order":21,"template":"aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee*","settings":{"index.analysis.analyzer.my_own_lowercase.tokenizer":"keyword","index.analysis.analyzer.my_own_lowercase.type":"custom","index.analysis.analyzer.my_own_lowercase.filter.0":"lowercase"},"mappings":{"message":{"properties":{"message":{"type":"string"},"tags":{"analyzer":"my_own_lowercase","type":"string"},"nick":{"analyzer":"my_own_lowercase","type":"string"}}}},"aliases":{}}}

Of course, you can also read a single template by running a command like this:

curl -XGET 'logsene-receiver.sematext.com/_template/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee_MyTemplate'

Deleting Templates

Sometimes you may need to remove templates you’ve defined.  Doing that is as simple as running HTTP DELETE command against the _template REST end-point of Logsene and specifying the template name, like this:

curl -XDELETE 'logsene-receiver.sematext.com/_template/18fcf616-7c1a-4bb5-840e-deaf9ad73d00_MyTemplate'

If deletion was successful Logsene will respond with the following message:

{"acknowledged":true}

If something went wrong, you will see an error:

{"error":"Only full template names can be used with deletes","errorId":"1944493021299","status":"400"}

If you still haven’t had a chance to try out Logsene, go to http://sematext.com/logsene/index.html and create a free account (or just add new Logsene application if you already have an account). You can also try a live demo of Logsene to quickly look how it works on common data.  If you can’t ship your logs to the cloud, you can also run Logsene On Premises or on your own cloud instances (e.g. on AWS EC2).

Leave a Reply