Slack Analytics & Search with Elasticsearch, Node.js and React

Sematext team is highly distributed. We are ex-Skype users who recently switched to Slack for team collaboration. We’ve been happy with Slack features and especially integrations for watching our Github repositories, Jenkins, or receiving SPM or Logsene Alerts from our production servers through their ChatOps support. The ability to add custom integrations is really awesome! Being search experts it is hard for us to accept any limitation in search functionality in tools we use. For example, I personally miss the ability to search over all teams and all channels and I really miss having no analytics on user activity or channel usage. Elasticsearch has become a popular data store for analytical queries.  What if we could take all Slack messages and index them into Elasticsearch? This would make it possible to perform advanced analytics with Kibana or Grafana, such as getting like top terms used, most active users or channels. Finally, a simple mobile web page to access only the indexed data from various Teams and Channels might be handy to have, too.

In this post we’re going to see how to build what we just described.  We’ll use the Slack API, Node.js, React and Elasticsearch in 3 steps:

• Index Data from Slack
• Analyse Data from Slack
• Create a custom Web-App for search

Index Data from Slack

The Slack API provides several ways to access data. For example, outgoing webhook. This looks useful at first, however, this needs a setup per channel or keywords as trigger. Then I discovered a better way – the Node.js Slack Client.  Simply log in with your Slack account and get all Slack messages! I wrote a little Node.js app to dump the relevant information as JSON to the console or to a file.  Having the JSON output, it can be piped to Logagent-js a smart log shipper written in Node.js. I packaged this as “slack-elasticsearch-indexer” so it’s super easy to run:

npm install slack-elasticsearch-indexer

# Set Elasticsearch Server, btw. the Logsene Receiver is the default
export LOGSENE_URL=https://logsene-receiver.sematext.com/_bulk
# 1 - Slack API Token from https://api.slack.com/web
# 2 - Index name or Logsene Token from https://apps.sematext.com
npm start SLACK_WEB_API_TOKEN LOGSENE_TOKEN

The LOGSENE_TOKEN is what you can get from Logsene – the “ELK log management service”.  Using Logsene means you don’t have to bother running your own Elasticsearch, plus the volume of most team’s Slack data is probably so small that it fits in Logsene’s free plan! 🙂

Once you run the above you should see new Slack Messages on the console.  At the same time the messages will also be sent to Logsene and you will see them in the Logsene UI (or your local Elasticsearch server or cluster) right away.

Analyze Slack Messages in Sematext

Now that our Slack messages are in Logsene we can build our Kibana Dashboards to visualize channel utilization, top terms, the chattiest people, and so on.  But … did you know, that Logsene comes with a nice ad-hoc charting function? Simply open one of the Slack messages in Logsene, and click on the little chart symbol in the field userName and channel (see below).

This will very quickly render top users and channels for you:

Slack Alerting

Imagine a support chat channel – wouldn’t it be nice to be notified when people start mentioning “Error”, “Problems” and “Broken” things increasingly frequently? This is where we can make use of Logsene Alerts and its ability to do anomaly detection. Any triggered alerts can be delivered via email, PagerDuty, Slack, HipChat or WebHooks:

While Logsene is great for alerts, analytics and Slack message search, as a general ‘data viewer’ the message rendering in Logsene does not show application-specific things like users’ profile pictures, which would allow much faster recognition of user messages. Thus, as our next step, we’ll create a simple Web Client with nice rendering of indexed Slack messages. Let’s see how this can be done very quickly using some cutting edge Web technology together with Logsene.

Create a Custom Web-App for Search

We recently started using Facebook’s React.js for rendering of various UI parts like the views for Top Database Operations and we came across a new set of React UI Components for Elasticsearch called SearchKit. Thanks to Logsene’s Elasticsearch API SearchKit works out of the box with Logsene!
After a few lines of CSS and some JavaScript a simple Slack Search UI is born. Check it out!

Edit the source code codepen.io

You just need to use your Logsene token as the Elasticsearch index name to run this app on your own data. For production we recommend adding a proxy to Elasticsearch (or Logsene) on the server side as described in the SearchKit UI documentation to hide connection details from the client application.

While this post shows how to index your Slack messages in Logsene for the purpose of archiving, searching, and analytics, we hope it also serves as an inspiration to build your own custom Search application with SearchKit, React, Node.js and Logsene?

If you haven’t used Logsene before, give it try – you can get a free account and have your logs and other event data in Logsene in no time. Drop us an email or hit us on Twitter with suggestions, questions or comments.