Registration is open - Live, Instructor-led Online Classes - Elasticsearch in March - Solr in April - OpenSearch in May. See all classes


Log Management for IBM Bluemix and Cloud Foundry

Enterprises without DevOps teams and culture typically experience long applications deployment cycles. Once developers make a new release the operations team needs to deploy it to one of the existing or new servers, and it often takes a long time (up to several weeks is not uncommon) to get a new application deployed to production. This process may include hardware purchases, configuration, and troubleshooting sessions with the development team. Especially the troubleshooting portion might take a while, with the typical e-mail ping-pong with commands for the administrators, sysadmins sending log files to the developers… Modern digital businesses don’t operate that way any more.  Instead, modern DevOps teams often utilize PaaS to boost productivity, without even thinking of server resources, ansible/chef/puppet scripts and related stuff. They push their application releases with a few commands to “the cloud” – typically a platform as a service (PaaS) like IBM Bluemix! Once deployments get simplified in such a way, the next logical step is to disrupt the e-mail ping-pong for troubleshooting by introducing centralized log managementLogs are the most important source of information for application troubleshooting. Central log management is crucial for providing real-time log access to relevant teammates.  

In this post, we’ll show how to do this by deploying a Node.js app to IBM Bluemix and using Logsene for Log Management.


Let’s start with the deployment of a Node.js app on IBM Bluemix. First, we’ll clone a git repository.

> git clone hello-world-app
> cd hello-world-app

Next, because IBM Bluemix is based on Cloud Foundry, we’ll use a few commands to get access to IBM Bluemix:

> cf api https://api.ng.bluemix.net> cf login

When prompted for username and password enter your IBM ID (email) and password. The final step is to push the app in the current directory to IBM Bluemix:

> cf push app1

This takes a while and the command prints the deployment status to the console. Once your app starts successfully, it will be available via the displayed URL (orgSpace-appName), e.g.

Showing health and status for app hello-world-seti123 in org ORG1 / space seti321 as stefan.thies@sematext.com…
OK
requested state: started
instances: 1/1
usage: 1G x 1 instances
urls: hello-world-seti123.mybluemix.net
last uploaded: Tue Dec 13 16:02:11 UTC 2016
stack: cflinuxfs2
buildpack: SDK for Node.js(TM) (ibm-node.js-4.6.2, buildpack-v3.9-20161128-1327)
     state     since                    cpu    memory         disk           details
#0   running   2016-12-13 05:03:27 PM   0.0%   114.9M of 1G   117.8M of 1G

Access the app via curl or the browser and check the most recent application logs in the IBM Bluemix user interface.

image02

While at first you may think Bluemix provides log management service and wonder why you’d need to ship your logs from Bluemix to a Log Management SaaS, you will quickly realize that IBM Bluemix shows only the most recent logs. That, of course, is a no-go for production, where we need logs for at least the last few days, several weeks, or even archived for a longer period.

The Cloud Foundry documentation shows the option to forward logs via so called log drains from any Cloud Foundry app to centralized Log Management tools like Logsene. Here you can choose the retention time, save search queries, define alerts on logs, or visualize them in Kibana! The setup for this is done with another small set of  Cloud Foundry commands:

  1. Create a Logsene Application to obtain the Logsene Application token
  2. Create the log drain to your Logsene App (note the Logsene token in the URL):
    cf cups logsene-log-drain -l https://logsene-cf-receiver.sematext.com/1a4a748f-33c3-4e73-9e25-f3f0c82ff0f2
  3. Bind the log drain to your app. You could optionally bind multiple apps to the same log drain:
    cf bind-service YOUR-APP-NAME logsene-log-drain
  4. Re-stage the app to pick up the newly bound service:
    cf restage YOUR-APP-NAME

A few seconds later your logs will start arriving to Logsene and you will be able to search and visualize them in Logsene’s built-in Kibana:image01

IBM Bluemix logs in Kibana

image00

Alert definition for logs in Logsene

Your logs are safe for the configured retention time, and if you need to archive your logs to Amazon S3, Logsene can archive your logs to S3 for you, too. Just another little time safer for modern DevOps teams!

SIGN UP – FREE TRIAL

Start Free Trial