HBase 0.98 Monitoring Support

HBase is a popular open-source, non-relational (NoSQL), column-oriented, distributed database that runs on top of the Hadoop Distributed File System (HDFS).  HBase is well suited for sparse data sets, which are common in many big data use cases.  Fortunately for all its users, SPM now supports monitoring, alerting and anomaly detection for HBase version 0.98.  Even those of you not running version 0.98 (here are the results for our HBase version distribution poll) are still in luck because a lot of HBase metrics captured by SPM are also in 0.94.x, 0.96.x, and even the recently released 1.0 version.  That said, HBase is one of those projects whose metrics change from version to version – some are deletes, some are added, others are modified.  If you have your own tools for monitoring HBase and are trying to monitor more than just the most basic HBase metrics, maintaining those tools must not be fun. Related to this common issue, we recently put together a “Build vs. Buy” post that weighs the pros and cons.

Here at Sematext we make heavy use of HBase.  We have recently moved from 0.94.x to 0.98.x and have been enjoying all its benefits.  Furthermore, we’ve recently updated SPM for HBase to monitor a pile of new HBase metrics.  Of course, we eat our own dog food and immediately got new and interesting insights about our own HBase clusters through some of the new metric charts.

For example, from https://apps.sematext.com/spm-reports/s/VhOltU14Cy we are now able to see the dramatic impact of major compactions on data locality (and thus HBase performance!):   (click to enlarge)

Local_Files_1

And from https://apps.sematext.com/spm-reports/s/7LU1qvs7ur we can see the number and size of HLog files over time:   (click to enlarge)

HLog

Alright, on to all the details!

Shiny, New HBase Metrics

In total, we’re talking 290 metrics: 195 for 0.98 and 95 for previous versions.  And lots of them changed in 0.98.  Here’s a summary of top-level SPM reports.  Each report listed below has one or more charts with one or more HBase metrics.  Juicy stuff.

Master:

  • Servers
  • Assign Manager
  • Balancer
  • FS
  • Snapshot

Region Server:

  • Regions & Stores
  • Requests
  • Files
  • Compact & Flush
  • Cache
  • Operations
  • Check & Mutate
  • WAL
  • Hedged Reads
  • MOB
  • Replication
  • Replication Source

Common / pre-0.98:

  • IPC
  • HBase JVM
  • UGI
  • Requests (pre-0.98)
  • Regions (pre-0.98)
  • Split (pre-0.98)
  • Memstore (pre-0.98)
  • Store (pre-0.98)
  • Compactions (pre-0.98)
  • FS (pre-0.98)
  • Block Cache (pre-0.98)

Screenshot: HBase Operation Calls & Time  (click to enlarge)

HBase_Ops_calls

Screenshot: HBase Slow Operations  (click to enlarge)

HBase_Slow_Ops

Screenshot: HBase Sync & Append Ops & Time  (click to enlarge)

HBase_Sync_Apend_Ops_Time

OK OK, how do I get all this stuff?

If you are not using SPM yet, simply sign up, create your first SPM App, and follow the directions in the UI.  You should see all your HBase metrics in a matter of minutes.  SPM is free for 30 days, requires no commitment or credit card and has no limit.  On Premises version is available as well.

If you are already using SPM, but not monitoring HBase, just create the SPM App for HBase, and follow the directions for installing the SPM agent on your HBase nodes.

If you are already using SPM for monitoring HBase, you just need to upgrade the SPM agent and configure it.

Leave a Reply