Are your GC logs enabled? Can you see any long pause in it?
On Fri, Aug 30, 2013 at 4:45 AM, Kiru Pakkirisamy <[EMAIL PROTECTED]
> I just moved from 0.94.10 to 0.94.11. Tremendous improvement in our app's
> query response. Went down to 1.3 sec from 1.7 sec.
> Concurrent tests are also good, but it still exponentially degrades from
> to 10 secs for 8 concurrent clients. There might a bug lurking in there
> somewhere that is probably affecting us.
> - kiru
> From: Federico Gaule <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Sent: Thursday, August 29, 2013 5:37 AM
> Subject: Re: experiencing high latency for few reads in HBase
> In 0.94.11 Release, has been included an optimization for MultiGets:
> What version have you deployed?
> On 08/29/2013 01:29 AM, lars hofhansl wrote:
> > A 1s SLA is tough in HBase (or any large memory JVM application).
> > Maybe, if you presplit your table, play with JDK7 and the G1 collector,
> but nobody here will vouch for such an SLA in the 99th percentile.
> > I heard some folks have experimented with 30GB heaps and G1 and have
> reported max GC times of 200ms, but I have not verified that.
> > -- Lars
> > ----- Original Message -----
> > From: Saurabh Yahoo <[EMAIL PROTECTED]>
> > To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> > Cc: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> > Sent: Wednesday, August 28, 2013 3:17 PM
> > Subject: Re: experiencing high latency for few reads in HBase
> > Hi Vlad,
> > Thanks for your response.
> > 1. Our SLA is less than one sec. we cannot afford latency more than 1
> > We can increase heap size if that help, we have enough memory on server.
> What would be the optimal heap size?
> > 2. Cache hit ratio is 95%. One thing I don't understand that we have
> allocated only 4gb for block cache out of 12gb. That left 8gb for rest of
> JVM. There is no write. Memcache is empty. Is 8gb not enough for hbase to
> process the requests? What are the most memory consuming objects in region
> > 3. We will change the cf to IN_memory and report back performance
> > Thanks,
> > Saurabh.
> > On Aug 28, 2013, at 3:15 PM, Vladimir Rodionov <[EMAIL PROTECTED]>
> >> 1. 4 sec max latency is not that bad taking into account 12GB heap. It
> can be much larger. What is your SLA?
> >> 2. Block evictions is the result of a poor cache hit rate and the root
> cause of a periodic stop-the-world GC pauses (max latencies
> >> latencies you have been observing in the test)
> >> 3. Block cache consists of 3 parts (25% young generation, 50% -
> tenured, 25% - permanent). Permanent part is for CF with
> >> IN_MEMORY = true (you can specify this when you create CF). Block
> first stored in 'young gen' space, then gets promoted to 'tenured gen' space
> >> (or gets evicted). May be your 'perm gen' space is underutilized? This
> is exact 25% of 4GB (1GB). Although HBase LruBlockCache should use all the
> space allocated for block cache -
> >> there is no guarantee (as usual). If you don have in_memory column
> families you may decrease
> >> Best regards,
> >> Vladimir Rodionov
> >> Principal Platform Engineer
> >> Carrier IQ, www.carrieriq.com
> >> e-mail: [EMAIL PROTECTED]
> >> ________________________________________
> >> From: Saurabh Yahoo [[EMAIL PROTECTED]]
> >> Sent: Wednesday, August 28, 2013 5:10 AM
> >> To: [EMAIL PROTECTED]
> >> Subject: experiencing high latency for few reads in HBase
> >> Hi,
> >> We are running a stress test in our 5 node cluster and we are getting
> the expected mean latency of 10ms. But we are seeing around 20 reads out of
> 25 million reads having latency more than 4 seconds. Can anyone provide the
> insight what we can do to meet below second SLA for each and every read?