Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - HBase Random Read latency > 100ms


Copy link to this message
-
Re: HBase Random Read latency > 100ms
Varun Sharma 2013-10-08, 19:14
How many reads per second per region server are you throwing at the system
- also is 100ms the average latency ?
On Mon, Oct 7, 2013 at 2:04 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:

> He still should not see 100ms latency. 20ms, sure. 100ms seems large;
> there are still 8 machines serving the requests.
>
> I agree this spec is far from optimal, but there is still something odd
> here.
>
>
> Ramu, this does not look like a GC issue. You'd see much larger (worst
> case) latencies if that were the case (dozens of seconds).
> Are you using 40 client from 40 different machines? Or from 40 different
> processes on the same machine? Or 40 threads in the same process?
>
> Thanks.
>
> -- Lars
>
>
>
> ________________________________
>  From: Vladimir Rodionov <[EMAIL PROTECTED]>
> To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>
> Sent: Monday, October 7, 2013 11:02 AM
> Subject: RE: HBase Random Read latency > 100ms
>
>
> Ramu, your HBase configuration (128GB of heap) is far from optimal.
> Nobody runs HBase with that amount of heap to my best knowledge.
> 32GB of RAM is the usual upper limit. We run 8-12GB in production.
>
> What else, your IO capacity is VERY low. 2 SATA drives in RAID 1 for
> mostly random reads load?
> You should have 8, better 12-16 drives per server. Forget about RAID. You
> have HDFS.
>
> Block cache in your case does not help much , as since your read
> amplification is at least x20 (16KB block and 724 B read) - its just waste
> RAM (heap). In your case you do not need LARGE heap and LARGE block cache.
>
> I advise you reconsidering your hardware spec, applying all optimizations
> mentioned already in this thread and lowering your expectations.
>
> With a right hardware you will be able to get 500-1000 truly random reads
> per server.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
>
> From: Ramu M S [[EMAIL PROTECTED]]
> Sent: Monday, October 07, 2013 5:23 AM
> To: [EMAIL PROTECTED]
> Subject: Re: HBase Random Read latency > 100ms
>
> Hi Bharath,
>
> I am little confused about the metrics displayed by Cloudera. Even when
> there are no oeprations, the gc_time metric is showing 2s constant in the
> graph. Is this the CMS gc_time (in that case no JVm pause) or the GC pause.
>
> GC timings reported earlier is the average taken for gc_time metric for all
> region servers.
>
> Regards,
> Ramu
>
>
> On Mon, Oct 7, 2013 at 9:10 PM, Ramu M S <[EMAIL PROTECTED]> wrote:
>
> > Jean,
> >
> > Yes. It is 2 drives.
> >
> > - Ramu
> >
> >
> > On Mon, Oct 7, 2013 at 8:45 PM, Jean-Marc Spaggiari <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Quick questionon the disk side.
> >>
> >> When you say:
> >> 800 GB SATA (7200 RPM) Disk
> >> Is it 1x800GB? It's raid 1, so might be 2 drives? What's the
> >> configuration?
> >>
> >> JM
> >>
> >>
> >> 2013/10/7 Ramu M S <[EMAIL PROTECTED]>
> >>
> >> > Lars, Bharath,
> >> >
> >> > Compression is disabled for the table. This was not intended from the
> >> > evaluation.
> >> > I forgot to mention that during table creation. I will enable snappy
> >> and do
> >> > major compaction again.
> >> >
> >> > Please suggest other options to try out and also suggestions for the
> >> > previous questions.
> >> >
> >> > Thanks,
> >> > Ramu
> >> >
> >> >
> >> > On Mon, Oct 7, 2013 at 6:35 PM, Ramu M S <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> > > Bharath,
> >> > >
> >> > > I was about to report this. Yes indeed there is too much of GC time.
> >> > > Just verified the GC time using Cloudera Manager statistics(Every
> >> minute
> >> > > update).
> >> > >
> >> > > For each Region Server,
> >> > >  - During Read: Graph shows 2s constant.
> >> > >  - During Compaction: Graph starts with 7s and goes as high as 20s
> >> during
> >> > > end.
> >> > >
> >> > > Few more questions,
> >> > > 1. For the current evaluation, since the reads are completely random