Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBase Random Read latency > 100ms


Copy link to this message
-
Re: HBase Random Read latency > 100ms
Vladimir,

Yes. I am fully aware of the HDD limitation and wrong configurations wrt
RAID.
Unfortunately, the hardware is leased from others for this work and I
wasn't consulted to decide the h/w specification for the tests that I am
doing now. Even the RAID cannot be turned off or set to RAID-0

Production system is according to the Hadoop needs (100 Nodes with 16 Core
CPU, 192 GB RAM, 24 X 600GB SAS Drives, RAID cannot be completely turned
off, so we are creating 1 Virtual Disk containing only 1 Physical Disk and
the VD RAID level set to* *RAID-0). These systems are still not available. If
you have any suggestion on the production setup, I will be glad to hear.

Also, as pointed out earlier, we are planning to use HBase also as an in
memory KV store to access the latest data.
That's why RAM was considered huge in this configuration. But looks like we
would run into more problems than any gains from this.

Keeping that aside, I was trying to get the maximum out of the current
cluster or as you said Is 500-1000 OPS the max I could get out of this
setup?

Regards,
Ramu
On Tue, Oct 8, 2013 at 3:02 AM, Vladimir Rodionov
<[EMAIL PROTECTED]>wrote:

> Ramu, your HBase configuration (128GB of heap) is far from optimal.
> Nobody runs HBase with that amount of heap to my best knowledge.
> 32GB of RAM is the usual upper limit. We run 8-12GB in production.
>
> What else, your IO capacity is VERY low. 2 SATA drives in RAID 1 for
> mostly random reads load?
> You should have 8, better 12-16 drives per server. Forget about RAID. You
> have HDFS.
>
> Block cache in your case does not help much , as since your read
> amplification is at least x20 (16KB block and 724 B read) - its just waste
> RAM (heap). In your case you do not need LARGE heap and LARGE block cache.
>
> I advise you reconsidering your hardware spec, applying all optimizations
> mentioned already in this thread and lowering your expectations.
>
> With a right hardware you will be able to get 500-1000 truly random reads
> per server.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Ramu M S [[EMAIL PROTECTED]]
> Sent: Monday, October 07, 2013 5:23 AM
> To: [EMAIL PROTECTED]
> Subject: Re: HBase Random Read latency > 100ms
>
> Hi Bharath,
>
> I am little confused about the metrics displayed by Cloudera. Even when
> there are no oeprations, the gc_time metric is showing 2s constant in the
> graph. Is this the CMS gc_time (in that case no JVm pause) or the GC pause.
>
> GC timings reported earlier is the average taken for gc_time metric for all
> region servers.
>
> Regards,
> Ramu
>
>
> On Mon, Oct 7, 2013 at 9:10 PM, Ramu M S <[EMAIL PROTECTED]> wrote:
>
> > Jean,
> >
> > Yes. It is 2 drives.
> >
> > - Ramu
> >
> >
> > On Mon, Oct 7, 2013 at 8:45 PM, Jean-Marc Spaggiari <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Quick questionon the disk side.
> >>
> >> When you say:
> >> 800 GB SATA (7200 RPM) Disk
> >> Is it 1x800GB? It's raid 1, so might be 2 drives? What's the
> >> configuration?
> >>
> >> JM
> >>
> >>
> >> 2013/10/7 Ramu M S <[EMAIL PROTECTED]>
> >>
> >> > Lars, Bharath,
> >> >
> >> > Compression is disabled for the table. This was not intended from the
> >> > evaluation.
> >> > I forgot to mention that during table creation. I will enable snappy
> >> and do
> >> > major compaction again.
> >> >
> >> > Please suggest other options to try out and also suggestions for the
> >> > previous questions.
> >> >
> >> > Thanks,
> >> > Ramu
> >> >
> >> >
> >> > On Mon, Oct 7, 2013 at 6:35 PM, Ramu M S <[EMAIL PROTECTED]>
> wrote:
> >> >
> >> > > Bharath,
> >> > >
> >> > > I was about to report this. Yes indeed there is too much of GC time.
> >> > > Just verified the GC time using Cloudera Manager statistics(Every
> >> minute
> >> > > update).
> >> > >
> >> > > For each Region Server,
> >> > >  - During Read: Graph shows 2s constant.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB