Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> Poor HBase map-reduce scan performance


+
Bryan Keller 2013-05-01, 04:01
+
Ted Yu 2013-05-01, 04:17
+
Bryan Keller 2013-05-01, 04:31
+
Ted Yu 2013-05-01, 04:56
+
Bryan Keller 2013-05-01, 05:01
Copy link to this message
-
Re: Poor HBase map-reduce scan performance
Your average row is 35k so scanner caching would not make a huge difference, although I would have expected some improvements by setting it to 10 or 50 since you have a wide 10ge pipe.

I assume your table is split sufficiently to touch all RegionServer... Do you see the same load/IO on all region servers?

A bunch of scan improvements went into HBase since 0.94.2.
I blogged about some of these changes here: http://hadoop-hbase.blogspot.com/2012/12/hbase-profiling.html

In your case - since you have many columns, each of which carry the rowkey - you might benefit a lot from HBASE-7279.

In the end HBase *is* slower than straight HDFS for full scans. How could it not be?
So I would start by looking at HDFS first. Make sure Nagle's is disbaled in both HBase and HDFS.

And lastly SSDs are somewhat new territory for HBase. Maybe Andy Purtell is listening, I think he did some tests with HBase on SSDs.
With rotating media you typically see an improvement with compression. With SSDs the added CPU needed for decompression might outweigh the benefits.

At the risk of starting a larger discussion here, I would posit that HBase's LSM based design, which trades random IO with sequential IO, might be a bit more questionable on SSDs.

If you can, it would be nice to run a profiler against one of the RegionServers (or maybe do it with the single RS setup) and see where it is bottlenecked.
(And if you send me a sample program to generate some data - not 700g, though :) - I'll try to do a bit of profiling during the next days as my day job permits, but I do not have any machines with SSDs).

-- Lars
________________________________
 From: Bryan Keller <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Tuesday, April 30, 2013 9:31 PM
Subject: Re: Poor HBase map-reduce scan performance
 

Yes, I have tried various settings for setCaching() and I have setCacheBlocks(false)

On Apr 30, 2013, at 9:17 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> From http://hbase.apache.org/book.html#mapreduce.example :
>
> scan.setCaching(500);        // 1 is the default in Scan, which will
> be bad for MapReduce jobs
> scan.setCacheBlocks(false);  // don't set to true for MR jobs
>
> I guess you have used the above setting.
>
> 0.94.x releases are compatible. Have you considered upgrading to, say
> 0.94.7 which was recently released ?
>
> Cheers
>
> On Tue, Apr 30, 2013 at 9:01 PM, Bryan Keller <[EMAIL PROTECTED]> wrote:
>
>> I have been attempting to speed up my HBase map-reduce scans for a while
>> now. I have tried just about everything without much luck. I'm running out
>> of ideas and was hoping for some suggestions. This is HBase 0.94.2 and
>> Hadoop 2.0.0 (CDH4.2.1).
>>
>> The table I'm scanning:
>> 20 mil rows
>> Hundreds of columns/row
>> Column keys can be 30-40 bytes
>> Column values are generally not large, 1k would be on the large side
>> 250 regions
>> Snappy compression
>> 8gb region size
>> 512mb memstore flush
>> 128k block size
>> 700gb of data on HDFS
>>
>> My cluster has 8 datanodes which are also regionservers. Each has 8 cores
>> (16 HT), 64gb RAM, and 2 SSDs. The network is 10gbit. I have a separate
>> machine acting as namenode, HMaster, and zookeeper (single instance). I
>> have disk local reads turned on.
>>
>> I'm seeing around 5 gbit/sec on average network IO. Each disk is getting
>> 400mb/sec read IO. Theoretically I could get 400mb/sec * 16 = 6.4gb/sec.
>>
>> Using Hadoop's TestDFSIO tool, I'm seeing around 1.4gb/sec read speed. Not
>> really that great compared to the theoretical I/O. However this is far
>> better than I am seeing with HBase map-reduce scans of my table.
>>
>> I have a simple no-op map-only job (using TableInputFormat) that scans the
>> table and does nothing with data. This takes 45 minutes. That's about
>> 260mb/sec read speed. This is over 5x slower than straight HDFS.
>> Basically, with HBase I'm seeing read performance of my 16 SSD cluster
>> performing nearly 35% slower than a single SSD.
>>
>> Here are some things I have changed to no avail:
+
Bryan Keller 2013-05-01, 06:02
+
Michael Segel 2013-05-01, 14:24
+
lars hofhansl 2013-05-01, 06:21
+
Bryan Keller 2013-05-01, 15:00
+
Bryan Keller 2013-05-02, 01:01
+
lars hofhansl 2013-05-02, 04:41
+
Bryan Keller 2013-05-02, 04:49
+
Bryan Keller 2013-05-02, 17:54
+
Nicolas Liochon 2013-05-02, 18:00
+
lars hofhansl 2013-05-03, 00:46
+
Bryan Keller 2013-05-03, 07:17
+
Bryan Keller 2013-05-03, 10:44
+
lars hofhansl 2013-05-05, 01:33
+
Bryan Keller 2013-05-08, 17:15
+
Bryan Keller 2013-05-10, 15:46
+
Sandy Pratt 2013-05-22, 20:29
+
Ted Yu 2013-05-22, 20:39
+
Sandy Pratt 2013-05-22, 22:33
+
Ted Yu 2013-05-22, 22:57
+
Bryan Keller 2013-05-23, 15:45
+
Sandy Pratt 2013-05-23, 22:42
+
Ted Yu 2013-05-23, 22:47
+
Sandy Pratt 2013-06-05, 01:11
+
Sandy Pratt 2013-06-05, 08:09
+
yonghu 2013-06-05, 14:55
+
Ted Yu 2013-06-05, 16:12
+
yonghu 2013-06-05, 18:14
+
Sandy Pratt 2013-06-05, 18:57
+
Sandy Pratt 2013-06-05, 17:58
+
lars hofhansl 2013-06-06, 01:03
+
Bryan Keller 2013-06-25, 08:56
+
lars hofhansl 2013-06-28, 17:56
+
Bryan Keller 2013-07-01, 04:23
+
Ted Yu 2013-07-01, 04:32
+
lars hofhansl 2013-07-01, 10:59
+
Enis Söztutar 2013-07-01, 21:23
+
Bryan Keller 2013-07-01, 21:35
+
lars hofhansl 2013-05-25, 05:50
+
Enis Söztutar 2013-05-29, 20:29
+
Bryan Keller 2013-06-04, 17:01
+
Michael Segel 2013-05-06, 03:09
+
Matt Corgan 2013-05-01, 06:52
+
Jean-Marc Spaggiari 2013-05-01, 10:56
+
Bryan Keller 2013-05-01, 16:39
+
Naidu MS 2013-05-01, 07:25
+
ramkrishna vasudevan 2013-05-01, 07:27
+
ramkrishna vasudevan 2013-05-01, 07:29
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB