Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # dev >> Poor HBase random read performance


Copy link to this message
-
Re: Poor HBase random read performance
I agree to both your points.

I have been trying to run tests in accordance with this post here:

http://hadoopblog.blogspot.com/2012/05/hadoop-and-solid-state-drives.html

My benchmarks look better than the blog post says - 35K but i dont know
their hardware configuration. It mentions that there is considerable
overhead in DFSClient code paths. Wonder if there is some low hanging fruit
there.

Varun
On Mon, Jul 1, 2013 at 11:26 AM, Vladimir Rodionov
<[EMAIL PROTECTED]>wrote:

> I think that "all data in block cache" test with 8K block size  will get
> significant boost as well.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: [EMAIL PROTECTED]
>
> ________________________________________
> From: Lars Hofhansl [[EMAIL PROTECTED]]
> Sent: Sunday, June 30, 2013 12:45 AM
> To: Varun Sharma
> Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]
> Subject: Re: Poor HBase random read performance
>
> Cool. That mingles well with the theory.
> If each read of a KV incurs a bock read reducing the block size by 4
> should lead to close to a 4x gain.
>
> W.r.t. the more store files... More KVs need to be read during the merge
> sorting, each likely incurring a block read of their own.
>
> Not sure what the leveldb folks were measuring. They also have a slightly
> different (simpler) problem as KVs cannot be client dated there, so older
> KVs are always in older files, which is not true for HBase.
>
> Still loading from the OS cache be faster in HBase. Ideally some memory
> mapped IO.
>
> -- Lars
>
>
> Varun Sharma <[EMAIL PROTECTED]> wrote:
>
> >Another update. I reduced the block size from 32K (it seems i was running
> with 32K initially not 64K) to 8K and bam, the throughput went from 4M
> requests to 11M.
> >
> >
> >One interesting thing to note however, is that when I had 3 store files
> per region, throughput on random reads was 1/3rd, this is understandable
> because u need to bring in 3X the amount of blocks and then merge. However,
> when I look at the leveldb benchmarks for non compacted vs compacted
> tables, I wonder why they are able to do 65K reads per second vs 80K reads
> per second when comparing compacted/non compacted files. It seems for their
> benchmark - performance does not fall proportionaly with # of store files
> (unless perhaps that benchmark includes bloom filters which I disabled).
> >
> >
> >Also, it seems the idLock issues was because of locking on IndexBlocks
> which are always hot. Now idLock does not seem to be an issue when its only
> locking up data blocks and for truly random reads, no data block is hot.
> >
> >
> >
> >On Sat, Jun 29, 2013 at 3:39 PM, Varun Sharma <[EMAIL PROTECTED]>
> wrote:
> >
> >So, I just major compacted the table which initially had 3 store files
> and performance went 3X from 1.6M to 4M+.
> >
> >
> >The tests I am running, have 8 byte keys with ~ 80-100 byte values. Right
> now i am working with 64K block size, I am going to make it 8K and see if
> that helps.
> >
> >
> >The one point though is the IdLock mechanism - that seems to add a huge
> amount of overhead 2x - however in that test I was not caching index blocks
> in the block cache, which means a lot higher contention on those blocks. I
> believe it was used so that we dont load the same block twice from disk. I
> am wondering, when IOPs are surplus (ssds for example), if we should have
> an option to disable it though I probably should reevaluate it, with index
> blocks in block cache.
> >
> >
> >
> >On Sat, Jun 29, 2013 at 3:24 PM, lars hofhansl <[EMAIL PROTECTED]> wrote:
> >
> >Should also say that random reads this way are somewhat of a worst case
> scenario.
> >
> >If the working set is much larger than the block cache and the reads are
> random, then each read will likely have to bring in an entirely new block
> from the OS cache,
> >even when the KVs are much smaller than a block.
> >
> >So in order to read a (say) 1k KV HBase needs to bring 64k (default block
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB