Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Performance between HBaseClient scan and HFileReaderV2


Copy link to this message
-
Re: Performance between HBaseClient scan and HFileReaderV2
Er, using MR over snapshots, which reads files directly...
https://issues.apache.org/jira/browse/HBASE-8369
However, it was only committed to 98.
There was interest in 94 port (HBASE-10076), but it never happened...
On Thu, Jan 2, 2014 at 1:42 PM, Sergey Shelukhin <[EMAIL PROTECTED]>wrote:

> You might be interested in using
> https://issues.apache.org/jira/browse/HBASE-8369
> However, it was only committed to 98.
> There was interest in 94 port (HBASE-10076), but it never happened...
>
>
> On Thu, Jan 2, 2014 at 1:32 PM, Jerry Lam <[EMAIL PROTECTED]> wrote:
>
>> Hello Vladimir,
>>
>> In my use case, I guarantee that a major compaction is executed before any
>> scan happens because the system we build is a read only system. There will
>> have no deleted cells. Additionally, I only need to read from a single
>> column family and therefore I don't need to access multiple HFiles.
>>
>> Filter conditions are nice to have because if I can read HFile 8x faster
>> than using HBaseClient, I can do the filter on the client side and still
>> perform faster than using HBaseClient.
>>
>> Thank you for your input!
>>
>> Jerry
>>
>>
>>
>> On Thu, Jan 2, 2014 at 1:30 PM, Vladimir Rodionov
>> <[EMAIL PROTECTED]>wrote:
>>
>> > HBase scanner MUST guarantee correct order of KeyValues (coming from
>> > different HFile's),
>> > filter condition+ filter condition on included column families and
>> > qualifiers, time range, max versions and correctly process deleted
>> cells.
>> > Direct HFileReader does nothing from the above list.
>> >
>> > Best regards,
>> > Vladimir Rodionov
>> > Principal Platform Engineer
>> > Carrier IQ, www.carrieriq.com
>> > e-mail: [EMAIL PROTECTED]
>> >
>> > ________________________________________
>> > From: Jerry Lam [[EMAIL PROTECTED]]
>> > Sent: Thursday, January 02, 2014 7:56 AM
>> > To: user
>> > Subject: Re: Performance between HBaseClient scan and HFileReaderV2
>> >
>> > Hi Tom,
>> >
>> > Good point. Note that I also ran the HBaseClient performance test
>> several
>> > times (as you can see from the chart). The caching should also benefit
>> the
>> > second time I ran the HBaseClient performance test not just benefitting
>> the
>> > HFileReaderV2 test.
>> >
>> > I still don't understand what makes the HBaseClient performs so poorly
>> in
>> > comparison to access directly HDFS. I can understand maybe a factor of 2
>> > (even that it is too much) but a factor of 8 is quite unreasonable.
>> >
>> > Any hint?
>> >
>> > Jerry
>> >
>> >
>> >
>> > On Sun, Dec 29, 2013 at 9:09 PM, Tom Hood <[EMAIL PROTECTED]> wrote:
>> >
>> > > I'm also new to HBase and am not familiar with HFileReaderV2.
>>  However,
>> > in
>> > > your description, you didn't mention anything about clearing the
>> linux OS
>> > > cache between tests.  That might be why you're seeing the big
>> difference
>> > if
>> > > you ran the HBaseClient test first, it may have warmed the OS cache
>> and
>> > > then HFileReaderV2 benefited from it.  Just a guess...
>> > >
>> > > -- Tom
>> > >
>> > >
>> > >
>> > > On Mon, Dec 23, 2013 at 12:18 PM, Jerry Lam <[EMAIL PROTECTED]>
>> > wrote:
>> > >
>> > > > Hello HBase users,
>> > > >
>> > > > I just ran a very simple performance test and would like to see if
>> > what I
>> > > > experienced make sense.
>> > > >
>> > > > The experiment is as follows:
>> > > > - I filled a hbase region with 700MB data (each row has roughly 45
>> > > columns
>> > > > and the size is 20KB for the entire row)
>> > > > - I configured the region to hold 4GB (therefore no split occurs)
>> > > > - I ran compactions after the data is loaded and make sure that
>> there
>> > is
>> > > > only 1 region in the table under test.
>> > > > - No other table exists in the hbase cluster because this is a DEV
>> > > > environment
>> > > > - I'm using HBase 0.92.1
>> > > >
>> > > > The test is very basic. I use HBaseClient to scan the entire region
>> to
>> > > > retrieve all rows and all columns in the table, just iterating all

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB