Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # dev - 30% random performance in 0.95+


Copy link to this message
-
Re: 30% random performance in 0.95+
Jean-Marc Spaggiari 2013-07-02, 19:38
If we can't rely on PE, then why do we keep it? ;)

I think that even if it loads the local hard drive, we can still
compare the 2 results because it does the same thing against the 2
versions. So if one version uses more the hard drive than another one,
then even with 12 drives and 100 nodes, it will still use more the
hard drive than the previous version. Goal here is really to compare 2
versions together, and not the performances.

Now, to make sure that I can run comparisons as good as possible, what
is the minimum hardware requirement? Are 3 dedicated servers enought?
(1RS+MS and 2RS) Or should I really go with 1 master + 3RS as a
minimum to have relevant results? I want to build something we will be
able to rely.

JM

2013/6/29 Varun Sharma <[EMAIL PROTECTED]>:
> Actually sending in a separate thread - since it does not really compare
> different versions of HBase but one version of Block Cache vs FS
> Cache(through hdfs).
>
>
> On Sat, Jun 29, 2013 at 12:00 PM, Varun Sharma <[EMAIL PROTECTED]> wrote:
>
>> I did some tests yesterday, on this. I will send them in a separate thread.
>>
>>
>> On Sat, Jun 29, 2013 at 5:10 AM, lars hofhansl <[EMAIL PROTECTED]> wrote:
>>
>>> In my measurements 0.94 has been getting faster with each release in both
>>> read and write performance.
>>> I wonder how representative PE is after all; it only tests via the local
>>> FS layer (not HDFS), among other issues.
>>>
>>> -- Lars
>>>
>>>
>>>
>>> ----- Original Message -----
>>> From: Jean-Marc Spaggiari <[EMAIL PROTECTED]>
>>> To: [EMAIL PROTECTED]; lars hofhansl <[EMAIL PROTECTED]>
>>> Cc:
>>> Sent: Friday, June 28, 2013 8:03 PM
>>> Subject: Re: 30% random performance in 0.95+
>>>
>>> I think we should do that on 0.94 as well. I don't see any good reason
>>> to not do it.
>>>
>>> JM
>>>
>>> 2013/6/28 lars hofhansl <[EMAIL PROTECTED]>:
>>> > Yep.
>>> > Now the question is: Make these changes to 0.94 as well? Or just
>>> document these better.
>>> >
>>> > -- Lars
>>> >
>>> >
>>> >
>>> > ----- Original Message -----
>>> > From: Andrew Purtell <[EMAIL PROTECTED]>
>>> > To: "[EMAIL PROTECTED]" <[EMAIL PROTECTED]>; lars hofhansl <
>>> [EMAIL PROTECTED]>
>>> > Cc:
>>> > Sent: Friday, June 28, 2013 2:08 PM
>>> > Subject: Re: 30% random performance in 0.95+
>>> >
>>> > I've been thinking about how to periodically search through some of our
>>> > parameter space to see what changes to defaults are better all the way
>>> > around. Probably will so something based on Bigtop.
>>> >
>>> >
>>> > On Friday, June 28, 2013, lars hofhansl wrote:
>>> >
>>> >> And indeed just this makes a tremendous difference. Unpatched 0.94 with
>>> >> 40% block cache configured is actually faster than 0.95 with the same
>>> block
>>> >> cache size.
>>> >>
>>> >> -- Lars
>>> >>
>>> >>
>>> >>
>>> >> ----- Original Message -----
>>> >> From: lars hofhansl <[EMAIL PROTECTED] <javascript:;>>
>>> >> To: "[EMAIL PROTECTED] <javascript:;>" <[EMAIL PROTECTED]
>>> <javascript:;>
>>> >> >
>>> >> Cc:
>>> >> Sent: Friday, June 28, 2013 1:34 PM
>>> >> Subject: Re: 30% random performance in 0.95+
>>> >>
>>> >> Thanks JM,
>>> >>
>>> >> HBASE-8450 (r1485562) is interesting. It increases (among other things)
>>> >> the block cache percentage from 24 to 40%, which would lead to a higher
>>> >> probability of a future random read to hit an already cached block.
>>> >>
>>> >>
>>> >> -- Lars
>>> >>
>>> >>
>>> >>
>>> >> ----- Original Message -----
>>> >> From: Jean-Marc Spaggiari <[EMAIL PROTECTED] <javascript:;>>
>>> >> To: [EMAIL PROTECTED] <javascript:;>; lars hofhansl <
>>> [EMAIL PROTECTED]<javascript:;>
>>> >> >
>>> >> Cc:
>>> >> Sent: Friday, June 28, 2013 1:18 PM
>>> >> Subject: Re: 30% random performance in 0.95+
>>> >>
>>> >> I have the script done to run over a list of "svn releases", so if
>>> >> required, just give me a bunch of them or a range and I can restart.
>>> >> Just keep me posted.
>>> >>
>>> >> JM
>>> >>
>>> >> 2013/6/28 lars hofhansl <[EMAIL PROTECTED] <javascript:;>>: