Hi,

In HBase we already have PerformanceEvaluation which gives us a good
way to validate that nothing broke HBase speed in the recent updates.

I can see in the JIRAs many improvements coming, like for the lazy
seeks, the bloom filters, etc. however, there is no tests for those
improvements.

Will it not be good to ask people to add some new tests in
PerformanceEvaluation when they are introducing an improvement which
is not covered there?

We should not touch existing tests because we need to have a way to
compare the baseline between the different versions, but we can still
add some new. Like in addition to RandomSeekScanTest we can add
RandomSeekScanBloomEnabledTest and so on. And even better if we can
back port those new tests to previous version.

The same way we add a test class when we introduce a new feature,
should we add a performance test method to test it too?

JM

@JM
I agree with you.  Mainly the perf improvement changes needs some
testcases.
But sometimes the scenario on which the perf improvments happens are bit
difficult to generate and we will be able to do in a standalone case only.
 May be overall if we need to get that perf improvment result we need a
real cluster with suitable data.  That is what i have experienced.  Just
telling.

Regards
Ram

On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <[EMAIL PROTECTED]
> wrote:

> Hi,
>
> In HBase we already have PerformanceEvaluation which gives us a good
> way to validate that nothing broke HBase speed in the recent updates.
>
> I can see in the JIRAs many improvements coming, like for the lazy
> seeks, the bloom filters, etc. however, there is no tests for those
> improvements.
>
> Will it not be good to ask people to add some new tests in
> PerformanceEvaluation when they are introducing an improvement which
> is not covered there?
>
> We should not touch existing tests because we need to have a way to
> compare the baseline between the different versions, but we can still
> add some new. Like in addition to RandomSeekScanTest we can add
> RandomSeekScanBloomEnabledTest and so on. And even better if we can
> back port those new tests to previous version.
>
> The same way we add a test class when we introduce a new feature,
> should we add a performance test method to test it too?
>
> JM
>

Tangentally: I think I prefer LoadTestTool over PerformanceEvaluation, it
doesn't depend on nor is influenced by MapReduce job startup.
On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
[EMAIL PROTECTED]> wrote:

> @JM
> I agree with you.  Mainly the perf improvement changes needs some
> testcases.
> But sometimes the scenario on which the perf improvments happens are bit
> difficult to generate and we will be able to do in a standalone case only.
>  May be overall if we need to get that perf improvment result we need a
> real cluster with suitable data.  That is what i have experienced.  Just
> telling.
>
> Regards
> Ram
>
> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]
> > wrote:
>
> > Hi,
> >
> > In HBase we already have PerformanceEvaluation which gives us a good
> > way to validate that nothing broke HBase speed in the recent updates.
> >
> > I can see in the JIRAs many improvements coming, like for the lazy
> > seeks, the bloom filters, etc. however, there is no tests for those
> > improvements.
> >
> > Will it not be good to ask people to add some new tests in
> > PerformanceEvaluation when they are introducing an improvement which
> > is not covered there?
> >
> > We should not touch existing tests because we need to have a way to
> > compare the baseline between the different versions, but we can still
> > add some new. Like in addition to RandomSeekScanTest we can add
> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
> > back port those new tests to previous version.
> >
> > The same way we add a test class when we introduce a new feature,
> > should we add a performance test method to test it too?
> >
> > JM
> >
>

--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

If you run only 1 client with PerformanceEvaluation, it's not running
it over MapReduce, so you don't have this overhead. But you can still
run it if you want to have something more distributed. Might be useful
to have the 2 options. But at the end, LoadTestTool or
PerformanceEvaluation, any of the 2 is good as long as we are adding
those tests.

I just finished to run all the PerformanceEvaluation tests on a
dedicated computer with all 0.9x.x HBase versions, and I found results
interesting. That gives us a good baseline to see if new HBase
improvements are really improving performances.

JM

2013/3/8 Andrew Purtell <[EMAIL PROTECTED]>:
> Tangentally: I think I prefer LoadTestTool over PerformanceEvaluation, it
> doesn't depend on nor is influenced by MapReduce job startup.
>
>
> On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
> [EMAIL PROTECTED]> wrote:
>
>> @JM
>> I agree with you.  Mainly the perf improvement changes needs some
>> testcases.
>> But sometimes the scenario on which the perf improvments happens are bit
>> difficult to generate and we will be able to do in a standalone case only.
>>  May be overall if we need to get that perf improvment result we need a
>> real cluster with suitable data.  That is what i have experienced.  Just
>> telling.
>>
>> Regards
>> Ram
>>
>> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
>> [EMAIL PROTECTED]
>> > wrote:
>>
>> > Hi,
>> >
>> > In HBase we already have PerformanceEvaluation which gives us a good
>> > way to validate that nothing broke HBase speed in the recent updates.
>> >
>> > I can see in the JIRAs many improvements coming, like for the lazy
>> > seeks, the bloom filters, etc. however, there is no tests for those
>> > improvements.
>> >
>> > Will it not be good to ask people to add some new tests in
>> > PerformanceEvaluation when they are introducing an improvement which
>> > is not covered there?
>> >
>> > We should not touch existing tests because we need to have a way to
>> > compare the baseline between the different versions, but we can still
>> > add some new. Like in addition to RandomSeekScanTest we can add
>> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
>> > back port those new tests to previous version.
>> >
>> > The same way we add a test class when we introduce a new feature,
>> > should we add a performance test method to test it too?
>> >
>> > JM
>> >
>>
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)

> I just finished to run all the PerformanceEvaluation tests on a
dedicated computer with all 0.9x.x HBase versions, and I found results
interesting.
Can you please provide your numbers if you can. What is interesting from
your findings?

Enis

On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> If you run only 1 client with PerformanceEvaluation, it's not running
> it over MapReduce, so you don't have this overhead. But you can still
> run it if you want to have something more distributed. Might be useful
> to have the 2 options. But at the end, LoadTestTool or
> PerformanceEvaluation, any of the 2 is good as long as we are adding
> those tests.
>
> I just finished to run all the PerformanceEvaluation tests on a
> dedicated computer with all 0.9x.x HBase versions, and I found results
> interesting. That gives us a good baseline to see if new HBase
> improvements are really improving performances.
>
> JM
>
> 2013/3/8 Andrew Purtell <[EMAIL PROTECTED]>:
> > Tangentally: I think I prefer LoadTestTool over PerformanceEvaluation, it
> > doesn't depend on nor is influenced by MapReduce job startup.
> >
> >
> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
> > [EMAIL PROTECTED]> wrote:
> >
> >> @JM
> >> I agree with you.  Mainly the perf improvement changes needs some
> >> testcases.
> >> But sometimes the scenario on which the perf improvments happens are bit
> >> difficult to generate and we will be able to do in a standalone case
> only.
> >>  May be overall if we need to get that perf improvment result we need a
> >> real cluster with suitable data.  That is what i have experienced.  Just
> >> telling.
> >>
> >> Regards
> >> Ram
> >>
> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
> >> [EMAIL PROTECTED]
> >> > wrote:
> >>
> >> > Hi,
> >> >
> >> > In HBase we already have PerformanceEvaluation which gives us a good
> >> > way to validate that nothing broke HBase speed in the recent updates.
> >> >
> >> > I can see in the JIRAs many improvements coming, like for the lazy
> >> > seeks, the bloom filters, etc. however, there is no tests for those
> >> > improvements.
> >> >
> >> > Will it not be good to ask people to add some new tests in
> >> > PerformanceEvaluation when they are introducing an improvement which
> >> > is not covered there?
> >> >
> >> > We should not touch existing tests because we need to have a way to
> >> > compare the baseline between the different versions, but we can still
> >> > add some new. Like in addition to RandomSeekScanTest we can add
> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
> >> > back port those new tests to previous version.
> >> >
> >> > The same way we add a test class when we introduce a new feature,
> >> > should we add a performance test method to test it too?
> >> >
> >> > JM
> >> >
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
>

Hi Enis,

"interesting" in the positive way ;)

Results are there:
http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477

The improvment on scan are impressive. sequentialRead and randomScan went down.

In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.

I will add HFilePerformanceEvaluation soon but I'm facinf some issues
with it on previous HBase version...

JM

2013/3/12 Enis Söztutar <[EMAIL PROTECTED]>:
>> I just finished to run all the PerformanceEvaluation tests on a
> dedicated computer with all 0.9x.x HBase versions, and I found results
> interesting.
> Can you please provide your numbers if you can. What is interesting from
> your findings?
>
> Enis
>
>
>
> On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]> wrote:
>
>> If you run only 1 client with PerformanceEvaluation, it's not running
>> it over MapReduce, so you don't have this overhead. But you can still
>> run it if you want to have something more distributed. Might be useful
>> to have the 2 options. But at the end, LoadTestTool or
>> PerformanceEvaluation, any of the 2 is good as long as we are adding
>> those tests.
>>
>> I just finished to run all the PerformanceEvaluation tests on a
>> dedicated computer with all 0.9x.x HBase versions, and I found results
>> interesting. That gives us a good baseline to see if new HBase
>> improvements are really improving performances.
>>
>> JM
>>
>> 2013/3/8 Andrew Purtell <[EMAIL PROTECTED]>:
>> > Tangentally: I think I prefer LoadTestTool over PerformanceEvaluation, it
>> > doesn't depend on nor is influenced by MapReduce job startup.
>> >
>> >
>> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
>> > [EMAIL PROTECTED]> wrote:
>> >
>> >> @JM
>> >> I agree with you.  Mainly the perf improvement changes needs some
>> >> testcases.
>> >> But sometimes the scenario on which the perf improvments happens are bit
>> >> difficult to generate and we will be able to do in a standalone case
>> only.
>> >>  May be overall if we need to get that perf improvment result we need a
>> >> real cluster with suitable data.  That is what i have experienced.  Just
>> >> telling.
>> >>
>> >> Regards
>> >> Ram
>> >>
>> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
>> >> [EMAIL PROTECTED]
>> >> > wrote:
>> >>
>> >> > Hi,
>> >> >
>> >> > In HBase we already have PerformanceEvaluation which gives us a good
>> >> > way to validate that nothing broke HBase speed in the recent updates.
>> >> >
>> >> > I can see in the JIRAs many improvements coming, like for the lazy
>> >> > seeks, the bloom filters, etc. however, there is no tests for those
>> >> > improvements.
>> >> >
>> >> > Will it not be good to ask people to add some new tests in
>> >> > PerformanceEvaluation when they are introducing an improvement which
>> >> > is not covered there?
>> >> >
>> >> > We should not touch existing tests because we need to have a way to
>> >> > compare the baseline between the different versions, but we can still
>> >> > add some new. Like in addition to RandomSeekScanTest we can add
>> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
>> >> > back port those new tests to previous version.
>> >> >
>> >> > The same way we add a test class when we introduce a new feature,
>> >> > should we add a performance test method to test it too?
>> >> >
>> >> > JM
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > Best regards,
>> >
>> >    - Andy
>> >
>> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> > (via Tom White)
>>

Thanks for sharing this result, Jean-Marc.

For randomRead, looks like the degradation happened between 0.94.0 and
0.94.1

Cheers

On Sat, Mar 16, 2013 at 7:03 PM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> Hi Enis,
>
> "interesting" in the positive way ;)
>
> Results are there:
>
> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>
> The improvment on scan are impressive. sequentialRead and randomScan went
> down.
>
> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>
> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
> with it on previous HBase version...
>
> JM
>
> 2013/3/12 Enis Söztutar <[EMAIL PROTECTED]>:
> >> I just finished to run all the PerformanceEvaluation tests on a
> > dedicated computer with all 0.9x.x HBase versions, and I found results
> > interesting.
> > Can you please provide your numbers if you can. What is interesting from
> > your findings?
> >
> > Enis
> >
> >
> >
> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
> > [EMAIL PROTECTED]> wrote:
> >
> >> If you run only 1 client with PerformanceEvaluation, it's not running
> >> it over MapReduce, so you don't have this overhead. But you can still
> >> run it if you want to have something more distributed. Might be useful
> >> to have the 2 options. But at the end, LoadTestTool or
> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
> >> those tests.
> >>
> >> I just finished to run all the PerformanceEvaluation tests on a
> >> dedicated computer with all 0.9x.x HBase versions, and I found results
> >> interesting. That gives us a good baseline to see if new HBase
> >> improvements are really improving performances.
> >>
> >> JM
> >>
> >> 2013/3/8 Andrew Purtell <[EMAIL PROTECTED]>:
> >> > Tangentally: I think I prefer LoadTestTool over
> PerformanceEvaluation, it
> >> > doesn't depend on nor is influenced by MapReduce job startup.
> >> >
> >> >
> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
> >> > [EMAIL PROTECTED]> wrote:
> >> >
> >> >> @JM
> >> >> I agree with you.  Mainly the perf improvement changes needs some
> >> >> testcases.
> >> >> But sometimes the scenario on which the perf improvments happens are
> bit
> >> >> difficult to generate and we will be able to do in a standalone case
> >> only.
> >> >>  May be overall if we need to get that perf improvment result we
> need a
> >> >> real cluster with suitable data.  That is what i have experienced.
>  Just
> >> >> telling.
> >> >>
> >> >> Regards
> >> >> Ram
> >> >>
> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
> >> >> [EMAIL PROTECTED]
> >> >> > wrote:
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> > In HBase we already have PerformanceEvaluation which gives us a
> good
> >> >> > way to validate that nothing broke HBase speed in the recent
> updates.
> >> >> >
> >> >> > I can see in the JIRAs many improvements coming, like for the lazy
> >> >> > seeks, the bloom filters, etc. however, there is no tests for those
> >> >> > improvements.
> >> >> >
> >> >> > Will it not be good to ask people to add some new tests in
> >> >> > PerformanceEvaluation when they are introducing an improvement
> which
> >> >> > is not covered there?
> >> >> >
> >> >> > We should not touch existing tests because we need to have a way to
> >> >> > compare the baseline between the different versions, but we can
> still
> >> >> > add some new. Like in addition to RandomSeekScanTest we can add
> >> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
> >> >> > back port those new tests to previous version.
> >> >> >
> >> >> > The same way we add a test class when we introduce a new feature,
> >> >> > should we add a performance test method to test it too?
> >> >> >
> >> >> > JM
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Best regards,
> >> >
> >> >    - Andy
> >> >
> >> > Problems worthy of attack prove their worth by hitting back. - Piet

Cool. The 0.94.3 scanning improvements seems almost unbelievable (especially since many of my improvements to reduce the internal friction went into 0.94.4).
I would like to track down the random read regression.

Can you send the commands you ran? Are you running this as M/R job or standalone client?

Thanks for doing this J-M.
-- Lars

________________________________
 From: Jean-Marc Spaggiari <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Saturday, March 16, 2013 7:03 PM
Subject: Re: Performances Tests
 
Hi Enis,

"interesting" in the positive way ;)

Results are there:
http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477

The improvment on scan are impressive. sequentialRead and randomScan went down.

In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.

I will add HFilePerformanceEvaluation soon but I'm facinf some issues
with it on previous HBase version...

JM

2013/3/12 Enis Söztutar <[EMAIL PROTECTED]>:
>> I just finished to run all the PerformanceEvaluation tests on a
> dedicated computer with all 0.9x.x HBase versions, and I found results
> interesting.
> Can you please provide your numbers if you can. What is interesting from
> your findings?
>
> Enis
>
>
>
> On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]> wrote:
>
>> If you run only 1 client with PerformanceEvaluation, it's not running
>> it over MapReduce, so you don't have this overhead. But you can still
>> run it if you want to have something more distributed. Might be useful
>> to have the 2 options. But at the end, LoadTestTool or
>> PerformanceEvaluation, any of the 2 is good as long as we are adding
>> those tests.
>>
>> I just finished to run all the PerformanceEvaluation tests on a
>> dedicated computer with all 0.9x.x HBase versions, and I found results
>> interesting. That gives us a good baseline to see if new HBase
>> improvements are really improving performances.
>>
>> JM
>>
>> 2013/3/8 Andrew Purtell <[EMAIL PROTECTED]>:
>> > Tangentally: I think I prefer LoadTestTool over PerformanceEvaluation, it
>> > doesn't depend on nor is influenced by MapReduce job startup.
>> >
>> >
>> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
>> > [EMAIL PROTECTED]> wrote:
>> >
>> >> @JM
>> >> I agree with you.  Mainly the perf improvement changes needs some
>> >> testcases.
>> >> But sometimes the scenario on which the perf improvments happens are bit
>> >> difficult to generate and we will be able to do in a standalone case
>> only.
>> >>  May be overall if we need to get that perf improvment result we need a
>> >> real cluster with suitable data.  That is what i have experienced.  Just
>> >> telling.
>> >>
>> >> Regards
>> >> Ram
>> >>
>> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
>> >> [EMAIL PROTECTED]
>> >> > wrote:
>> >>
>> >> > Hi,
>> >> >
>> >> > In HBase we already have PerformanceEvaluation which gives us a good
>> >> > way to validate that nothing broke HBase speed in the recent updates.
>> >> >
>> >> > I can see in the JIRAs many improvements coming, like for the lazy
>> >> > seeks, the bloom filters, etc. however, there is no tests for those
>> >> > improvements.
>> >> >
>> >> > Will it not be good to ask people to add some new tests in
>> >> > PerformanceEvaluation when they are introducing an improvement which
>> >> > is not covered there?
>> >> >
>> >> > We should not touch existing tests because we need to have a way to
>> >> > compare the baseline between the different versions, but we can still
>> >> > add some new. Like in addition to RandomSeekScanTest we can add
>> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
>> >> > back port those new tests to previous version.
>> >> >
>> >> > The same way we add a test class when we introduce a new feature,
>> >> > should we add a performance test method to test it too?
>> >> >
>> >> > JM
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > Best regards,

I'm not sure I trust the results of PerformanceEvauation.

LoadTestTool and YCSB have their own issues but seem to produce more
consistent results. I've been thinking on adding scanning and filtering
tests to LoadTestTool.

On Sunday, March 17, 2013, lars hofhansl wrote:

> Cool. The 0.94.3 scanning improvements seems almost unbelievable
> (especially since many of my improvements to reduce the internal friction
> went into 0.94.4).
> I would like to track down the random read regression.
>
> Can you send the commands you ran? Are you running this as M/R job or
> standalone client?
>
> Thanks for doing this J-M.
>
>
> -- Lars
>
>
>
> ________________________________
>  From: Jean-Marc Spaggiari <[EMAIL PROTECTED] <javascript:;>>
> To: [EMAIL PROTECTED] <javascript:;>
> Sent: Saturday, March 16, 2013 7:03 PM
> Subject: Re: Performances Tests
>
> Hi Enis,
>
> "interesting" in the positive way ;)
>
> Results are there:
>
> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>
> The improvment on scan are impressive. sequentialRead and randomScan went
> down.
>
> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>
> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
> with it on previous HBase version...
>
> JM
>
> 2013/3/12 Enis Söztutar <[EMAIL PROTECTED]>:
> >> I just finished to run all the PerformanceEvaluation tests on a
> > dedicated computer with all 0.9x.x HBase versions, and I found results
> > interesting.
> > Can you please provide your numbers if you can. What is interesting from
> > your findings?
> >
> > Enis
> >
> >
> >
> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
> > [EMAIL PROTECTED]> wrote:
> >
> >> If you run only 1 client with PerformanceEvaluation, it's not running
> >> it over MapReduce, so you don't have this overhead. But you can still
> >> run it if you want to have something more distributed. Might be useful
> >> to have the 2 options. But at the end, LoadTestTool or
> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
> >> those tests.
> >>
> >> I just finished to run all the PerformanceEvaluation tests on a
> >> dedicated computer with all 0.9x.x HBase versions, and I found results
> >> interesting. That gives us a good baseline to see if new HBase
> >> improvements are really improving performances.
> >>
> >> JM
> >>
> >> 2013/3/8 Andrew Purtell <[EMAIL PROTECTED]>:
> >> > Tangentally: I think I prefer LoadTestTool over
> PerformanceEvaluation, it
> >> > doesn't depend on nor is influenced by MapReduce job startup.
> >> >
> >> >
> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
> >> > [EMAIL PROTECTED]> wrote:
> >> >
> >> >> @JM
> >> >> I agree with you.  Mainly the perf improvement changes needs some
> >> >> testcases.
> >> >> But sometimes the scenario on which the perf improvments happens are
> bit
> >> >> difficult to generate and we will be able to do in a standalone case
> >> only.
> >> >>  May be overall if we need to get that perf improvment result we
> need a
> >> >> real cluster with suitable data.  That is what i have experienced.
> Just
> >> >> telling.
> >> >>
> >> >> Regards
> >> >> Ram
> >> >>
> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
> >> >> [EMAIL PROTECTED]
> >> >> > wrote:
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> > In HBase we already have PerformanceEvaluation which gives us a
> good
> >> >> > way to validate that nothing broke HBase speed in the recent
> updates.
> >> >> >
> >> >> > I can see in the JIRAs many improvements coming, like for the lazy
> >> >> > seeks, the bloom filters, etc. however, there is no tests for those
> >> >> > improvements.
> >> >> >
> >> >> > Will it not be good to ask people to add some new tests in
> >> >> > PerformanceEvaluation when they are introducing an improvement
> which
> >> >> > is not covered there?
> >> >> >
> >> >> > We should not touch existing tests because we need to have a way to

Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

@Lars: here is an example of what I use
for i in {1..10}; do echo; echo -n $i ; rm -rf /tmp/*;
bin/start-hbase.sh; sleep 60; bin/hbase
org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 1;
bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=100
filterScan 1; bin/stop-hbase.sh; done &>> output.txt

Giving only 1 as the readers allow to not launch the test as a MR. I
have not try the same tests with a bigger number, but I can.

@Andy: LoadTestTool is also on my list, but for YCSB it's a but harder
since I will need more than one dedicated computer. But I will look at
it too. I might be able to remove 1 node from my cluster and dedicate
it to the tests when required...
I will be pretty buzy next week, but before next week-end I will try
to prepare the LoadTestTool scripts to run something similar. With
multiple scenarios, like with and whithout bloom, etc.

If yu have any recommandation/request, feel free! I will come back
soon with some additionnal numbers.

JM

2013/3/17 Andrew Purtell <[EMAIL PROTECTED]>:
> I'm not sure I trust the results of PerformanceEvauation.
>
> LoadTestTool and YCSB have their own issues but seem to produce more
> consistent results. I've been thinking on adding scanning and filtering
> tests to LoadTestTool.
>
> On Sunday, March 17, 2013, lars hofhansl wrote:
>
>> Cool. The 0.94.3 scanning improvements seems almost unbelievable
>> (especially since many of my improvements to reduce the internal friction
>> went into 0.94.4).
>> I would like to track down the random read regression.
>>
>> Can you send the commands you ran? Are you running this as M/R job or
>> standalone client?
>>
>> Thanks for doing this J-M.
>>
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>>  From: Jean-Marc Spaggiari <[EMAIL PROTECTED] <javascript:;>>
>> To: [EMAIL PROTECTED] <javascript:;>
>> Sent: Saturday, March 16, 2013 7:03 PM
>> Subject: Re: Performances Tests
>>
>> Hi Enis,
>>
>> "interesting" in the positive way ;)
>>
>> Results are there:
>>
>> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>>
>> The improvment on scan are impressive. sequentialRead and randomScan went
>> down.
>>
>> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>>
>> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
>> with it on previous HBase version...
>>
>> JM
>>
>> 2013/3/12 Enis Söztutar <[EMAIL PROTECTED]>:
>> >> I just finished to run all the PerformanceEvaluation tests on a
>> > dedicated computer with all 0.9x.x HBase versions, and I found results
>> > interesting.
>> > Can you please provide your numbers if you can. What is interesting from
>> > your findings?
>> >
>> > Enis
>> >
>> >
>> >
>> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
>> > [EMAIL PROTECTED]> wrote:
>> >
>> >> If you run only 1 client with PerformanceEvaluation, it's not running
>> >> it over MapReduce, so you don't have this overhead. But you can still
>> >> run it if you want to have something more distributed. Might be useful
>> >> to have the 2 options. But at the end, LoadTestTool or
>> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
>> >> those tests.
>> >>
>> >> I just finished to run all the PerformanceEvaluation tests on a
>> >> dedicated computer with all 0.9x.x HBase versions, and I found results
>> >> interesting. That gives us a good baseline to see if new HBase
>> >> improvements are really improving performances.
>> >>
>> >> JM
>> >>
>> >> 2013/3/8 Andrew Purtell <[EMAIL PROTECTED]>:
>> >> > Tangentally: I think I prefer LoadTestTool over
>> PerformanceEvaluation, it
>> >> > doesn't depend on nor is influenced by MapReduce job startup.
>> >> >
>> >> >
>> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
>> >> > [EMAIL PROTECTED]> wrote:
>> >> >
>> >> >> @JM
>> >> >> I agree with you.  Mainly the perf improvement changes needs some
>> >> >> testcases.
>> >> >> But sometimes the scenario on which the perf improvments happens are

Anything at all would be great JM. Looking forward to it if you have the
time and interest. Thanks!

On Sunday, March 17, 2013, Jean-Marc Spaggiari wrote:

> @Lars: here is an example of what I use
> for i in {1..10}; do echo; echo -n $i ; rm -rf /tmp/*;
> bin/start-hbase.sh; sleep 60; bin/hbase
> org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 1;
> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=100
> filterScan 1; bin/stop-hbase.sh; done &>> output.txt
>
> Giving only 1 as the readers allow to not launch the test as a MR. I
> have not try the same tests with a bigger number, but I can.
>
> @Andy: LoadTestTool is also on my list, but for YCSB it's a but harder
> since I will need more than one dedicated computer. But I will look at
> it too. I might be able to remove 1 node from my cluster and dedicate
> it to the tests when required...
>
>
> I will be pretty buzy next week, but before next week-end I will try
> to prepare the LoadTestTool scripts to run something similar. With
> multiple scenarios, like with and whithout bloom, etc.
>
> If yu have any recommandation/request, feel free! I will come back
> soon with some additionnal numbers.
>
> JM
>
> 2013/3/17 Andrew Purtell <[EMAIL PROTECTED] <javascript:;>>:
> > I'm not sure I trust the results of PerformanceEvauation.
> >
> > LoadTestTool and YCSB have their own issues but seem to produce more
> > consistent results. I've been thinking on adding scanning and filtering
> > tests to LoadTestTool.
> >
> > On Sunday, March 17, 2013, lars hofhansl wrote:
> >
> >> Cool. The 0.94.3 scanning improvements seems almost unbelievable
> >> (especially since many of my improvements to reduce the internal
> friction
> >> went into 0.94.4).
> >> I would like to track down the random read regression.
> >>
> >> Can you send the commands you ran? Are you running this as M/R job or
> >> standalone client?
> >>
> >> Thanks for doing this J-M.
> >>
> >>
> >> -- Lars
> >>
> >>
> >>
> >> ________________________________
> >>  From: Jean-Marc Spaggiari <[EMAIL PROTECTED] <javascript:;><javascript:;>>
> >> To: [EMAIL PROTECTED] <javascript:;> <javascript:;>
> >> Sent: Saturday, March 16, 2013 7:03 PM
> >> Subject: Re: Performances Tests
> >>
> >> Hi Enis,
> >>
> >> "interesting" in the positive way ;)
> >>
> >> Results are there:
> >>
> >>
> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
> >>
> >> The improvment on scan are impressive. sequentialRead and randomScan
> went
> >> down.
> >>
> >> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
> >>
> >> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
> >> with it on previous HBase version...
> >>
> >> JM
> >>
> >> 2013/3/12 Enis Söztutar <[EMAIL PROTECTED]>:
> >> >> I just finished to run all the PerformanceEvaluation tests on a
> >> > dedicated computer with all 0.9x.x HBase versions, and I found results
> >> > interesting.
> >> > Can you please provide your numbers if you can. What is interesting
> from
> >> > your findings?
> >> >
> >> > Enis
> >> >
> >> >
> >> >
> >> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
> >> > [EMAIL PROTECTED]> wrote:
> >> >
> >> >> If you run only 1 client with PerformanceEvaluation, it's not running
> >> >> it over MapReduce, so you don't have this overhead. But you can still
> >> >> run it if you want to have something more distributed. Might be
> useful
> >> >> to have the 2 options. But at the end, LoadTestTool or
> >> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
> >> >> those tests.
> >> >>
> >> >> I just finished to run all the PerformanceEvaluation tests on a
> >> >> dedicated computer with all 0.9x.x HBase versions, and I found
> results
> >> >> interesting. That gives us a good baseline to see if new HBase
> >> >> improvements are really improving performances.
> >> >>
> >> >> JM
> >> >>
> >> >> 2013/3/8 Andrew Purtell <[EMAIL PROTECTED]>:
>
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Thanks J-M, so that should be easy to reproduce. I'll try.

-- Lars

________________________________
 From: Jean-Marc Spaggiari <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Sunday, March 17, 2013 9:47 AM
Subject: Re: Performances Tests
 
@Lars: here is an example of what I use
for i in {1..10}; do echo; echo -n $i ; rm -rf /tmp/*;
bin/start-hbase.sh; sleep 60; bin/hbase
org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 1;
bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=100
filterScan 1; bin/stop-hbase.sh; done &>> output.txt

Giving only 1 as the readers allow to not launch the test as a MR. I
have not try the same tests with a bigger number, but I can.

@Andy: LoadTestTool is also on my list, but for YCSB it's a but harder
since I will need more than one dedicated computer. But I will look at
it too. I might be able to remove 1 node from my cluster and dedicate
it to the tests when required...
I will be pretty buzy next week, but before next week-end I will try
to prepare the LoadTestTool scripts to run something similar. With
multiple scenarios, like with and whithout bloom, etc.

If yu have any recommandation/request, feel free! I will come back
soon with some additionnal numbers.

JM

2013/3/17 Andrew Purtell <[EMAIL PROTECTED]>:
> I'm not sure I trust the results of PerformanceEvauation.
>
> LoadTestTool and YCSB have their own issues but seem to produce more
> consistent results. I've been thinking on adding scanning and filtering
> tests to LoadTestTool.
>
> On Sunday, March 17, 2013, lars hofhansl wrote:
>
>> Cool. The 0.94.3 scanning improvements seems almost unbelievable
>> (especially since many of my improvements to reduce the internal friction
>> went into 0.94.4).
>> I would like to track down the random read regression.
>>
>> Can you send the commands you ran? Are you running this as M/R job or
>> standalone client?
>>
>> Thanks for doing this J-M.
>>
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>>  From: Jean-Marc Spaggiari <[EMAIL PROTECTED] <javascript:;>>
>> To: [EMAIL PROTECTED] <javascript:;>
>> Sent: Saturday, March 16, 2013 7:03 PM
>> Subject: Re: Performances Tests
>>
>> Hi Enis,
>>
>> "interesting" in the positive way ;)
>>
>> Results are there:
>>
>> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>>
>> The improvment on scan are impressive. sequentialRead and randomScan went
>> down.
>>
>> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>>
>> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
>> with it on previous HBase version...
>>
>> JM
>>
>> 2013/3/12 Enis Söztutar <[EMAIL PROTECTED]>:
>> >> I just finished to run all the PerformanceEvaluation tests on a
>> > dedicated computer with all 0.9x.x HBase versions, and I found results
>> > interesting.
>> > Can you please provide your numbers if you can. What is interesting from
>> > your findings?
>> >
>> > Enis
>> >
>> >
>> >
>> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
>> > [EMAIL PROTECTED]> wrote:
>> >
>> >> If you run only 1 client with PerformanceEvaluation, it's not running
>> >> it over MapReduce, so you don't have this overhead. But you can still
>> >> run it if you want to have something more distributed. Might be useful
>> >> to have the 2 options. But at the end, LoadTestTool or
>> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
>> >> those tests.
>> >>
>> >> I just finished to run all the PerformanceEvaluation tests on a
>> >> dedicated computer with all 0.9x.x HBase versions, and I found results
>> >> interesting. That gives us a good baseline to see if new HBase
>> >> improvements are really improving performances.
>> >>
>> >> JM
>> >>
>> >> 2013/3/8 Andrew Purtell <[EMAIL PROTECTED]>:
>> >> > Tangentally: I think I prefer LoadTestTool over
>> PerformanceEvaluation, it
>> >> > doesn't depend on nor is influenced by MapReduce job startup.
>> >> >

I did my own tests (single threaded hacked together test code - not PE or LT) and I do not measure scan perf improvements between 0.94.2 and 0.94.3.
But I do see improvements between 0.94.3 and 0.94.4.
So I am starting to either:
1. Not trust my test code
2. Not trust PE
3. Not trust your methodology

Or maybe PE exercises some code that I don't.
;)

-- Lars

________________________________
 From: Jean-Marc Spaggiari <[EMAIL PROTECTED]>
To: [EMAIL PROTECTED]
Sent: Sunday, March 17, 2013 9:47 AM
Subject: Re: Performances Tests
 
@Lars: here is an example of what I use
for i in {1..10}; do echo; echo -n $i ; rm -rf /tmp/*;
bin/start-hbase.sh; sleep 60; bin/hbase
org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 1;
bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=100
filterScan 1; bin/stop-hbase.sh; done &>> output.txt

Giving only 1 as the readers allow to not launch the test as a MR. I
have not try the same tests with a bigger number, but I can.

@Andy: LoadTestTool is also on my list, but for YCSB it's a but harder
since I will need more than one dedicated computer. But I will look at
it too. I might be able to remove 1 node from my cluster and dedicate
it to the tests when required...
I will be pretty buzy next week, but before next week-end I will try
to prepare the LoadTestTool scripts to run something similar. With
multiple scenarios, like with and whithout bloom, etc.

If yu have any recommandation/request, feel free! I will come back
soon with some additionnal numbers.

JM

2013/3/17 Andrew Purtell <[EMAIL PROTECTED]>:
> I'm not sure I trust the results of PerformanceEvauation.
>
> LoadTestTool and YCSB have their own issues but seem to produce more
> consistent results. I've been thinking on adding scanning and filtering
> tests to LoadTestTool.
>
> On Sunday, March 17, 2013, lars hofhansl wrote:
>
>> Cool. The 0.94.3 scanning improvements seems almost unbelievable
>> (especially since many of my improvements to reduce the internal friction
>> went into 0.94.4).
>> I would like to track down the random read regression.
>>
>> Can you send the commands you ran? Are you running this as M/R job or
>> standalone client?
>>
>> Thanks for doing this J-M.
>>
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>>  From: Jean-Marc Spaggiari <[EMAIL PROTECTED] <javascript:;>>
>> To: [EMAIL PROTECTED] <javascript:;>
>> Sent: Saturday, March 16, 2013 7:03 PM
>> Subject: Re: Performances Tests
>>
>> Hi Enis,
>>
>> "interesting" in the positive way ;)
>>
>> Results are there:
>>
>> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>>
>> The improvment on scan are impressive. sequentialRead and randomScan went
>> down.
>>
>> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>>
>> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
>> with it on previous HBase version...
>>
>> JM
>>
>> 2013/3/12 Enis Söztutar <[EMAIL PROTECTED]>:
>> >> I just finished to run all the PerformanceEvaluation tests on a
>> > dedicated computer with all 0.9x.x HBase versions, and I found results
>> > interesting.
>> > Can you please provide your numbers if you can. What is interesting from
>> > your findings?
>> >
>> > Enis
>> >
>> >
>> >
>> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
>> > [EMAIL PROTECTED]> wrote:
>> >
>> >> If you run only 1 client with PerformanceEvaluation, it's not running
>> >> it over MapReduce, so you don't have this overhead. But you can still
>> >> run it if you want to have something more distributed. Might be useful
>> >> to have the 2 options. But at the end, LoadTestTool or
>> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
>> >> those tests.
>> >>
>> >> I just finished to run all the PerformanceEvaluation tests on a
>> >> dedicated computer with all 0.9x.x HBase versions, and I found results
>> >> interesting. That gives us a good baseline to see if new HBase

What is the y axis's unit?  seconds or operations per second etc?  (nit:
would be nice to have on the axis.. )

Based on the context, I believe it is ops/s.

Jon.

On Sat, Mar 16, 2013 at 7:03 PM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> Hi Enis,
>
> "interesting" in the positive way ;)
>
> Results are there:
>
> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>
> The improvment on scan are impressive. sequentialRead and randomScan went
> down.
>
> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>
> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
> with it on previous HBase version...
>
> JM
>
> 2013/3/12 Enis Söztutar <[EMAIL PROTECTED]>:
> >> I just finished to run all the PerformanceEvaluation tests on a
> > dedicated computer with all 0.9x.x HBase versions, and I found results
> > interesting.
> > Can you please provide your numbers if you can. What is interesting from
> > your findings?
> >
> > Enis
> >
> >
> >
> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
> > [EMAIL PROTECTED]> wrote:
> >
> >> If you run only 1 client with PerformanceEvaluation, it's not running
> >> it over MapReduce, so you don't have this overhead. But you can still
> >> run it if you want to have something more distributed. Might be useful
> >> to have the 2 options. But at the end, LoadTestTool or
> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
> >> those tests.
> >>
> >> I just finished to run all the PerformanceEvaluation tests on a
> >> dedicated computer with all 0.9x.x HBase versions, and I found results
> >> interesting. That gives us a good baseline to see if new HBase
> >> improvements are really improving performances.
> >>
> >> JM
> >>
> >> 2013/3/8 Andrew Purtell <[EMAIL PROTECTED]>:
> >> > Tangentally: I think I prefer LoadTestTool over
> PerformanceEvaluation, it
> >> > doesn't depend on nor is influenced by MapReduce job startup.
> >> >
> >> >
> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
> >> > [EMAIL PROTECTED]> wrote:
> >> >
> >> >> @JM
> >> >> I agree with you.  Mainly the perf improvement changes needs some
> >> >> testcases.
> >> >> But sometimes the scenario on which the perf improvments happens are
> bit
> >> >> difficult to generate and we will be able to do in a standalone case
> >> only.
> >> >>  May be overall if we need to get that perf improvment result we
> need a
> >> >> real cluster with suitable data.  That is what i have experienced.
>  Just
> >> >> telling.
> >> >>
> >> >> Regards
> >> >> Ram
> >> >>
> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
> >> >> [EMAIL PROTECTED]
> >> >> > wrote:
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> > In HBase we already have PerformanceEvaluation which gives us a
> good
> >> >> > way to validate that nothing broke HBase speed in the recent
> updates.
> >> >> >
> >> >> > I can see in the JIRAs many improvements coming, like for the lazy
> >> >> > seeks, the bloom filters, etc. however, there is no tests for those
> >> >> > improvements.
> >> >> >
> >> >> > Will it not be good to ask people to add some new tests in
> >> >> > PerformanceEvaluation when they are introducing an improvement
> which
> >> >> > is not covered there?
> >> >> >
> >> >> > We should not touch existing tests because we need to have a way to
> >> >> > compare the baseline between the different versions, but we can
> still
> >> >> > add some new. Like in addition to RandomSeekScanTest we can add
> >> >> > RandomSeekScanBloomEnabledTest and so on. And even better if we can
> >> >> > back port those new tests to previous version.
> >> >> >
> >> >> > The same way we add a test class when we introduce a new feature,
> >> >> > should we add a performance test method to test it too?
> >> >> >
> >> >> > JM
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Best regards,
> >> >
> >> >    - Andy
> >> >
> >> > Problems worthy of attack prove their worth by hitting back. - Piet

// Jonathan Hsieh (shay)
// Software Engineer, Cloudera
// [EMAIL PROTECTED]

Hi Lars,

Can you share the code you are using so I can compate with PE? Also, I
will re-run all for my scanRange100 tests today and update the
spreadsheet again to make sure it's correct. Also also re-download all
the HBase versions to make sure they are all clean. I'm not doing any
configuration with them. Simply reducing the logs and tmp pointing to
memory file system.

I will keep you posted when it's done.

Hi Jonathan,

It's usually rows per seconds, but with a factor 10. Sometime I had to
divide by 100000, sometime to multiply to get numbers bigger... I will
take a look at th formulas and add the legend for each of the charts.

JM

2013/3/19 Jonathan Hsieh <[EMAIL PROTECTED]>:
> What is the y axis's unit?  seconds or operations per second etc?  (nit:
> would be nice to have on the axis.. )
>
> Based on the context, I believe it is ops/s.
>
> Jon.
>
> On Sat, Mar 16, 2013 at 7:03 PM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]> wrote:
>
>> Hi Enis,
>>
>> "interesting" in the positive way ;)
>>
>> Results are there:
>>
>> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>>
>> The improvment on scan are impressive. sequentialRead and randomScan went
>> down.
>>
>> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>>
>> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
>> with it on previous HBase version...
>>
>> JM
>>
>> 2013/3/12 Enis Söztutar <[EMAIL PROTECTED]>:
>> >> I just finished to run all the PerformanceEvaluation tests on a
>> > dedicated computer with all 0.9x.x HBase versions, and I found results
>> > interesting.
>> > Can you please provide your numbers if you can. What is interesting from
>> > your findings?
>> >
>> > Enis
>> >
>> >
>> >
>> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
>> > [EMAIL PROTECTED]> wrote:
>> >
>> >> If you run only 1 client with PerformanceEvaluation, it's not running
>> >> it over MapReduce, so you don't have this overhead. But you can still
>> >> run it if you want to have something more distributed. Might be useful
>> >> to have the 2 options. But at the end, LoadTestTool or
>> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
>> >> those tests.
>> >>
>> >> I just finished to run all the PerformanceEvaluation tests on a
>> >> dedicated computer with all 0.9x.x HBase versions, and I found results
>> >> interesting. That gives us a good baseline to see if new HBase
>> >> improvements are really improving performances.
>> >>
>> >> JM
>> >>
>> >> 2013/3/8 Andrew Purtell <[EMAIL PROTECTED]>:
>> >> > Tangentally: I think I prefer LoadTestTool over
>> PerformanceEvaluation, it
>> >> > doesn't depend on nor is influenced by MapReduce job startup.
>> >> >
>> >> >
>> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
>> >> > [EMAIL PROTECTED]> wrote:
>> >> >
>> >> >> @JM
>> >> >> I agree with you.  Mainly the perf improvement changes needs some
>> >> >> testcases.
>> >> >> But sometimes the scenario on which the perf improvments happens are
>> bit
>> >> >> difficult to generate and we will be able to do in a standalone case
>> >> only.
>> >> >>  May be overall if we need to get that perf improvment result we
>> need a
>> >> >> real cluster with suitable data.  That is what i have experienced.
>>  Just
>> >> >> telling.
>> >> >>
>> >> >> Regards
>> >> >> Ram
>> >> >>
>> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
>> >> >> [EMAIL PROTECTED]
>> >> >> > wrote:
>> >> >>
>> >> >> > Hi,
>> >> >> >
>> >> >> > In HBase we already have PerformanceEvaluation which gives us a
>> good
>> >> >> > way to validate that nothing broke HBase speed in the recent
>> updates.
>> >> >> >
>> >> >> > I can see in the JIRAs many improvements coming, like for the lazy
>> >> >> > seeks, the bloom filters, etc. however, there is no tests for those
>> >> >> > improvements.
>> >> >> >
>> >> >> > Will it not be good to ask people to add some new tests in

I am curious to know how trunk stands in the performance comparison.
There have been many optimizations going into trunk. Getting hold of
overall improvement would be nice.

Cheers

On Wed, Mar 20, 2013 at 5:02 AM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> Hi Lars,
>
> Can you share the code you are using so I can compate with PE? Also, I
> will re-run all for my scanRange100 tests today and update the
> spreadsheet again to make sure it's correct. Also also re-download all
> the HBase versions to make sure they are all clean. I'm not doing any
> configuration with them. Simply reducing the logs and tmp pointing to
> memory file system.
>
> I will keep you posted when it's done.
>
> Hi Jonathan,
>
> It's usually rows per seconds, but with a factor 10. Sometime I had to
> divide by 100000, sometime to multiply to get numbers bigger... I will
> take a look at th formulas and add the legend for each of the charts.
>
> JM
>
> 2013/3/19 Jonathan Hsieh <[EMAIL PROTECTED]>:
> > What is the y axis's unit?  seconds or operations per second etc?  (nit:
> > would be nice to have on the axis.. )
> >
> > Based on the context, I believe it is ops/s.
> >
> > Jon.
> >
> > On Sat, Mar 16, 2013 at 7:03 PM, Jean-Marc Spaggiari <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Hi Enis,
> >>
> >> "interesting" in the positive way ;)
> >>
> >> Results are there:
> >>
> >>
> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
> >>
> >> The improvment on scan are impressive. sequentialRead and randomScan
> went
> >> down.
> >>
> >> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
> >>
> >> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
> >> with it on previous HBase version...
> >>
> >> JM
> >>
> >> 2013/3/12 Enis Söztutar <[EMAIL PROTECTED]>:
> >> >> I just finished to run all the PerformanceEvaluation tests on a
> >> > dedicated computer with all 0.9x.x HBase versions, and I found results
> >> > interesting.
> >> > Can you please provide your numbers if you can. What is interesting
> from
> >> > your findings?
> >> >
> >> > Enis
> >> >
> >> >
> >> >
> >> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
> >> > [EMAIL PROTECTED]> wrote:
> >> >
> >> >> If you run only 1 client with PerformanceEvaluation, it's not running
> >> >> it over MapReduce, so you don't have this overhead. But you can still
> >> >> run it if you want to have something more distributed. Might be
> useful
> >> >> to have the 2 options. But at the end, LoadTestTool or
> >> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
> >> >> those tests.
> >> >>
> >> >> I just finished to run all the PerformanceEvaluation tests on a
> >> >> dedicated computer with all 0.9x.x HBase versions, and I found
> results
> >> >> interesting. That gives us a good baseline to see if new HBase
> >> >> improvements are really improving performances.
> >> >>
> >> >> JM
> >> >>
> >> >> 2013/3/8 Andrew Purtell <[EMAIL PROTECTED]>:
> >> >> > Tangentally: I think I prefer LoadTestTool over
> >> PerformanceEvaluation, it
> >> >> > doesn't depend on nor is influenced by MapReduce job startup.
> >> >> >
> >> >> >
> >> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
> >> >> > [EMAIL PROTECTED]> wrote:
> >> >> >
> >> >> >> @JM
> >> >> >> I agree with you.  Mainly the perf improvement changes needs some
> >> >> >> testcases.
> >> >> >> But sometimes the scenario on which the perf improvments happens
> are
> >> bit
> >> >> >> difficult to generate and we will be able to do in a standalone
> case
> >> >> only.
> >> >> >>  May be overall if we need to get that perf improvment result we
> >> need a
> >> >> >> real cluster with suitable data.  That is what i have experienced.
> >>  Just
> >> >> >> telling.
> >> >> >>
> >> >> >> Regards
> >> >> >> Ram
> >> >> >>
> >> >> >> On Fri, Mar 8, 2013 at 7:28 PM, Jean-Marc Spaggiari <
> >> >> >> [EMAIL PROTECTED]
> >> >

Hi Ted,

I will try to build the trunk version and add it in the list....

So I have to:
- Re-download all HBase version to make sure PE is using the right one;
- Re-run the scanRange100 tests to validate the values already found;
- Add the trunk;
- Add LoadTestTool

Than will keep my free time buzy ;)

I keep you all posted as soon as it's done.

JM

2013/3/20 Ted Yu <[EMAIL PROTECTED]>:
> I am curious to know how trunk stands in the performance comparison.
> There have been many optimizations going into trunk. Getting hold of
> overall improvement would be nice.
>
> Cheers
>
> On Wed, Mar 20, 2013 at 5:02 AM, Jean-Marc Spaggiari <
> [EMAIL PROTECTED]> wrote:
>
>> Hi Lars,
>>
>> Can you share the code you are using so I can compate with PE? Also, I
>> will re-run all for my scanRange100 tests today and update the
>> spreadsheet again to make sure it's correct. Also also re-download all
>> the HBase versions to make sure they are all clean. I'm not doing any
>> configuration with them. Simply reducing the logs and tmp pointing to
>> memory file system.
>>
>> I will keep you posted when it's done.
>>
>> Hi Jonathan,
>>
>> It's usually rows per seconds, but with a factor 10. Sometime I had to
>> divide by 100000, sometime to multiply to get numbers bigger... I will
>> take a look at th formulas and add the legend for each of the charts.
>>
>> JM
>>
>> 2013/3/19 Jonathan Hsieh <[EMAIL PROTECTED]>:
>> > What is the y axis's unit?  seconds or operations per second etc?  (nit:
>> > would be nice to have on the axis.. )
>> >
>> > Based on the context, I believe it is ops/s.
>> >
>> > Jon.
>> >
>> > On Sat, Mar 16, 2013 at 7:03 PM, Jean-Marc Spaggiari <
>> > [EMAIL PROTECTED]> wrote:
>> >
>> >> Hi Enis,
>> >>
>> >> "interesting" in the positive way ;)
>> >>
>> >> Results are there:
>> >>
>> >>
>> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
>> >>
>> >> The improvment on scan are impressive. sequentialRead and randomScan
>> went
>> >> down.
>> >>
>> >> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
>> >>
>> >> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
>> >> with it on previous HBase version...
>> >>
>> >> JM
>> >>
>> >> 2013/3/12 Enis Söztutar <[EMAIL PROTECTED]>:
>> >> >> I just finished to run all the PerformanceEvaluation tests on a
>> >> > dedicated computer with all 0.9x.x HBase versions, and I found results
>> >> > interesting.
>> >> > Can you please provide your numbers if you can. What is interesting
>> from
>> >> > your findings?
>> >> >
>> >> > Enis
>> >> >
>> >> >
>> >> >
>> >> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
>> >> > [EMAIL PROTECTED]> wrote:
>> >> >
>> >> >> If you run only 1 client with PerformanceEvaluation, it's not running
>> >> >> it over MapReduce, so you don't have this overhead. But you can still
>> >> >> run it if you want to have something more distributed. Might be
>> useful
>> >> >> to have the 2 options. But at the end, LoadTestTool or
>> >> >> PerformanceEvaluation, any of the 2 is good as long as we are adding
>> >> >> those tests.
>> >> >>
>> >> >> I just finished to run all the PerformanceEvaluation tests on a
>> >> >> dedicated computer with all 0.9x.x HBase versions, and I found
>> results
>> >> >> interesting. That gives us a good baseline to see if new HBase
>> >> >> improvements are really improving performances.
>> >> >>
>> >> >> JM
>> >> >>
>> >> >> 2013/3/8 Andrew Purtell <[EMAIL PROTECTED]>:
>> >> >> > Tangentally: I think I prefer LoadTestTool over
>> >> PerformanceEvaluation, it
>> >> >> > doesn't depend on nor is influenced by MapReduce job startup.
>> >> >> >
>> >> >> >
>> >> >> > On Fri, Mar 8, 2013 at 10:05 PM, ramkrishna vasudevan <
>> >> >> > [EMAIL PROTECTED]> wrote:
>> >> >> >
>> >> >> >> @JM
>> >> >> >> I agree with you.  Mainly the perf improvement changes needs some
>> >> >> >> testcases.
>> >> >> >> But sometimes the scenario on which the perf improvments happens

Thanks so much for doing this J-M.

Enis
On Wed, Mar 20, 2013 at 11:44 AM, Jean-Marc Spaggiari <
[EMAIL PROTECTED]> wrote:

> Hi Ted,
>
> I will try to build the trunk version and add it in the list....
>
> So I have to:
> - Re-download all HBase version to make sure PE is using the right one;
> - Re-run the scanRange100 tests to validate the values already found;
> - Add the trunk;
> - Add LoadTestTool
>
> Than will keep my free time buzy ;)
>
> I keep you all posted as soon as it's done.
>
> JM
>
> 2013/3/20 Ted Yu <[EMAIL PROTECTED]>:
> > I am curious to know how trunk stands in the performance comparison.
> > There have been many optimizations going into trunk. Getting hold of
> > overall improvement would be nice.
> >
> > Cheers
> >
> > On Wed, Mar 20, 2013 at 5:02 AM, Jean-Marc Spaggiari <
> > [EMAIL PROTECTED]> wrote:
> >
> >> Hi Lars,
> >>
> >> Can you share the code you are using so I can compate with PE? Also, I
> >> will re-run all for my scanRange100 tests today and update the
> >> spreadsheet again to make sure it's correct. Also also re-download all
> >> the HBase versions to make sure they are all clean. I'm not doing any
> >> configuration with them. Simply reducing the logs and tmp pointing to
> >> memory file system.
> >>
> >> I will keep you posted when it's done.
> >>
> >> Hi Jonathan,
> >>
> >> It's usually rows per seconds, but with a factor 10. Sometime I had to
> >> divide by 100000, sometime to multiply to get numbers bigger... I will
> >> take a look at th formulas and add the legend for each of the charts.
> >>
> >> JM
> >>
> >> 2013/3/19 Jonathan Hsieh <[EMAIL PROTECTED]>:
> >> > What is the y axis's unit?  seconds or operations per second etc?
>  (nit:
> >> > would be nice to have on the axis.. )
> >> >
> >> > Based on the context, I believe it is ops/s.
> >> >
> >> > Jon.
> >> >
> >> > On Sat, Mar 16, 2013 at 7:03 PM, Jean-Marc Spaggiari <
> >> > [EMAIL PROTECTED]> wrote:
> >> >
> >> >> Hi Enis,
> >> >>
> >> >> "interesting" in the positive way ;)
> >> >>
> >> >> Results are there:
> >> >>
> >> >>
> >>
> http://www.spaggiari.org/media/blogs/hbase/pictures/performances-1.pdf?mtime=1363484477
> >> >>
> >> >> The improvment on scan are impressive. sequentialRead and randomScan
> >> went
> >> >> down.
> >> >>
> >> >> In ran the 0.94.6 tests with RC2. If we have a RC3 I will rerun them.
> >> >>
> >> >> I will add HFilePerformanceEvaluation soon but I'm facinf some issues
> >> >> with it on previous HBase version...
> >> >>
> >> >> JM
> >> >>
> >> >> 2013/3/12 Enis Söztutar <[EMAIL PROTECTED]>:
> >> >> >> I just finished to run all the PerformanceEvaluation tests on a
> >> >> > dedicated computer with all 0.9x.x HBase versions, and I found
> results
> >> >> > interesting.
> >> >> > Can you please provide your numbers if you can. What is interesting
> >> from
> >> >> > your findings?
> >> >> >
> >> >> > Enis
> >> >> >
> >> >> >
> >> >> >
> >> >> > On Tue, Mar 12, 2013 at 5:41 PM, Jean-Marc Spaggiari <
> >> >> > [EMAIL PROTECTED]> wrote:
> >> >> >
> >> >> >> If you run only 1 client with PerformanceEvaluation, it's not
> running
> >> >> >> it over MapReduce, so you don't have this overhead. But you can
> still
> >> >> >> run it if you want to have something more distributed. Might be
> >> useful
> >> >> >> to have the 2 options. But at the end, LoadTestTool or
> >> >> >> PerformanceEvaluation, any of the 2 is good as long as we are
> adding
> >> >> >> those tests.
> >> >> >>
> >> >> >> I just finished to run all the PerformanceEvaluation tests on a
> >> >> >> dedicated computer with all 0.9x.x HBase versions, and I found
> >> results
> >> >> >> interesting. That gives us a good baseline to see if new HBase
> >> >> >> improvements are really improving performances.
> >> >> >>
> >> >> >> JM
> >> >> >>
> >> >> >> 2013/3/8 Andrew Purtell <[EMAIL PROTECTED]>:
> >> >> >> > Tangentally: I think I prefer LoadTestTool over
> >> >> PerformanceEvaluation, it
> >> >> >> > doesn't depend on nor is influenced by MapReduce job startup.