Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Performance degrades on moving from desktop to blade environment


Copy link to this message
-
Re: Performance degrades on moving from desktop to blade environment
Jack Levin 2011-05-16, 20:26
What is the clock rate of your CPUs (desktop vs blade)?

-Jack

On Mon, May 16, 2011 at 1:24 PM, Himanish Kushary <[EMAIL PROTECTED]> wrote:
> Yes, it is only the HW that was changed . All the configurations are kept at
> default from the cloudera installer.
>
> The regionserver logs semms ok.
>
> On Mon, May 16, 2011 at 3:20 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:
>
>> Ok I see... so the only thing that changed is the HW right? No
>> upgrades to a new version? Also could it be possible that you changed
>> some configs (or missed them)? BTW counting has a parameter for
>> scanner caching, like you would write: count "myTable", CACHE = 1000
>>
>> and it should stream through your data.
>>
>> Anything weird in the region server logs?
>>
>> J-D
>>
>> On Mon, May 16, 2011 at 12:13 PM, Himanish Kushary <[EMAIL PROTECTED]>
>> wrote:
>> > Thanks for the reply. We ran the TestDFSIO benchmark on both the
>> development
>> > and production and found the production to be better.The statistics are
>> > shown below.
>> >
>> > But once we bring HBase into the picture things gets reversed :-(
>> >
>> > The count operation,map-reduces etc becomes less performing on the
>> > production box.We are using Pseudo Distribution mode in both the
>> development
>> > and production servers for both hadoop and hbase.
>> >
>> > *DEVELOPMENT SERVER*
>> >
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO:            Date & time: Sun May 15
>> > 21:26:26 EDT 2011
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO:        Number of files: 10
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO: Total MBytes processed: 10000
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO:      Throughput mb/sec:
>> > 58.09495038691237
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO: Average IO rate mb/sec:
>> > 59.699485778808594
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO:  IO rate std deviation:
>> > 10.54547265175703
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO:     Test exec time sec: 163.354
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO:
>> >
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO:            Date & time: Sun May 15
>> > 21:28:44 EDT 2011
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO:        Number of files: 10
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO: Total MBytes processed: 10000
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO:      Throughput mb/sec:
>> > 682.4075337791729
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO: Average IO rate mb/sec:
>> > 755.5845947265625
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO:  IO rate std deviation:
>> > 229.60029445080488
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO:     Test exec time sec: 63.896
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO:
>> >
>> >
>> >
>> >
>> > *PRODUCTION SERVER*
>> >
>> > 5/16 01:00:43 INFO fs.TestDFSIO: ----- TestDFSIO ----- : *WRITE
>> PERFORMANCE*
>> >
>> > 11/05/16 01:00:43 INFO fs.TestDFSIO: Date & time: Mon May 16 01:00:43
>> > GMT+00:00 2011
>> >
>> > 11/05/16 01:00:43 INFO fs.TestDFSIO: Number of files: 10
>> >
>> > 11/05/16 01:00:43 INFO fs.TestDFSIO: Total MBytes processed: 10000
>> >
>> > 11/05/16 01:00:43 INFO fs.TestDFSIO: Throughput mb/sec: 69.25447557048375
>> >
>> > 11/05/16 01:00:43 INFO fs.TestDFSIO: Average IO rate mb/sec:
>> > 70.06581115722656
>> >
>> > 11/05/16 01:00:43 INFO fs.TestDFSIO: IO rate std deviation:
>> > 7.243961483443693
>> >
>> > 11/05/16 01:00:43 INFO fs.TestDFSIO: Test exec time sec: 126.896
>> >
>> >
>> > 5/16 01:25:01 INFO fs.TestDFSIO: ----- TestDFSIO ----- : *READ
>> PERFORMANCE*
>> >
>> > 11/05/16 01:25:01 INFO fs.TestDFSIO: Date & time: Mon May 16 01:25:01
>> > GMT+00:00 2011
>> >
>> > 11/05/16 01:25:01 INFO fs.TestDFSIO: Number of files: 10
>> >
>> > 11/05/16 01:25:01 INFO fs.TestDFSIO: Total MBytes processed: 10000
>> >
>> > 11/05/16 01:25:01 INFO fs.TestDFSIO: Throughput mb/sec: 1487.20999405116
>> >
>> > 11/05/16 01:25:01 INFO fs.TestDFSIO: Average IO rate mb/sec:
>> > 1525.230712890625
>> >