Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Performance degrades on moving from desktop to blade environment


Copy link to this message
-
Re: Performance degrades on moving from desktop to blade environment
What is the clock rate of your CPUs (desktop vs blade)?

-Jack

On Mon, May 16, 2011 at 1:24 PM, Himanish Kushary <[EMAIL PROTECTED]> wrote:
> Yes, it is only the HW that was changed . All the configurations are kept at
> default from the cloudera installer.
>
> The regionserver logs semms ok.
>
> On Mon, May 16, 2011 at 3:20 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:
>
>> Ok I see... so the only thing that changed is the HW right? No
>> upgrades to a new version? Also could it be possible that you changed
>> some configs (or missed them)? BTW counting has a parameter for
>> scanner caching, like you would write: count "myTable", CACHE = 1000
>>
>> and it should stream through your data.
>>
>> Anything weird in the region server logs?
>>
>> J-D
>>
>> On Mon, May 16, 2011 at 12:13 PM, Himanish Kushary <[EMAIL PROTECTED]>
>> wrote:
>> > Thanks for the reply. We ran the TestDFSIO benchmark on both the
>> development
>> > and production and found the production to be better.The statistics are
>> > shown below.
>> >
>> > But once we bring HBase into the picture things gets reversed :-(
>> >
>> > The count operation,map-reduces etc becomes less performing on the
>> > production box.We are using Pseudo Distribution mode in both the
>> development
>> > and production servers for both hadoop and hbase.
>> >
>> > *DEVELOPMENT SERVER*
>> >
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO: ----- TestDFSIO ----- : write
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO:            Date & time: Sun May 15
>> > 21:26:26 EDT 2011
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO:        Number of files: 10
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO: Total MBytes processed: 10000
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO:      Throughput mb/sec:
>> > 58.09495038691237
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO: Average IO rate mb/sec:
>> > 59.699485778808594
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO:  IO rate std deviation:
>> > 10.54547265175703
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO:     Test exec time sec: 163.354
>> > 11/05/15 21:26:26 INFO fs.TestDFSIO:
>> >
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO: ----- TestDFSIO ----- : read
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO:            Date & time: Sun May 15
>> > 21:28:44 EDT 2011
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO:        Number of files: 10
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO: Total MBytes processed: 10000
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO:      Throughput mb/sec:
>> > 682.4075337791729
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO: Average IO rate mb/sec:
>> > 755.5845947265625
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO:  IO rate std deviation:
>> > 229.60029445080488
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO:     Test exec time sec: 63.896
>> > 11/05/15 21:28:44 INFO fs.TestDFSIO:
>> >
>> >
>> >
>> >
>> > *PRODUCTION SERVER*
>> >
>> > 5/16 01:00:43 INFO fs.TestDFSIO: ----- TestDFSIO ----- : *WRITE
>> PERFORMANCE*
>> >
>> > 11/05/16 01:00:43 INFO fs.TestDFSIO: Date & time: Mon May 16 01:00:43
>> > GMT+00:00 2011
>> >
>> > 11/05/16 01:00:43 INFO fs.TestDFSIO: Number of files: 10
>> >
>> > 11/05/16 01:00:43 INFO fs.TestDFSIO: Total MBytes processed: 10000
>> >
>> > 11/05/16 01:00:43 INFO fs.TestDFSIO: Throughput mb/sec: 69.25447557048375
>> >
>> > 11/05/16 01:00:43 INFO fs.TestDFSIO: Average IO rate mb/sec:
>> > 70.06581115722656
>> >
>> > 11/05/16 01:00:43 INFO fs.TestDFSIO: IO rate std deviation:
>> > 7.243961483443693
>> >
>> > 11/05/16 01:00:43 INFO fs.TestDFSIO: Test exec time sec: 126.896
>> >
>> >
>> > 5/16 01:25:01 INFO fs.TestDFSIO: ----- TestDFSIO ----- : *READ
>> PERFORMANCE*
>> >
>> > 11/05/16 01:25:01 INFO fs.TestDFSIO: Date & time: Mon May 16 01:25:01
>> > GMT+00:00 2011
>> >
>> > 11/05/16 01:25:01 INFO fs.TestDFSIO: Number of files: 10
>> >
>> > 11/05/16 01:25:01 INFO fs.TestDFSIO: Total MBytes processed: 10000
>> >
>> > 11/05/16 01:25:01 INFO fs.TestDFSIO: Throughput mb/sec: 1487.20999405116
>> >
>> > 11/05/16 01:25:01 INFO fs.TestDFSIO: Average IO rate mb/sec:
>> > 1525.230712890625
>> >
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB