Hi,

My cluster started being incredibly slow in the past 2 days.
I've seen many Blocking updates on the region server logs, which lead me to believe HDFS creates is the bottleneck.
I ran a small test (hadoop fs -copyFromLocal big3_3Giga.tz.gz /tmp) which copies a 3.3G file, and I was surprised to see a rate of 11MB/sec, when it was previously around 28MB/sec.

I have 3 RS computers, hosting DataNode and embedded zookeeper as well.
And 1 Master computer, hosting NameNode as well.

The slowest computer has Core 2 Duo 2.93GHz, 4GB.

What steps can I take to find out the source of this slowdown?
Thanks!

Asaf Mesika

On Mon, Jul 9, 2012 at 8:35 PM, Asaf Mesika <[EMAIL PROTECTED]> wrote:
> Hi,
>
> My cluster started being incredibly slow in the past 2 days.
> I've seen many Blocking updates on the region server logs, which lead me to believe HDFS creates is the bottleneck.
> I ran a small test (hadoop fs -copyFromLocal big3_3Giga.tz.gz /tmp) which copies a 3.3G file, and I was surprised to see a rate of 11MB/sec, when it was previously around 28MB/sec.
>
> I have 3 RS computers, hosting DataNode and embedded zookeeper as well.
> And 1 Master computer, hosting NameNode as well.
>
> The slowest computer has Core 2 Duo 2.93GHz, 4GB.
>
> What steps can I take to find out the source of this slowdown?

Install ganglia or opentsdb and figure what your cluster is up to when its slow.

St.Ack

I got JMX counters hooked up to JConsole (couple of them opened).
Do you have any advice from your experience on what metrics I should focus on  to spot this issue?

On Jul 9, 2012, at 22:19 PM, Stack wrote:

> On Mon, Jul 9, 2012 at 8:35 PM, Asaf Mesika <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>> My cluster started being incredibly slow in the past 2 days.
>> I've seen many Blocking updates on the region server logs, which lead me to believe HDFS creates is the bottleneck.
>> I ran a small test (hadoop fs -copyFromLocal big3_3Giga.tz.gz /tmp) which copies a 3.3G file, and I was surprised to see a rate of 11MB/sec, when it was previously around 28MB/sec.
>>
>> I have 3 RS computers, hosting DataNode and embedded zookeeper as well.
>> And 1 Master computer, hosting NameNode as well.
>>
>> The slowest computer has Core 2 Duo 2.93GHz, 4GB.
>>
>> What steps can I take to find out the source of this slowdown?
>
> Install ganglia or opentsdb and figure what your cluster is up to when its slow.
>
> St.Ack

Asaf - check the URL in my signature - it's a service for monitoring various HBase metrics and may visually show you what is going on with various parts of your 3 servers.

Otis 
----
Performance Monitoring for Solr / ElasticSearch / HBase - http://sematext.com/spm 

----- Original Message -----
> From: Asaf Mesika <[EMAIL PROTECTED]>
> To: [EMAIL PROTECTED]
> Cc:
> Sent: Monday, July 9, 2012 2:35 PM
> Subject: HBase becomes ultra-slaggy
>
> Hi,
>
> My cluster started being incredibly slow in the past 2 days.
> I've seen many Blocking updates on the region server logs, which lead me to
> believe HDFS creates is the bottleneck.
> I ran a small test (hadoop fs -copyFromLocal big3_3Giga.tz.gz /tmp) which copies
> a 3.3G file, and I was surprised to see a rate of 11MB/sec, when it was
> previously around 28MB/sec.
>
> I have 3 RS computers, hosting DataNode and embedded zookeeper as well.
> And 1 Master computer, hosting NameNode as well.
>
> The slowest computer has Core 2 Duo 2.93GHz, 4GB.
>
> What steps can I take to find out the source of this slowdown?
>
>
> Thanks!
>
> Asaf Mesika
>

I used it but I didn't like the fact that the graph was updating every
minute and sometimes every 5 minutes.

Sent from my iPhone

On 10 ביול 2012, at 06:57, Otis Gospodnetic <[EMAIL PROTECTED]>
wrote:

Asaf - check the URL in my signature - it's a service for monitoring
various HBase metrics and may visually show you what is going on with
various parts of your 3 servers.

Otis
----
Performance Monitoring for Solr / ElasticSearch / HBase -
http://sematext.com/spm

----- Original Message -----

From: Asaf Mesika <[EMAIL PROTECTED]>

To: [EMAIL PROTECTED]

Cc:

Sent: Monday, July 9, 2012 2:35 PM

Subject: HBase becomes ultra-slaggy
Hi,
My cluster started being incredibly slow in the past 2 days.

I've seen many Blocking updates on the region server logs, which lead me to

believe HDFS creates is the bottleneck.

I ran a small test (hadoop fs -copyFromLocal big3_3Giga.tz.gz /tmp) which
copies

a 3.3G file, and I was surprised to see a rate of 11MB/sec, when it was

previously around 28MB/sec.
I have 3 RS computers, hosting DataNode and embedded zookeeper as well.

And 1 Master computer, hosting NameNode as well.
The slowest computer has Core 2 Duo 2.93GHz, 4GB.
What steps can I take to find out the source of this slowdown?

Thanks!
Asaf Mesika