Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> High load on datanode startup


Copy link to this message
-
Re: High load on datanode startup
Darrell

Are the new dn,nn and mapred directories on the same physical disk? Nothing on NFS , correct?

Could you be having some hardware issue? Any clue in /var/log/messages or dmesg?

A non responsive system indicates a CPU that is really busy either doing something or waiting for something and the fact that it happens only on some nodes indicates a local problem.

Raj

>________________________________
> From: Darrell Taylor <[EMAIL PROTECTED]>
>To: [EMAIL PROTECTED]
>Cc: Raj Vishwanathan <[EMAIL PROTECTED]>
>Sent: Thursday, May 10, 2012 3:57 AM
>Subject: Re: High load on datanode startup
>
>On Thu, May 10, 2012 at 9:33 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> That's real weird..
>>
>> If you can reproduce this after a reboot, I'd recommend letting the DN
>> run for a minute, and then capturing a "jstack <pid of dn>" as well as
>> the output of "top -H -p <pid of dn> -b -n 5" and send it to the list.
>
>
>What I did after the reboot this morning was to move the my dn, nn, and
>mapred directories out of the the way, create a new one, formatted it, and
>restarted the node, it's now happy.
>
>I'll try moving the directories back later and do the jstack as you suggest.
>
>
>>
>> What JVM/JDK are you using? What OS version?
>>
>
>root@pl446:/# dpkg --get-selections | grep java
>java-common                                     install
>libjaxp1.3-java                                 install
>libjaxp1.3-java-gcj                             install
>libmysql-java                                   install
>libxerces2-java                                 install
>libxerces2-java-gcj                             install
>sun-java6-bin                                   install
>sun-java6-javadb                                install
>sun-java6-jdk                                   install
>sun-java6-jre                                   install
>
>root@pl446:/# java -version
>java version "1.6.0_26"
>Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
>Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
>
>root@pl446:/# cat /etc/issue
>Debian GNU/Linux 6.0 \n \l
>
>
>
>>
>> -Todd
>>
>>
>> On Wed, May 9, 2012 at 11:57 PM, Darrell Taylor
>> <[EMAIL PROTECTED]> wrote:
>> > On Wed, May 9, 2012 at 10:52 PM, Raj Vishwanathan <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >> The picture either too small or too pixelated for my eyes :-)
>> >>
>> >
>> > There should be a zoom option in the top right of the page that allows
>> you
>> > to view it full size
>> >
>> >
>> >>
>> >> Can you login to the box and send the output of top? If the system is
>> >> unresponsive, it has to be something more than an unbalanced hdfs
>> cluster,
>> >> methinks.
>> >>
>> >
>> > Sorry, I'm unable to login to the box, it's completely unresponsive.
>> >
>> >
>> >>
>> >> Raj
>> >>
>> >>
>> >>
>> >> >________________________________
>> >> > From: Darrell Taylor <[EMAIL PROTECTED]>
>> >> >To: [EMAIL PROTECTED]; Raj Vishwanathan <[EMAIL PROTECTED]
>> >
>> >> >Sent: Wednesday, May 9, 2012 2:40 PM
>> >> >Subject: Re: High load on datanode startup
>> >> >
>> >> >On Wed, May 9, 2012 at 10:23 PM, Raj Vishwanathan <[EMAIL PROTECTED]>
>> >> wrote:
>> >> >
>> >> >> When you say 'load', what do you mean? CPU load or something else?
>> >> >>
>> >> >
>> >> >I mean in the unix sense of load average, i.e. top would show a load of
>> >> >(currently) 376.
>> >> >
>> >> >Looking at Ganglia stats for the box it's not CPU load as such, the
>> graphs
>> >> >shows actual CPU usage as 30%, but the number of running processes is
>> >> >simply growing in a linear manner - screen shot of ganglia page here :
>> >> >
>> >> >
>> >>
>> https://picasaweb.google.com/lh/photo/Q0uFSzyLiriDuDnvyRUikXVR0iWwMibMfH0upnTwi28?feat=directlink
>> >> >
>> >> >
>> >> >
>> >> >>
>> >> >> Raj
>> >> >>
>> >> >>
>> >> >>
>> >> >> >________________________________
>> >> >> > From: Darrell Taylor <[EMAIL PROTECTED]>
>> >> >> >To: [EMAIL PROTECTED]
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB