Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> High load on datanode startup


Copy link to this message
-
Re: High load on datanode startup

Whats the response from fsck look like?
hadoop fsck /
It might be the case that some of the blocks are misreplicated
Serge

Hadoopway.blogspot.com

On 5/9/12 9:58 AM, "Darrell Taylor" <[EMAIL PROTECTED]> wrote:

>On Wed, May 9, 2012 at 5:56 PM, Serge Blazhiyevskyy <
>[EMAIL PROTECTED]> wrote:
>
>> Take a look at your data distribution for that cluster. Maybe, it is
>> unbalanced.
>>
>>
>> Run balancer, if it isŠ
>>
>
>The cluster is balanced, I ran balancer yesterday.  Oddly enough the
>problem started after I had run the balancer.
>
>I'm running CDH3 btw.
>
>
>
>>
>> Regards,
>> Serge
>>
>> hadoopway.blogspot.com
>>
>>
>>
>> On 5/9/12 9:52 AM, "Darrell Taylor" <[EMAIL PROTECTED]> wrote:
>>
>> >Hi,
>> >
>> >I wonder if someone could give some pointers with a problem I'm having?
>> >
>> >I have a 7 machine cluster setup for testing and we have been pouring
>>data
>> >into it for a week without issue, have learnt several thing along the
>>way
>> >and solved all the problems up to now by searching online, but now I'm
>> >stuck.  One of the data nodes decided to have a load of 70+ this
>>morning,
>> >stopping datanode and tasktracker brought it back to normal, but every
>> >time
>> >I start the datanode again the load shoots through the roof, and all I
>>get
>> >in the logs is :
>> >
>> >STARTUP_MSG: Starting DataNode
>> >
>> >
>> >STARTUP_MSG:   host = pl464/10.20.16.64
>> >
>> >
>> >STARTUP_MSG:   args = []
>> >
>> >
>> >STARTUP_MSG:   version = 0.20.2-cdh3u3
>> >
>> >
>> >STARTUP_MSG:   build >>
>>>file:///data/1/tmp/nightly_2012-03-20_13-13-48_3/hadoop-0.20-0.20.2+923.
>>>19
>> >7-1~squeeze
>> >-************************************************************/
>> >
>> >
>> >2012-05-09 16:12:05,925 INFO
>> >org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
>> >already
>> >set up for Hadoop, not re-installing.
>> >
>> >2012-05-09 16:12:06,139 INFO
>> >org.apache.hadoop.security.UserGroupInformation: JAAS Configuration
>> >already
>> >set up for Hadoop, not re-installing.
>> >
>> >Nothing else.
>> >
>> >The load seems to max out only 1 of the CPUs, but the machine becomes
>> >*very* unresponsive
>> >
>> >Anybody got any pointers of things I can try?
>> >
>> >Thanks
>> >Darrell.
>>
>>