Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - one of our datanodes stops working after few hours


Copy link to this message
-
Re: one of our datanodes stops working after few hours
Jack Levin 2011-05-02, 19:56
Tried removing yourkit and run on javasun, same thing.  We have some
threads blocked, does anyone know what they block on?

-Jack

On Mon, May 2, 2011 at 7:53 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> Hi Jack,
>
> Does this happen even if you aren't running Yourkit on the DN?
>
> Can you try using a Sun JDK instead of OpenJDK?
>
> -Todd
>
> On Sun, May 1, 2011 at 7:34 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>
>> Version:         0.20.2+320 hdfs
>> .89 HBASE
>>
>> ulimit is 32k
>> xcievers is 5k
>>
>> Note from the jstack, I am not exceeding xcievers.
>>
>> -Jack
>>
>> On Sun, May 1, 2011 at 6:19 PM, Michael Segel <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >
>> > What's your xceivers set to?
>> > What's the ulimit -n  set for hdfs/hadoop user... (You didn't say which
>> release/version you were using.)
>> >
>> >> Date: Sun, 1 May 2011 17:47:18 -0700
>> >> Subject: one of our datanodes stops working after few hours
>> >> From: [EMAIL PROTECTED]
>> >> To: [EMAIL PROTECTED]
>> >>
>> >> I took a jstack (http://pastebin.com/5v6mHg3t).   After few hours, its
>> >> literally staggers to a halt and gets very very slow... Any ideas
>> >> whats its blocking on?
>> >> (main issue is that fsreads for RS get really slow when that happens).
>> >>
>> >> -Jack
>> >
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>