Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> one of our datanodes stops working after few hours


Copy link to this message
-
Re: one of our datanodes stops working after few hours
Tried removing yourkit and run on javasun, same thing.  We have some
threads blocked, does anyone know what they block on?

-Jack

On Mon, May 2, 2011 at 7:53 AM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> Hi Jack,
>
> Does this happen even if you aren't running Yourkit on the DN?
>
> Can you try using a Sun JDK instead of OpenJDK?
>
> -Todd
>
> On Sun, May 1, 2011 at 7:34 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>
>> Version:         0.20.2+320 hdfs
>> .89 HBASE
>>
>> ulimit is 32k
>> xcievers is 5k
>>
>> Note from the jstack, I am not exceeding xcievers.
>>
>> -Jack
>>
>> On Sun, May 1, 2011 at 6:19 PM, Michael Segel <[EMAIL PROTECTED]>
>> wrote:
>> >
>> >
>> > What's your xceivers set to?
>> > What's the ulimit -n  set for hdfs/hadoop user... (You didn't say which
>> release/version you were using.)
>> >
>> >> Date: Sun, 1 May 2011 17:47:18 -0700
>> >> Subject: one of our datanodes stops working after few hours
>> >> From: [EMAIL PROTECTED]
>> >> To: [EMAIL PROTECTED]
>> >>
>> >> I took a jstack (http://pastebin.com/5v6mHg3t).   After few hours, its
>> >> literally staggers to a halt and gets very very slow... Any ideas
>> >> whats its blocking on?
>> >> (main issue is that fsreads for RS get really slow when that happens).
>> >>
>> >> -Jack
>> >
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB