Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # user >> Datanodes going down frequently


Copy link to this message
-
Re: Datanodes going down frequently
You have only 2 GB of ram? Have you cheked if you are swapping?

Raj

Sent from my iPad

On Sep 15, 2011, at 4:27 PM, Aaron Baff <[EMAIL PROTECTED]> wrote:

> Do they eventually recover and get added back into the cluster? Do you have a ton of blocks? I've noticed that sometimes the block checker will take so long, and will tie up so much CPU (and memory, which hits a GC cycle) that it stops reporting to the NN for a while, but when it finishes its check, it resumes talking to the NN and the NN adds it back in.
>
> --Aaron
> -----Original Message-----
> From: john smith [mailto:[EMAIL PROTECTED]]
> Sent: Thursday, September 15, 2011 3:07 PM
> To: [EMAIL PROTECTED]
> Subject: Datanodes going down frequently
>
> Hi all,
>
> I am running a 10 node cluster (1NN + 9DN, ubuntu server 10.04, 2GB RAM
> each). I am facing a strange problem. My datanodes go down randomly and
> nothing showup in the logs. They lose their network connectivity suddenly
> and NN declares them as dead. Any one faced this problem? Is it because of
> hadoop or is it some problem with my infrastructure?
>
> The worst part of the problem is, I need to manually go to the remote
> machine and restart networking. Can someone help me with this? Did any one
> face a similar kind of a problem
>
> Btw: my had version : 0.20.2
>
> Thanks,
> jS
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB