Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> Hadoop efficient resource isolation


Copy link to this message
-
Hadoop efficient resource isolation
Hi Guys,

It's possible isolation job submission for hadoop cluster, we currently
running 48 machine cluster. we  monitor Hadoop is not provides efficient
resource isolation. In my case we ran for tech and research pool, When tech
job some memory leak will haven, It's occupy the hole cluster.  Finally
we figure out  issue with tech job. It's  screwed up hole hadoop cluster.
finally 10 data node  are dead.

Any prevention of job submission efficient way resource allocation. When
something wrong in   particular job, effect particular pool, Not effect
others job. Any way to archive this

Please guide me guys.

My idea is, When tech user submit job means only apply job in for my
case submit 24 machine. other machine only for research user.

It's will prevent the memory leak problem.
-Dhanasekaran.
Did I learn something today? If not, I wasted it.
+
Hemanth Yamijala 2013-02-21, 16:51
+
Michel Segel 2013-02-21, 16:37
+
Arun C Murthy 2013-02-26, 14:02
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB