Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> hadoop imbalanced topology


+
Jameson Li 2013-01-28, 15:29
Copy link to this message
-
Re: hadoop imbalanced topology
Hi Jameson,

Inline reply below for a specific point of yours ("scheduler deadlock"):

On Mon, Jan 28, 2013 at 8:59 PM, Jameson Li <[EMAIL PROTECTED]> wrote:
> Hi,
>
> Is it right that hadoop Network Topology design treat an imbalanced topology
> as an invalid topology?
>
> This is mentioned here
> https://issues.apache.org/jira/browse/MAPREDUCE-4095(TestJobInProgress#testLocality
> uses a bogus topology.)
> said:"(https://issues.apache.org/jira/secure/attachment/12345251/Rack_aware_HDFS_proposal.pdf)
> which does not support imbalanced topology. "
>
> But when I re-read the desigh pdf, there just draw a balanced topology but
> really not mentioned it will not support imbalanced topology.
>
> Also I found this JIRA:
> https://issues.apache.org/jira/browse/HADOOP-8159(NetworkTopology: getLeaf
> should check for invalid topologies).
>
> Just no why for my question, because of the design(Right?):
> Why below is a invalid topology(except imbalanced), for hadoop rack
> awareness:
> serverA /default-rack    ##default rack
> serverB /dc1/rack2    ##actually the node has a pre "dc1"
> Should all of the hadoop datanode have the same level number? If not, there
> will be some issues for namenode and jt(like scheduler deadlock)?

This was a problem in earlier releases, where an imbalanced or
improper config for topology levels could lead to a jarring halt in
the JobTracker's schedulers. This has been fixed in 1.1.0 already via
https://issues.apache.org/jira/browse/MAPREDUCE-1740.

--
Harsh J
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB