Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
MapReduce >> mail # user >> rack topology data update


Copy link to this message
-
rack topology data update
Our hadoop version is hadoop-0.20-append+4.

We have configured the rack awareness in the namenode.
But when I add new datanode, and update the topology data file, and restart
the datanode, I just see the log in the namenode that:
2012-09-13 10:35:25,074 INFO org.apache.hadoop.net.NetworkTopology: Adding
a new node: /default-rack/ipc:50010
So should I restart the namenode?
Is there some command like 'hadoop dfsadmin -refreshtopology'?

My configuration:

*core-site.xml:*
<property>
<name>topology.script.file.name</name>
<value>conf/rack-awareness.sh</value>
</property>
<property>
<name>topology.script.number.args</name>
<value>1000</value>
</property>

*conf/rack-awareness.sh:*
#!/bin/sh

HADOOP_CONF=/opt/hadoop/conf

while [ $# -gt 0 ] ; do
  nodeArg=$1
  exec< ${HADOOP_CONF}/topology.data
  result=""
  while read line ; do
    ar=( $line )
    if [ "${ar[0]}" = "$nodeArg" ] ; then
      result="${ar[1]}"
    fi
  done
  shift
  if [ -z "$result" ] ; then
    echo -n "/default-rack "
  else
    echo -n "$result "
  fi
done

topology.data:
ipa rackA
ipb rackA
ipc rackB
And also I have search the mailing list "topology.script.file update":
I found a mail that:
Tom Hall 2011-10-27, 16:07
I was hoping that if I updated the file it would give new answers as
datanodes were restarted and reconnected but that does not seem to be
the case.
Surely I dont need to restart the namenode...

But there is not replying.
So somebody can help me?
专注于Mysql,MSSQL,Oracle,Hadoop
+
Harsh J 2012-09-13, 03:32
+
Jameson Li 2012-09-13, 05:51
+
Viji R 2012-09-13, 05:55
+
Saurabh bhutyani 2012-09-13, 06:22
+
Jameson Li 2012-09-13, 07:19
+
Harsh J 2012-09-13, 08:03
+
Steve Loughran 2012-09-13, 09:10
+
Harsh J 2012-09-13, 09:15
+
Harsh J 2012-09-14, 04:28
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB