Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce >> mail # user >> rack topology data update


Copy link to this message
-
rack topology data update
Our hadoop version is hadoop-0.20-append+4.

We have configured the rack awareness in the namenode.
But when I add new datanode, and update the topology data file, and restart
the datanode, I just see the log in the namenode that:
2012-09-13 10:35:25,074 INFO org.apache.hadoop.net.NetworkTopology: Adding
a new node: /default-rack/ipc:50010
So should I restart the namenode?
Is there some command like 'hadoop dfsadmin -refreshtopology'?

My configuration:

*core-site.xml:*
<property>
<name>topology.script.file.name</name>
<value>conf/rack-awareness.sh</value>
</property>
<property>
<name>topology.script.number.args</name>
<value>1000</value>
</property>

*conf/rack-awareness.sh:*
#!/bin/sh

HADOOP_CONF=/opt/hadoop/conf

while [ $# -gt 0 ] ; do
  nodeArg=$1
  exec< ${HADOOP_CONF}/topology.data
  result=""
  while read line ; do
    ar=( $line )
    if [ "${ar[0]}" = "$nodeArg" ] ; then
      result="${ar[1]}"
    fi
  done
  shift
  if [ -z "$result" ] ; then
    echo -n "/default-rack "
  else
    echo -n "$result "
  fi
done

topology.data:
ipa rackA
ipb rackA
ipc rackB
And also I have search the mailing list "topology.script.file update":
I found a mail that:
Tom Hall 2011-10-27, 16:07
I was hoping that if I updated the file it would give new answers as
datanodes were restarted and reconnected but that does not seem to be
the case.
Surely I dont need to restart the namenode...

But there is not replying.
So somebody can help me?
专注于Mysql,MSSQL,Oracle,Hadoop