I am Decommissioning data nodes for an OS upgrade on a HPC cluster . Currently, users can run jobs that use data stored on /hdfs. They are able to access all datanodes/compute nodes except the one being decommissioned.
Is this safe to do? Will edited files affect the decommissioning node?
I've been adding the nodes to /usr/lib/hadoop-0.20/conf/hosts_exclude and running 'hadoop dfsadmin -refreshNodes' on the name name node. Then I simply wait for log files to report completion. After upgrade, I simply remove the node from hosts_exlude and start hadoop again on the datanode.
Also: Under the namenode web interface I just noticed that the node I have decommissioned previously now has 0 Configured capacity, Used, Remaining memory and is now 100% Used.
I used the same /etc/sysconfig/hadoop file from before the upgrade, removed the node from hosts_exclude, and ran '-refreshNodes' afterwards.
What steps have I missed in the decommissioning process or while bringing the data node back online?
"...it will not bring in exactly the same blocks like before" Is that what usually happens when adding nodes back in? Should I expect any data loss due to starting the data node process before running the balancing tool?
On Thu, Jul 24, 2014 at 11:37 AM, Mirko Kämpf <[EMAIL PROTECTED]> wrote:
You should not face any data loss. The replicas were just moved away from that node to other nodes in the cluster during decommission. Once you recommission the node and re-balance your cluster, HDFS will re-distribute replicas between the nodes evenly, and the recommissioned node will receive replicas from other nodes, but there is no guarantee that exact the same replicas that were stored on this node before it was decommissioned will be assigned to this node again, after recommission and rebalance.
Cheers, Wellington. On 24 Jul 2014, at 17:55, andrew touchet <[EMAIL PROTECTED]> wrote:
That sounds wonderful! I appreciate everyone's help.
Andrew Touchet On Thu, Jul 24, 2014 at 12:01 PM, Wellington Chevreuil < [EMAIL PROTECTED]> wrote:
NEW: Monitor These Apps!
Apache Lucene, Apache Solr and all other Apache Software Foundation project and their respective logos are trademarks of the Apache Software Foundation.
Elasticsearch, Kibana, Logstash, and Beats are trademarks of Elasticsearch BV, registered in the U.S. and in other countries. This site and Sematext Group is in no way affiliated with Elasticsearch BV.
Service operated by Sematext