Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop >> mail # general >> Fwd: adding hdfs volume


Copy link to this message
-
Fwd: adding hdfs volume
Hi,

I have installed hadoop cluster . I tried to add other drives in hdfs but
didn't get success. I tried to add  following parameter in
/etc/hadoop/conf/hdfs-site.xml. but it is not working i get service

  <property>
     <name>dfs.datanode.data.dir</name>
<!--     <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/data</value> -->
  <value>/mnt/san1/hdfs/${user.name}/dfs/data,/mnt/san2/hdfs/${user.name
}/dfs/data,/mnt/san3/hdfs/${user.name}/dfs/data,/mnt/san4/hdfs/${user.name
}/dfs/data</value>
  </property>

]# for service in /etc/init.d/hadoop-hdfs-* ; do $service status; done
Hadoop datanode is running                                 [  OK  ]
Hadoop namenode is dead and pid file exists                [FAILED]
Hadoop secondarynamenode is running                        [  OK  ]
Logs

# tail -n 30 hadoop-hdfs-namenode-OPERA-MAST1.log
2012-12-27 13:04:42,526 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner             hdfs (auth:SIMPLE)
2012-12-27 13:04:42,526 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup          hadmin
2012-12-27 13:04:42,526 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled true
2012-12-27 13:04:42,526 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
2012-12-27 13:04:42,528 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
2012-12-27 13:04:42,555 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
occuring more than 10 times
2012-12-27 13:04:42,559 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2012-12-27 13:04:42,559 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.min.datanodes = 0
2012-12-27 13:04:42,559 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
dfs.namenode.safemode.extension     = 0
2012-12-27 13:04:42,566 INFO org.apache.hadoop.hdfs.server.common.Storage:
Lock on /var/lib/hadoop-hdfs/cache/hdfs/dfs/name/in_use.lock acquired by
nodename [EMAIL PROTECTED]l
2012-12-27 13:04:42,568 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2012-12-27 13:04:42,568 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
stopped.
2012-12-27 13:04:42,568 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
shutdown complete.
2012-12-27 13:04:42,569 FATAL
org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: NameNode is not formatted.
        at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:211)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
        at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
        at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
2012-12-27 13:04:42,571 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1
2012-12-27 13:04:42,574 INFO
org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at OPERA-MAST1.ny.os.local/172.20.3.119

===================================================

But if i add in following way means one property for one drive then it pick
last drive .

  <property>
     <name>dfs.datanode.data.dir</name>
<!--     <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/data</value> -->
  <value>/mnt/san2/hdfs/${user.name}/dfs/data</value>
  </property>
  <property>
     <name>dfs.datanode.data.dir</name>
<!--     <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/data</value> -->
  <value>/mnt/san3/hdfs/${user.name}/dfs/data</value>
  </property>
    <property>
     <name>dfs.datanode.data.dir</name>
<!--     <value>/var/lib/hadoop-hdfs/cache/${user.name}/dfs/data</value> -->
  <value>/mnt/san4/hdfs/${user.name}/dfs/data</value>
  </property>
Can you please let me know the issue.

Thanks,
Mohit
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB