Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HDFS >> mail # user >> DFSClient not able to write


Copy link to this message
-
DFSClient not able to write
Hi,

When DFS cluster;
The datanodes keep on dumping the logs:
2012-06-19 04:05:04,842 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-1233167859-10.10.35.8-1340093005960
2012-06-19 04:05:04,861 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 beginning handshake with NN
2012-06-19 04:05:04,979 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 successfully registered with NN
2012-06-19 04:05:04,980 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 using DELETEREPORT_INTERVAL of 300000 msec  BLOCKREPORT_INTERVAL of 21600000msec Initial delay: 0msec; heartBeatInterval=3000
2012-06-19 04:05:05,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Namenode Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000 trying to claim ACTIVE state with txid=24
2012-06-19 04:05:05,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE Namenode Block pool BP-1233167859-10.10.35.8-1340093005960 (storage id DS-1255092928-10.244.15.192-50010-1340093104449) service to ec2-107-22-39-220.compute-1.amazonaws.com/10.10.35.8:9000
2012-06-19 04:05:05,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0 blocks took 1 msec to generate and 135 msecs for RPC and NN processing
2012-06-19 04:05:05,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: sent block report, processed command:org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@49d31859
2012-06-19 04:05:05,396 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Periodic Block Verification Scanner initialized with interval 504 hours for block pool BP-1233167859-10.10.35.8-1340093005960.
2012-06-19 04:05:05,400 INFO org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Added bpid=BP-1233167859-10.10.35.8-1340093005960 to blockPoolScannerMap, new size=1
2012-06-19 04:05:09,845 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:14,848 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:19,849 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:24,850 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:29,851 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:34,852 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:39,853 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:44,854 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%
2012-06-19 04:05:49,855 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00%

The DFSClient operations are also not taking place properly:
2012-06-19 04:04:56,653 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception
java.io.IOException: File /hbase/hbase.version could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and no node(s) are excluded in this operation.
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1256)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1977)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:470)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:292)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:42602)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:427)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)

Please let me know if there is any configuration which needs to be set…..
The hadoop version being used is: 2.0.0

Cheers,
Subroto Sanyal
+
Todd Lipcon 2012-06-19, 16:22
+
Subroto 2012-06-20, 07:06
+
qhdpp 2012-06-20, 10:16
+
Subroto 2012-06-20, 10:19
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB