Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS, mail # user - High IO Usage in Datanodes due to Replication


Copy link to this message
-
High IO Usage in Datanodes due to Replication
selva 2013-04-27, 18:03
Hi All,

I have lost amazon instances of my hadoop cluster. But i had all the data
in aws EBS volumes. So i launched new instances and attached volumes.

But all of the datanode logs keep on print the below lines it cauased to
high IO rate. Due to IO usage i am not able to run any jobs.

Can anyone help me to understand what it is doing? Thanks in advance.

2013-04-27 17:51:40,197 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
10.157.10.242:10013,
storageID=DS-407656544-10.28.217.27-10013-1353165843727, infoPort=15075,
ipcPort=10014) Starting thread to transfer block
blk_2440813767266473910_11564425 to 10.168.18.178:10013
2013-04-27 17:51:40,230 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(
10.157.10.242:10013,
storageID=DS-407656544-10.28.217.27-10013-1353165843727, infoPort=15075,
ipcPort=10014):Transmitted block blk_2440813767266473910_11564425 to /
10.168.18.178:10013
2013-04-27 17:51:40,433 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_2442656050740605335_10906493 src: /10.171.11.11:60744 dest: /
10.157.10.242:10013
2013-04-27 17:51:40,450 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Received block
blk_2442656050740605335_10906493 src: /10.171.11.11:60744 dest: /
10.157.10.242:10013 of size 25431

Thanks
Selva