Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
Hadoop, mail # user - java.io.IOException: Bad connect ack with firstBadLink


Copy link to this message
-
java.io.IOException: Bad connect ack with firstBadLink
Oleg Ruchovets 2010-11-09, 10:58
Hi , running hadoop map/reduce got such exception?

  1) Why does it happen?

  2) Job didn't failed and continue it's execution? Does this
exception cause losing data or map/reduce

uses recovery mechanism?

2010-11-09 05:10:08,735 INFO org.apache.hadoop.hdfs.DFSClient:
Exception in createBlockOutputStream java.io.IOException: Bad connect
ack with firstBadLink 10.11.87.65:50010
2010-11-09 05:10:08,735 INFO org.apache.hadoop.hdfs.DFSClient:
Abandoning block blk_-9208177033562590356_775948
2010-11-09 05:10:08,739 INFO org.apache.hadoop.hdfs.DFSClient: Waiting
to find target node: 10.11.87.61:50010
2010-11-09 05:11:23,743 INFO org.apache.hadoop.hdfs.DFSClient:
Exception in createBlockOutputStream java.net.SocketTimeoutException:
69000 millis timeout while waiting for channel to be ready for read.
ch : java.nio.channels.SocketChannel[connected
local=/10.11.87.61:55309 remote=/10.11.87.61:50010]
2010-11-09 05:11:23,743 INFO org.apache.hadoop.hdfs.DFSClient:
Abandoning block blk_-10251707095594311_775950
2010-11-09 05:11:23,744 INFO org.apache.hadoop.hdfs.DFSClient: Waiting
to find target node: 10.11.87.61:50010
2010-11-09 05:12:29,815 INFO org.apache.hadoop.hdfs.DFSClient:
Exception in createBlockOutputStream java.io.IOException: Bad connect
ack with firstBadLink 10.11.87.65:50010
2010-11-09 05:12:29,816 INFO org.apache.hadoop.hdfs.DFSClient:
Abandoning block blk_3509928762116143133_775950
2010-11-09 05:12:29,818 INFO org.apache.hadoop.hdfs.DFSClient: Waiting
to find target node: 10.11.87.61:50010
2010-11-09 05:13:35,949 INFO org.apache.hadoop.hdfs.DFSClient:
Exception in createBlockOutputStream java.io.IOException: Bad connect
ack with firstBadLink 10.11.87.65:50010
2010-11-09 05:13:35,949 INFO org.apache.hadoop.hdfs.DFSClient:
Abandoning block blk_3138002906377068146_775950
2010-11-09 05:13:35,950 INFO org.apache.hadoop.hdfs.DFSClient: Waiting
to find target node: 10.11.87.61:50010
2010-11-09 05:13:51,757 WARN org.apache.hadoop.hdfs.DFSClient:
DataStreamer Exception: java.io.IOException: Unable to create new
block.
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2812)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)

2010-11-09 05:13:51,757 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_3138002906377068146_775950 bad datanode[2]
nodes == null
2010-11-09 05:13:51,758 WARN org.apache.hadoop.hdfs.DFSClient: Could
not get block locations. Source file
"/user/hadoop/requests_logs/merged/2010-11-08/_temporary/_attempt_201011081008_0002_r_000008_0/part-r-00008"
- Aborting...
2010-11-09 05:13:51,760 WARN org.apache.hadoop.mapred.TaskTracker:
Error running child
java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2870)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2793)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
2010-11-09 05:13:54,888 INFO org.apache.hadoop.mapred.TaskRunner:
Runnning cleanup for the task
Thanks

Oleg.

------------------------------