Hi , running hadoop map/reduce got such exception?

  1) Why does it happen?

  2) Job didn't failed and continue it's execution? Does this
exception cause losing data or map/reduce

uses recovery mechanism?

2010-11-09 05:10:08,735 INFO org.apache.hadoop.hdfs.DFSClient:
Exception in createBlockOutputStream java.io.IOException: Bad connect
ack with firstBadLink 10.11.87.65:50010
2010-11-09 05:10:08,735 INFO org.apache.hadoop.hdfs.DFSClient:
Abandoning block blk_-9208177033562590356_775948
2010-11-09 05:10:08,739 INFO org.apache.hadoop.hdfs.DFSClient: Waiting
to find target node: 10.11.87.61:50010
2010-11-09 05:11:23,743 INFO org.apache.hadoop.hdfs.DFSClient:
Exception in createBlockOutputStream java.net.SocketTimeoutException:
69000 millis timeout while waiting for channel to be ready for read.
ch : java.nio.channels.SocketChannel[connected
local=/10.11.87.61:55309 remote=/10.11.87.61:50010]
2010-11-09 05:11:23,743 INFO org.apache.hadoop.hdfs.DFSClient:
Abandoning block blk_-10251707095594311_775950
2010-11-09 05:11:23,744 INFO org.apache.hadoop.hdfs.DFSClient: Waiting
to find target node: 10.11.87.61:50010
2010-11-09 05:12:29,815 INFO org.apache.hadoop.hdfs.DFSClient:
Exception in createBlockOutputStream java.io.IOException: Bad connect
ack with firstBadLink 10.11.87.65:50010
2010-11-09 05:12:29,816 INFO org.apache.hadoop.hdfs.DFSClient:
Abandoning block blk_3509928762116143133_775950
2010-11-09 05:12:29,818 INFO org.apache.hadoop.hdfs.DFSClient: Waiting
to find target node: 10.11.87.61:50010
2010-11-09 05:13:35,949 INFO org.apache.hadoop.hdfs.DFSClient:
Exception in createBlockOutputStream java.io.IOException: Bad connect
ack with firstBadLink 10.11.87.65:50010
2010-11-09 05:13:35,949 INFO org.apache.hadoop.hdfs.DFSClient:
Abandoning block blk_3138002906377068146_775950
2010-11-09 05:13:35,950 INFO org.apache.hadoop.hdfs.DFSClient: Waiting
to find target node: 10.11.87.61:50010
2010-11-09 05:13:51,757 WARN org.apache.hadoop.hdfs.DFSClient:
DataStreamer Exception: java.io.IOException: Unable to create new
block.
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2812)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)

2010-11-09 05:13:51,757 WARN org.apache.hadoop.hdfs.DFSClient: Error
Recovery for block blk_3138002906377068146_775950 bad datanode[2]
nodes == null
2010-11-09 05:13:51,758 WARN org.apache.hadoop.hdfs.DFSClient: Could
not get block locations. Source file
"/user/hadoop/requests_logs/merged/2010-11-08/_temporary/_attempt_201011081008_0002_r_000008_0/part-r-00008"
- Aborting...
2010-11-09 05:13:51,760 WARN org.apache.hadoop.mapred.TaskTracker:
Error running child
java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2870)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2793)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
2010-11-09 05:13:54,888 INFO org.apache.hadoop.mapred.TaskRunner:
Runnning cleanup for the task
Thanks

Oleg.

------------------------------

What does it mean:   It looks like second attempt to process data after the
first time it failed?
All Task AttemptsTask AttemptsMachineStatusProgressStart TimeShuffle
FinishedSort FinishedFinish TimeErrorsTask LogsCountersActions
attempt_201011081008_0002_r_000008_0/default-rack/<http://hadoop1.infolinks.local:8022>
FAILED0.00%9-Nov-2010 04:36:159-Nov-2010 05:09:08 (32mins, 52sec)9-Nov-2010
05:09:08 (0sec)9-Nov-2010 05:14:07 (37mins, 51sec)

java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2870)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2793)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)

Last 4KB<http://hadoop1.infolinks.local:8022/tasklog?taskid=attempt_201011081008_0002_r_000008_0&start=-4097>
Last 8KB<http://hadoop1.infolinks.local:8022/tasklog?taskid=attempt_201011081008_0002_r_000008_0&start=-8193>
All<http://hadoop1.infolinks.local:8022/tasklog?taskid=attempt_201011081008_0002_r_000008_0&all=true>
10</taskstats.jsp?jobid=job_201011081008_0002&tipid=task_201011081008_0002_r_000008&taskid=attempt_201011081008_0002_r_000008_0>

attempt_201011081008_0002_r_000008_1/default-rack/<http://hadoop-transfer.infolinks.local:8022>
RUNNING24.98%9-Nov-2010 05:50:21
On Tue, Nov 9, 2010 at 12:58 PM, Oleg Ruchovets <[EMAIL PROTECTED]>wrote:

> Hi , running hadoop map/reduce got such exception?
>
>   1) Why does it happen?
>
>   2) Job didn't failed and continue it's execution? Does this exception cause losing data or map/reduce
>
> uses recovery mechanism?
>
>
>
> 2010-11-09 05:10:08,735 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010
> 2010-11-09 05:10:08,735 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-9208177033562590356_775948
> 2010-11-09 05:10:08,739 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target node: 10.11.87.61:50010
> 2010-11-09 05:11:23,743 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.net.SocketTimeoutException: 69000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.11.87.61:55309 remote=/10.11.87.61:50010]
> 2010-11-09 05:11:23,743 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-10251707095594311_775950
> 2010-11-09 05:11:23,744 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target node: 10.11.87.61:50010
> 2010-11-09 05:12:29,815 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010
> 2010-11-09 05:12:29,816 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_3509928762116143133_775950
> 2010-11-09 05:12:29,818 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target node: 10.11.87.61:50010
> 2010-11-09 05:13:35,949 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Bad connect ack with firstBadLink 10.11.87.65:50010
> 2010-11-09 05:13:35,949 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_3138002906377068146_775950
> 2010-11-09 05:13:35,950 INFO org.apache.hadoop.hdfs.DFSClient: Waiting to find target node: 10.11.87.61:50010
> 2010-11-09 05:13:51,757 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Unable to create new block.
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2812)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:2076)
> at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2262)
>
> 2010-11-09 05:13:51,757 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_3138002906377068146_775950 bad datanode[2] nodes == null