Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
MapReduce, mail # user - Strange error on Datanodes


Copy link to this message
-
RE: Strange error on Datanodes
Siddharth Tiwari 2013-12-02, 15:55
Here are few more logs and confs Jeet
Thank you so much for help

*------------------------*

Cheers !!!

Siddharth Tiwari

Have a refreshing day !!!
"Every duty is holy, and devotion to duty is the highest form of worship of God.”

"Maybe other people will try to limit me but I don't limit myself"
Date: Mon, 2 Dec 2013 21:09:03 +0530
Subject: Re: Strange error on Datanodes
From: [EMAIL PROTECTED]
To: [EMAIL PROTECTED]

Hi,
Can you share some more logs from Data nodes? could you please also share the conf and cluster size?
RegardsJitendra  

On Mon, Dec 2, 2013 at 8:49 PM, Siddharth Tiwari <[EMAIL PROTECTED]> wrote:
Hi team
I see following errors on datanodes. What is the reason for this and how can it will be resolved:-

2013-12-02 13:11:36,441 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception  for block BP-1854340821-10.238.9.151-1385733655875:blk_-2927699636194035560_63092
java.net.SocketTimeoutException: 65000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.238.10.43:54040 remote=/10.238.10.43:50010]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:165)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:117)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:169)
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:114)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:694)
2013-12-02 13:12:06,572 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-12-02 13:12:06,581 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hadoop (auth:SIMPLE) cause:java.io.IOException: All datanodes 10.238.10.43:50010 are bad. Aborting...
2013-12-02 13:12:06,581 WARN org.apache.hadoop.mapred.Child: Error running child
java.io.IOException: All datanodes 10.238.10.43:50010 are bad. Aborting...
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:959)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:779)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:448)
2013-12-02 13:12:06,583 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task

*------------------------*

Cheers !!!

Siddharth Tiwari

Have a refreshing day !!!
"Every duty is holy, and devotion to duty is the highest form of worship of God.”
"Maybe other people will try to limit me but I don't limit myself"