|
|
-
IOException in offerServiceYanbo Liang 2012-11-13, 10:51
DataNode may received IOException in heartbeat sending process. When
running TestStartup, I got the following warn log: 2012-11-13 01:28:19,217 WARN datanode.DataNode (BPServiceActor.java:offerService(596)) - IOException in offerService java.io.IOException: Failed on local exception: java.io.IOException: Response is null.; Host Details : local host is: "orange/127.0.0.1"; destination host is: "orange":38705; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:759) at org.apache.hadoop.ipc.Client.call(Client.java:1163) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202) at $Proxy14.sendHeartbeat(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:163) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:82) at $Proxy14.sendHeartbeat(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:167) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:445) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:525) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:676) at java.lang.Thread.run(Thread.java:722) Caused by: java.io.IOException: Response is null. at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:884) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:812) >From the log, I can infer that DN send heartbeat to NN and got response is null. But the workflow of process the response at DN just log it and without other action. I wonder which caused this result? The NameNode has no DatanodeCommand to send for this DataNode, protocolbuffer related problem or other reason? Is it make sense that the DN just log this Exception and without other action? Thanks, Yanbo |