Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HDFS >> mail # user >> Exceptions in Name node and Data node logs


Copy link to this message
-
Re: Exceptions in Name node and Data node logs
Hi,

One of your DN is marked as dead because NN is not able to get heartbeat
message from DN but NN still getting block information from dead node. This
error is similar to a bug *HDFS-1250* reported 2 years back and fixed in
0.20 release.

Can you please check the status of DN's in cluster.

#bin/hadoop dfsadmin -report

Thanks
On Tue, Aug 13, 2013 at 1:53 PM, Vimal Jain <[EMAIL PROTECTED]> wrote:

>   Hi,
> I have configured Hadoop and Hbase in pseudo distributed mode.
> So far things were working fine , but suddenly i started receiving some
> exceptions in my namenode and datanode log files.
> It keeps repeating and thus fills up my disk space.
> Please help here.
>
> *Exception in data node :-*
>
> 2013-07-31 19:39:51,094 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: Got blockRec
> eived message from unregistered or dead node blk_-4787262105551508952_28369
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>     at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
>     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387)
>
>     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>     at sun.proxy.$Proxy5.blockReceived(Unknown Source)
>     at
> org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:1006)
>     at
> org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1527)
>     at java.lang.Thread.run(Thread.java:662)
>
> *Exception in name node :- *
>
> 2013-07-31 19:39:50,671 WARN org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.blockReceived: blk_-4787262105551508952_28369 is received from
> dead
>  or unregistered node 192.168.20.30:50010
> 2013-07-31 19:39:50,671 ERROR
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
> as:hadoop cause:java.io.IOException: Got blo
> ckReceived message from unregistered or dead node
> blk_-4787262105551508952_28369
> 2013-07-31 19:39:50,671 INFO org.apache.hadoop.ipc.Server: IPC Server
> handler 5 on 9000, call blockReceived(DatanodeRegistration(
> 192.168.20.30:50010,
> storageID=DS-1816106352-192.168.20.30-50010-1369314076237, infoPort=50075,
> ipcPort=50020), [Lorg.apache.hadoop.hdfs.protocol.Block;@64f2d559,
> [Ljava.l
> ang.String;@294f9d6) from 192.168.20.30:59764: error:
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551
> 508952_28369
> java.io.IOException: Got blockReceived message from unregistered or dead
> node blk_-4787262105551508952_28369
>     at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.blockReceived(FSNamesystem.java:4188)
>     at
> org.apache.hadoop.hdfs.server.namenode.NameNode.blockReceived(NameNode.java:1069)
>     at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393)
>     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389)
>     at java.security.AccessController.doPrivileged(Native Method)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB