Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - Namenode won't start after unclean shutdown of RS


Copy link to this message
-
Re: Namenode won't start after unclean shutdown of RS
Ted Yu 2013-12-07, 07:26
What version of Hadoop / HBase are you using ?

Cheers
On Sat, Dec 7, 2013 at 9:38 AM, iain wright <[EMAIL PROTECTED]> wrote:

> Hi folks,
>
> One of our RS/DN/TT nodes went down dirty (kernel panic). Users contacted
> about reports failing, and i saw wierd logs in the jobtracker for "pending
> shutdown" etc.
>
> Proceeded to stop jobtracker/tt nodes, hbase, and hdfs.
>
> On attempting to turn back on the NN + DN's our namenode is failing to
> start, complaining about being unable to move one of our hbase region files
> because source file does exist.
>
> Has anyone come across this/have suggestions??
>
> 2013-12-07 01:20:43,156 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Image file of size 149098688 loaded in 14 seconds.
> 2013-12-07 01:20:43,253 WARN org.apache.hadoop.hdfs.StateChange: DIR*
> FSDirectory.unprotectedRenameTo: failed to rename
>
> /hbase_prod/prod_transactions/d3fdbd9b9883b2f8efe6d0f8b7790f41/c/df40b2db7d9345ab9384a77a2c573de0
> to
>
> /hbase_prod/.archive/prod_transactions/d3fdbd9b9883b2f8efe6d0f8b7790f41/c/df40b2db7d9345ab9384a77a2c573de0
> because source does not exist
> 2013-12-07 01:20:43,348 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode:
> java.lang.NullPointerException
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetTimes(FSDirectory.java:1343)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetTimes(FSDirectory.java:1332)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:774)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1008)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:826)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:362)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:97)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:386)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:360)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
>
> 2013-12-07 01:20:43,349 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at
> namenode0.hadoop.compe.tv/10.10.11.10
>
> Thank you,
>
> --
> iain
>
> <http://www.labctsi.org/>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>