Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Namenode won't start after unclean shutdown of RS


Copy link to this message
-
Re: Namenode won't start after unclean shutdown of RS
What version of Hadoop / HBase are you using ?

Cheers
On Sat, Dec 7, 2013 at 9:38 AM, iain wright <[EMAIL PROTECTED]> wrote:

> Hi folks,
>
> One of our RS/DN/TT nodes went down dirty (kernel panic). Users contacted
> about reports failing, and i saw wierd logs in the jobtracker for "pending
> shutdown" etc.
>
> Proceeded to stop jobtracker/tt nodes, hbase, and hdfs.
>
> On attempting to turn back on the NN + DN's our namenode is failing to
> start, complaining about being unable to move one of our hbase region files
> because source file does exist.
>
> Has anyone come across this/have suggestions??
>
> 2013-12-07 01:20:43,156 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Image file of size 149098688 loaded in 14 seconds.
> 2013-12-07 01:20:43,253 WARN org.apache.hadoop.hdfs.StateChange: DIR*
> FSDirectory.unprotectedRenameTo: failed to rename
>
> /hbase_prod/prod_transactions/d3fdbd9b9883b2f8efe6d0f8b7790f41/c/df40b2db7d9345ab9384a77a2c573de0
> to
>
> /hbase_prod/.archive/prod_transactions/d3fdbd9b9883b2f8efe6d0f8b7790f41/c/df40b2db7d9345ab9384a77a2c573de0
> because source does not exist
> 2013-12-07 01:20:43,348 ERROR
> org.apache.hadoop.hdfs.server.namenode.NameNode:
> java.lang.NullPointerException
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetTimes(FSDirectory.java:1343)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.unprotectedSetTimes(FSDirectory.java:1332)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.loadFSEdits(FSEditLog.java:774)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSEdits(FSImage.java:1008)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:826)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:362)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:97)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:386)
> at
>
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:360)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:276)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:496)
> at
>
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1279)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1288)
>
> 2013-12-07 01:20:43,349 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at
> namenode0.hadoop.compe.tv/10.10.11.10
>
> Thank you,
>
> --
> iain
>
> <http://www.labctsi.org/>
> This email message is confidential, intended only for the recipient(s)
> named above and may contain information that is privileged, exempt from
> disclosure under applicable law. If you are not the intended recipient, do
> not disclose or disseminate the message to anyone except the intended
> recipient. If you have received this message in error, or are not the named
> recipient(s), please immediately notify the sender by return email, and
> delete all copies of this message.
>
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB