Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HMaster fails to start up, Failed construction of Master exception

Copy link to this message
Re: HMaster fails to start up, Failed construction of Master exception
Sorry for not including that information in my original email.

Cluster Info:
I'm running the hadoop-0.20-append branch and HBase 0.90.1, and java 1.6.
All machines are 64-bit running Red Hat 5.5.

I have a small cluster of 4 nodes all acting as datanodes and regionservers.
Replication in my cluster is set to 3.

As an update, I removed all regionservers except my master from the
regionservers list and from the zookeeper quorum list in hbase-site.xml. I
started up HBase again and was no longer seeing the "Failed Construction of
Master" errors I mentioned in my previous email. HMaster started up more
normally this time and began reading HLog files. It then printed a message
about not being able to contact some of my regionservers and quit again.

I added all the regionservers back again to the regionservers list and the
zookeeper qurom list. Now the master starts up, spends several minutes
printing messages about HLog files, and then fails again with the following

2011-03-11 14:17:58,197 FATAL org.apache.hadoop.hbase.master.HMaster:
Unhandled exception. Starting shutdown.
java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.hbase.KeyValue.readFields(KeyValue.java:1970)
 at org.apache.hadoop.hbase.KeyValue.readFields(KeyValue.java:1977)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1886)
 at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:278)

On Fri, Mar 11, 2011 at 1:20 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:

> Please include relevant basic information when asking that sort of
> question, such as hbase/hadoop version, hardware, OS, java version,
> cluster setup, etc.
> The exceptions seems to indicate that it's having a hard time getting
> data from zookeeper? Have you checked the zookeeper log(s)?
> Maybe that's a red herring tho, but without any context those lines of
> log could mean anything.
> J-D
> On Fri, Mar 11, 2011 at 8:49 AM, Nichole Treadway <[EMAIL PROTECTED]>
> wrote:
> > Last night I was putting pretty heavy load on my HBase cluster. One of
> the
> > region servers shut down unexpectedly, and I restarted the regionserver,
> but
> > HBase still wasn't assigning regions to it. I attempted to move regions
> > using the HBase shell but regions were still not being assigned to it. In
> > the past when this has happened, I've just restarted HBase and it's been
> > fine. I attempted to do this, but now HBase is failing to start up at
> all.
> >
> > In my HMaster logs, here's the message I'm getting.
> >
> > 2011-03-11 11:30:51,014 INFO org.apache.zookeeper.ClientCnxn: Socket
> > connection established to myip1/myip1:2181, initiating session
> >
> > 2011-03-11 11:31:04,004 INFO org.apache.zookeeper.ClientCnxn: Unable to
> read
> > additional data from server sessionid 0x0, likely server has closed
> socket,
> > closing socket connection and attempting reconnect
> >
> > 2011-03-11 11:31:04,107 ERROR
> > org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
> >
> > java.lang.RuntimeException: Failed construction of Master: class
> > org.apache.hadoop.hbase.master.HMaster