Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - HMaster fails to start up, Failed construction of Master exception


Copy link to this message
-
Re: HMaster fails to start up, Failed construction of Master exception
Nichole Treadway 2011-03-11, 19:38
Alright, I think I've got it working now. I increased the HBASE_HEAPSIZE
value in hbase-env.sh and the HMaster finally started up and it looks like
its working as normal now.

I'm not really sure what caused this problem in the first place though since
I've never encountered this problem before.

My cells aren't fat but my table is very large, ~400 columns, two column
families.

Thank you for your help.

On Fri, Mar 11, 2011 at 2:28 PM, Nichole Treadway <[EMAIL PROTECTED]>wrote:

> Sorry for not including that information in my original email.
>
> Cluster Info:
> I'm running the hadoop-0.20-append branch and HBase 0.90.1, and java 1.6.
> All machines are 64-bit running Red Hat 5.5.
>
> I have a small cluster of 4 nodes all acting as datanodes and
> regionservers. Replication in my cluster is set to 3.
>
> As an update, I removed all regionservers except my master from the
> regionservers list and from the zookeeper quorum list in hbase-site.xml. I
> started up HBase again and was no longer seeing the "Failed Construction of
> Master" errors I mentioned in my previous email. HMaster started up more
> normally this time and began reading HLog files. It then printed a message
> about not being able to contact some of my regionservers and quit again.
>
> I added all the regionservers back again to the regionservers list and the
> zookeeper qurom list. Now the master starts up, spends several minutes
> printing messages about HLog files, and then fails again with the following
> error:
>
> 2011-03-11 14:17:58,197 FATAL org.apache.hadoop.hbase.master.HMaster:
> Unhandled exception. Starting shutdown.
> java.lang.OutOfMemoryError: Java heap space
> at org.apache.hadoop.hbase.KeyValue.readFields(KeyValue.java:1970)
>  at org.apache.hadoop.hbase.KeyValue.readFields(KeyValue.java:1977)
> at
> org.apache.hadoop.hbase.regionserver.wal.WALEdit.readFields(WALEdit.java:118)
>  at
> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1758)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1886)
>  at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:198)
> at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.next(SequenceFileLogReader.java:172)
>  at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.parseHLog(HLogSplitter.java:429)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:262)
>  at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLog(HLogSplitter.java:188)
> at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:196)
>  at
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAfterStartup(MasterFileSystem.java:180)
> at
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:379)
>  at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:278)
>
> On Fri, Mar 11, 2011 at 1:20 PM, Jean-Daniel Cryans <[EMAIL PROTECTED]>wrote:
>
>> Please include relevant basic information when asking that sort of
>> question, such as hbase/hadoop version, hardware, OS, java version,
>> cluster setup, etc.
>>
>> The exceptions seems to indicate that it's having a hard time getting
>> data from zookeeper? Have you checked the zookeeper log(s)?
>>
>> Maybe that's a red herring tho, but without any context those lines of
>> log could mean anything.
>>
>> J-D
>>
>> On Fri, Mar 11, 2011 at 8:49 AM, Nichole Treadway <[EMAIL PROTECTED]>
>> wrote:
>> > Last night I was putting pretty heavy load on my HBase cluster. One of
>> the
>> > region servers shut down unexpectedly, and I restarted the regionserver,
>> but
>> > HBase still wasn't assigning regions to it. I attempted to move regions
>> > using the HBase shell but regions were still not being assigned to it.
>> In
>> > the past when this has happened, I've just restarted HBase and it's been
>> > fine. I attempted to do this, but now HBase is failing to start up at
>> all.
>> >
>> > In my HMaster logs, here's the message I'm getting.