Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HMaster fails to start up, Failed construction of Master exception


Copy link to this message
-
HMaster fails to start up, Failed construction of Master exception
Last night I was putting pretty heavy load on my HBase cluster. One of the
region servers shut down unexpectedly, and I restarted the regionserver, but
HBase still wasn't assigning regions to it. I attempted to move regions
using the HBase shell but regions were still not being assigned to it. In
the past when this has happened, I've just restarted HBase and it's been
fine. I attempted to do this, but now HBase is failing to start up at all.

In my HMaster logs, here's the message I'm getting.

2011-03-11 11:30:51,014 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to myip1/myip1:2181, initiating session

2011-03-11 11:31:04,004 INFO org.apache.zookeeper.ClientCnxn: Unable to read
additional data from server sessionid 0x0, likely server has closed socket,
closing socket connection and attempting reconnect

2011-03-11 11:31:04,107 ERROR
org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master

java.lang.RuntimeException: Failed construction of Master: class
org.apache.hadoop.hbase.master.HMaster

        at
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1064)

        at
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:142)
        at
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:102)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

        at
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1078)

Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase

        at
org.apache.zookeeper.KeeperException.create(KeeperException.java:90)

        at
org.apache.zookeeper.KeeperException.create(KeeperException.java:42)

        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:637)

        at
org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:902)
        at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
        at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:218)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)

        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)

        at
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1059)

        ... 5 more
-------------------
Errors I'm seeing in the Zookeeper logs:
2011-03-11 11:30:47,479 WARN org.apache.zookeeper.server.quorum.Learner:
Unexpected exception, tries=0, connecting to /myip:2888

java.net.ConnectException: Connection refused

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)

        at
java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)

        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

        at java.net.Socket.connect(Socket.java:529)

        at
org.apache.zookeeper.server.quorum.Learner.connectToLeader(Learner.java:212)
        at
org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:65)

        at
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:644)

2011-03-11 11:32:37,091 WARN
org.apache.zookeeper.server.quorum.QuorumCnxManager: Interrupted while
waiting for message on queue java.lang.InterruptedException

        at
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1961)

        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2038)

        at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:342)

        at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:601)

2011-03-11 11:32:18,671 ERROR
org.apache.zookeeper.server.quorum.QuorumCnxManager: Failed to send last
message. Shutting down thread.java.nio.channels.AsynchronousCloseException

        at
java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)

        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:341)

        at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.send(QuorumCnxManager.java:579)

        at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:588
)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB