Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> Fwd: Hbase region server disconnecting with master after some time.


Copy link to this message
-
Fwd: Hbase region server disconnecting with master after some time.
Reply Inline..

---------- Forwarded message ----------
From: Vamshi Krishna <[EMAIL PROTECTED]>
Date: Wed, Aug 21, 2013 at 10:42 PM
Subject: Hbase region server disconnecting with master after some time.
To: [EMAIL PROTECTED]
Hi all,
 Facing problem with Hbase region server disconnecting with master after
some time. I set up Hbase cluster with 2 machines where Machine-1 (M1) is
master and Region server and M2 is only Region server.
After running hbase-start.sh , all the daemons are started perfectly but
after some time i see M2 region server is dead. I am running zookeeper on
M1 alone.

The error i found is M2 region server log is pasted below.

2013-08-22 10:31:38,554 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=vamshi_RS:2181 sessionTimeout=180000
watcher=regionserver:60020
*Anil: Above line means that RS & ZK and were unable to communicate for 3
minutes(180 sec). Hence RS is deemed dead by ZK. It seems like you have
some networking/firewall problem. *
2013-08-22 10:31:38,564 INFO
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of
this process is 4076@vamshi
2013-08-22 10:31:38,568 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server vamshi_RS/192.168.1.57:2181. Will not attempt
to authenticate using SASL (Unable to locate a login configuration)
2013-08-22 10:32:41,675 WARN org.apache.zookeeper.ClientCnxn: Session 0x0
for server null, unexpected error, closing socket connection and attempting
reconnect
java.net.ConnectException: Connection timed out
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:599)
    at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
2013-08-22 10:32:41,791 WARN
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient
ZooKeeper exception:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
2013-08-22 10:32:41,791 INFO org.apache.hadoop.hbase.util.RetryCounter:
Sleeping 2000ms before retry #1...
2013-08-22 10:32:42,789 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server vamshi_RS/192.168.1.57:2181. Will not attempt
to authenticate using SASL (Unable to locate a login configuration)
2013-08-22 10:33:45,929 WARN org.apache.zookeeper.ClientCnxn: Session 0x0
for server null, unexpected error, closing socket connection and attempting
reconnect
.
.
..
2013-08-22 10:35:54,542 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
regionserver:60020 Unable to set watcher on znode /hbase/master
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/master
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
    at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:172)
    at
org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:420)
    at
org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.start(ZooKeeperNodeTracker.java:76)
    at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:648)
    at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:609)
    at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:735)
    at java.lang.Thread.run(Thread.java:662)
.
..
2013-08-22 10:35:57,549 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
vamshi,60020,1377147698472: Initialization of RS failed.  Hence aborting RS.
java.io.IOException: Received the shutdown message while waiting.
    at
org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:680)
    at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:649)
    at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:609)
    at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:735)
    at java.lang.Thread.run(Thread.java:662)
2013-08-22 10:35:57,550 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort:
loaded coprocessors are: []
2013-08-22 10:35:57,550 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Initialization
of RS failed.  Hence aborting RS.
2013-08-22 10:35:57,552 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Registered RegionServer
MXBean
2013-08-22 10:35:57,553 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
vamshi,60020,1377147698472: Unhandled exception: null
java.lang.NullPointerException
    at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:756)
    at java.lang.Thread.run(Thread.java:662)
2013-08-22 10:35:57,553 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort:
loaded coprocessors are: []
2013-08-22 10:35:57,553 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unhandled
exception: null
2013-08-22 10:35:57,554 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting;
hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-5,5,main]
2013-08-22 10:35:57,554 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown hook
2013-08-22 10:35:57,555 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown
hook thread.
2013-08-22 10:35:57,556 INFO
org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook finished..
hbase-site.xml :

Please find my hbase-site.xml content which which is same in both M1 and M2.

<property>
        <name>hbase.rootdir</name>

<!--value>hdfs://vam
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB