Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBASE -- HMaster Aborts after 28 minutes.


Copy link to this message
-
Re: HBASE -- HMaster Aborts after 28 minutes.
Session expired usually results from a long GC that exceeds the
zookeeper.session.timeout.

> 2012-07-01 18:20:00,961 FATAL org.apache.hadoop.hbase.master.HMaster:master:60000-0x238444cf77e0000 master:60000-0x238444cf77e0000 received
expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException:KeeperErrorCode
= Session expired
See http://hbase.apache.org/book.html#trouble.log.gc

--Suraj
On Sun, Jul 1, 2012 at 6:34 PM, Jay Wilson
<[EMAIL PROTECTED]> wrote:
> I assume a timer may need to be increased based on these messages.  Any
> thoughts as to which one or is it something else?
>
> 2012-07-01 18:17:16,461 INFO
> org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.
> servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=0
> 2012-07-01 18:17:16,466 DEBUG
> org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s)
> and gc'd 0 unreferenced parent region(s)
> 2012-07-01 18:20:00,281 INFO org.apache.zookeeper.ClientCnxn: Unable to
> read additional data from server sessionid 0x38444cf77f0000, likely
> server has closed socket, closing socket connection and attempting reconnect
> 2012-07-01 18:20:00,281 INFO org.apache.zookeeper.ClientCnxn: Unable to
> read additional data from server sessionid 0x238444cf77e0000, likely
> server has closed socket, closing socket connection and attempting reconnect
> 2012-07-01 18:20:00,384 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> hconnection-0x38444cf77f0000 Unable to set watcher on znode /hbase/master
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:815)
>         at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232)
>         at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.nodeDeleted(ZooKeeperNodeTracker.java:178)
>         at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:294)
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> 2012-07-01 18:20:00,386 ERROR
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher:
> hconnection-0x38444cf77f0000 Received unexpected KeeperException,
> re-throwing exception
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:815)
>         at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232)
>         at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.nodeDeleted(ZooKeeperNodeTracker.java:178)
>         at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:294)
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> 2012-07-01 18:20:00,387 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> This client just lost it's session with ZooKeeper, trying to reconnect.
> 2012-07-01 18:20:00,387 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Trying to reconnect to zookeeper.
> 2012-07-01 18:20:00,957 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server devrackA-05/172.18.0.6:2181
> 2012-07-01 18:20:00,958 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to devrackA-05/172.18.0.6:2181, initiating session