Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> HBASE -- HMaster Aborts after 28 minutes.


Copy link to this message
-
Re: HBASE -- HMaster Aborts after 28 minutes.
Session expired usually results from a long GC that exceeds the
zookeeper.session.timeout.

> 2012-07-01 18:20:00,961 FATAL org.apache.hadoop.hbase.master.HMaster:master:60000-0x238444cf77e0000 master:60000-0x238444cf77e0000 received
expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException:KeeperErrorCode
= Session expired
See http://hbase.apache.org/book.html#trouble.log.gc

--Suraj
On Sun, Jul 1, 2012 at 6:34 PM, Jay Wilson
<[EMAIL PROTECTED]> wrote:
> I assume a timer may need to be increased based on these messages.  Any
> thoughts as to which one or is it something else?
>
> 2012-07-01 18:17:16,461 INFO
> org.apache.hadoop.hbase.master.LoadBalancer: Skipping load balancing.
> servers=3 regions=2 average=0.6666667 mostloaded=1 leastloaded=0
> 2012-07-01 18:17:16,466 DEBUG
> org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 0 catalog row(s)
> and gc'd 0 unreferenced parent region(s)
> 2012-07-01 18:20:00,281 INFO org.apache.zookeeper.ClientCnxn: Unable to
> read additional data from server sessionid 0x38444cf77f0000, likely
> server has closed socket, closing socket connection and attempting reconnect
> 2012-07-01 18:20:00,281 INFO org.apache.zookeeper.ClientCnxn: Unable to
> read additional data from server sessionid 0x238444cf77e0000, likely
> server has closed socket, closing socket connection and attempting reconnect
> 2012-07-01 18:20:00,384 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil:
> hconnection-0x38444cf77f0000 Unable to set watcher on znode /hbase/master
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:815)
>         at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232)
>         at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.nodeDeleted(ZooKeeperNodeTracker.java:178)
>         at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:294)
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> 2012-07-01 18:20:00,386 ERROR
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher:
> hconnection-0x38444cf77f0000 Received unexpected KeeperException,
> re-throwing exception
> org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase/master
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
>         at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
>         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:815)
>         at
> org.apache.hadoop.hbase.zookeeper.ZKUtil.watchAndCheckExists(ZKUtil.java:232)
>         at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.nodeDeleted(ZooKeeperNodeTracker.java:178)
>         at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:294)
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> 2012-07-01 18:20:00,387 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> This client just lost it's session with ZooKeeper, trying to reconnect.
> 2012-07-01 18:20:00,387 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Trying to reconnect to zookeeper.
> 2012-07-01 18:20:00,957 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server devrackA-05/172.18.0.6:2181
> 2012-07-01 18:20:00,958 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to devrackA-05/172.18.0.6:2181, initiating session
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB