Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Plain View
HBase >> mail # user >> hbase-master-server slept


+
So Hibino 2013-02-08, 08:55
Copy link to this message
-
Re: hbase-master-server slept
Regards, So,
Can you provide more information about your setup?
- HBase version
- Hadoop version
- Operating System
- Java version

On 02/08/2013 03:55 AM, So Hibino wrote:
> Our hbase-master-server was shutdown with following message.
> Hbase is runnig in Distributed mode in a single node.
Can you share your .conf files?
> I checked that GC completed in a very short time at the time of output the
> WARN.
> In addition the other system that is running in the same architecture
> doesn't output the following WARN messsage and works well.
> So I think that this is not due to a long GC pause.
>
> Do you have any idea about the problem?
>
> 2013-01-30 03:07:48,582 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
> 28970ms instead of 1000ms, this is likely due to a long garbage collecting
> pause and it's usually bad, see
> http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
Did you check the link?
Todd wrote a series of posts in Cloudera�s blog about Java Long GC
pauses, HBase and Zookeeper.
It�s a great read:
http://www.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-1/
http://www.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-2/
> 2013-01-30 03:07:48,583 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
> 36902ms instead of 10000ms, this is likely due to a long garbage collecting
> pause and it's usually bad, see
> http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
> 2013-01-30 03:07:48,585 INFO org.apache.zookeeper.ClientCnxn: Client session
> timed out, have not heard from server in 39989ms for sessionid
> 0x13c84cebfce0000, closing socket connection and attempting reconnect
> 2013-01-30 03:07:48,586 INFO org.apache.zookeeper.ClientCnxn: Client session
> timed out, have not heard from server in 39987ms for sessionid
> 0x13c84cebfce0001, closing socket connection and attempting reconnect
> 2013-01-30 03:07:52,779 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> connection to server VM_11/192.168.152.1:2181
> 2013-01-30 03:07:52,789 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to VM_11/192.168.152.1:2181, initiating session
> 2013-01-30 03:07:52,777 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> connection to server VM_11/192.168.152.1:2181
> 2013-01-30 03:07:52,793 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to VM_11/192.168.152.1:2181, initiating session
> 2013-01-30 03:07:52,794 INFO org.apache.zookeeper.ClientCnxn: Unable to
> reconnect to ZooKeeper service, session 0x13c84cebfce0001 has expired,
> closing socket connection
> 2013-01-30 03:07:52,794 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> This client just lost it's session with ZooKeeper, trying to reconnect.
> 2013-01-30 03:07:52,794 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Trying to reconnect to zookeeper.
> 2013-01-30 03:07:52,795 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, connectString=VM_11:2181 sessionTimeout=180000
> watcher=hconnection
> 2013-01-30 03:07:52,812 INFO org.apache.zookeeper.ClientCnxn: Unable to
> reconnect to ZooKeeper service, session 0x13c84cebfce0000 has expired,
> closing socket connection
> 2013-01-30 03:07:52,813 FATAL org.apache.hadoop.hbase.master.HMaster:
> master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000
> master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000
> received expired from ZooKeeper, aborting
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:361)

Marcos Ortiz Valmaseda,
Product Manager && Data Scientist at UCI
Blog: http://marcosluis2186.posterous.com
Twitter: @marcosluis2186 <http://twitter.com/marcosluis2186>
+
Jean-Daniel Cryans 2013-02-08, 17:47
+
Ted Yu 2013-02-08, 18:48
+
So Hibino 2013-02-12, 04:06
+
Marcos Ortiz Valmaseda 2013-02-12, 04:35
+
So Hibino 2013-02-12, 06:33
+
Jean-Daniel Cryans 2013-02-12, 18:59
+
So Hibino 2013-02-13, 00:10
+
Jean-Daniel Cryans 2013-02-13, 00:31
+
So Hibino 2013-02-14, 06:16
+
Michel Segel 2013-02-14, 12:41
+
So Hibino 2013-02-14, 06:10
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB