Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> hbase-master-server slept


Copy link to this message
-
Re: hbase-master-server slept
What zookeeper version are you using ?
Is the ensemble managed by HBase ?

Can you check the zookeeper log on 192.168.152.1<http://192.168.152.1:2181/>
 ?
Use pastebin to show us the log if necessary.

Thanks

On Fri, Feb 8, 2013 at 12:55 AM, So Hibino <[EMAIL PROTECTED]> wrote:

> Our hbase-master-server was shutdown with following message.
> Hbase is runnig in Distributed mode in a single node.
> I checked that GC completed in a very short time at the time of output the
> WARN.
> In addition the other system that is running in the same architecture
> doesn't output the following WARN messsage and works well.
> So I think that this is not due to a long GC pause.
>
> Do you have any idea about the problem?
>
> 2013-01-30 03:07:48,582 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
> 28970ms instead of 1000ms, this is likely due to a long garbage collecting
> pause and it's usually bad, see
> http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
> 2013-01-30 03:07:48,583 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
> 36902ms instead of 10000ms, this is likely due to a long garbage collecting
> pause and it's usually bad, see
> http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
> 2013-01-30 03:07:48,585 INFO org.apache.zookeeper.ClientCnxn: Client
> session
> timed out, have not heard from server in 39989ms for sessionid
> 0x13c84cebfce0000, closing socket connection and attempting reconnect
> 2013-01-30 03:07:48,586 INFO org.apache.zookeeper.ClientCnxn: Client
> session
> timed out, have not heard from server in 39987ms for sessionid
> 0x13c84cebfce0001, closing socket connection and attempting reconnect
> 2013-01-30 03:07:52,779 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket
> connection to server VM_11/192.168.152.1:2181
> 2013-01-30 03:07:52,789 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to VM_11/192.168.152.1:2181, initiating session
> 2013-01-30 03:07:52,777 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket
> connection to server VM_11/192.168.152.1:2181
> 2013-01-30 03:07:52,793 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to VM_11/192.168.152.1:2181, initiating session
> 2013-01-30 03:07:52,794 INFO org.apache.zookeeper.ClientCnxn: Unable to
> reconnect to ZooKeeper service, session 0x13c84cebfce0001 has expired,
> closing socket connection
> 2013-01-30 03:07:52,794 INFO
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> This client just lost it's session with ZooKeeper, trying to reconnect.
> 2013-01-30 03:07:52,794 INFO
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Trying to reconnect to zookeeper.
> 2013-01-30 03:07:52,795 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, connectString=VM_11:2181 sessionTimeout=180000
> watcher=hconnection
> 2013-01-30 03:07:52,812 INFO org.apache.zookeeper.ClientCnxn: Unable to
> reconnect to ZooKeeper service, session 0x13c84cebfce0000 has expired,
> closing socket connection
> 2013-01-30 03:07:52,813 FATAL org.apache.hadoop.hbase.master.HMaster:
>
> master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000
>
> master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000
> received expired from ZooKeeper, aborting
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired
>         at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:361)
>         at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:279)
>         at
>
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB