Our hbase-master-server was shutdown with following message.
Hbase is runnig in Distributed mode in a single node.
I checked that GC completed in a very short time at the time of output the
WARN.
In addition the other system that is running in the same architecture
doesn't output the following WARN messsage and works well.
So I think that this is not due to a long GC pause.

Do you have any idea about the problem?

2013-01-30 03:07:48,582 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
28970ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-01-30 03:07:48,583 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
36902ms instead of 10000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-01-30 03:07:48,585 INFO org.apache.zookeeper.ClientCnxn: Client session
timed out, have not heard from server in 39989ms for sessionid
0x13c84cebfce0000, closing socket connection and attempting reconnect
2013-01-30 03:07:48,586 INFO org.apache.zookeeper.ClientCnxn: Client session
timed out, have not heard from server in 39987ms for sessionid
0x13c84cebfce0001, closing socket connection and attempting reconnect
2013-01-30 03:07:52,779 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server VM_11/192.168.152.1:2181
2013-01-30 03:07:52,789 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to VM_11/192.168.152.1:2181, initiating session
2013-01-30 03:07:52,777 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server VM_11/192.168.152.1:2181
2013-01-30 03:07:52,793 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to VM_11/192.168.152.1:2181, initiating session
2013-01-30 03:07:52,794 INFO org.apache.zookeeper.ClientCnxn: Unable to
reconnect to ZooKeeper service, session 0x13c84cebfce0001 has expired,
closing socket connection
2013-01-30 03:07:52,794 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
This client just lost it's session with ZooKeeper, trying to reconnect.
2013-01-30 03:07:52,794 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Trying to reconnect to zookeeper.
2013-01-30 03:07:52,795 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=VM_11:2181 sessionTimeout=180000
watcher=hconnection
2013-01-30 03:07:52,812 INFO org.apache.zookeeper.ClientCnxn: Unable to
reconnect to ZooKeeper service, session 0x13c84cebfce0000 has expired,
closing socket connection
2013-01-30 03:07:52,813 FATAL org.apache.hadoop.hbase.master.HMaster:
master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000
master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000
received expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:361)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:279)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
2013-01-30 03:07:52,813 INFO org.apache.hadoop.hbase.master.HMaster:
Aborting
2013-01-30 03:07:52,813 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2013-01-30 03:07:52,813 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server VM_11/192.168.152.1:2181
2013-01-30 03:07:52,814 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to VM_11/192.168.152.1:2181, initiating session
2013-01-30 03:07:52,815 ERROR org.apache.hadoop.hbase.master.HMaster: Region
server serverName=VM_11,60020,1359437833300, load=(requests=0, regions=3,
usedHeap=45, maxHeap=997) reported a fatal error:
ABORTING region server serverName=VM_11,60020,1359437833300,
load=(requests=0, regions=3, usedHeap=45, maxHeap=997):
regionserver:60020-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002
regionserver:60020-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002-0x13c84cebfce0002
received expired from ZooKeeper, aborting
Cause:
org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:361)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:279)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)

2013-01-30 03:07:52,820 INFO org.apache.zookeeper.ClientCnxn: Session
establishment complete on server VM_11/192.168.152.1:2181, sessionid 0x13c84cebfce0005, negotiated timeout = 40000
2013-01-30 03:07:52,841 INFO
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Reconnected successfully. This disconnect could have been caused by a
network partition or a long-running GC pause, either way it's recommended
that you verify your environment.
2013-01-30 03:07:52,841 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2013-01-30 03:07:53,614 INFO org.apache.hadoop.hbase.master.LogCleaner:
master-VM_11:60000.oldLogCleaner exiting
2013-01-30 03:07:54,251 INFO org.apache.hadoop.hbase.master.HMaster$2:
VM_11:60000-BalancerChor

Regards, So,
Can you provide more information about your setup?
- HBase version
- Hadoop version
- Operating System
- Java version

On 02/08/2013 03:55 AM, So Hibino wrote:
> Our hbase-master-server was shutdown with following message.
> Hbase is runnig in Distributed mode in a single node.
Can you share your .conf files?
> I checked that GC completed in a very short time at the time of output the
> WARN.
> In addition the other system that is running in the same architecture
> doesn't output the following WARN messsage and works well.
> So I think that this is not due to a long GC pause.
>
> Do you have any idea about the problem?
>
> 2013-01-30 03:07:48,582 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
> 28970ms instead of 1000ms, this is likely due to a long garbage collecting
> pause and it's usually bad, see
> http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
Did you check the link?
Todd wrote a series of posts in Cloudera�s blog about Java Long GC
pauses, HBase and Zookeeper.
It�s a great read:
http://www.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-1/
http://www.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-2/
> 2013-01-30 03:07:48,583 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
> 36902ms instead of 10000ms, this is likely due to a long garbage collecting
> pause and it's usually bad, see
> http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
> 2013-01-30 03:07:48,585 INFO org.apache.zookeeper.ClientCnxn: Client session
> timed out, have not heard from server in 39989ms for sessionid
> 0x13c84cebfce0000, closing socket connection and attempting reconnect
> 2013-01-30 03:07:48,586 INFO org.apache.zookeeper.ClientCnxn: Client session
> timed out, have not heard from server in 39987ms for sessionid
> 0x13c84cebfce0001, closing socket connection and attempting reconnect
> 2013-01-30 03:07:52,779 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> connection to server VM_11/192.168.152.1:2181
> 2013-01-30 03:07:52,789 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to VM_11/192.168.152.1:2181, initiating session
> 2013-01-30 03:07:52,777 INFO org.apache.zookeeper.ClientCnxn: Opening socket
> connection to server VM_11/192.168.152.1:2181
> 2013-01-30 03:07:52,793 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to VM_11/192.168.152.1:2181, initiating session
> 2013-01-30 03:07:52,794 INFO org.apache.zookeeper.ClientCnxn: Unable to
> reconnect to ZooKeeper service, session 0x13c84cebfce0001 has expired,
> closing socket connection
> 2013-01-30 03:07:52,794 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> This client just lost it's session with ZooKeeper, trying to reconnect.
> 2013-01-30 03:07:52,794 INFO
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Trying to reconnect to zookeeper.
> 2013-01-30 03:07:52,795 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, connectString=VM_11:2181 sessionTimeout=180000
> watcher=hconnection
> 2013-01-30 03:07:52,812 INFO org.apache.zookeeper.ClientCnxn: Unable to
> reconnect to ZooKeeper service, session 0x13c84cebfce0000 has expired,
> closing socket connection
> 2013-01-30 03:07:52,813 FATAL org.apache.hadoop.hbase.master.HMaster:
> master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000
> master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000
> received expired from ZooKeeper, aborting
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:361)

Marcos Ortiz Valmaseda,
Product Manager && Data Scientist at UCI
Blog: http://marcosluis2186.posterous.com
Twitter: @marcosluis2186 <http://twitter.com/marcosluis2186>

On Fri, Feb 8, 2013 at 1:26 AM, Marcos Ortiz <[EMAIL PROTECTED]> wrote:
> Regards, So,
> Can you provide more information about your setup?
> - HBase version
> - Hadoop version
> - Operating System
> - Java version

This would be helpful.

I would also like to see that GC log please.

> Did you check the link?
> Todd wrote a series of posts in Cloudera´s blog about Java Long GC pauses,
> HBase and Zookeeper.
> It´s a great read:
> http://www.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-1/
> http://www.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-2/

The master doesn't have memstores so this wouldn't help. In fact it's
pretty rare that we see the master with GC issues. I recall seing
issues with time travelling (machine clock's too slow and ntpd resets
it) or on EC2 where sometimes you'd see random machine pauses out of
nowhere (although that was a long time ago and haven't used EC2
since).

J-D

What zookeeper version are you using ?
Is the ensemble managed by HBase ?

Can you check the zookeeper log on 192.168.152.1<http://192.168.152.1:2181/>
 ?
Use pastebin to show us the log if necessary.

Thanks

On Fri, Feb 8, 2013 at 12:55 AM, So Hibino <[EMAIL PROTECTED]> wrote:

> Our hbase-master-server was shutdown with following message.
> Hbase is runnig in Distributed mode in a single node.
> I checked that GC completed in a very short time at the time of output the
> WARN.
> In addition the other system that is running in the same architecture
> doesn't output the following WARN messsage and works well.
> So I think that this is not due to a long GC pause.
>
> Do you have any idea about the problem?
>
> 2013-01-30 03:07:48,582 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
> 28970ms instead of 1000ms, this is likely due to a long garbage collecting
> pause and it's usually bad, see
> http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
> 2013-01-30 03:07:48,583 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
> 36902ms instead of 10000ms, this is likely due to a long garbage collecting
> pause and it's usually bad, see
> http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
> 2013-01-30 03:07:48,585 INFO org.apache.zookeeper.ClientCnxn: Client
> session
> timed out, have not heard from server in 39989ms for sessionid
> 0x13c84cebfce0000, closing socket connection and attempting reconnect
> 2013-01-30 03:07:48,586 INFO org.apache.zookeeper.ClientCnxn: Client
> session
> timed out, have not heard from server in 39987ms for sessionid
> 0x13c84cebfce0001, closing socket connection and attempting reconnect
> 2013-01-30 03:07:52,779 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket
> connection to server VM_11/192.168.152.1:2181
> 2013-01-30 03:07:52,789 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to VM_11/192.168.152.1:2181, initiating session
> 2013-01-30 03:07:52,777 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket
> connection to server VM_11/192.168.152.1:2181
> 2013-01-30 03:07:52,793 INFO org.apache.zookeeper.ClientCnxn: Socket
> connection established to VM_11/192.168.152.1:2181, initiating session
> 2013-01-30 03:07:52,794 INFO org.apache.zookeeper.ClientCnxn: Unable to
> reconnect to ZooKeeper service, session 0x13c84cebfce0001 has expired,
> closing socket connection
> 2013-01-30 03:07:52,794 INFO
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> This client just lost it's session with ZooKeeper, trying to reconnect.
> 2013-01-30 03:07:52,794 INFO
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
> Trying to reconnect to zookeeper.
> 2013-01-30 03:07:52,795 INFO org.apache.zookeeper.ZooKeeper: Initiating
> client connection, connectString=VM_11:2181 sessionTimeout=180000
> watcher=hconnection
> 2013-01-30 03:07:52,812 INFO org.apache.zookeeper.ClientCnxn: Unable to
> reconnect to ZooKeeper service, session 0x13c84cebfce0000 has expired,
> closing socket connection
> 2013-01-30 03:07:52,813 FATAL org.apache.hadoop.hbase.master.HMaster:
>
> master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000
>
> master:60000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000-0x13c84cebfce0000
> received expired from ZooKeeper, aborting
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired
>         at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:361)
>         at
>
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:279)
>         at
>
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:526)
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)

Hi,
>The master doesn't have memstores so this wouldn't help. In fact it's
>pretty rare that we see the master with GC issues. I recall seing
>issues with time travelling (machine clock's too slow and ntpd resets
>it) or on EC2 where sometimes you'd see random machine pauses out of
>nowhere (although that was a long time ago and haven't used EC2
>since).
We doesn't use EC2,but this server works with KVM.

The software version, the logs, the conf files are shown below.

software version
----------------------------------------
HBase version: 0.90.6-cdh3u4
Hadoop version: 0.20.2+923.256-1
Zookeeper version: 3.3.5+19.1-1
Operating System: CentOS release 5.8
Linux kernel version: 2.6.18-308.el5
Java version: 1.6.0_31
----------------------------------------

master log
------------------
2013-02-12 00:10:24,309 DEBUG org.apache.hadoop.hbase.master.LoadBalancer:
Server information: VM_11,60020,1359691508001=3
2013-02-12 00:10:24,310 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing.  servers=1 regions=3 average=3.0 mostloaded=3
leastloaded=3
2013-02-12 00:10:24,318 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
2013-02-12 00:13:21,105 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
13417ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:13:55,239 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
34132ms instead of 10000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:13:55,242 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
24949ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:14:18,441 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
73255ms instead of 60000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:14:18,442 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
23203ms instead of 10000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:14:18,444 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
14017ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:15:24,358 DEBUG org.apache.hadoop.hbase.master.LoadBalancer:
Server information: VM_11,60020,1359691508001=3
2013-02-12 00:15:24,358 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing.  servers=1 regions=3 average=3.0 mostloaded=3
leastloaded=3
2013-02-12 00:15:24,361 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
------------------
master GC log
------------------
2013-02-11T23:46:37.285+0900: 902498.189: [GC 902498.189: [DefNew:
17041K->16K(19136K), 0.0017450 secs] 20049K->3025K(83008K) icms_dc=0 ,
0.0018270 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2013-02-12T00:35:25.628+0900: 905426.532: [GC 905426.532: [DefNew:
17040K->18K(19136K), 0.0017430 secs] 20049K->3026K(83008K) icms_dc=0 ,
0.0018370 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2013-02-12T01:20:26.110+0900: 908127.014: [GC 908127.014: [DefNew:
17034K->27K(19136K), 0.0023420 secs] 20043K->3036K(83008K) icms_dc=0 ,
0.0025090 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
-------------------
region log
-------------------
2013-02-12 00:00:09,968 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB,
free=197.94 MB, max=199.59 MB, blocks=3, accesses=3022, hits=3015,
hitRatio=99.76%%, cachingAccesses=3015, cachingHits=3012,
cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN
2013-02-12 00:05:09,971 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB,
free=197.94 MB, max=199.59 MB, blocks=3, accesses=3023, hits=3016,
hitRatio=99.76%%, cachingAccesses=3016, cachingHits=3013,
cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN
2013-02-12 00:10:12,109 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB,
free=197.94 MB, max=199.59 MB, blocks=3, accesses=3024, hits=3017,
hitRatio=99.76%%, cachingAccesses=3017, cachingHits=3014,
cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN
2013-02-12 00:15:09,969 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB,
free=197.94 MB, max=199.59 MB, blocks=3, accesses=3025, hits=3018,
hitRatio=99.76%%, cachingAccesses=3018, cachingHits=3015,
cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN
2013-02-12 00:20:09,970 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB,
free=197.94 MB, max=199.59 MB, blocks=3, accesses=3026, hits=3019,
hitRatio=99.76%%, cachingAccesses=3019, cachingHits=3016,
cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN
region GC log
2013-02-11T22:31:11.315+0900: 897964.350: [GC 897964.350: [DefNew:
17062K->35K(19136K), 0.0036000 secs] 40262K->23234K(83008K) icms_dc=0 ,
0.0037710 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2013-02-12T00:27:13.313+0900: 904926.348: [GC 904926.348: [DefNew:
17059K->43K(19136K), 0.0020250 secs] 40258K->23243K(83008K) icms_dc=0 ,
0.0021130 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2013-02-12T02:23:52.114+0900: 911925.149: [GC 911925.149: [DefNew:
17067K->43K(19136K), 0.0018170 secs] 40267K->23243K(83008K) icms_dc=0 ,
0.0019330 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
zookeeper log
no logs at that time
hbase-site.xml
<configura

Well my friend, my first advice is to update your completed infrastructure:
- Update your Hadoop to 1.x branch
- Update HBase to 0.94.4
- Update Zookeeper to 3.4.5

Or simply update your CDH version to 4.1 or 4.2

----- Mensaje original -----

De: "So Hibino" <[EMAIL PROTECTED]>
Para: [EMAIL PROTECTED]
Enviados: Lunes, 11 de Febrero 2013 23:06:25
Asunto: Re: hbase-master-server slept

Hi,
>The master doesn't have memstores so this wouldn't help. In fact it's
>pretty rare that we see the master with GC issues. I recall seing
>issues with time travelling (machine clock's too slow and ntpd resets
>it) or on EC2 where sometimes you'd see random machine pauses out of
>nowhere (although that was a long time ago and haven't used EC2
>since).
We doesn't use EC2,but this server works with KVM.

The software version, the logs, the conf files are shown below.

software version
----------------------------------------
HBase version: 0.90.6-cdh3u4
Hadoop version: 0.20.2+923.256-1
Zookeeper version: 3.3.5+19.1-1
Operating System: CentOS release 5.8
Linux kernel version: 2.6.18-308.el5
Java version: 1.6.0_31
----------------------------------------

master log
------------------
2013-02-12 00:10:24,309 DEBUG org.apache.hadoop.hbase.master.LoadBalancer:
Server information: VM_11,60020,1359691508001=3
2013-02-12 00:10:24,310 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing. servers=1 regions=3 average=3.0 mostloaded=3
leastloaded=3
2013-02-12 00:10:24,318 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
2013-02-12 00:13:21,105 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
13417ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:13:55,239 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
34132ms instead of 10000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:13:55,242 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
24949ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:14:18,441 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
73255ms instead of 60000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:14:18,442 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
23203ms instead of 10000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:14:18,444 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
14017ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:15:24,358 DEBUG org.apache.hadoop.hbase.master.LoadBalancer:
Server information: VM_11,60020,1359691508001=3
2013-02-12 00:15:24,358 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing. servers=1 regions=3 average=3.0 mostloaded=3
leastloaded=3
2013-02-12 00:15:24,361 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
------------------
master GC log
------------------
2013-02-11T23:46:37.285+0900: 902498.189: [GC 902498.189: [DefNew:
17041K->16K(19136K), 0.0017450 secs] 20049K->3025K(83008K) icms_dc=0 ,
0.0018270 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2013-02-12T00:35:25.628+0900: 905426.532: [GC 905426.532: [DefNew:
17040K->18K(19136K), 0.0017430 secs] 20049K->3026K(83008K) icms_dc=0 ,
0.0018370 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2013-02-12T01:20:26.110+0900: 908127.014: [GC 908127.014: [DefNew:
17034K->27K(19136K), 0.0023420 secs] 20043K->3036K(83008K) icms_dc=0 ,
0.0025090 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
region log
2013-02-12 00:00:09,968 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB,
free=197.94 MB, max=199.59 MB, blocks=3, accesses=3022, hits=3015,
hitRatio=99.76%%, cachingAccesses=3015, cachingHits=3012,
cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN
2013-02-12 00:05:09,971 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB,
free=197.94 MB, max=199.59 MB, blocks=3, accesses=3023, hits=3016,
hitRatio=99.76%%, cachingAccesses=3016, cachingHits=3013,
cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN
2013-02-12 00:10:12,109 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB,
free=197.94 MB, max=199.59 MB, blocks=3, accesses=3024, hits=3017,
hitRatio=99.76%%, cachingAccesses=3017, cachingHits=3014,
cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN
2013-02-12 00:15:09,969 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB,
free=197.94 MB, max=199.59 MB, blocks=3, accesses=3025, hits=3018,
hitRatio=99.76%%, cachingAccesses=3018, cachingHits=3015,
cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN
2013-02-12 00:20:09,970 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB,
free=197.94 MB, max=199.59 MB, blocks=3, accesses=3026, hits=3019,
hitRatio=99.76%%, cachingAccesses=3019, cachingHits=3016,
cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN
region GC log
2013-02-11T22:31:11.315+0900: 897964.350: [GC 897964.350: [DefNew:
17062K->35K(19136K), 0.0036000 secs] 40262K->23234K(83008K) icms_dc=0 ,
0.0037710 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2013-02-

Hi,

The other system with the same version works well and don't output any WARN
message.
So I think this isn't problem of CDH version.

Thank you,
So

--
View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038416.html
Sent from the HBase User mailing list archive at Nabble.com.

On Mon, Feb 11, 2013 at 8:06 PM, So Hibino <[EMAIL PROTECTED]> wrote:

> master log

Unfortunately there's not much in there apart from the weird sleep
times. At this point I'd consider OS-level issues.

> We doesn't use EC2,but this server works with KVM.

Can you post the dmesg and syslog spanning the time of the issue?
Please use pastebin.com as it can be really hard to read the logs in
emails.

Thx,

J-D

Hi,

> Can you post the dmesg and syslog spanning the time of the issue?
Is the syslog you mean /var/log/message ?
On 2013/02/13 4:00, Jean-Daniel Cryans [via Apache HBase] wrote:
> On Mon, Feb 11, 2013 at 8:06 PM, So Hibino <[hidden email] </user/SendEmail.jtp?type=node&node=4038442&i=0>> wrote:
>
>  > master log
>
> Unfortunately there's not much in there apart from the weird sleep
> times. At this point I'd consider OS-level issues.
>
>  > We doesn't use EC2,but this server works with KVM.
>
> Can you post the dmesg and syslog spanning the time of the issue?
> Please use pastebin.com as it can be really hard to read the logs in
> emails.
>
> Thx,
>
> J-D
>
>
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------!
 ---
> If you reply to this email, your message will be added to the discussion below:
> http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038442.html
> To unsubscribe from hbase-master-server slept, click here <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4038192&code=aGliaW5vLnNvQGxhYi5udHQuY28uanB8NDAzODE5MnwtNjAxOTg5MzI0>.
> NAML <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
--
H24.07より組織名が変わりました。

NTTサービスエボリューション研究所
日比野　壮　（So Hibino）

〒239-0847 横須賀市光の丘1-1 (320C)
TEL : 046-859-4582  FAX : 046-855-1282
e-mail : [EMAIL PROTECTED]

--
View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038459.html
Sent from the HBase User mailing list archive at Nabble.com.

messages and syslog please, just for the time spanning the issue.

J-D

On Tue, Feb 12, 2013 at 4:10 PM, So Hibino <[EMAIL PROTECTED]> wrote:
> Hi,
>
>> Can you post the dmesg and syslog spanning the time of the issue?
> Is the syslog you mean /var/log/message ?
>
>
> On 2013/02/13 4:00, Jean-Daniel Cryans [via Apache HBase] wrote:
>> On Mon, Feb 11, 2013 at 8:06 PM, So Hibino <[hidden email] </user/SendEmail.jtp?type=node&node=4038442&i=0>> wrote:
>>
>>  > master log
>>
>> Unfortunately there's not much in there apart from the weird sleep
>> times. At this point I'd consider OS-level issues.
>>
>>  > We doesn't use EC2,but this server works with KVM.
>>
>> Can you post the dmesg and syslog spanning the time of the issue?
>> Please use pastebin.com as it can be really hard to read the logs in
>> emails.
>>
>> Thx,
>>
>> J-D
>>
>>
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------!
>  ---
>> If you reply to this email, your message will be added to the discussion below:
>> http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038442.html
>> To unsubscribe from hbase-master-server slept, click here <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4038192&code=aGliaW5vLnNvQGxhYi5udHQuY28uanB8NDAzODE5MnwtNjAxOTg5MzI0>.
>> NAML <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
>
> --
> H24.07より組織名が変わりました。
>
> NTTサービスエボリューション研究所
> 日比野　壮　（So Hibino）
>
> 〒239-0847 横須賀市光の丘1-1 (320C)
> TEL : 046-859-4582  FAX : 046-855-1282
> e-mail : [EMAIL PROTECTED]
>
>
>
>
>
> --
> View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038459.html
> Sent from the HBase User mailing list archive at Nabble.com.

Hi,

I pasted dmseg and syslog on pastebin.com.

URL is below.

dmseg
http://pastebin.com/t3i3KRJ8

syslog
http://pastebin.com/LcBRYMyX

Any messages wasn't output for the time spanning the isuue.

thank you,
So
On 2013/02/13 9:31, Jean-Daniel Cryans [via Apache HBase] wrote:
>
>
> messages and syslog please, just for the time spanning the issue.
>
> J-D
>
> On Tue, Feb 12, 2013 at 4:10 PM, So Hibino <[EMAIL PROTECTED]> wrote:
>> Hi,
>>
>>> Can you post the dmesg and syslog spanning the time of the issue?
>> Is the syslog you mean /var/log/message ?
>>
>>
>> On 2013/02/13 4:00, Jean-Daniel Cryans [via Apache HBase] wrote:
>>> On Mon, Feb 11, 2013 at 8:06 PM, So Hibino <[hidden email] </user/SendEmail.jtp?type=node&node=4038442&i=0>> wrote:
>>>
>>>   > master log
>>>
>>> Unfortunately there's not much in there apart from the weird sleep
>>> times. At this point I'd consider OS-level issues.
>>>
>>>   > We doesn't use EC2,but this server works with KVM.
>>>
>>> Can you post the dmesg and syslog spanning the time of the issue?
>>> Please use pastebin.com as it can be really hard to read the logs in
>>> emails.
>>>
>>> Thx,
>>>
>>> J-D
>>>
>>>
>>> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------!
 --!
>>   ---
>>> If you reply to this email, your message will be added to the discussion below:
>>> http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038442.html
>>> To unsubscribe from hbase-master-server slept, click here <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4038192&code=aGliaW5vLnNvQGxhYi5udHQuY28uanB8NDAzODE5MnwtNjAxOTg5MzI0>.
>>> NAML <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>>
>> --
>> H24.07より組織名が変わりました。
>>
>> NTTサービスエボリューション研究所
>> 日比野　壮　（So Hibino）
>>
>> 〒239-0847 横須賀市光の丘1-1 (320C)
>> TEL : 046-859-4582  FAX : 046-855-1282
>> e-mail : [EMAIL PROTECTED]
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038459.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>
>
>
>
> _______________________________________________
> If you reply to this email, your message will be added to the discussion below:
> http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038461.html
>
> To unsubscribe from hbase-master-server slept, visit http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4038192&code=aGliaW5vLnNvQGxhYi5udHQuY28uanB8NDAzODE5MnwtNjAxOTg5MzI0
>
--
H24.07より組織名が変わりました。

NTTサービスエボリューション研究所
日比野　壮　（So Hibino）

〒239-0847 横須賀市光の丘1-1 (320C)
TEL : 046-859-4582  FAX : 046-855-1282
e-mail : [EMAIL PROTECTED]

--
View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038515.html
Sent from the HBase User mailing list archive at Nabble.com.

Hi,

The VM was rebooted last night because of the host server memory error.
So the dmesg was cleared at the time of the isuue.

The below dmesg is new and the issue has not been happened yet.

Thank you,
So

On 2013/02/14 15:10, So Hibino wrote:
> Hi,
>
> I pasted dmseg and syslog on pastebin.com.
>
> URL is below.
>
> dmseg
> http://pastebin.com/t3i3KRJ8
>
> syslog
> http://pastebin.com/LcBRYMyX
>
> Any messages wasn't output for the time spanning the isuue.
>
> thank you,
> So
>
>
> On 2013/02/13 9:31, Jean-Daniel Cryans [via Apache HBase] wrote:
>>
>>
>> messages and syslog please, just for the time spanning the issue.
>>
>> J-D
>>
>> On Tue, Feb 12, 2013 at 4:10 PM, So Hibino <[EMAIL PROTECTED]> wrote:
>>> Hi,
>>>
>>>> Can you post the dmesg and syslog spanning the time of the issue?
>>> Is the syslog you mean /var/log/message ?
>>>
>>>
>>> On 2013/02/13 4:00, Jean-Daniel Cryans [via Apache HBase] wrote:
>>>> On Mon, Feb 11, 2013 at 8:06 PM, So Hibino <[hidden email] </user/SendEmail.jtp?type=node&node=4038442&i=0>> wrote:
>>>>
>>>>    > master log
>>>>
>>>> Unfortunately there's not much in there apart from the weird sleep
>>>> times. At this point I'd consider OS-level issues.
>>>>
>>>>    > We doesn't use EC2,but this server works with KVM.
>>>>
>>>> Can you post the dmesg and syslog spanning the time of the issue?
>>>> Please use pastebin.com as it can be really hard to read the logs in
>>>> emails.
>>>>
>>>> Thx,
>>>>
>>>> J-D
>>>>
>>>>
>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------!
 -!
>   --!
>>>    ---
>>>> If you reply to this email, your message will be added to the discussion below:
>>>> http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038442.html
>>>> To unsubscribe from hbase-master-server slept, click here <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4038192&code=aGliaW5vLnNvQGxhYi5udHQuY28uanB8NDAzODE5MnwtNjAxOTg5MzI0>.
>>>> NAML <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>
>>>
>>> --
>>> H24.07より組織名が変わりました。
>>>
>>> NTTサービスエボリューション研究所
>>> 日比野　壮　（So Hibino）
>>>
>>> 〒239-0847 横須賀市光の丘1-1 (320C)
>>> TEL : 046-859-4582  FAX : 046-855-1282
>>> e-mail : [EMAIL PROTECTED]
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038459.html
>>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
>>
>>
>> _______________________________________________
>> If you reply to this email, your message will be added to the discussion below:
>> http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038461.html
>>
>> To unsubscribe from hbase-master-server slept, visit http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4038192&code=aGliaW5vLnNvQGxhYi5udHQuY28uanB8NDAzODE5MnwtNjAxOTg5MzI0
H24.07より組織名が変わりました。

NTTサービスエボリューション研究所
日比野　壮　（So Hibino）

〒239-0847 横須賀市光の丘1-1 (320C)
TEL : 046-859-4582  FAX : 046-855-1282
e-mail : [EMAIL PROTECTED]

View this message in context: http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038516.html
Sent from the HBase User mailing list archive at Nabble.com.

First,
In the world of Hadoop, if it ain't broke don't fix it, may not be the best advice.
HBase is still evolving at a good pace and you want to be closer to the latest releases.
CDH4 is stable so that I would agree that going to CDH4 would be best.

Second.
You are running this as a single machine within the VM.

What does the hardware look like?
Number of cores, physical or virtual ?
How much memory?

What type of disks and where are they?
( attached or SAN?)
Sent from a remote device. Please excuse any typos...

Mike Segel

On Feb 14, 2013, at 1:16 AM, So Hibino <[EMAIL PROTECTED]> wrote:

> Hi,
>
> The VM was rebooted last night because of the host server memory error.
> So the dmesg was cleared at the time of the isuue.
>
> The below dmesg is new and the issue has not been happened yet.
>
> Thank you,
> So
>
> On 2013/02/14 15:10, So Hibino wrote:
>> Hi,
>>
>> I pasted dmseg and syslog on pastebin.com.
>>
>> URL is below.
>>
>> dmseg
>> http://pastebin.com/t3i3KRJ8
>>
>> syslog
>> http://pastebin.com/LcBRYMyX
>>
>> Any messages wasn't output for the time spanning the isuue.
>>
>> thank you,
>> So
>>
>>
>> On 2013/02/13 9:31, Jean-Daniel Cryans [via Apache HBase] wrote:
>>>
>>>
>>> messages and syslog please, just for the time spanning the issue.
>>>
>>> J-D
>>>
>>> On Tue, Feb 12, 2013 at 4:10 PM, So Hibino <[EMAIL PROTECTED]> wrote:
>>>> Hi,
>>>>
>>>>> Can you post the dmesg and syslog spanning the time of the issue?
>>>> Is the syslog you mean /var/log/message ?
>>>>
>>>>
>>>> On 2013/02/13 4:00, Jean-Daniel Cryans [via Apache HBase] wrote:
>>>>> On Mon, Feb 11, 2013 at 8:06 PM, So Hibino <[hidden email] </user/SendEmail.jtp?type=node&node=4038442&i=0>> wrote:
>>>>>
>>>>>> master log
>>>>>
>>>>> Unfortunately there's not much in there apart from the weird sleep
>>>>> times. At this point I'd consider OS-level issues.
>>>>>
>>>>>> We doesn't use EC2,but this server works with KVM.
>>>>>
>>>>> Can you post the dmesg and syslog spanning the time of the issue?
>>>>> Please use pastebin.com as it can be really hard to read the logs in
>>>>> emails.
>>>>>
>>>>> Thx,
>>>>>
>>>>> J-D
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------!
> -!
>>  --!
>>>>   ---
>>>>> If you reply to this email, your message will be added to the discussion below:
>>>>> http://apache-hbase.679495.n3.nabble.com/hbase-master-server-slept-tp4038192p4038442.html
>>>>> To unsubscribe from hbase-master-server slept, click here <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4038192&code=aGliaW5vLnNvQGxhYi5udHQuY28uanB8NDAzODE5MnwtNjAxOTg5MzI0>.
>>>>> NAML <http://apache-hbase.679495.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>