Home | About | Sematext search-lucene.com search-hadoop.com
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase, mail # user - hbase-master-server slept


Copy link to this message
-
Re: hbase-master-server slept
Marcos Ortiz Valmaseda 2013-02-12, 04:35
Well my friend, my first advice is to update your completed infrastructure:
- Update your Hadoop to 1.x branch
- Update HBase to 0.94.4
- Update Zookeeper to 3.4.5

Or simply update your CDH version to 4.1 or 4.2

----- Mensaje original -----

De: "So Hibino" <[EMAIL PROTECTED]>
Para: [EMAIL PROTECTED]
Enviados: Lunes, 11 de Febrero 2013 23:06:25
Asunto: Re: hbase-master-server slept

Hi,
>The master doesn't have memstores so this wouldn't help. In fact it's
>pretty rare that we see the master with GC issues. I recall seing
>issues with time travelling (machine clock's too slow and ntpd resets
>it) or on EC2 where sometimes you'd see random machine pauses out of
>nowhere (although that was a long time ago and haven't used EC2
>since).
We doesn't use EC2,but this server works with KVM.

The software version, the logs, the conf files are shown below.

software version
----------------------------------------
HBase version: 0.90.6-cdh3u4
Hadoop version: 0.20.2+923.256-1
Zookeeper version: 3.3.5+19.1-1
Operating System: CentOS release 5.8
Linux kernel version: 2.6.18-308.el5
Java version: 1.6.0_31
----------------------------------------

master log
------------------
2013-02-12 00:10:24,309 DEBUG org.apache.hadoop.hbase.master.LoadBalancer:
Server information: VM_11,60020,1359691508001=3
2013-02-12 00:10:24,310 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing. servers=1 regions=3 average=3.0 mostloaded=3
leastloaded=3
2013-02-12 00:10:24,318 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
2013-02-12 00:13:21,105 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
13417ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:13:55,239 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
34132ms instead of 10000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:13:55,242 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
24949ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:14:18,441 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
73255ms instead of 60000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:14:18,442 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
23203ms instead of 10000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:14:18,444 WARN org.apache.hadoop.hbase.util.Sleeper: We slept
14017ms instead of 1000ms, this is likely due to a long garbage collecting
pause and it's usually bad, see
http://hbase.apache.org/book.html#trouble.rs.runtime.zkexpired
2013-02-12 00:15:24,358 DEBUG org.apache.hadoop.hbase.master.LoadBalancer:
Server information: VM_11,60020,1359691508001=3
2013-02-12 00:15:24,358 INFO org.apache.hadoop.hbase.master.LoadBalancer:
Skipping load balancing. servers=1 regions=3 average=3.0 mostloaded=3
leastloaded=3
2013-02-12 00:15:24,361 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 1 catalog row(s) and gc'd 0 unreferenced parent region(s)
------------------
master GC log
------------------
2013-02-11T23:46:37.285+0900: 902498.189: [GC 902498.189: [DefNew:
17041K->16K(19136K), 0.0017450 secs] 20049K->3025K(83008K) icms_dc=0 ,
0.0018270 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
2013-02-12T00:35:25.628+0900: 905426.532: [GC 905426.532: [DefNew:
17040K->18K(19136K), 0.0017430 secs] 20049K->3026K(83008K) icms_dc=0 ,
0.0018370 secs] [Times: user=0.00 sys=0.00, real=0.01 secs]
2013-02-12T01:20:26.110+0900: 908127.014: [GC 908127.014: [DefNew:
17034K->27K(19136K), 0.0023420 secs] 20043K->3036K(83008K) icms_dc=0 ,
0.0025090 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
region log
2013-02-12 00:00:09,968 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB,
free=197.94 MB, max=199.59 MB, blocks=3, accesses=3022, hits=3015,
hitRatio=99.76%%, cachingAccesses=3015, cachingHits=3012,
cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN
2013-02-12 00:05:09,971 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB,
free=197.94 MB, max=199.59 MB, blocks=3, accesses=3023, hits=3016,
hitRatio=99.76%%, cachingAccesses=3016, cachingHits=3013,
cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN
2013-02-12 00:10:12,109 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB,
free=197.94 MB, max=199.59 MB, blocks=3, accesses=3024, hits=3017,
hitRatio=99.76%%, cachingAccesses=3017, cachingHits=3014,
cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN
2013-02-12 00:15:09,969 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB,
free=197.94 MB, max=199.59 MB, blocks=3, accesses=3025, hits=3018,
hitRatio=99.76%%, cachingAccesses=3018, cachingHits=3015,
cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN
2013-02-12 00:20:09,970 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 MB,
free=197.94 MB, max=199.59 MB, blocks=3, accesses=3026, hits=3019,
hitRatio=99.76%%, cachingAccesses=3019, cachingHits=3016,
cachingHitsRatio=99.90%%, evictions=0, evicted=0, evictedPerRun=NaN
region GC log
2013-02-11T22:31:11.315+0900: 897964.350: [GC 897964.350: [DefNew:
17062K->35K(19136K), 0.0036000 secs] 40262K->23234K(83008K) icms_dc=0 ,
0.0037710 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2013-02-