Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> GC and High CPU


Copy link to this message
-
Re: GC and High CPU
How do you tell?  This is the log entries when we had 100% cpu:

2011-05-14T15:48:58.240-0700: 5128.407: [GC 5128.407: [ParNew:
17723K->780K(19136K), 0.0199350 secs] 4309804K->4292973K(5777060K),
0.0200660 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]
2011-05-14T15:48:58.349-0700: 5128.515: [GC 5128.515: [ParNew:
17804K->886K(19136K), 0.0188930 secs] 4309997K->4293119K(5777060K),
0.0189990 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]
2011-05-14T15:48:58.456-0700: 5128.623: [GC 5128.623: [ParNew:
17910K->900K(19136K), 0.0144800 secs] 4310143K->4293237K(5777060K),
0.0145890 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
2011-05-14T15:48:58.563-0700: 5128.729: [GC 5128.729: [ParNew:
17924K->945K(19136K), 0.0144940 secs] 4310261K->4293469K(5777060K),
0.0146210 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2011-05-14T15:48:58.668-0700: 5128.835: [GC 5128.835: [ParNew:
17969K->731K(19136K), 0.0149060 secs] 4310493K->4293412K(5777060K),
0.0150210 secs] [Times: user=0.06 sys=0.00, real=0.01 secs]
2011-05-14T15:48:58.773-0700: 5128.940: [GC 5128.940: [ParNew:
17755K->906K(19136K), 0.0248650 secs] 4310436K->4293724K(5777060K),
0.0249820 secs] [Times: user=0.07 sys=0.00, real=0.03 secs]
2011-05-14T15:48:58.891-0700: 5129.057: [GC 5129.057: [ParNew:
17930K->1031K(19136K), 0.0194370 secs] 4310748K->4293961K(5777060K),
0.0196490 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]
2011-05-14T15:48:59.004-0700: 5129.171: [GC 5129.171: [ParNew:
18055K->960K(19136K), 0.0145450 secs] 4310985K->4293993K(5777060K),
0.0146530 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
2011-05-14T15:48:59.115-0700: 5129.281: [GC 5129.282: [ParNew:
17984K->1098K(19136K), 0.0143740 secs] 4311017K->4294296K(5777060K),
0.0144780 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
2011-05-14T15:48:59.224-0700: 5129.391: [GC 5129.391: [ParNew:
18122K->867K(19136K), 0.0150360 secs] 4311320K->4294248K(5777060K),
0.0151500 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2011-05-14T15:48:59.331-0700: 5129.498: [GC 5129.498: [ParNew:
17891K->1011K(19136K), 0.0220280 secs] 4311272K->4294544K(5777060K),
0.0221320 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2011-05-14T15:48:59.441-0700: 5129.608: [GC 5129.608: [ParNew:
18035K->1064K(19136K), 0.0164780 secs] 4311568K->4294803K(5777060K),
0.0165820 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]
2011-05-14T15:48:59.548-0700: 5129.714: [GC 5129.715: [ParNew:
18088K->1193K(19136K), 0.0141640 secs] 4311827K->4295008K(5777060K),
0.0142730 secs] [Times: user=0.06 sys=0.00, real=0.01 secs]
2011-05-14T15:48:59.653-0700: 5129.820: [GC 5129.820: [ParNew:
18217K->1078K(19136K), 0.0150800 secs] 4312032K->4295035K(5777060K),
0.0151780 secs] [Times: user=0.06 sys=0.00, real=0.02 secs]
2011-05-14T15:48:59.754-0700: 5129.921: [GC 5129.921: [ParNew:
18102K->906K(19136K), 0.0149160 secs] 4312059K->4295145K(5777060K),
0.0150130 secs] [Times: user=0.05 sys=0.00, real=0.02 secs]
2011-05-14T15:48:59.858-0700: 5130.025: [GC 5130.025: [ParNew:
17930K->749K(19136K), 0.0177820 secs] 4312169K->4295108K(5777060K),
0.0178890 secs] [Times: user=0.07 sys=0.00, real=0.01 secs]
2011-05-14T15:48:59.962-0700: 5130.129: [GC 5130.129: [ParNew:
17773K->970K(19136K), 0.0145400 secs] 4312132K->4295460K(5777060K),
0.0146530 secs] [Times: user=0.06 sys=0.00, real=0.01 secs]

-Jack

On Mon, May 16, 2011 at 12:44 PM, Stack <[EMAIL PROTECTED]> wrote:
> What is the size of your new gen?  Is it growing?  Does it level off?
> St.Ack
>
> On Mon, May 16, 2011 at 12:39 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>> There HEAP would be 8G used out of 12G total.  The gc-log would be
>> full of ParNew, no FULL GC at all, ParNew just starts at hight rate
>> (10-15 lines per second).  Even if you remove queries, the JVM will
>> not recover, and user CPU will remain 100%.  Only thing left to do is
>> to kill the process.  We are not writing in nearly as fast to case
>> such an issue.  This looks like a systemic problem, e.g. low enough
>> load when it comes to QPS, and the problem effectively kills HBASE.
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB