Home | About | Sematext search-lucene.com search-hadoop.com
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB
 Search Hadoop and all its subprojects:

Switch to Threaded View
HBase >> mail # user >> GC pause issues


Copy link to this message
-
Re: 答复: GC pause issues
How much memory are you flushing?  Can you paste the log here?  The
large the chunk of the flush the longer  your GC is going to be.

-Jack

On Fri, Jan 25, 2013 at 9:55 AM, Varun Sharma <[EMAIL PROTECTED]> wrote:
> My total heap size is 12G and my young gen is 1G. I am getting 200ms pauses
> every few seconds - the machine has 4 cores (its m1.xlarge on ec2), if I
> set the young gen low. The one thing I noted today was that the pauses are
> spaced @10 seconds until a minor compaction kicks in - and then then the
> pauses happen every 3-4 seconds. I found this to be highly correlated on a
> region server. These minor compactions are not huge - they are typically
> compacting 3 files - upto 200M in total size. They are lasting for 20-30
> seconds. Once they are over the GC pauses recover back to the 10 second
> frequency.
>
> I have the following settings enabled (index.cacheonwrite and
> bloom.cacheonwrite) to cache the index/bloom blocks upon hfile writes
> though i find it unlikely that they could be impacting my setup.
>
> On Thu, Jan 24, 2013 at 11:46 PM, Jack Levin <[EMAIL PROTECTED]> wrote:
>
>> Generally, the larger the flush to harder the GC will work. Flush more
>> often to avoid this. What is your total heap size set at?
>> On Jan 24, 2013 9:02 PM, "Varun Sharma" <[EMAIL PROTECTED]> wrote:
>>
>> > I do have significant block cache churn and this issue is typical
>> > correlated with a huge increase in read latencies - could that be the
>> > reason for this - mslab should be taking care of the memstore related
>> heap
>> > fragmentation ? Has anyone seen issues with block cache churn ?
>> >
>> > On Thu, Jan 24, 2013 at 6:30 PM, Varun Sharma <[EMAIL PROTECTED]>
>> wrote:
>> >
>> > > I am curious how reducing the memstore size would help - I have 6
>> regions
>> > > and 3G total for memstore - so I would like max out on that by having a
>> > > bigger flush size per region. Are you asking me to have more regions
>> and
>> > > smaller memstore flush size instead ? How is that likely to help
>> > >
>> > >
>> > > On Thu, Jan 24, 2013 at 6:07 PM, 谢良 <[EMAIL PROTECTED]> wrote:
>> > >
>> > >> Hi Varun,
>> > >>
>> > >> Please note if you try to increase new generation size, then the
>> > "ParNew"
>> > >> time will be up accordingly, and CMS YGC is also a STW.
>> > >> could you have a try to reduce memstore size to a smaller value, e.g.
>> > >> 128m or 256m ?
>> > >>
>> > >> Regards,
>> > >> Liang
>> > >> ________________________________________
>> > >> 发件人: Varun Sharma [[EMAIL PROTECTED]]
>> > >> 发送时间: 2013年1月25日 1:40
>> > >> 收件人: [EMAIL PROTECTED]
>> > >> 主题: GC pause issues
>> > >>
>> > >> Hi,
>> > >>
>> > >> I have a region server which has the following logs. As you can see
>> from
>> > >> the log, ParNew is sufficiently big (450M) and there are heavy writes
>> > >> going
>> > >> in. I am seeing 200ms pauses which eventually build up and there is a
>> > >> promotion failure. There is a parnew collection every 2-3 seconds so
>> it
>> > >> fills up real fast. My memstore size is bigger 512m for flushes and 4
>> > >> regions per server. (overall size is 3G for all memstores) - I have
>> > mslab
>> > >> enabled
>> > >>
>> > >> 2013-01-24T13:08:16.870+0000: 63533.964: [GC 63533.964: [ParNew:
>> > >> 471841K->52416K(471872K), 0.2251880 secs]
>> 8733008K->8445039K(12727104K),
>> > >> 0.2254100 secs] [Times: user=0.50 sys=0.18, real=0.22 secs]
>> > >> 2013-01-24T13:08:19.546+0000: 63536.639: [GC 63536.639: [ParNew:
>> > >> 461593K->52416K(471872K), 0.2812690 secs]
>> 8854216K->8557572K(12727104K),
>> > >> 0.2814870 secs] [Times: user=0.66 sys=0.09, real=0.29 secs]
>> > >> 013-01-24T13:08:21.824+0000: 63538.917: [GC 63538.918: [ParNew:
>> > >> 442836K->52416K(471872K), 0.2781490 secs]
>> 8947992K->8705355K(12727104K),
>> > >> 0.2783810 secs] [Times: user=0.58 sys=0.14, real=0.28 secs]
>> > >> 2013-01-24T13:08:22.122+0000: 63539.216: [GC [1 CMS-initial-mark:
>> > >> 8652939K(12255232K)] 8752914K(12727104K), 0.0365000 secs] [Times:
NEW: Monitor These Apps!
elasticsearch, apache solr, apache hbase, hadoop, redis, casssandra, amazon cloudwatch, mysql, memcached, apache kafka, apache zookeeper, apache storm, ubuntu, centOS, red hat, debian, puppet labs, java, senseiDB