I have switched to using the mslab enabled java setting to try to avoid GC
causing nodes to go awol but it almost appears to be worse. Below is the
latest problem with the JVM apparently actually crashing. I am using 0.90.1
with an 8GB heap. Is there a recommended JVM and recommended settings to be
used? As it stands right now we can not run 24 hours under heavy write load
without a node being taken out by zookeeper for GCing > 60 sec or other
problems like below.

Any help would be greatly appreciated.
2011-05-23T02:34:51.626+0000: 13902.361: [GC 13902.361: [ParNew:
249216K->27648K(249216K), 0.1119520 secs] 7546544K->7433319K(8360960K),
0.1120390 secs] [Times: user=1.14 sys=0.05, real=0.11 secs]
2011-05-23T02:34:52.292+0000: 13903.027: [GC 13903.027: [ParNew:
249216K->27648K(249216K), 0.0732800 secs] 7654887K->7506032K(8360960K),
0.0733690 secs] [Times: user=0.76 sys=0.02, real=0.08 secs]
2011-05-23T02:34:52.721+0000: 13903.456: [CMS-concurrent-mark: 8.137/10.065
secs] [Times: user=60.86 sys=2.98, real=10.06 secs]
2011-05-23T02:34:52.721+0000: 13903.456: [CMS-concurrent-preclean-start]
2011-05-23T02:34:52.839+0000: 13903.574: [GC 13903.574: [ParNew:
249216K->27648K(249216K), 0.0575510 secs] 7727600K->7562758K(8360960K),
0.0576420 secs] [Times: user=0.62 sys=0.02, real=0.06 secs]
2011-05-23T02:34:53.190+0000: 13903.925: [GC 13903.925: [ParNew:
249171K->27648K(249216K), 0.1108480 secs] 7784281K->7661505K(8360960K),
0.1109440 secs] [Times: user=1.10 sys=0.03, real=0.11 secs]
2011-05-23T02:34:53.539+0000: 13904.274: [GC 13904.274: [ParNew (promotion
failed): 249216K->249216K(249216K), 0.1207770 secs]13904.395:
[CMS2011-05-23T02:34:54.310+0000: 13905.045: [CMS-concurrent-preclean:
1.245/1.589 secs] [Times: user=5.99 sys=0.13, real=1.59 secs]
 (concurrent mode failure)#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00002b19debbe665, pid=25868, tid=1078290752
#
# JRE version: 6.0_17-b17
# Java VM: OpenJDK 64-Bit Server VM (14.0-b16 mixed mode linux-amd64 )
# Derivative: IcedTea6 1.7.10
# Distribution: Custom build (Wed May  4 23:17:24 EDT 2011)
# Problematic frame:
# V  [libjvm.so+0x29d665]
#
# An error report file with more information is saved as:
# .../hbase-0.90.1/hs_err_pid25868.log
#
# If you would like to submit a bug report, please include
# instructions how to reproduce the bug and visit:
#   http://icedtea.classpath.org/bugzilla

Do you have the same problem with a more recent JVM?

On Mon, May 23, 2011 at 4:52 AM, Wayne <[EMAIL PROTECTED]> wrote:

> I have switched to using the mslab enabled java setting to try to avoid GC
> causing nodes to go awol but it almost appears to be worse. Below is the
> latest problem with the JVM apparently actually crashing. I am using 0.90.1
> with an 8GB heap. Is there a recommended JVM and recommended settings to be
> used? As it stands right now we can not run 24 hours under heavy write load
> without a node being taken out by zookeeper for GCing > 60 sec or other
> problems like below.
>
> Any help would be greatly appreciated.
>
>
> 2011-05-23T02:34:51.626+0000: 13902.361: [GC 13902.361: [ParNew:
> 249216K->27648K(249216K), 0.1119520 secs] 7546544K->7433319K(8360960K),
> 0.1120390 secs] [Times: user=1.14 sys=0.05, real=0.11 secs]
> 2011-05-23T02:34:52.292+0000: 13903.027: [GC 13903.027: [ParNew:
> 249216K->27648K(249216K), 0.0732800 secs] 7654887K->7506032K(8360960K),
> 0.0733690 secs] [Times: user=0.76 sys=0.02, real=0.08 secs]
> 2011-05-23T02:34:52.721+0000: 13903.456: [CMS-concurrent-mark: 8.137/10.065
> secs] [Times: user=60.86 sys=2.98, real=10.06 secs]
> 2011-05-23T02:34:52.721+0000: 13903.456: [CMS-concurrent-preclean-start]
> 2011-05-23T02:34:52.839+0000: 13903.574: [GC 13903.574: [ParNew:
> 249216K->27648K(249216K), 0.0575510 secs] 7727600K->7562758K(8360960K),
> 0.0576420 secs] [Times: user=0.62 sys=0.02, real=0.06 secs]
> 2011-05-23T02:34:53.190+0000: 13903.925: [GC 13903.925: [ParNew:
> 249171K->27648K(249216K), 0.1108480 secs] 7784281K->7661505K(8360960K),
> 0.1109440 secs] [Times: user=1.10 sys=0.03, real=0.11 secs]
> 2011-05-23T02:34:53.539+0000: 13904.274: [GC 13904.274: [ParNew (promotion
> failed): 249216K->249216K(249216K), 0.1207770 secs]13904.395:
> [CMS2011-05-23T02:34:54.310+0000: 13905.045: [CMS-concurrent-preclean:
> 1.245/1.589 secs] [Times: user=5.99 sys=0.13, real=1.59 secs]
>  (concurrent mode failure)#
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x00002b19debbe665, pid=25868, tid=1078290752
> #
> # JRE version: 6.0_17-b17
> # Java VM: OpenJDK 64-Bit Server VM (14.0-b16 mixed mode linux-amd64 )
> # Derivative: IcedTea6 1.7.10
> # Distribution: Custom build (Wed May  4 23:17:24 EDT 2011)
> # Problematic frame:
> # V  [libjvm.so+0x29d665]
> #
> # An error report file with more information is saved as:
> # .../hbase-0.90.1/hs_err_pid25868.log
> #
> # If you would like to submit a bug report, please include
> # instructions how to reproduce the bug and visit:
> #   http://icedtea.classpath.org/bugzilla
>

Besides this...
JRE version: 6.0_17-b17

Just a silly question ...
What happens if you double the zookeeper time out to 120 seconds?

Also I'm going to assume that you're not running your ZK on the same nodes as your data nodes, but you know what they say about assumptions...
> From: [EMAIL PROTECTED]
> Date: Mon, 23 May 2011 07:33:05 -0700
> Subject: Re: mslab enabled jvm crash
> To: [EMAIL PROTECTED]
>
> Do you have the same problem with a more recent JVM?
>
> On Mon, May 23, 2011 at 4:52 AM, Wayne <[EMAIL PROTECTED]> wrote:
>
> > I have switched to using the mslab enabled java setting to try to avoid GC
> > causing nodes to go awol but it almost appears to be worse. Below is the
> > latest problem with the JVM apparently actually crashing. I am using 0.90.1
> > with an 8GB heap. Is there a recommended JVM and recommended settings to be
> > used? As it stands right now we can not run 24 hours under heavy write load
> > without a node being taken out by zookeeper for GCing > 60 sec or other
> > problems like below.
> >
> > Any help would be greatly appreciated.
> >
> >
> > 2011-05-23T02:34:51.626+0000: 13902.361: [GC 13902.361: [ParNew:
> > 249216K->27648K(249216K), 0.1119520 secs] 7546544K->7433319K(8360960K),
> > 0.1120390 secs] [Times: user=1.14 sys=0.05, real=0.11 secs]
> > 2011-05-23T02:34:52.292+0000: 13903.027: [GC 13903.027: [ParNew:
> > 249216K->27648K(249216K), 0.0732800 secs] 7654887K->7506032K(8360960K),
> > 0.0733690 secs] [Times: user=0.76 sys=0.02, real=0.08 secs]
> > 2011-05-23T02:34:52.721+0000: 13903.456: [CMS-concurrent-mark: 8.137/10.065
> > secs] [Times: user=60.86 sys=2.98, real=10.06 secs]
> > 2011-05-23T02:34:52.721+0000: 13903.456: [CMS-concurrent-preclean-start]
> > 2011-05-23T02:34:52.839+0000: 13903.574: [GC 13903.574: [ParNew:
> > 249216K->27648K(249216K), 0.0575510 secs] 7727600K->7562758K(8360960K),
> > 0.0576420 secs] [Times: user=0.62 sys=0.02, real=0.06 secs]
> > 2011-05-23T02:34:53.190+0000: 13903.925: [GC 13903.925: [ParNew:
> > 249171K->27648K(249216K), 0.1108480 secs] 7784281K->7661505K(8360960K),
> > 0.1109440 secs] [Times: user=1.10 sys=0.03, real=0.11 secs]
> > 2011-05-23T02:34:53.539+0000: 13904.274: [GC 13904.274: [ParNew (promotion
> > failed): 249216K->249216K(249216K), 0.1207770 secs]13904.395:
> > [CMS2011-05-23T02:34:54.310+0000: 13905.045: [CMS-concurrent-preclean:
> > 1.245/1.589 secs] [Times: user=5.99 sys=0.13, real=1.59 secs]
> >  (concurrent mode failure)#
> > # A fatal error has been detected by the Java Runtime Environment:
> > #
> > #  SIGSEGV (0xb) at pc=0x00002b19debbe665, pid=25868, tid=1078290752
> > #
> > # JRE version: 6.0_17-b17
> > # Java VM: OpenJDK 64-Bit Server VM (14.0-b16 mixed mode linux-amd64 )
> > # Derivative: IcedTea6 1.7.10
> > # Distribution: Custom build (Wed May  4 23:17:24 EDT 2011)
> > # Problematic frame:
> > # V  [libjvm.so+0x29d665]
> > #
> > # An error report file with more information is saved as:
> > # .../hbase-0.90.1/hs_err_pid25868.log
> > #
> > # If you would like to submit a bug report, please include
> > # instructions how to reproduce the bug and visit:
> > #   http://icedtea.classpath.org/bugzilla
> >
     

We have not used a more recent JVM with the mslab enabled. In the past (pre
0.90.1) we had a TON of problems (CMF) with more recent JVMs so we avoid
them.

What is the recommended JVM / settings to use with mslab enabled?

Thanks.
On Mon, May 23, 2011 at 10:33 AM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Do you have the same problem with a more recent JVM?
>
> On Mon, May 23, 2011 at 4:52 AM, Wayne <[EMAIL PROTECTED]> wrote:
>
> > I have switched to using the mslab enabled java setting to try to avoid
> GC
> > causing nodes to go awol but it almost appears to be worse. Below is the
> > latest problem with the JVM apparently actually crashing. I am using
> 0.90.1
> > with an 8GB heap. Is there a recommended JVM and recommended settings to
> be
> > used? As it stands right now we can not run 24 hours under heavy write
> load
> > without a node being taken out by zookeeper for GCing > 60 sec or other
> > problems like below.
> >
> > Any help would be greatly appreciated.
> >
> >
> > 2011-05-23T02:34:51.626+0000: 13902.361: [GC 13902.361: [ParNew:
> > 249216K->27648K(249216K), 0.1119520 secs] 7546544K->7433319K(8360960K),
> > 0.1120390 secs] [Times: user=1.14 sys=0.05, real=0.11 secs]
> > 2011-05-23T02:34:52.292+0000: 13903.027: [GC 13903.027: [ParNew:
> > 249216K->27648K(249216K), 0.0732800 secs] 7654887K->7506032K(8360960K),
> > 0.0733690 secs] [Times: user=0.76 sys=0.02, real=0.08 secs]
> > 2011-05-23T02:34:52.721+0000: 13903.456: [CMS-concurrent-mark:
> 8.137/10.065
> > secs] [Times: user=60.86 sys=2.98, real=10.06 secs]
> > 2011-05-23T02:34:52.721+0000: 13903.456: [CMS-concurrent-preclean-start]
> > 2011-05-23T02:34:52.839+0000: 13903.574: [GC 13903.574: [ParNew:
> > 249216K->27648K(249216K), 0.0575510 secs] 7727600K->7562758K(8360960K),
> > 0.0576420 secs] [Times: user=0.62 sys=0.02, real=0.06 secs]
> > 2011-05-23T02:34:53.190+0000: 13903.925: [GC 13903.925: [ParNew:
> > 249171K->27648K(249216K), 0.1108480 secs] 7784281K->7661505K(8360960K),
> > 0.1109440 secs] [Times: user=1.10 sys=0.03, real=0.11 secs]
> > 2011-05-23T02:34:53.539+0000: 13904.274: [GC 13904.274: [ParNew
> (promotion
> > failed): 249216K->249216K(249216K), 0.1207770 secs]13904.395:
> > [CMS2011-05-23T02:34:54.310+0000: 13905.045: [CMS-concurrent-preclean:
> > 1.245/1.589 secs] [Times: user=5.99 sys=0.13, real=1.59 secs]
> >  (concurrent mode failure)#
> > # A fatal error has been detected by the Java Runtime Environment:
> > #
> > #  SIGSEGV (0xb) at pc=0x00002b19debbe665, pid=25868, tid=1078290752
> > #
> > # JRE version: 6.0_17-b17
> > # Java VM: OpenJDK 64-Bit Server VM (14.0-b16 mixed mode linux-amd64 )
> > # Derivative: IcedTea6 1.7.10
> > # Distribution: Custom build (Wed May  4 23:17:24 EDT 2011)
> > # Problematic frame:
> > # V  [libjvm.so+0x29d665]
> > #
> > # An error report file with more information is saved as:
> > # .../hbase-0.90.1/hs_err_pid25868.log
> > #
> > # If you would like to submit a bug report, please include
> > # instructions how to reproduce the bug and visit:
> > #   http://icedtea.classpath.org/bugzilla
> >
>

Zookeeper is not on the same nodes...and yes we could up to 120 seconds but
then we are back to AWOL nodes for 118 seconds is OK which it is not.

Bottom line is the JVM is our enemy here (as it always has been) and we had
high hopes for Todd's fix, and it is not panning out for us...yet.
On Mon, May 23, 2011 at 11:07 AM, Michael Segel
<[EMAIL PROTECTED]>wrote:

>
> Besides this...
> JRE version: 6.0_17-b17
>
> Just a silly question ...
> What happens if you double the zookeeper time out to 120 seconds?
>
> Also I'm going to assume that you're not running your ZK on the same nodes
> as your data nodes, but you know what they say about assumptions...
>
>
> > From: [EMAIL PROTECTED]
> > Date: Mon, 23 May 2011 07:33:05 -0700
> > Subject: Re: mslab enabled jvm crash
> > To: [EMAIL PROTECTED]
> >
> > Do you have the same problem with a more recent JVM?
> >
> > On Mon, May 23, 2011 at 4:52 AM, Wayne <[EMAIL PROTECTED]> wrote:
> >
> > > I have switched to using the mslab enabled java setting to try to avoid
> GC
> > > causing nodes to go awol but it almost appears to be worse. Below is
> the
> > > latest problem with the JVM apparently actually crashing. I am using
> 0.90.1
> > > with an 8GB heap. Is there a recommended JVM and recommended settings
> to be
> > > used? As it stands right now we can not run 24 hours under heavy write
> load
> > > without a node being taken out by zookeeper for GCing > 60 sec or other
> > > problems like below.
> > >
> > > Any help would be greatly appreciated.
> > >
> > >
> > > 2011-05-23T02:34:51.626+0000: 13902.361: [GC 13902.361: [ParNew:
> > > 249216K->27648K(249216K), 0.1119520 secs] 7546544K->7433319K(8360960K),
> > > 0.1120390 secs] [Times: user=1.14 sys=0.05, real=0.11 secs]
> > > 2011-05-23T02:34:52.292+0000: 13903.027: [GC 13903.027: [ParNew:
> > > 249216K->27648K(249216K), 0.0732800 secs] 7654887K->7506032K(8360960K),
> > > 0.0733690 secs] [Times: user=0.76 sys=0.02, real=0.08 secs]
> > > 2011-05-23T02:34:52.721+0000: 13903.456: [CMS-concurrent-mark:
> 8.137/10.065
> > > secs] [Times: user=60.86 sys=2.98, real=10.06 secs]
> > > 2011-05-23T02:34:52.721+0000: 13903.456:
> [CMS-concurrent-preclean-start]
> > > 2011-05-23T02:34:52.839+0000: 13903.574: [GC 13903.574: [ParNew:
> > > 249216K->27648K(249216K), 0.0575510 secs] 7727600K->7562758K(8360960K),
> > > 0.0576420 secs] [Times: user=0.62 sys=0.02, real=0.06 secs]
> > > 2011-05-23T02:34:53.190+0000: 13903.925: [GC 13903.925: [ParNew:
> > > 249171K->27648K(249216K), 0.1108480 secs] 7784281K->7661505K(8360960K),
> > > 0.1109440 secs] [Times: user=1.10 sys=0.03, real=0.11 secs]
> > > 2011-05-23T02:34:53.539+0000: 13904.274: [GC 13904.274: [ParNew
> (promotion
> > > failed): 249216K->249216K(249216K), 0.1207770 secs]13904.395:
> > > [CMS2011-05-23T02:34:54.310+0000: 13905.045: [CMS-concurrent-preclean:
> > > 1.245/1.589 secs] [Times: user=5.99 sys=0.13, real=1.59 secs]
> > >  (concurrent mode failure)#
> > > # A fatal error has been detected by the Java Runtime Environment:
> > > #
> > > #  SIGSEGV (0xb) at pc=0x00002b19debbe665, pid=25868, tid=1078290752
> > > #
> > > # JRE version: 6.0_17-b17
> > > # Java VM: OpenJDK 64-Bit Server VM (14.0-b16 mixed mode linux-amd64 )
> > > # Derivative: IcedTea6 1.7.10
> > > # Distribution: Custom build (Wed May  4 23:17:24 EDT 2011)
> > > # Problematic frame:
> > > # V  [libjvm.so+0x29d665]
> > > #
> > > # An error report file with more information is saved as:
> > > # .../hbase-0.90.1/hs_err_pid25868.log
> > > #
> > > # If you would like to submit a bug report, please include
> > > # instructions how to reproduce the bug and visit:
> > > #   http://icedtea.classpath.org/bugzilla
> > >
>
>

u17 was release a year and a half ago.  Latest is u25 (we run u24).
What kind of 'crash' are you seeing?  What is your OS?
St.Ack

On Mon, May 23, 2011 at 8:19 AM, Wayne <[EMAIL PROTECTED]> wrote:
> Zookeeper is not on the same nodes...and yes we could up to 120 seconds but
> then we are back to AWOL nodes for 118 seconds is OK which it is not.
>
> Bottom line is the JVM is our enemy here (as it always has been) and we had
> high hopes for Todd's fix, and it is not panning out for us...yet.
>
>
> On Mon, May 23, 2011 at 11:07 AM, Michael Segel
> <[EMAIL PROTECTED]>wrote:
>
>>
>> Besides this...
>> JRE version: 6.0_17-b17
>>
>> Just a silly question ...
>> What happens if you double the zookeeper time out to 120 seconds?
>>
>> Also I'm going to assume that you're not running your ZK on the same nodes
>> as your data nodes, but you know what they say about assumptions...
>>
>>
>> > From: [EMAIL PROTECTED]
>> > Date: Mon, 23 May 2011 07:33:05 -0700
>> > Subject: Re: mslab enabled jvm crash
>> > To: [EMAIL PROTECTED]
>> >
>> > Do you have the same problem with a more recent JVM?
>> >
>> > On Mon, May 23, 2011 at 4:52 AM, Wayne <[EMAIL PROTECTED]> wrote:
>> >
>> > > I have switched to using the mslab enabled java setting to try to avoid
>> GC
>> > > causing nodes to go awol but it almost appears to be worse. Below is
>> the
>> > > latest problem with the JVM apparently actually crashing. I am using
>> 0.90.1
>> > > with an 8GB heap. Is there a recommended JVM and recommended settings
>> to be
>> > > used? As it stands right now we can not run 24 hours under heavy write
>> load
>> > > without a node being taken out by zookeeper for GCing > 60 sec or other
>> > > problems like below.
>> > >
>> > > Any help would be greatly appreciated.
>> > >
>> > >
>> > > 2011-05-23T02:34:51.626+0000: 13902.361: [GC 13902.361: [ParNew:
>> > > 249216K->27648K(249216K), 0.1119520 secs] 7546544K->7433319K(8360960K),
>> > > 0.1120390 secs] [Times: user=1.14 sys=0.05, real=0.11 secs]
>> > > 2011-05-23T02:34:52.292+0000: 13903.027: [GC 13903.027: [ParNew:
>> > > 249216K->27648K(249216K), 0.0732800 secs] 7654887K->7506032K(8360960K),
>> > > 0.0733690 secs] [Times: user=0.76 sys=0.02, real=0.08 secs]
>> > > 2011-05-23T02:34:52.721+0000: 13903.456: [CMS-concurrent-mark:
>> 8.137/10.065
>> > > secs] [Times: user=60.86 sys=2.98, real=10.06 secs]
>> > > 2011-05-23T02:34:52.721+0000: 13903.456:
>> [CMS-concurrent-preclean-start]
>> > > 2011-05-23T02:34:52.839+0000: 13903.574: [GC 13903.574: [ParNew:
>> > > 249216K->27648K(249216K), 0.0575510 secs] 7727600K->7562758K(8360960K),
>> > > 0.0576420 secs] [Times: user=0.62 sys=0.02, real=0.06 secs]
>> > > 2011-05-23T02:34:53.190+0000: 13903.925: [GC 13903.925: [ParNew:
>> > > 249171K->27648K(249216K), 0.1108480 secs] 7784281K->7661505K(8360960K),
>> > > 0.1109440 secs] [Times: user=1.10 sys=0.03, real=0.11 secs]
>> > > 2011-05-23T02:34:53.539+0000: 13904.274: [GC 13904.274: [ParNew
>> (promotion
>> > > failed): 249216K->249216K(249216K), 0.1207770 secs]13904.395:
>> > > [CMS2011-05-23T02:34:54.310+0000: 13905.045: [CMS-concurrent-preclean:
>> > > 1.245/1.589 secs] [Times: user=5.99 sys=0.13, real=1.59 secs]
>> > >  (concurrent mode failure)#
>> > > # A fatal error has been detected by the Java Runtime Environment:
>> > > #
>> > > #  SIGSEGV (0xb) at pc=0x00002b19debbe665, pid=25868, tid=1078290752
>> > > #
>> > > # JRE version: 6.0_17-b17
>> > > # Java VM: OpenJDK 64-Bit Server VM (14.0-b16 mixed mode linux-amd64 )
>> > > # Derivative: IcedTea6 1.7.10
>> > > # Distribution: Custom build (Wed May  4 23:17:24 EDT 2011)
>> > > # Problematic frame:
>> > > # V  [libjvm.so+0x29d665]
>> > > #
>> > > # An error report file with more information is saved as:
>> > > # .../hbase-0.90.1/hs_err_pid25868.log
>> > > #
>> > > # If you would like to submit a bug report, please include
>> > > # instructions how to reproduce the bug and visit:
>> > > #   http://icedtea.classpath.org/bugzilla
>> > >
>>
>>
>

Our experience with any newer JVM was that fragmentation was much much worse
and Concurrent Mode Failures were rampant. We kept moving back in releases
 to get to what we use now. We are on CentOS 5.5. We will try to use u24.

Thanks.

On Mon, May 23, 2011 at 11:29 AM, Stack <[EMAIL PROTECTED]> wrote:

> u17 was release a year and a half ago.  Latest is u25 (we run u24).
> What kind of 'crash' are you seeing?  What is your OS?
> St.Ack
>
> On Mon, May 23, 2011 at 8:19 AM, Wayne <[EMAIL PROTECTED]> wrote:
> > Zookeeper is not on the same nodes...and yes we could up to 120 seconds
> but
> > then we are back to AWOL nodes for 118 seconds is OK which it is not.
> >
> > Bottom line is the JVM is our enemy here (as it always has been) and we
> had
> > high hopes for Todd's fix, and it is not panning out for us...yet.
> >
> >
> > On Mon, May 23, 2011 at 11:07 AM, Michael Segel
> > <[EMAIL PROTECTED]>wrote:
> >
> >>
> >> Besides this...
> >> JRE version: 6.0_17-b17
> >>
> >> Just a silly question ...
> >> What happens if you double the zookeeper time out to 120 seconds?
> >>
> >> Also I'm going to assume that you're not running your ZK on the same
> nodes
> >> as your data nodes, but you know what they say about assumptions...
> >>
> >>
> >> > From: [EMAIL PROTECTED]
> >> > Date: Mon, 23 May 2011 07:33:05 -0700
> >> > Subject: Re: mslab enabled jvm crash
> >> > To: [EMAIL PROTECTED]
> >> >
> >> > Do you have the same problem with a more recent JVM?
> >> >
> >> > On Mon, May 23, 2011 at 4:52 AM, Wayne <[EMAIL PROTECTED]> wrote:
> >> >
> >> > > I have switched to using the mslab enabled java setting to try to
> avoid
> >> GC
> >> > > causing nodes to go awol but it almost appears to be worse. Below is
> >> the
> >> > > latest problem with the JVM apparently actually crashing. I am using
> >> 0.90.1
> >> > > with an 8GB heap. Is there a recommended JVM and recommended
> settings
> >> to be
> >> > > used? As it stands right now we can not run 24 hours under heavy
> write
> >> load
> >> > > without a node being taken out by zookeeper for GCing > 60 sec or
> other
> >> > > problems like below.
> >> > >
> >> > > Any help would be greatly appreciated.
> >> > >
> >> > >
> >> > > 2011-05-23T02:34:51.626+0000: 13902.361: [GC 13902.361: [ParNew:
> >> > > 249216K->27648K(249216K), 0.1119520 secs]
> 7546544K->7433319K(8360960K),
> >> > > 0.1120390 secs] [Times: user=1.14 sys=0.05, real=0.11 secs]
> >> > > 2011-05-23T02:34:52.292+0000: 13903.027: [GC 13903.027: [ParNew:
> >> > > 249216K->27648K(249216K), 0.0732800 secs]
> 7654887K->7506032K(8360960K),
> >> > > 0.0733690 secs] [Times: user=0.76 sys=0.02, real=0.08 secs]
> >> > > 2011-05-23T02:34:52.721+0000: 13903.456: [CMS-concurrent-mark:
> >> 8.137/10.065
> >> > > secs] [Times: user=60.86 sys=2.98, real=10.06 secs]
> >> > > 2011-05-23T02:34:52.721+0000: 13903.456:
> >> [CMS-concurrent-preclean-start]
> >> > > 2011-05-23T02:34:52.839+0000: 13903.574: [GC 13903.574: [ParNew:
> >> > > 249216K->27648K(249216K), 0.0575510 secs]
> 7727600K->7562758K(8360960K),
> >> > > 0.0576420 secs] [Times: user=0.62 sys=0.02, real=0.06 secs]
> >> > > 2011-05-23T02:34:53.190+0000: 13903.925: [GC 13903.925: [ParNew:
> >> > > 249171K->27648K(249216K), 0.1108480 secs]
> 7784281K->7661505K(8360960K),
> >> > > 0.1109440 secs] [Times: user=1.10 sys=0.03, real=0.11 secs]
> >> > > 2011-05-23T02:34:53.539+0000: 13904.274: [GC 13904.274: [ParNew
> >> (promotion
> >> > > failed): 249216K->249216K(249216K), 0.1207770 secs]13904.395:
> >> > > [CMS2011-05-23T02:34:54.310+0000: 13905.045:
> [CMS-concurrent-preclean:
> >> > > 1.245/1.589 secs] [Times: user=5.99 sys=0.13, real=1.59 secs]
> >> > >  (concurrent mode failure)#
> >> > > # A fatal error has been detected by the Java Runtime Environment:
> >> > > #
> >> > > #  SIGSEGV (0xb) at pc=0x00002b19debbe665, pid=25868, tid=1078290752
> >> > > #
> >> > > # JRE version: 6.0_17-b17
> >> > > # Java VM: OpenJDK 64-Bit Server VM (14.0-b16 mixed mode linux-amd64
> )
> >> > > # Derivative: IcedTea6 1.7.10

On Mon, May 23, 2011 at 8:42 AM, Wayne <[EMAIL PROTECTED]> wrote:
> Our experience with any newer JVM was that fragmentation was much much worse
> and Concurrent Mode Failures were rampant. We kept moving back in releases
>  to get to what we use now. We are on CentOS 5.5. We will try to use u24.
>

CMS's you should be able to configure around.  u21 was supposed to
make improvements to put off frag but apparently made it worse.  Try
u25, the latest.  Also google for other's experience with JVMs up on
CentOS 5.5.

St.Ack

We switched to u25 and reverted the JVM settings to those recommended. Now
we have concurrent mode failures that occur lasting more than 60 seconds
while not under hardly any load....

Below are the entries from the JVM log. Of course we can up the zookeeper
timeout to 2 min or 10 min for that matter but it does not address the
underlying issue. Sorry but I can not confirm that the changes for the new
GC settings have any affect. It appears no better or even worse as this
problem below occurred while the cluster was almost idle.
2011-05-25T14:15:45.518+0000: 150358.023: [GC 150358.023: [ParNew:
230155K->27648K(249216K), 0.0653880 secs] 7754007K->7586719K(8360960K)
icms_dc=100 , 0.0654900 secs] [Times: user=0.78 sys=0.00, real=0.06 secs]
2011-05-25T14:15:45.906+0000: 150358.410: [GC 150358.410: [ParNew (promotion
failed): 249216K->249216K(249216K), 0.5768350 secs]150358.987:
[CMS2011-05-25T14:16:44.404+0000: 150416.909: [CMS-concurrent-sweep:
87.667/92.820 secs] [Times: user=182.64 sys=1.37, real=92.80 secs]
 (concurrent mode failure)[Unloading class
sun.reflect.GeneratedMethodAccessor20]
[Unloading class sun.reflect.GeneratedMethodAccessor29]
[Unloading class sun.reflect.GeneratedMethodAccessor31]
[Unloading class sun.reflect.GeneratedMethodAccessor30]
[Unloading class sun.reflect.GeneratedMethodAccessor32]
[Unloading class sun.reflect.GeneratedMethodAccessor1]
[Unloading class sun.reflect.GeneratedMethodAccessor17]
[Unloading class sun.reflect.GeneratedMethodAccessor28]
: 7621159K->2503625K(8111744K), 63.3195660 secs]
7798327K->2503625K(8360960K), [CMS Perm : 20128K->20106K(33580K)]
icms_dc=100 , 63.8965450 secs] [Times: user=69.50 sys=0.01, real=63.89
secs]

On Mon, May 23, 2011 at 12:04 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Mon, May 23, 2011 at 8:42 AM, Wayne <[EMAIL PROTECTED]> wrote:
> > Our experience with any newer JVM was that fragmentation was much much
> worse
> > and Concurrent Mode Failures were rampant. We kept moving back in
> releases
> >  to get to what we use now. We are on CentOS 5.5. We will try to use u24.
> >
>
> CMS's you should be able to configure around.  u21 was supposed to
> make improvements to put off frag but apparently made it worse.  Try
> u25, the latest.  Also google for other's experience with JVMs up on
> CentOS 5.5.
>
> St.Ack
>

Hi Wayne,

Looks like your RAM might be oversubscribed. Could you paste your
hbase-site.xml and hbase-env.sh files? Also looks like you have some
strange GC settings on (eg perm gen collection which we don't really
need)

If you can paste a larger segment of GC logs (enough to include at
least two or three of the full gc pauses) that would be helpful.

-Todd

On Wed, May 25, 2011 at 10:32 AM, Wayne <[EMAIL PROTECTED]> wrote:
> We switched to u25 and reverted the JVM settings to those recommended. Now
> we have concurrent mode failures that occur lasting more than 60 seconds
> while not under hardly any load....
>
> Below are the entries from the JVM log. Of course we can up the zookeeper
> timeout to 2 min or 10 min for that matter but it does not address the
> underlying issue. Sorry but I can not confirm that the changes for the new
> GC settings have any affect. It appears no better or even worse as this
> problem below occurred while the cluster was almost idle.
>
>
> 2011-05-25T14:15:45.518+0000: 150358.023: [GC 150358.023: [ParNew:
> 230155K->27648K(249216K), 0.0653880 secs] 7754007K->7586719K(8360960K)
> icms_dc=100 , 0.0654900 secs] [Times: user=0.78 sys=0.00, real=0.06 secs]
> 2011-05-25T14:15:45.906+0000: 150358.410: [GC 150358.410: [ParNew (promotion
> failed): 249216K->249216K(249216K), 0.5768350 secs]150358.987:
> [CMS2011-05-25T14:16:44.404+0000: 150416.909: [CMS-concurrent-sweep:
> 87.667/92.820 secs] [Times: user=182.64 sys=1.37, real=92.80 secs]
>  (concurrent mode failure)[Unloading class
> sun.reflect.GeneratedMethodAccessor20]
> [Unloading class sun.reflect.GeneratedMethodAccessor29]
> [Unloading class sun.reflect.GeneratedMethodAccessor31]
> [Unloading class sun.reflect.GeneratedMethodAccessor30]
> [Unloading class sun.reflect.GeneratedMethodAccessor32]
> [Unloading class sun.reflect.GeneratedMethodAccessor1]
> [Unloading class sun.reflect.GeneratedMethodAccessor17]
> [Unloading class sun.reflect.GeneratedMethodAccessor28]
> : 7621159K->2503625K(8111744K), 63.3195660 secs]
> 7798327K->2503625K(8360960K), [CMS Perm : 20128K->20106K(33580K)]
> icms_dc=100 , 63.8965450 secs] [Times: user=69.50 sys=0.01, real=63.89
> secs]
>
>
>
> On Mon, May 23, 2011 at 12:04 PM, Stack <[EMAIL PROTECTED]> wrote:
>
>> On Mon, May 23, 2011 at 8:42 AM, Wayne <[EMAIL PROTECTED]> wrote:
>> > Our experience with any newer JVM was that fragmentation was much much
>> worse
>> > and Concurrent Mode Failures were rampant. We kept moving back in
>> releases
>> >  to get to what we use now. We are on CentOS 5.5. We will try to use u24.
>> >
>>
>> CMS's you should be able to configure around.  u21 was supposed to
>> make improvements to put off frag but apparently made it worse.  Try
>> u25, the latest.  Also google for other's experience with JVMs up on
>> CentOS 5.5.
>>
>> St.Ack
>>
>

--
Todd Lipcon
Software Engineer, Cloudera

I tried to turn off all special JVM settings we have tried in the past.
Below are link to the requested configs. I will try to find more logs for
the full GC. We just made the switch and on this node it has
only occurred once in the scope of the current log (it may have rolled?).

Thanks.

http://pastebin.com/ca13aMRu

http://pastebin.com/9KfRZFBW
On Wed, May 25, 2011 at 1:42 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> Hi Wayne,
>
> Looks like your RAM might be oversubscribed. Could you paste your
> hbase-site.xml and hbase-env.sh files? Also looks like you have some
> strange GC settings on (eg perm gen collection which we don't really
> need)
>
> If you can paste a larger segment of GC logs (enough to include at
> least two or three of the full gc pauses) that would be helpful.
>
> -Todd
>
> On Wed, May 25, 2011 at 10:32 AM, Wayne <[EMAIL PROTECTED]> wrote:
> > We switched to u25 and reverted the JVM settings to those recommended.
> Now
> > we have concurrent mode failures that occur lasting more than 60 seconds
> > while not under hardly any load....
> >
> > Below are the entries from the JVM log. Of course we can up the zookeeper
> > timeout to 2 min or 10 min for that matter but it does not address the
> > underlying issue. Sorry but I can not confirm that the changes for the
> new
> > GC settings have any affect. It appears no better or even worse as this
> > problem below occurred while the cluster was almost idle.
> >
> >
> > 2011-05-25T14:15:45.518+0000: 150358.023: [GC 150358.023: [ParNew:
> > 230155K->27648K(249216K), 0.0653880 secs] 7754007K->7586719K(8360960K)
> > icms_dc=100 , 0.0654900 secs] [Times: user=0.78 sys=0.00, real=0.06 secs]
> > 2011-05-25T14:15:45.906+0000: 150358.410: [GC 150358.410: [ParNew
> (promotion
> > failed): 249216K->249216K(249216K), 0.5768350 secs]150358.987:
> > [CMS2011-05-25T14:16:44.404+0000: 150416.909: [CMS-concurrent-sweep:
> > 87.667/92.820 secs] [Times: user=182.64 sys=1.37, real=92.80 secs]
> >  (concurrent mode failure)[Unloading class
> > sun.reflect.GeneratedMethodAccessor20]
> > [Unloading class sun.reflect.GeneratedMethodAccessor29]
> > [Unloading class sun.reflect.GeneratedMethodAccessor31]
> > [Unloading class sun.reflect.GeneratedMethodAccessor30]
> > [Unloading class sun.reflect.GeneratedMethodAccessor32]
> > [Unloading class sun.reflect.GeneratedMethodAccessor1]
> > [Unloading class sun.reflect.GeneratedMethodAccessor17]
> > [Unloading class sun.reflect.GeneratedMethodAccessor28]
> > : 7621159K->2503625K(8111744K), 63.3195660 secs]
> > 7798327K->2503625K(8360960K), [CMS Perm : 20128K->20106K(33580K)]
> > icms_dc=100 , 63.8965450 secs] [Times: user=69.50 sys=0.01, real=63.89
> > secs]
> >
> >
> >
> > On Mon, May 23, 2011 at 12:04 PM, Stack <[EMAIL PROTECTED]> wrote:
> >
> >> On Mon, May 23, 2011 at 8:42 AM, Wayne <[EMAIL PROTECTED]> wrote:
> >> > Our experience with any newer JVM was that fragmentation was much much
> >> worse
> >> > and Concurrent Mode Failures were rampant. We kept moving back in
> >> releases
> >> >  to get to what we use now. We are on CentOS 5.5. We will try to use
> u24.
> >> >
> >>
> >> CMS's you should be able to configure around.  u21 was supposed to
> >> make improvements to put off frag but apparently made it worse.  Try
> >> u25, the latest.  Also google for other's experience with JVMs up on
> >> CentOS 5.5.
> >>
> >> St.Ack
> >>
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

For your GC settings:
- i wouldn't tune newratio or survivor ratio at all
- if you want to tame your young GC pauses, use -Xmn to pick a new
size - eg -Xmn256m
- turn off CMS Incremental Mode if you're running on real server hardware

HBase settings:
- 1% of heap to block cache seems strange. maybe you should just be
turning it off at the table level?
- mslab is definitely experimental. If you can compare with it on vs
with it off, that would be a good data point.
-Todd

On Wed, May 25, 2011 at 11:08 AM, Wayne <[EMAIL PROTECTED]> wrote:
> I tried to turn off all special JVM settings we have tried in the past.
> Below are link to the requested configs. I will try to find more logs for
> the full GC. We just made the switch and on this node it has
> only occurred once in the scope of the current log (it may have rolled?).
>
> Thanks.
>
> http://pastebin.com/ca13aMRu
>
> http://pastebin.com/9KfRZFBW
>
>
> On Wed, May 25, 2011 at 1:42 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> Hi Wayne,
>>
>> Looks like your RAM might be oversubscribed. Could you paste your
>> hbase-site.xml and hbase-env.sh files? Also looks like you have some
>> strange GC settings on (eg perm gen collection which we don't really
>> need)
>>
>> If you can paste a larger segment of GC logs (enough to include at
>> least two or three of the full gc pauses) that would be helpful.
>>
>> -Todd
>>
>> On Wed, May 25, 2011 at 10:32 AM, Wayne <[EMAIL PROTECTED]> wrote:
>> > We switched to u25 and reverted the JVM settings to those recommended.
>> Now
>> > we have concurrent mode failures that occur lasting more than 60 seconds
>> > while not under hardly any load....
>> >
>> > Below are the entries from the JVM log. Of course we can up the zookeeper
>> > timeout to 2 min or 10 min for that matter but it does not address the
>> > underlying issue. Sorry but I can not confirm that the changes for the
>> new
>> > GC settings have any affect. It appears no better or even worse as this
>> > problem below occurred while the cluster was almost idle.
>> >
>> >
>> > 2011-05-25T14:15:45.518+0000: 150358.023: [GC 150358.023: [ParNew:
>> > 230155K->27648K(249216K), 0.0653880 secs] 7754007K->7586719K(8360960K)
>> > icms_dc=100 , 0.0654900 secs] [Times: user=0.78 sys=0.00, real=0.06 secs]
>> > 2011-05-25T14:15:45.906+0000: 150358.410: [GC 150358.410: [ParNew
>> (promotion
>> > failed): 249216K->249216K(249216K), 0.5768350 secs]150358.987:
>> > [CMS2011-05-25T14:16:44.404+0000: 150416.909: [CMS-concurrent-sweep:
>> > 87.667/92.820 secs] [Times: user=182.64 sys=1.37, real=92.80 secs]
>> >  (concurrent mode failure)[Unloading class
>> > sun.reflect.GeneratedMethodAccessor20]
>> > [Unloading class sun.reflect.GeneratedMethodAccessor29]
>> > [Unloading class sun.reflect.GeneratedMethodAccessor31]
>> > [Unloading class sun.reflect.GeneratedMethodAccessor30]
>> > [Unloading class sun.reflect.GeneratedMethodAccessor32]
>> > [Unloading class sun.reflect.GeneratedMethodAccessor1]
>> > [Unloading class sun.reflect.GeneratedMethodAccessor17]
>> > [Unloading class sun.reflect.GeneratedMethodAccessor28]
>> > : 7621159K->2503625K(8111744K), 63.3195660 secs]
>> > 7798327K->2503625K(8360960K), [CMS Perm : 20128K->20106K(33580K)]
>> > icms_dc=100 , 63.8965450 secs] [Times: user=69.50 sys=0.01, real=63.89
>> > secs]
>> >
>> >
>> >
>> > On Mon, May 23, 2011 at 12:04 PM, Stack <[EMAIL PROTECTED]> wrote:
>> >
>> >> On Mon, May 23, 2011 at 8:42 AM, Wayne <[EMAIL PROTECTED]> wrote:
>> >> > Our experience with any newer JVM was that fragmentation was much much
>> >> worse
>> >> > and Concurrent Mode Failures were rampant. We kept moving back in
>> >> releases
>> >> >  to get to what we use now. We are on CentOS 5.5. We will try to use
>> u24.
>> >> >
>> >>
>> >> CMS's you should be able to configure around.  u21 was supposed to
>> >> make improvements to put off frag but apparently made it worse.  Try
>> >> u25, the latest.  Also google for other's experience with JVMs up on
>> >> CentOS 5.5.
>> >>
>> >> St.Ack

Todd Lipcon
Software Engineer, Cloudera

We have the line commented out with the new ratio. I will turn off the
incremental mode. We do have cache turned off on the table level and have
set to 1% for .meta. only. We do not use the block cache.

I will keep testing. Frankly u25 scares as well as older JVMs seem much
better based on previous testing.

Thanks.

On Wed, May 25, 2011 at 2:13 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> For your GC settings:
> - i wouldn't tune newratio or survivor ratio at all
> - if you want to tame your young GC pauses, use -Xmn to pick a new
> size - eg -Xmn256m
> - turn off CMS Incremental Mode if you're running on real server hardware
>
> HBase settings:
> - 1% of heap to block cache seems strange. maybe you should just be
> turning it off at the table level?
> - mslab is definitely experimental. If you can compare with it on vs
> with it off, that would be a good data point.
>
>
> -Todd
>
> On Wed, May 25, 2011 at 11:08 AM, Wayne <[EMAIL PROTECTED]> wrote:
> > I tried to turn off all special JVM settings we have tried in the past.
> > Below are link to the requested configs. I will try to find more logs for
> > the full GC. We just made the switch and on this node it has
> > only occurred once in the scope of the current log (it may have rolled?).
> >
> > Thanks.
> >
> > http://pastebin.com/ca13aMRu
> >
> > http://pastebin.com/9KfRZFBW
> >
> >
> > On Wed, May 25, 2011 at 1:42 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> >
> >> Hi Wayne,
> >>
> >> Looks like your RAM might be oversubscribed. Could you paste your
> >> hbase-site.xml and hbase-env.sh files? Also looks like you have some
> >> strange GC settings on (eg perm gen collection which we don't really
> >> need)
> >>
> >> If you can paste a larger segment of GC logs (enough to include at
> >> least two or three of the full gc pauses) that would be helpful.
> >>
> >> -Todd
> >>
> >> On Wed, May 25, 2011 at 10:32 AM, Wayne <[EMAIL PROTECTED]> wrote:
> >> > We switched to u25 and reverted the JVM settings to those recommended.
> >> Now
> >> > we have concurrent mode failures that occur lasting more than 60
> seconds
> >> > while not under hardly any load....
> >> >
> >> > Below are the entries from the JVM log. Of course we can up the
> zookeeper
> >> > timeout to 2 min or 10 min for that matter but it does not address the
> >> > underlying issue. Sorry but I can not confirm that the changes for the
> >> new
> >> > GC settings have any affect. It appears no better or even worse as
> this
> >> > problem below occurred while the cluster was almost idle.
> >> >
> >> >
> >> > 2011-05-25T14:15:45.518+0000: 150358.023: [GC 150358.023: [ParNew:
> >> > 230155K->27648K(249216K), 0.0653880 secs] 7754007K->7586719K(8360960K)
> >> > icms_dc=100 , 0.0654900 secs] [Times: user=0.78 sys=0.00, real=0.06
> secs]
> >> > 2011-05-25T14:15:45.906+0000: 150358.410: [GC 150358.410: [ParNew
> >> (promotion
> >> > failed): 249216K->249216K(249216K), 0.5768350 secs]150358.987:
> >> > [CMS2011-05-25T14:16:44.404+0000: 150416.909: [CMS-concurrent-sweep:
> >> > 87.667/92.820 secs] [Times: user=182.64 sys=1.37, real=92.80 secs]
> >> >  (concurrent mode failure)[Unloading class
> >> > sun.reflect.GeneratedMethodAccessor20]
> >> > [Unloading class sun.reflect.GeneratedMethodAccessor29]
> >> > [Unloading class sun.reflect.GeneratedMethodAccessor31]
> >> > [Unloading class sun.reflect.GeneratedMethodAccessor30]
> >> > [Unloading class sun.reflect.GeneratedMethodAccessor32]
> >> > [Unloading class sun.reflect.GeneratedMethodAccessor1]
> >> > [Unloading class sun.reflect.GeneratedMethodAccessor17]
> >> > [Unloading class sun.reflect.GeneratedMethodAccessor28]
> >> > : 7621159K->2503625K(8111744K), 63.3195660 secs]
> >> > 7798327K->2503625K(8360960K), [CMS Perm : 20128K->20106K(33580K)]
> >> > icms_dc=100 , 63.8965450 secs] [Times: user=69.50 sys=0.01, real=63.89
> >> > secs]
> >> >
> >> >
> >> >
> >> > On Mon, May 23, 2011 at 12:04 PM, Stack <[EMAIL PROTECTED]> wrote:
> >> >
> >> >> On Mon, May 23, 2011 at 8:42 AM, Wayne <[EMAIL PROTECTED]> wrote:

Wayne,

It should be recognized that your experiences are a bit out of the norm
here.  Many hbase installations use more recent JVM's without problems.

As such, it may be premature to point the finger at the JVM as opposed to
the workload or environmental factors.  Such a premature diagnosis can make
getting at the true cause of problems more  difficult.

On Wed, May 25, 2011 at 11:21 AM, Wayne <[EMAIL PROTECTED]> wrote:

> I will keep testing. Frankly u25 scares as well as older JVMs seem much
> better based on previous testing.
>

Most hbase installations also seem to recommend bulk inserts for loading
data. We are pushing it more than most in terms of actually using the client
API to load large volumes of data. We keep delaying putting hbase into
production as nodes going awol for as much as 2+ minutes we can not accept
as normal. Cassandra had the exact same problems for us (plus a lot of other
issues), and we all know what is common between the two.

On Wed, May 25, 2011 at 2:39 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> Wayne,
>
> It should be recognized that your experiences are a bit out of the norm
> here.  Many hbase installations use more recent JVM's without problems.
>
> As such, it may be premature to point the finger at the JVM as opposed to
> the workload or environmental factors.  Such a premature diagnosis can make
> getting at the true cause of problems more  difficult.
>
> On Wed, May 25, 2011 at 11:21 AM, Wayne <[EMAIL PROTECTED]> wrote:
>
> > I will keep testing. Frankly u25 scares as well as older JVMs seem much
> > better based on previous testing.
> >
>

On Wed, May 25, 2011 at 11:08 AM, Wayne <[EMAIL PROTECTED]> wrote:
> I tried to turn off all special JVM settings we have tried in the past.
> Below are link to the requested configs. I will try to find more logs for
> the full GC. We just made the switch and on this node it has
> only occurred once in the scope of the current log (it may have rolled?).
>
> Thanks.
>
> http://pastebin.com/ca13aMRu
>

You are running w/ the defaults.  Would suggest you do as Todd says;
turn off incremental and can you try giving hbase more RAM (or cap
your parnew)?  From the above, if representative, it would seem that
your young gen is riding at about 256M?  Is that so?   Try capping it
at something smaller?  192M or 128M?  Another thing to try is
startting the CMS earlier.  Usually this is recommended as means of
putting off concurrent mode failures as opposed to promotion failures
but it might be enough to ensure space to do the parnew promotion to
old gen (This is an oldie but might I'd say basic premise holds:
http://blogs.oracle.com/jonthecollector/entry/when_the_sum_of_the).

How long do you run before you hit this pause?  Or is this only the
first instance?

> http://pastebin.com/9KfRZFBW

It looks like you are keeping your zk logs in /tmp, is that so?  See
hbase.zookeeper.property.dataDir.

You are flushing at 4x the default.  You might try running at defaults
to see if it lesser retention brings some relief.

St.Ack
P.S. We do not do bulk loading.
>
>
> On Wed, May 25, 2011 at 1:42 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> Hi Wayne,
>>
>> Looks like your RAM might be oversubscribed. Could you paste your
>> hbase-site.xml and hbase-env.sh files? Also looks like you have some
>> strange GC settings on (eg perm gen collection which we don't really
>> need)
>>
>> If you can paste a larger segment of GC logs (enough to include at
>> least two or three of the full gc pauses) that would be helpful.
>>
>> -Todd
>>
>> On Wed, May 25, 2011 at 10:32 AM, Wayne <[EMAIL PROTECTED]> wrote:
>> > We switched to u25 and reverted the JVM settings to those recommended.
>> Now
>> > we have concurrent mode failures that occur lasting more than 60 seconds
>> > while not under hardly any load....
>> >
>> > Below are the entries from the JVM log. Of course we can up the zookeeper
>> > timeout to 2 min or 10 min for that matter but it does not address the
>> > underlying issue. Sorry but I can not confirm that the changes for the
>> new
>> > GC settings have any affect. It appears no better or even worse as this
>> > problem below occurred while the cluster was almost idle.
>> >
>> >
>> > 2011-05-25T14:15:45.518+0000: 150358.023: [GC 150358.023: [ParNew:
>> > 230155K->27648K(249216K), 0.0653880 secs] 7754007K->7586719K(8360960K)
>> > icms_dc=100 , 0.0654900 secs] [Times: user=0.78 sys=0.00, real=0.06 secs]
>> > 2011-05-25T14:15:45.906+0000: 150358.410: [GC 150358.410: [ParNew
>> (promotion
>> > failed): 249216K->249216K(249216K), 0.5768350 secs]150358.987:
>> > [CMS2011-05-25T14:16:44.404+0000: 150416.909: [CMS-concurrent-sweep:
>> > 87.667/92.820 secs] [Times: user=182.64 sys=1.37, real=92.80 secs]
>> >  (concurrent mode failure)[Unloading class
>> > sun.reflect.GeneratedMethodAccessor20]
>> > [Unloading class sun.reflect.GeneratedMethodAccessor29]
>> > [Unloading class sun.reflect.GeneratedMethodAccessor31]
>> > [Unloading class sun.reflect.GeneratedMethodAccessor30]
>> > [Unloading class sun.reflect.GeneratedMethodAccessor32]
>> > [Unloading class sun.reflect.GeneratedMethodAccessor1]
>> > [Unloading class sun.reflect.GeneratedMethodAccessor17]
>> > [Unloading class sun.reflect.GeneratedMethodAccessor28]
>> > : 7621159K->2503625K(8111744K), 63.3195660 secs]
>> > 7798327K->2503625K(8360960K), [CMS Perm : 20128K->20106K(33580K)]
>> > icms_dc=100 , 63.8965450 secs] [Times: user=69.50 sys=0.01, real=63.89
>> > secs]
>> >
>> >
>> >
>> > On Mon, May 23, 2011 at 12:04 PM, Stack <[EMAIL PROTECTED]> wrote:
>> >
>> >> On Mon, May 23, 2011 at 8:42 AM, Wayne <[EMAIL PROTECTED]> wrote:

I have restarted kicking in CMS earlier (65%) and turning off the
incremental. We have an 8g heap...should we go to 10g (24g in box)? More
memory for the JVM has never seemed to be better...though maybe with lots of
hot regions and our flush size we might be pushing it? Should we up the 50%
for memstore higher given that we do not use cache?

The 4x default flushing I would prefer even higher. We have the up to 90s
pause due to too many files occurring all of the time as it is. We could up
the file count from 15 to ?? but then not sure that is good either. If a
region is getting hammered with writes nothing we do seems to stop it with
getting paused very very frequently.

I will look into the zookeeper log location, never looked at those...

Thanks for the help.

On Wed, May 25, 2011 at 3:23 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Wed, May 25, 2011 at 11:08 AM, Wayne <[EMAIL PROTECTED]> wrote:
> > I tried to turn off all special JVM settings we have tried in the past.
> > Below are link to the requested configs. I will try to find more logs for
> > the full GC. We just made the switch and on this node it has
> > only occurred once in the scope of the current log (it may have rolled?).
> >
> > Thanks.
> >
> > http://pastebin.com/ca13aMRu
> >
>
> You are running w/ the defaults.  Would suggest you do as Todd says;
> turn off incremental and can you try giving hbase more RAM (or cap
> your parnew)?  From the above, if representative, it would seem that
> your young gen is riding at about 256M?  Is that so?   Try capping it
> at something smaller?  192M or 128M?  Another thing to try is
> startting the CMS earlier.  Usually this is recommended as means of
> putting off concurrent mode failures as opposed to promotion failures
> but it might be enough to ensure space to do the parnew promotion to
> old gen (This is an oldie but might I'd say basic premise holds:
> http://blogs.oracle.com/jonthecollector/entry/when_the_sum_of_the).
>
> How long do you run before you hit this pause?  Or is this only the
> first instance?
>
> > http://pastebin.com/9KfRZFBW
>
> It looks like you are keeping your zk logs in /tmp, is that so?  See
> hbase.zookeeper.property.dataDir.
>
> You are flushing at 4x the default.  You might try running at defaults
> to see if it lesser retention brings some relief.
>
> St.Ack
> P.S. We do not do bulk loading.
>
>
> >
> >
> > On Wed, May 25, 2011 at 1:42 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> >
> >> Hi Wayne,
> >>
> >> Looks like your RAM might be oversubscribed. Could you paste your
> >> hbase-site.xml and hbase-env.sh files? Also looks like you have some
> >> strange GC settings on (eg perm gen collection which we don't really
> >> need)
> >>
> >> If you can paste a larger segment of GC logs (enough to include at
> >> least two or three of the full gc pauses) that would be helpful.
> >>
> >> -Todd
> >>
> >> On Wed, May 25, 2011 at 10:32 AM, Wayne <[EMAIL PROTECTED]> wrote:
> >> > We switched to u25 and reverted the JVM settings to those recommended.
> >> Now
> >> > we have concurrent mode failures that occur lasting more than 60
> seconds
> >> > while not under hardly any load....
> >> >
> >> > Below are the entries from the JVM log. Of course we can up the
> zookeeper
> >> > timeout to 2 min or 10 min for that matter but it does not address the
> >> > underlying issue. Sorry but I can not confirm that the changes for the
> >> new
> >> > GC settings have any affect. It appears no better or even worse as
> this
> >> > problem below occurred while the cluster was almost idle.
> >> >
> >> >
> >> > 2011-05-25T14:15:45.518+0000: 150358.023: [GC 150358.023: [ParNew:
> >> > 230155K->27648K(249216K), 0.0653880 secs] 7754007K->7586719K(8360960K)
> >> > icms_dc=100 , 0.0654900 secs] [Times: user=0.78 sys=0.00, real=0.06
> secs]
> >> > 2011-05-25T14:15:45.906+0000: 150358.410: [GC 150358.410: [ParNew
> >> (promotion
> >> > failed): 249216K->249216K(249216K), 0.5768350 secs]150358.987:
> >> > [CMS2011-05-25T14:16:44.404+0000: 150416.909: [CMS-concurrent-sweep:

On Wed, May 25, 2011 at 11:39 AM, Ted Dunning <[EMAIL PROTECTED]> wrote:
> It should be recognized that your experiences are a bit out of the norm
> here.  Many hbase installations use more recent JVM's without problems.

Indeed, we run u25 on CentOS 5.6 and over several days uptime it's
common to never see a full GC across an 8GB heap.

What we never see is a ParNew taking .1 seconds, they're usually .01
and we never have full collections lasting 92 seconds. The only time
I've ever seen a JVM at 8GB take that long is when running on puny
(read virtualized) cores or when there are Ubuntu kernel bugs at play.
The same is true for our Cassandra deploys as well.

What are your write levels? We are pushing 30-40k writes/sec/node on 10
nodes for 24-36-48-72 hours straight. We have only 4 writers per node so we
are hardly overwhelming the nodes. Disk utilization runs at 10-20%, load is
max 50% including some app code, and memory is the 8g JVM out of 24G. We run
our production as 20TB in MySQL and see 90% disk utilization for hours every
day. A database must be able to accept being pounded 24/7. If the hardware
can handle it so should the database...this is not true for Java based
databases. The reality is that Java is not nearly ready for what a real
database will expect of it. Sure we could "back off" the volume and add 100
more nodes like Cassandra requires but then we might as well have used
something else given that hardware spend.

Our problem is that we have invested so much time with Hbase that it is hard
for us to walk away and go to the sharded PostgreSQL we should have used 9
months back. Sorry for the negativity but considering giving up after having
invested all of this time is painful.
On Wed, May 25, 2011 at 4:21 PM, Erik Onnen <[EMAIL PROTECTED]> wrote:

> On Wed, May 25, 2011 at 11:39 AM, Ted Dunning <[EMAIL PROTECTED]>
> wrote:
> > It should be recognized that your experiences are a bit out of the norm
> > here.  Many hbase installations use more recent JVM's without problems.
>
> Indeed, we run u25 on CentOS 5.6 and over several days uptime it's
> common to never see a full GC across an 8GB heap.
>
> What we never see is a ParNew taking .1 seconds, they're usually .01
> and we never have full collections lasting 92 seconds. The only time
> I've ever seen a JVM at 8GB take that long is when running on puny
> (read virtualized) cores or when there are Ubuntu kernel bugs at play.
> The same is true for our Cassandra deploys as well.
>

We know several things that are in common with your hbase and your
cassandra.

a) the jvm

b) the machines

c) the os

d) the (necessary) prejudices of the implementors and op staff
On the other hand, we know of other hbase (and cassandra) installations
running similar
volumes on the same JVM.

If you look only at the evidence you have at hand, then, yes, the JVM is one
of a relatively
short list of possible causes.  If you look more widely, it recedes a bit in
prominence.

On Wed, May 25, 2011 at 11:53 AM, Wayne <[EMAIL PROTECTED]> wrote:

> Most hbase installations also seem to recommend bulk inserts for loading
> data. We are pushing it more than most in terms of actually using the
> client
> API to load large volumes of data. We keep delaying putting hbase into
> production as nodes going awol for as much as 2+ minutes we can not accept
> as normal. Cassandra had the exact same problems for us (plus a lot of
> other
> issues), and we all know what is common between the two.
>
> On Wed, May 25, 2011 at 2:39 PM, Ted Dunning <[EMAIL PROTECTED]>
> wrote:
>
> > Wayne,
> >
> > It should be recognized that your experiences are a bit out of the norm
> > here.  Many hbase installations use more recent JVM's without problems.
> >
> > As such, it may be premature to point the finger at the JVM as opposed to
> > the workload or environmental factors.  Such a premature diagnosis can
> make
> > getting at the true cause of problems more  difficult.
> >
> > On Wed, May 25, 2011 at 11:21 AM, Wayne <[EMAIL PROTECTED]> wrote:
> >
> > > I will keep testing. Frankly u25 scares as well as older JVMs seem much
> > > better based on previous testing.
> > >
> >
>

How large are these writes?

Are you using asynchbase or other alternative client implementation?

Are you batching updates?

On Wed, May 25, 2011 at 2:44 PM, Wayne <[EMAIL PROTECTED]> wrote:

> What are your write levels? We are pushing 30-40k writes/sec/node on 10
> nodes for 24-36-48-72 hours straight. We have only 4 writers per node so we
> are hardly overwhelming the nodes. Disk utilization runs at 10-20%, load is
> max 50% including some app code, and memory is the 8g JVM out of 24G. We
> run
> our production as 20TB in MySQL and see 90% disk utilization for hours
> every
> day. A database must be able to accept being pounded 24/7. If the hardware
> can handle it so should the database...this is not true for Java based
> databases. The reality is that Java is not nearly ready for what a real
> database will expect of it. Sure we could "back off" the volume and add 100
> more nodes like Cassandra requires but then we might as well have used
> something else given that hardware spend.
>
> Our problem is that we have invested so much time with Hbase that it is
> hard
> for us to walk away and go to the sharded PostgreSQL we should have used 9
> months back. Sorry for the negativity but considering giving up after
> having
> invested all of this time is painful.
>
>
> On Wed, May 25, 2011 at 4:21 PM, Erik Onnen <[EMAIL PROTECTED]> wrote:
>
> > On Wed, May 25, 2011 at 11:39 AM, Ted Dunning <[EMAIL PROTECTED]>
> > wrote:
> > > It should be recognized that your experiences are a bit out of the norm
> > > here.  Many hbase installations use more recent JVM's without problems.
> >
> > Indeed, we run u25 on CentOS 5.6 and over several days uptime it's
> > common to never see a full GC across an 8GB heap.
> >
> > What we never see is a ParNew taking .1 seconds, they're usually .01
> > and we never have full collections lasting 92 seconds. The only time
> > I've ever seen a JVM at 8GB take that long is when running on puny
> > (read virtualized) cores or when there are Ubuntu kernel bugs at play.
> > The same is true for our Cassandra deploys as well.
> >
>

We are using std thrift from python. All writes are batched into usually 30k
writes per batch. The writes are small double/varchar(100) type values. Our
current write performance is fine for our needs...our concern is that they
are not sustainable over time given the GC timeouts.

Per the 4 items above, yes the staff is biased (obviously), the OS is what
2/3+ linux servers are running, the machines are Super Micro twins with 6
sata 1TB RE4 disks (5 for hdfs) and 24G of ECC memory (hardly non-standard).
We are plain vanilla. Our workload may be non-standard in that it is NOT
map-reduce. It is an evenly distributed q based home spun parallel
processing framework.

We are not a Java shop, and do not want to become one. I think to push the
limits and do well with hadoop+hdfs you have to buy into Java and have deep
skills there. We are database experts looking only for an open source
scalable database. We are a python shop and have no interest in digging deep
into java and the jvm.

Thanks.

On Wed, May 25, 2011 at 8:07 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> How large are these writes?
>
> Are you using asynchbase or other alternative client implementation?
>
> Are you batching updates?
>
> On Wed, May 25, 2011 at 2:44 PM, Wayne <[EMAIL PROTECTED]> wrote:
>
> > What are your write levels? We are pushing 30-40k writes/sec/node on 10
> > nodes for 24-36-48-72 hours straight. We have only 4 writers per node so
> we
> > are hardly overwhelming the nodes. Disk utilization runs at 10-20%, load
> is
> > max 50% including some app code, and memory is the 8g JVM out of 24G. We
> > run
> > our production as 20TB in MySQL and see 90% disk utilization for hours
> > every
> > day. A database must be able to accept being pounded 24/7. If the
> hardware
> > can handle it so should the database...this is not true for Java based
> > databases. The reality is that Java is not nearly ready for what a real
> > database will expect of it. Sure we could "back off" the volume and add
> 100
> > more nodes like Cassandra requires but then we might as well have used
> > something else given that hardware spend.
> >
> > Our problem is that we have invested so much time with Hbase that it is
> > hard
> > for us to walk away and go to the sharded PostgreSQL we should have used
> 9
> > months back. Sorry for the negativity but considering giving up after
> > having
> > invested all of this time is painful.
> >
> >
> > On Wed, May 25, 2011 at 4:21 PM, Erik Onnen <[EMAIL PROTECTED]> wrote:
> >
> > > On Wed, May 25, 2011 at 11:39 AM, Ted Dunning <[EMAIL PROTECTED]>
> > > wrote:
> > > > It should be recognized that your experiences are a bit out of the
> norm
> > > > here.  Many hbase installations use more recent JVM's without
> problems.
> > >
> > > Indeed, we run u25 on CentOS 5.6 and over several days uptime it's
> > > common to never see a full GC across an 8GB heap.
> > >
> > > What we never see is a ParNew taking .1 seconds, they're usually .01
> > > and we never have full collections lasting 92 seconds. The only time
> > > I've ever seen a JVM at 8GB take that long is when running on puny
> > > (read virtualized) cores or when there are Ubuntu kernel bugs at play.
> > > The same is true for our Cassandra deploys as well.
> > >
> >
>

This may be the most important detail of all.

It is important to go with your deep skills.  I would be a round peg in your
square shop and you would be a square one in my round one.

On Wed, May 25, 2011 at 5:55 PM, Wayne <[EMAIL PROTECTED]> wrote:

> We are not a Java shop, and do not want to become one. I think to push the
> limits and do well with hadoop+hdfs you have to buy into Java and have deep
> skills there. We are database experts looking only for an open source
> scalable database. We are a python shop and have no interest in digging
> deep
> into java and the jvm.
>

That may be the best advice I ever got...although I would say 9 months we
didn't have 1 line of python and now we have a fantastic mpp framework built
with python with a team most of which never wrote a line of python before.

But...java is not python...

We have shredded our relational past and frankly the sharded postgresql
design mirrors the key store of hbase. We are not looking to make hbase a
relational database...we expect it to work without going awol. It works
great 99% of the time but that 1% might be when our bigest customer is
logging in to see important data and gets hung with a 2 minute pause because
it was not in memcache and the region has gone awol in a full GC.
On Wed, May 25, 2011 at 9:09 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:

> This may be the most important detail of all.
>
> It is important to go with your deep skills.  I would be a round peg in
> your
> square shop and you would be a square one in my round one.
>
> On Wed, May 25, 2011 at 5:55 PM, Wayne <[EMAIL PROTECTED]> wrote:
>
> > We are not a Java shop, and do not want to become one. I think to push
> the
> > limits and do well with hadoop+hdfs you have to buy into Java and have
> deep
> > skills there. We are database experts looking only for an open source
> > scalable database. We are a python shop and have no interest in digging
> > deep
> > into java and the jvm.
> >
>

On Wed, May 25, 2011 at 2:44 PM, Wayne <[EMAIL PROTECTED]> wrote:
> What are your write levels? We are pushing 30-40k writes/sec/node on 10
> nodes for 24-36-48-72 hours straight. We have only 4 writers per node so we
> are hardly overwhelming the nodes. Disk utilization runs at 10-20%, load is
> max 50% including some app code, and memory is the 8g JVM out of 24G.

Our write load isn't as even as that. At peak we're higher than that
sustained for up to 30min, at lower points we're much lower, closer to
5K/sec, all across an 11 node cluster but that cluster does other work
including job trackers.

In neither case do we run into GC times you describe. In no properly
configured and tuned Java service I've ever run have I encountered a
90 second GC, database or otherwise.

Python is great.

If you can hold your nose a little longer, you are either almost
there, or its a lost cause so bare with us a little longer.

Did the configs. above make a difference?  (Initiating compaction at
65% is conservative -- you'll be burning lots of CPU -- but probably
good to start here).

Thanks,
St.Ack
On Wed, May 25, 2011 at 6:20 PM, Wayne <[EMAIL PROTECTED]> wrote:
> That may be the best advice I ever got...although I would say 9 months we
> didn't have 1 line of python and now we have a fantastic mpp framework built
> with python with a team most of which never wrote a line of python before.
>
> But...java is not python...
>
> We have shredded our relational past and frankly the sharded postgresql
> design mirrors the key store of hbase. We are not looking to make hbase a
> relational database...we expect it to work without going awol. It works
> great 99% of the time but that 1% might be when our bigest customer is
> logging in to see important data and gets hung with a 2 minute pause because
> it was not in memcache and the region has gone awol in a full GC.
>
>
>
>
> On Wed, May 25, 2011 at 9:09 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:
>
>> This may be the most important detail of all.
>>
>> It is important to go with your deep skills.  I would be a round peg in
>> your
>> square shop and you would be a square one in my round one.
>>
>> On Wed, May 25, 2011 at 5:55 PM, Wayne <[EMAIL PROTECTED]> wrote:
>>
>> > We are not a Java shop, and do not want to become one. I think to push
>> the
>> > limits and do well with hadoop+hdfs you have to buy into Java and have
>> deep
>> > skills there. We are database experts looking only for an open source
>> > scalable database. We are a python shop and have no interest in digging
>> > deep
>> > into java and the jvm.
>> >
>>
>

Wayne, I think you are hitting fragmentation, how often do you flush?
 Can you share memstore flush graphs?
Here is ours:  http://img851.yfrog.com/img851/9814/screenshot20110526at124.png

We run at 12G Heap, 20% memstore size, 50% blockcache, have recently
added incremental mode to combat too frequent ParNew GCs:

export HBASE_OPTS="$HBASE_OPTS -verbose:gc -Xms12000m
-XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
-Xloggc:$HBASE_HOME/logs/gc-hbase.log \
-XX:+CMSIncrementalMode \
-XX:+CMSIncrementalPacing \
-XX:-TraceClassUnloading
"

-Jack

On Wed, May 25, 2011 at 5:55 PM, Wayne <[EMAIL PROTECTED]> wrote:
> We are using std thrift from python. All writes are batched into usually 30k
> writes per batch. The writes are small double/varchar(100) type values. Our
> current write performance is fine for our needs...our concern is that they
> are not sustainable over time given the GC timeouts.
>
> Per the 4 items above, yes the staff is biased (obviously), the OS is what
> 2/3+ linux servers are running, the machines are Super Micro twins with 6
> sata 1TB RE4 disks (5 for hdfs) and 24G of ECC memory (hardly non-standard).
> We are plain vanilla. Our workload may be non-standard in that it is NOT
> map-reduce. It is an evenly distributed q based home spun parallel
> processing framework.
>
> We are not a Java shop, and do not want to become one. I think to push the
> limits and do well with hadoop+hdfs you have to buy into Java and have deep
> skills there. We are database experts looking only for an open source
> scalable database. We are a python shop and have no interest in digging deep
> into java and the jvm.
>
> Thanks.
>
> On Wed, May 25, 2011 at 8:07 PM, Ted Dunning <[EMAIL PROTECTED]> wrote:
>
>> How large are these writes?
>>
>> Are you using asynchbase or other alternative client implementation?
>>
>> Are you batching updates?
>>
>> On Wed, May 25, 2011 at 2:44 PM, Wayne <[EMAIL PROTECTED]> wrote:
>>
>> > What are your write levels? We are pushing 30-40k writes/sec/node on 10
>> > nodes for 24-36-48-72 hours straight. We have only 4 writers per node so
>> we
>> > are hardly overwhelming the nodes. Disk utilization runs at 10-20%, load
>> is
>> > max 50% including some app code, and memory is the 8g JVM out of 24G. We
>> > run
>> > our production as 20TB in MySQL and see 90% disk utilization for hours
>> > every
>> > day. A database must be able to accept being pounded 24/7. If the
>> hardware
>> > can handle it so should the database...this is not true for Java based
>> > databases. The reality is that Java is not nearly ready for what a real
>> > database will expect of it. Sure we could "back off" the volume and add
>> 100
>> > more nodes like Cassandra requires but then we might as well have used
>> > something else given that hardware spend.
>> >
>> > Our problem is that we have invested so much time with Hbase that it is
>> > hard
>> > for us to walk away and go to the sharded PostgreSQL we should have used
>> 9
>> > months back. Sorry for the negativity but considering giving up after
>> > having
>> > invested all of this time is painful.
>> >
>> >
>> > On Wed, May 25, 2011 at 4:21 PM, Erik Onnen <[EMAIL PROTECTED]> wrote:
>> >
>> > > On Wed, May 25, 2011 at 11:39 AM, Ted Dunning <[EMAIL PROTECTED]>
>> > > wrote:
>> > > > It should be recognized that your experiences are a bit out of the
>> norm
>> > > > here.  Many hbase installations use more recent JVM's without
>> problems.
>> > >
>> > > Indeed, we run u25 on CentOS 5.6 and over several days uptime it's
>> > > common to never see a full GC across an 8GB heap.
>> > >
>> > > What we never see is a ParNew taking .1 seconds, they're usually .01
>> > > and we never have full collections lasting 92 seconds. The only time
>> > > I've ever seen a JVM at 8GB take that long is when running on puny
>> > > (read virtualized) cores or when there are Ubuntu kernel bugs at play.
>> > > The same is true for our Cassandra deploys as well.

Attached is our memstore size graph...not sure it will make it to the post.
Ours it definitely not as gracefull as yours. You can see where we restarted
last 16 hours ago. We have not had any issues since, but we usually don't
have problems until 24-48 hours into loads. Stack yes the 65% seams to help
again but it has not been long enough.

I think our problem is the load pattern. Since we use a very controlled q
based method to do work our Python code is relentless in terms of keeping
the pressure up. In our testing we will Q up 500k messages with 10k writes
per message that all get written to 3 column families (primary plus 2
secondary indexes). This means there will be 15 billion writes waiting to go
into hbase. It will take us days to load this and the JVM will eventually
crumble. If we are lucky enough to avoid too long of a GC or along with it
we also see OOM problems crop up eventually.

Again our hardware is taking it easy during this process except for the JVM
and its 8g heap. The heat should be on the disk and it is not, as we are not
really pushing it at all. I could and would like to throttle it up and have
6 writers per node instead of 4 but I know the nodes can not sustain that.
What we need to find is what level of writes can our cluster sustain for
weeks at a time and still be fast with reads and not go AWOL. Once we find
that sweat spot then we can try to turn up the heat...but we never seem to
find it. Between GC pauses and OOMs we have never run under load for long
enough to gain "confidence".

Thanks.

On Thu, May 26, 2011 at 3:43 AM, Jack Levin <[EMAIL PROTECTED]> wrote:

> Wayne, I think you are hitting fragmentation, how often do you flush?
>  Can you share memstore flush graphs?
> Here is ours:
> http://img851.yfrog.com/img851/9814/screenshot20110526at124.png
>
> We run at 12G Heap, 20% memstore size, 50% blockcache, have recently
> added incremental mode to combat too frequent ParNew GCs:
>
> export HBASE_OPTS="$HBASE_OPTS -verbose:gc -Xms12000m
> -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails
> -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
> -Xloggc:$HBASE_HOME/logs/gc-hbase.log \
> -XX:+CMSIncrementalMode \
> -XX:+CMSIncrementalPacing \
> -XX:-TraceClassUnloading
> "
>
> -Jack
>
> On Wed, May 25, 2011 at 5:55 PM, Wayne <[EMAIL PROTECTED]> wrote:
> > We are using std thrift from python. All writes are batched into usually
> 30k
> > writes per batch. The writes are small double/varchar(100) type values.
> Our
> > current write performance is fine for our needs...our concern is that
> they
> > are not sustainable over time given the GC timeouts.
> >
> > Per the 4 items above, yes the staff is biased (obviously), the OS is
> what
> > 2/3+ linux servers are running, the machines are Super Micro twins with 6
> > sata 1TB RE4 disks (5 for hdfs) and 24G of ECC memory (hardly
> non-standard).
> > We are plain vanilla. Our workload may be non-standard in that it is NOT
> > map-reduce. It is an evenly distributed q based home spun parallel
> > processing framework.
> >
> > We are not a Java shop, and do not want to become one. I think to push
> the
> > limits and do well with hadoop+hdfs you have to buy into Java and have
> deep
> > skills there. We are database experts looking only for an open source
> > scalable database. We are a python shop and have no interest in digging
> deep
> > into java and the jvm.
> >
> > Thanks.
> >
> > On Wed, May 25, 2011 at 8:07 PM, Ted Dunning <[EMAIL PROTECTED]>
> wrote:
> >
> >> How large are these writes?
> >>
> >> Are you using asynchbase or other alternative client implementation?
> >>
> >> Are you batching updates?
> >>
> >> On Wed, May 25, 2011 at 2:44 PM, Wayne <[EMAIL PROTECTED]> wrote:
> >>
> >> > What are your write levels? We are pushing 30-40k writes/sec/node on
> 10
> >> > nodes for 24-36-48-72 hours straight. We have only 4 writers per node
> so
> >> we
> >> > are hardly overwhelming the nodes. Disk utilization runs at 10-20%,
> load
> >> is

Looking more closely I can see that we are still
getting Concurrent Mode Failures on some of the nodes but they are only
lasting for 10s so the nodes don't go away. Is this considered "normal"?
With CMSInitiatingOccupancyFraction=65 I would suspect this is not normal??

Here is a link to some GC logs.

http://pastebin.com/kJyJHgQc

Thanks.
On Thu, May 26, 2011 at 9:08 AM, Wayne <[EMAIL PROTECTED]> wrote:

> Attached is our memstore size graph...not sure it will make it to the post.
> Ours it definitely not as gracefull as yours. You can see where we restarted
> last 16 hours ago. We have not had any issues since, but we usually don't
> have problems until 24-48 hours into loads. Stack yes the 65% seams to help
> again but it has not been long enough.
>
> I think our problem is the load pattern. Since we use a very controlled q
> based method to do work our Python code is relentless in terms of keeping
> the pressure up. In our testing we will Q up 500k messages with 10k writes
> per message that all get written to 3 column families (primary plus 2
> secondary indexes). This means there will be 15 billion writes waiting to go
> into hbase. It will take us days to load this and the JVM will eventually
> crumble. If we are lucky enough to avoid too long of a GC or along with it
> we also see OOM problems crop up eventually.
>
> Again our hardware is taking it easy during this process except for the JVM
> and its 8g heap. The heat should be on the disk and it is not, as we are not
> really pushing it at all. I could and would like to throttle it up and have
> 6 writers per node instead of 4 but I know the nodes can not sustain that.
> What we need to find is what level of writes can our cluster sustain for
> weeks at a time and still be fast with reads and not go AWOL. Once we find
> that sweat spot then we can try to turn up the heat...but we never seem to
> find it. Between GC pauses and OOMs we have never run under load for long
> enough to gain "confidence".
>
> Thanks.
>
> On Thu, May 26, 2011 at 3:43 AM, Jack Levin <[EMAIL PROTECTED]> wrote:
>
>> Wayne, I think you are hitting fragmentation, how often do you flush?
>>  Can you share memstore flush graphs?
>> Here is ours:
>> http://img851.yfrog.com/img851/9814/screenshot20110526at124.png
>>
>> We run at 12G Heap, 20% memstore size, 50% blockcache, have recently
>> added incremental mode to combat too frequent ParNew GCs:
>>
>> export HBASE_OPTS="$HBASE_OPTS -verbose:gc -Xms12000m
>> -XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails
>> -XX:+PrintGCDateStamps -XX:+HeapDumpOnOutOfMemoryError
>> -Xloggc:$HBASE_HOME/logs/gc-hbase.log \
>> -XX:+CMSIncrementalMode \
>> -XX:+CMSIncrementalPacing \
>> -XX:-TraceClassUnloading
>> "
>>
>> -Jack
>>
>> On Wed, May 25, 2011 at 5:55 PM, Wayne <[EMAIL PROTECTED]> wrote:
>> > We are using std thrift from python. All writes are batched into usually
>> 30k
>> > writes per batch. The writes are small double/varchar(100) type values.
>> Our
>> > current write performance is fine for our needs...our concern is that
>> they
>> > are not sustainable over time given the GC timeouts.
>> >
>> > Per the 4 items above, yes the staff is biased (obviously), the OS is
>> what
>> > 2/3+ linux servers are running, the machines are Super Micro twins with
>> 6
>> > sata 1TB RE4 disks (5 for hdfs) and 24G of ECC memory (hardly
>> non-standard).
>> > We are plain vanilla. Our workload may be non-standard in that it is NOT
>> > map-reduce. It is an evenly distributed q based home spun parallel
>> > processing framework.
>> >
>> > We are not a Java shop, and do not want to become one. I think to push
>> the
>> > limits and do well with hadoop+hdfs you have to buy into Java and have
>> deep
>> > skills there. We are database experts looking only for an open source
>> > scalable database. We are a python shop and have no interest in digging
>> deep
>> > into java and the jvm.
>> >
>> > Thanks.
>> >
>> > On Wed, May 25, 2011 at 8:07 PM, Ted Dunning <[EMAIL PROTECTED]>

On Thu, May 26, 2011 at 9:00 AM, Wayne <[EMAIL PROTECTED]> wrote:
> Looking more closely I can see that we are still
> getting Concurrent Mode Failures on some of the nodes but they are only
> lasting for 10s so the nodes don't go away. Is this considered "normal"?
> With CMSInitiatingOccupancyFraction=65 I would suspect this is not normal??
>

What configs. are you running with now?  It looks like you either left
parnew as unbounded or else you set it to 256M max?   So you did not
change the parnew size?  Did you up your heap size?  I see you are
getting 'promotion failed'/'concurrent mode failure'.  How long has it
been running now?

St.Ack

On Thu, May 26, 2011 at 6:08 AM, Wayne <[EMAIL PROTECTED]> wrote:
> I think our problem is the load pattern. Since we use a very controlled q
> based method to do work our Python code is relentless in terms of keeping
> the pressure up. In our testing we will Q up 500k messages with 10k writes
> per message that all get written to 3 column families (primary plus 2
> secondary indexes).

Whats the insert batch size do you think?

> This means there will be 15 billion writes waiting to go
> into hbase. It will take us days to load this and the JVM will eventually
> crumble. If we are lucky enough to avoid too long of a GC or along with it
> we also see OOM problems crop up eventually.

What version of hbase?  If your batches are large and they get backed
up, they could be hanging out in queues in hbase that will hold number
of handlers * 100 requests (See HBASE-3813 mitigated in 0.90.3).

Bulk load is out of the question getting you off the ground because
you have these secondary indices?
> Again our hardware is taking it easy during this process except for the JVM
> and its 8g heap. The heat should be on the disk and it is not, as we are not
> really pushing it at all. I could and would like to throttle it up and have
> 6 writers per node instead of 4 but I know the nodes can not sustain that.
> What we need to find is what level of writes can our cluster sustain for
> weeks at a time and still be fast with reads and not go AWOL. Once we find
> that sweat spot then we can try to turn up the heat...but we never seem to
> find it. Between GC pauses and OOMs we have never run under load for long
> enough to gain "confidence".

Yes. We have a bunch of work to do still (I'll spare you the lecture
on this being an open-source project, blah, blah, volunteers...).

St.Ack

I left parnew alone (did not add any settings). I also did not increase the
heap. 8g with 50% for memstore. Below are the JVM settings.

The errors I pasted occurred after running for only maybe 12 hours. The
cluster as a whole has been running for 24 hours with dropping a node, but
short time span CMFs are occurring.

Any recommendations?

export HBASE_OPTS="-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=65 -XX:+CMSParallelRemarkEnabled
-XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC"

Thanks.
On Thu, May 26, 2011 at 1:30 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Thu, May 26, 2011 at 9:00 AM, Wayne <[EMAIL PROTECTED]> wrote:
> > Looking more closely I can see that we are still
> > getting Concurrent Mode Failures on some of the nodes but they are only
> > lasting for 10s so the nodes don't go away. Is this considered "normal"?
> > With CMSInitiatingOccupancyFraction=65 I would suspect this is not
> normal??
> >
>
> What configs. are you running with now?  It looks like you either left
> parnew as unbounded or else you set it to 256M max?   So you did not
> change the parnew size?  Did you up your heap size?  I see you are
> getting 'promotion failed'/'concurrent mode failure'.  How long has it
> been running now?
>
> St.Ack
>

Wayne, we get CMS failures also, I am pretty sure they are
fragmentation related:

2011-05-26T09:20:00.304-0700: 206371.599: [GC 206371.599: [ParNew
(promotion failed): 76633K->76023K(76672K), 0.0924180 secs]206371.692:
[CMS: 11452308K->7142504K(122
02816K), 13.5870310 secs] 11525447K->7142504K(12279488K), [CMS Perm :
18254K->18254K(30436K)] icms_dc=0 , 13.6796820 secs] [Times:
user=13.17 sys=0.64, real=13.68 sec
s]

The RS does not go away when this happens.   If your disks are not
overloaded, you should consider flushing sooner and deeper, e.g. flush
larger chunks of memory, and offload the load to the disks, this way,
you run will free up more memstore cache, and promotion of YG to
tenured has more chances to succeed without CMS Failure.

-Jack.

PS.  Wayne, are you on IM, I am "jacklevin74" on both AIM and Skype, lets chat.

On Thu, May 26, 2011 at 10:42 AM, Wayne <[EMAIL PROTECTED]> wrote:
> I left parnew alone (did not add any settings). I also did not increase the
> heap. 8g with 50% for memstore. Below are the JVM settings.
>
> The errors I pasted occurred after running for only maybe 12 hours. The
> cluster as a whole has been running for 24 hours with dropping a node, but
> short time span CMFs are occurring.
>
> Any recommendations?
>
> export HBASE_OPTS="-XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingOccupancyFraction=65 -XX:+CMSParallelRemarkEnabled
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC"
>
> Thanks.
>
>
> On Thu, May 26, 2011 at 1:30 PM, Stack <[EMAIL PROTECTED]> wrote:
>
>> On Thu, May 26, 2011 at 9:00 AM, Wayne <[EMAIL PROTECTED]> wrote:
>> > Looking more closely I can see that we are still
>> > getting Concurrent Mode Failures on some of the nodes but they are only
>> > lasting for 10s so the nodes don't go away. Is this considered "normal"?
>> > With CMSInitiatingOccupancyFraction=65 I would suspect this is not
>> normal??
>> >
>>
>> What configs. are you running with now?  It looks like you either left
>> parnew as unbounded or else you set it to 256M max?   So you did not
>> change the parnew size?  Did you up your heap size?  I see you are
>> getting 'promotion failed'/'concurrent mode failure'.  How long has it
>> been running now?
>>
>> St.Ack
>>
>

On Thu, May 26, 2011 at 1:41 PM, Stack <[EMAIL PROTECTED]> wrote:

> On Thu, May 26, 2011 at 6:08 AM, Wayne <[EMAIL PROTECTED]> wrote:
> > I think our problem is the load pattern. Since we use a very controlled q
> > based method to do work our Python code is relentless in terms of keeping
> > the pressure up. In our testing we will Q up 500k messages with 10k
> writes
> > per message that all get written to 3 column families (primary plus 2
> > secondary indexes).
>
> Whats the insert batch size do you think?
>

It is 10k row/col/ts/vals per batch mutation call (exactly).
>
> > This means there will be 15 billion writes waiting to go
> > into hbase. It will take us days to load this and the JVM will eventually
> > crumble. If we are lucky enough to avoid too long of a GC or along with
> it
> > we also see OOM problems crop up eventually.
>
> What version of hbase?  If your batches are large and they get backed
> up, they could be hanging out in queues in hbase that will hold number
> of handlers * 100 requests (See HBASE-3813 mitigated in 0.90.3).
>

0.90.1 right now. I do not think our batches are backing up as we track
everything and we can issue 3 batch mutates of 10k each in 3 seconds. We
have a huge amount of audit around all of this and the batches are going
through consistently.

>
> Bulk load is out of the question getting you off the ground because
> you have these secondary indices?
>

Home built secondary indexes so yes we could move to bulk insert. Bulk
inserts are like a cheat to us in that if the database can't handle our data
volumes through the front door then we probably shouldn't be using it...

>
>
> > Again our hardware is taking it easy during this process except for the
> JVM
> > and its 8g heap. The heat should be on the disk and it is not, as we are
> not
> > really pushing it at all. I could and would like to throttle it up and
> have
> > 6 writers per node instead of 4 but I know the nodes can not sustain
> that.
> > What we need to find is what level of writes can our cluster sustain for
> > weeks at a time and still be fast with reads and not go AWOL. Once we
> find
> > that sweat spot then we can try to turn up the heat...but we never seem
> to
> > find it. Between GC pauses and OOMs we have never run under load for long
> > enough to gain "confidence".
>
> Yes. We have a bunch of work to do still (I'll spare you the lecture
> on this being an open-source project, blah, blah, volunteers...).
>

Hey I would like to be on board for the ride...as long as we can actually
use this in production now. We will have 50TB+ volumes of data after RF=3
and the database is our business. We need to "trust" it...CMF is an evil
preventing me from trusting the JVM which prevents me from trusting Hbase.
>
> St.Ack
>

What JVM configs are you running Erik?
St.Ack

On Wed, May 25, 2011 at 6:31 PM, Erik Onnen <[EMAIL PROTECTED]> wrote:
> On Wed, May 25, 2011 at 2:44 PM, Wayne <[EMAIL PROTECTED]> wrote:
>> What are your write levels? We are pushing 30-40k writes/sec/node on 10
>> nodes for 24-36-48-72 hours straight. We have only 4 writers per node so we
>> are hardly overwhelming the nodes. Disk utilization runs at 10-20%, load is
>> max 50% including some app code, and memory is the 8g JVM out of 24G.
>
> Our write load isn't as even as that. At peak we're higher than that
> sustained for up to 30min, at lower points we're much lower, closer to
> 5K/sec, all across an 11 node cluster but that cluster does other work
> including job trackers.
>
> In neither case do we run into GC times you describe. In no properly
> configured and tuned Java service I've ever run have I encountered a
> 90 second GC, database or otherwise.
>

It might sound crazy, but if you have plenty of CPU, consider lowering
your NewSize to like 30MB, if you do that your ParNews will be more
frequent, but hitting CMS failure will be less likely, this is what we
seen.

-Jack

On Thu, May 26, 2011 at 10:51 AM, Jack Levin <[EMAIL PROTECTED]> wrote:
> Wayne, we get CMS failures also, I am pretty sure they are
> fragmentation related:
>
> 2011-05-26T09:20:00.304-0700: 206371.599: [GC 206371.599: [ParNew
> (promotion failed): 76633K->76023K(76672K), 0.0924180 secs]206371.692:
> [CMS: 11452308K->7142504K(122
> 02816K), 13.5870310 secs] 11525447K->7142504K(12279488K), [CMS Perm :
> 18254K->18254K(30436K)] icms_dc=0 , 13.6796820 secs] [Times:
> user=13.17 sys=0.64, real=13.68 sec
> s]
>
> The RS does not go away when this happens.   If your disks are not
> overloaded, you should consider flushing sooner and deeper, e.g. flush
> larger chunks of memory, and offload the load to the disks, this way,
> you run will free up more memstore cache, and promotion of YG to
> tenured has more chances to succeed without CMS Failure.
>
> -Jack.
>
> PS.  Wayne, are you on IM, I am "jacklevin74" on both AIM and Skype, lets chat.
>
> On Thu, May 26, 2011 at 10:42 AM, Wayne <[EMAIL PROTECTED]> wrote:
>> I left parnew alone (did not add any settings). I also did not increase the
>> heap. 8g with 50% for memstore. Below are the JVM settings.
>>
>> The errors I pasted occurred after running for only maybe 12 hours. The
>> cluster as a whole has been running for 24 hours with dropping a node, but
>> short time span CMFs are occurring.
>>
>> Any recommendations?
>>
>> export HBASE_OPTS="-XX:+UseCMSInitiatingOccupancyOnly
>> -XX:CMSInitiatingOccupancyFraction=65 -XX:+CMSParallelRemarkEnabled
>> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC"
>>
>> Thanks.
>>
>>
>> On Thu, May 26, 2011 at 1:30 PM, Stack <[EMAIL PROTECTED]> wrote:
>>
>>> On Thu, May 26, 2011 at 9:00 AM, Wayne <[EMAIL PROTECTED]> wrote:
>>> > Looking more closely I can see that we are still
>>> > getting Concurrent Mode Failures on some of the nodes but they are only
>>> > lasting for 10s so the nodes don't go away. Is this considered "normal"?
>>> > With CMSInitiatingOccupancyFraction=65 I would suspect this is not
>>> normal??
>>> >
>>>
>>> What configs. are you running with now?  It looks like you either left
>>> parnew as unbounded or else you set it to 256M max?   So you did not
>>> change the parnew size?  Did you up your heap size?  I see you are
>>> getting 'promotion failed'/'concurrent mode failure'.  How long has it
>>> been running now?
>>>
>>> St.Ack
>>>
>>
>

On Thu, May 26, 2011 at 10:42 AM, Wayne <[EMAIL PROTECTED]> wrote:
> I left parnew alone (did not add any settings). I also did not increase the
> heap. 8g with 50% for memstore. Below are the JVM settings.
>
> The errors I pasted occurred after running for only maybe 12 hours. The
> cluster as a whole has been running for 24 hours with dropping a node, but
> short time span CMFs are occurring.
>

You mean 'without' in the above?  Is this an improvement over the past?
> Any recommendations?
>
> export HBASE_OPTS="-XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingOccupancyFraction=65 -XX:+CMSParallelRemarkEnabled
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC"
>

Just the ones I made yesterday about upping the heap and putting bound
on parnew but my guess is that you are not of the experimental mindset
at the moment so I am afraid to make suggestions that I am not for
sure will improve your situation.

Does seem strange that you do run into CMFs though you are triggering
CMS super early.

Lets gather more data first.

St.Ack

Bulk load is just another front door.

It is very reasonable to have an adaptive policy that throttles uploads and
switches to fairly frequent bulk loading when the load gets very high.
 Whether this is an option depends on your real-time SLA's.

On Thu, May 26, 2011 at 10:55 AM, Wayne <[EMAIL PROTECTED]> wrote:

> >
> > Bulk load is out of the question getting you off the ground because
> > you have these secondary indices?
> >
>
> Home built secondary indexes so yes we could move to bulk insert. Bulk
> inserts are like a cheat to us in that if the database can't handle our
> data
> volumes through the front door then we probably shouldn't be using it...
>

On Thu, May 26, 2011 at 11:01 AM, Stack <[EMAIL PROTECTED]> wrote:
> What JVM configs are you running Erik?
> St.Ack

Omitting some of the irrelevant ones...

JAVA_OPTS="-XX:+UseLargePages -Xms8192M -Xmx8192M
-XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
-Xloggc:/mnt/services/hbaseregion/var/log/hbaseregion-gc.log
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps"

For our Cassandra servers we also explicitly pin the new gen tenuring
and occupancy thresholds but that hasn't been necessary for the HBase
workloads so far:

"-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly"

I have finally been able to spend enough time to digest/test
all recommendations and get this under control. I wanted to thank Stack,
Jack Levin, and Ted Dunning for their input.

Basically our memory was being pushed to the limit and the JVM does not
like/can not handle this. We are successfully using Todd's MSLAB enabled on
u25 and have set parnew to 128m, increased the heap to 10g, and increased
our block size 4x to 256k to reduce the size of the store file index (thanks
Jack for pointing this out). The combination of all of these changes reduce
significantly the pressure on memory and now we are getting more throughput
(40k writes/sec/node) sustained with no CMF errors (I expect to see one 5
min after hitting send on this email...).

We are also going to move to bulk inserts where we can, which should help as
well. The decreased performance of reads due to the block size increase is
going to be the next challenge. Thanks everyone for your help, I am
a believer again.
On Fri, May 27, 2011 at 12:17 AM, Erik Onnen <[EMAIL PROTECTED]> wrote:

> On Thu, May 26, 2011 at 11:01 AM, Stack <[EMAIL PROTECTED]> wrote:
> > What JVM configs are you running Erik?
> > St.Ack
>
> Omitting some of the irrelevant ones...
>
> JAVA_OPTS="-XX:+UseLargePages -Xms8192M -Xmx8192M
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> -Xloggc:/mnt/services/hbaseregion/var/log/hbaseregion-gc.log
> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
>
> For our Cassandra servers we also explicitly pin the new gen tenuring
> and occupancy thresholds but that hasn't been necessary for the HBase
> workloads so far:
>
> "-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly"
>

Is there any information from this thread that we should make sure gets into the hbase book?  it
seem like Wayne went through a lot of work to get good performance and it would be nice if all the
information he gleaned from the community were recorded somewhere.  If it doesn't make sense to put
the information in a general performance section, it might make sense to have a performance "case
study" section where for Wayne's case study these are what the performance settings he used to get
good performance.

~Jeff

On 6/2/2011 9:09 AM, Wayne wrote:
> I have finally been able to spend enough time to digest/test
> all recommendations and get this under control. I wanted to thank Stack,
> Jack Levin, and Ted Dunning for their input.
>
> Basically our memory was being pushed to the limit and the JVM does not
> like/can not handle this. We are successfully using Todd's MSLAB enabled on
> u25 and have set parnew to 128m, increased the heap to 10g, and increased
> our block size 4x to 256k to reduce the size of the store file index (thanks
> Jack for pointing this out). The combination of all of these changes reduce
> significantly the pressure on memory and now we are getting more throughput
> (40k writes/sec/node) sustained with no CMF errors (I expect to see one 5
> min after hitting send on this email...).
>
> We are also going to move to bulk inserts where we can, which should help as
> well. The decreased performance of reads due to the block size increase is
> going to be the next challenge. Thanks everyone for your help, I am
> a believer again.
>
>
> On Fri, May 27, 2011 at 12:17 AM, Erik Onnen<[EMAIL PROTECTED]>  wrote:
>
>> On Thu, May 26, 2011 at 11:01 AM, Stack<[EMAIL PROTECTED]>  wrote:
>>> What JVM configs are you running Erik?
>>> St.Ack
>> Omitting some of the irrelevant ones...
>>
>> JAVA_OPTS="-XX:+UseLargePages -Xms8192M -Xmx8192M
>> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC
>> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
>> -Xloggc:/mnt/services/hbaseregion/var/log/hbaseregion-gc.log
>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
>>
>> For our Cassandra servers we also explicitly pin the new gen tenuring
>> and occupancy thresholds but that hasn't been necessary for the HBase
>> workloads so far:
>>
>> "-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
>> -XX:+UseCMSInitiatingOccupancyOnly"
>>

--
Jeff Whiting
Qualtrics Senior Software Engineer
[EMAIL PROTECTED]

I'd be particularly interested how you guys came to the conclusion for
increasing block size and how you arrived at the size you chose. For
example, what metrics were you looking at that indicated the block
size was too small and what tests did you run to arrive at 256k for
the correct size?

Thanks for writing back to the list Wayne.  Hopefully this message
hits you before the next CMF does.  Would you mind pasting your final
JVM args and any other configs you think one of us could use writing
up your war story for the 'book' as per Jeff Whiting's suggestion?

Good stuff,
St.Ack
On Thu, Jun 2, 2011 at 8:09 AM, Wayne <[EMAIL PROTECTED]> wrote:
> I have finally been able to spend enough time to digest/test
> all recommendations and get this under control. I wanted to thank Stack,
> Jack Levin, and Ted Dunning for their input.
>
> Basically our memory was being pushed to the limit and the JVM does not
> like/can not handle this. We are successfully using Todd's MSLAB enabled on
> u25 and have set parnew to 128m, increased the heap to 10g, and increased
> our block size 4x to 256k to reduce the size of the store file index (thanks
> Jack for pointing this out). The combination of all of these changes reduce
> significantly the pressure on memory and now we are getting more throughput
> (40k writes/sec/node) sustained with no CMF errors (I expect to see one 5
> min after hitting send on this email...).
>
> We are also going to move to bulk inserts where we can, which should help as
> well. The decreased performance of reads due to the block size increase is
> going to be the next challenge. Thanks everyone for your help, I am
> a believer again.
>
>
> On Fri, May 27, 2011 at 12:17 AM, Erik Onnen <[EMAIL PROTECTED]> wrote:
>
>> On Thu, May 26, 2011 at 11:01 AM, Stack <[EMAIL PROTECTED]> wrote:
>> > What JVM configs are you running Erik?
>> > St.Ack
>>
>> Omitting some of the irrelevant ones...
>>
>> JAVA_OPTS="-XX:+UseLargePages -Xms8192M -Xmx8192M
>> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC
>> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
>> -Xloggc:/mnt/services/hbaseregion/var/log/hbaseregion-gc.log
>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
>>
>> For our Cassandra servers we also explicitly pin the new gen tenuring
>> and occupancy thresholds but that hasn't been necessary for the HBase
>> workloads so far:
>>
>> "-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
>> -XX:+UseCMSInitiatingOccupancyOnly"
>>
>

Our storefileindex was pushing 3g. We used the hfile tool to see that we had
very large keys (50-70 bytes) and small values (5-7 bytes). Jack pointed me
to a great Jira about this: https://issues.apache.org/jira/browse/HBASE-3551
.

We HAD to increase from the default and we picked 256k to reduce the index
size. It was chosen more from an index reduction required vs. a testing of
affected read performance. We are still dealing with the ramifications of
this change...being reads are slower. How much slower we are not sure yet.
In the end we might go to 128k or 512k, but it needs to be bigger than 56k.
We have stabilized our writes, but very well might have done so at the cost
of reads. We are going to be thoroughly testing reads next.
On Thu, Jun 2, 2011 at 11:40 AM, Erik Onnen <[EMAIL PROTECTED]> wrote:

> I'd be particularly interested how you guys came to the conclusion for
> increasing block size and how you arrived at the size you chose. For
> example, what metrics were you looking at that indicated the block
> size was too small and what tests did you run to arrive at 256k for
> the correct size?
>

JVM w/ 10g Heap settings below. Once we are "bored" with stability we will
try to up the 65 to 70 which seems to be standard.

-XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=65
-XX:+CMSParallelRemarkEnabled -XX:+UseConcMarkSweepGC -XX:NewSize=128m
-XX:MaxNewSize=128m -XX:+UseParNewG

Our Memstore settings are default lower/upper is .35/.4.

We do not currently use block cache. That may change in the future...

Our table block size is 256k to keep the store file index size down. It was
averaging ~2.75G+ per node and now has gone down to ~1.5G which should go
even lower as regions are compacted.

We have enabled the memstore MSLAB option. Not sure it is relevant but we
have a 5G region size and 256m memstore flush size.

Thanks.

On Thu, Jun 2, 2011 at 11:48 AM, Stack <[EMAIL PROTECTED]> wrote:

> Thanks for writing back to the list Wayne.  Hopefully this message
> hits you before the next CMF does.  Would you mind pasting your final
> JVM args and any other configs you think one of us could use writing
> up your war story for the 'book' as per Jeff Whiting's suggestion?
>
> Good stuff,
> St.Ack
>
>
> On Thu, Jun 2, 2011 at 8:09 AM, Wayne <[EMAIL PROTECTED]> wrote:
> > I have finally been able to spend enough time to digest/test
> > all recommendations and get this under control. I wanted to thank Stack,
> > Jack Levin, and Ted Dunning for their input.
> >
> > Basically our memory was being pushed to the limit and the JVM does not
> > like/can not handle this. We are successfully using Todd's MSLAB enabled
> on
> > u25 and have set parnew to 128m, increased the heap to 10g, and increased
> > our block size 4x to 256k to reduce the size of the store file index
> (thanks
> > Jack for pointing this out). The combination of all of these changes
> reduce
> > significantly the pressure on memory and now we are getting more
> throughput
> > (40k writes/sec/node) sustained with no CMF errors (I expect to see one 5
> > min after hitting send on this email...).
> >
> > We are also going to move to bulk inserts where we can, which should help
> as
> > well. The decreased performance of reads due to the block size increase
> is
> > going to be the next challenge. Thanks everyone for your help, I am
> > a believer again.
> >
> >
> > On Fri, May 27, 2011 at 12:17 AM, Erik Onnen <[EMAIL PROTECTED]> wrote:
> >
> >> On Thu, May 26, 2011 at 11:01 AM, Stack <[EMAIL PROTECTED]> wrote:
> >> > What JVM configs are you running Erik?
> >> > St.Ack
> >>
> >> Omitting some of the irrelevant ones...
> >>
> >> JAVA_OPTS="-XX:+UseLargePages -Xms8192M -Xmx8192M
> >> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC
> >> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
> >> -Xloggc:/mnt/services/hbaseregion/var/log/hbaseregion-gc.log
> >> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
> >>
> >> For our Cassandra servers we also explicitly pin the new gen tenuring
> >> and occupancy thresholds but that hasn't been necessary for the HBase
> >> workloads so far:
> >>
> >> "-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
> >> -XX:+UseCMSInitiatingOccupancyOnly"
> >>
> >
>

I had 25 sec CMF failure this morning...looks like bulk inserts are required
along with possibly weekly/daily scheduled rolling restarts. Do most
production clusters run rolling restarts on a regular basis to give the JVM
a fresh start?

Thanks.

On Thu, Jun 2, 2011 at 1:56 PM, Wayne <[EMAIL PROTECTED]> wrote:

> JVM w/ 10g Heap settings below. Once we are "bored" with stability we will
> try to up the 65 to 70 which seems to be standard.
>
> -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=65
> -XX:+CMSParallelRemarkEnabled -XX:+UseConcMarkSweepGC -XX:NewSize=128m
> -XX:MaxNewSize=128m -XX:+UseParNewG
>
> Our Memstore settings are default lower/upper is .35/.4.
>
> We do not currently use block cache. That may change in the future...
>
> Our table block size is 256k to keep the store file index size down. It was
> averaging ~2.75G+ per node and now has gone down to ~1.5G which should go
> even lower as regions are compacted.
>
> We have enabled the memstore MSLAB option. Not sure it is relevant but we
> have a 5G region size and 256m memstore flush size.
>
> Thanks.
>
>
>
> On Thu, Jun 2, 2011 at 11:48 AM, Stack <[EMAIL PROTECTED]> wrote:
>
>> Thanks for writing back to the list Wayne.  Hopefully this message
>> hits you before the next CMF does.  Would you mind pasting your final
>> JVM args and any other configs you think one of us could use writing
>> up your war story for the 'book' as per Jeff Whiting's suggestion?
>>
>> Good stuff,
>> St.Ack
>>
>>
>> On Thu, Jun 2, 2011 at 8:09 AM, Wayne <[EMAIL PROTECTED]> wrote:
>> > I have finally been able to spend enough time to digest/test
>> > all recommendations and get this under control. I wanted to thank Stack,
>> > Jack Levin, and Ted Dunning for their input.
>> >
>> > Basically our memory was being pushed to the limit and the JVM does not
>> > like/can not handle this. We are successfully using Todd's MSLAB enabled
>> on
>> > u25 and have set parnew to 128m, increased the heap to 10g, and
>> increased
>> > our block size 4x to 256k to reduce the size of the store file index
>> (thanks
>> > Jack for pointing this out). The combination of all of these changes
>> reduce
>> > significantly the pressure on memory and now we are getting more
>> throughput
>> > (40k writes/sec/node) sustained with no CMF errors (I expect to see one
>> 5
>> > min after hitting send on this email...).
>> >
>> > We are also going to move to bulk inserts where we can, which should
>> help as
>> > well. The decreased performance of reads due to the block size increase
>> is
>> > going to be the next challenge. Thanks everyone for your help, I am
>> > a believer again.
>> >
>> >
>> > On Fri, May 27, 2011 at 12:17 AM, Erik Onnen <[EMAIL PROTECTED]> wrote:
>> >
>> >> On Thu, May 26, 2011 at 11:01 AM, Stack <[EMAIL PROTECTED]> wrote:
>> >> > What JVM configs are you running Erik?
>> >> > St.Ack
>> >>
>> >> Omitting some of the irrelevant ones...
>> >>
>> >> JAVA_OPTS="-XX:+UseLargePages -Xms8192M -Xmx8192M
>> >> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseParNewGC
>> >> -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
>> >> -Xloggc:/mnt/services/hbaseregion/var/log/hbaseregion-gc.log
>> >> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
>> >>
>> >> For our Cassandra servers we also explicitly pin the new gen tenuring
>> >> and occupancy thresholds but that hasn't been necessary for the HBase
>> >> workloads so far:
>> >>
>> >> "-XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75
>> >> -XX:+UseCMSInitiatingOccupancyOnly"
>> >>
>> >
>>
>
>

On Mon, Jun 6, 2011 at 10:06 AM, Wayne <[EMAIL PROTECTED]> wrote:
> I had 25 sec CMF failure this morning...looks like bulk inserts are required
> along with possibly weekly/daily scheduled rolling restarts. Do most
> production clusters run rolling restarts on a regular basis to give the JVM
> a fresh start?
>

We don't do it (maybe we should!).   Here is our bit of doc. on the
decommission script: http://hbase.apache.org/book/decommission.html
Its been working well for us; i.e. config. changes and upgrades while
under load.

St.Ack

We have two production clusters, and we don't on either. We also have
days and days worth of no CMF reported.

Here is my config that works great for us:

export HBASE_OPTS="$HBASE_OPTS -XX:+UseConcMarkSweepGC
-XX:MaxDirectMemorySize=2G"

# Uncomment below to enable java garbage collection logging.
export HBASE_OPTS="$HBASE_OPTS -verbose:gc -Xms12000m
-XX:CMSInitiatingOccupancyFraction=70 -XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-XX:+HeapDumpOnOutOfMemoryError -Xloggc:$HBASE_HOME/logs/gc-hbase.log \
-XX:MaxTenuringThreshold=15 -XX:SurvivorRatio=8 \
-XX:+UseParNewGC \
-XX:NewSize=128m -XX:MaxNewSize=128m \
-XX:+CMSParallelRemarkEnabled \
-XX:-TraceClassUnloading
"

Also, just reduce the size of your RAM flushes to minimum, we are
running on 0.19 and 0.20 for lower to high memstore limits, so our
flushes are usually small enough not to cause major fragmentation
issues.

-Jack

On Mon, Jun 6, 2011 at 10:24 AM, Stack <[EMAIL PROTECTED]> wrote:
> On Mon, Jun 6, 2011 at 10:06 AM, Wayne <[EMAIL PROTECTED]> wrote:
>> I had 25 sec CMF failure this morning...looks like bulk inserts are required
>> along with possibly weekly/daily scheduled rolling restarts. Do most
>> production clusters run rolling restarts on a regular basis to give the JVM
>> a fresh start?
>>
>
> We don't do it (maybe we should!).   Here is our bit of doc. on the
> decommission script: http://hbase.apache.org/book/decommission.html
> Its been working well for us; i.e. config. changes and upgrades while
> under load.
>
> St.Ack
>