Hi,
I found the following in
https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92-security/lastCompletedBuild/console
:

Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
Exception in thread "ThreadedStreamConsumer"
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuffer.append(StringBuffer.java:224)
at org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201)
at org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139)
at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:112)
Running org.apache.hadoop.hbase.master.TestMasterFailover
Exception in thread "ThreadedStreamConsumer"
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuffer.append(StringBuffer.java:224)
at org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201)
at org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139)

Currently each unit test is given 1.9GB of heap.
I think we should give more heap to the tests.

Cheers

Usually IME this means that some test is generating many GB of logs,
etc... before raising heap let's look and see if some of the tests are
being excessively noisy

-Todd

On Tue, Nov 22, 2011 at 2:01 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> Hi,
> I found the following in
> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92-security/lastCompletedBuild/console
> :
>
> Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
> Exception in thread "ThreadedStreamConsumer"
> java.lang.OutOfMemoryError: Java heap space
>        at java.util.Arrays.copyOf(Arrays.java:2882)
>        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
>        at java.lang.StringBuffer.append(StringBuffer.java:224)
>        at org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201)
>        at org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139)
>        at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:112)
> Running org.apache.hadoop.hbase.master.TestMasterFailover
> Exception in thread "ThreadedStreamConsumer"
> java.lang.OutOfMemoryError: Java heap space
>        at java.util.Arrays.copyOf(Arrays.java:2882)
>        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
>        at java.lang.StringBuffer.append(StringBuffer.java:224)
>        at org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201)
>        at org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139)
>
> Currently each unit test is given 1.9GB of heap.
> I think we should give more heap to the tests.
>
> Cheers
>

--
Todd Lipcon
Software Engineer, Cloudera

The interesting thing is that their counterparts in HBase-0.92 build didn't
run out of heap :-)

On Tue, Nov 22, 2011 at 2:05 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:

> Usually IME this means that some test is generating many GB of logs,
> etc... before raising heap let's look and see if some of the tests are
> being excessively noisy
>
> -Todd
>
> On Tue, Nov 22, 2011 at 2:01 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > Hi,
> > I found the following in
> >
> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92-security/lastCompletedBuild/console
> > :
> >
> > Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
> > Exception in thread "ThreadedStreamConsumer"
> > java.lang.OutOfMemoryError: Java heap space
> >        at java.util.Arrays.copyOf(Arrays.java:2882)
> >        at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> >        at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> >        at java.lang.StringBuffer.append(StringBuffer.java:224)
> >        at
> org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201)
> >        at
> org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139)
> >        at
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:112)
> > Running org.apache.hadoop.hbase.master.TestMasterFailover
> > Exception in thread "ThreadedStreamConsumer"
> > java.lang.OutOfMemoryError: Java heap space
> >        at java.util.Arrays.copyOf(Arrays.java:2882)
> >        at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> >        at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> >        at java.lang.StringBuffer.append(StringBuffer.java:224)
> >        at
> org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201)
> >        at
> org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139)
> >
> > Currently each unit test is given 1.9GB of heap.
> > I think we should give more heap to the tests.
> >
> > Cheers
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

The heap space error happened again for build #9

TestMasterFailover hung in build 9 too.

On Tue, Nov 22, 2011 at 2:07 PM, Ted Yu <[EMAIL PROTECTED]> wrote:

> The interesting thing is that their counterparts in HBase-0.92 build
> didn't run out of heap :-)
>
>
> On Tue, Nov 22, 2011 at 2:05 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>
>> Usually IME this means that some test is generating many GB of logs,
>> etc... before raising heap let's look and see if some of the tests are
>> being excessively noisy
>>
>> -Todd
>>
>> On Tue, Nov 22, 2011 at 2:01 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> > Hi,
>> > I found the following in
>> >
>> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92-security/lastCompletedBuild/console
>> > :
>> >
>> > Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
>> > Exception in thread "ThreadedStreamConsumer"
>> > java.lang.OutOfMemoryError: Java heap space
>> >        at java.util.Arrays.copyOf(Arrays.java:2882)
>> >        at
>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>> >        at
>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
>> >        at java.lang.StringBuffer.append(StringBuffer.java:224)
>> >        at
>> org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201)
>> >        at
>> org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139)
>> >        at
>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:112)
>> > Running org.apache.hadoop.hbase.master.TestMasterFailover
>> > Exception in thread "ThreadedStreamConsumer"
>> > java.lang.OutOfMemoryError: Java heap space
>> >        at java.util.Arrays.copyOf(Arrays.java:2882)
>> >        at
>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>> >        at
>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
>> >        at java.lang.StringBuffer.append(StringBuffer.java:224)
>> >        at
>> org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201)
>> >        at
>> org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139)
>> >
>> > Currently each unit test is given 1.9GB of heap.
>> > I think we should give more heap to the tests.
>> >
>> > Cheers
>> >
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>

On Wed, Nov 23, 2011 at 8:53 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> The heap space error happened again for build #9
>
> TestMasterFailover hung in build 9 too.
>

Yes.  You mean the pom setting?  Want to make an issue Ted w/ patch?
(Maybe more crap going on when security in the mix pushing is over
edge; we should dig in on Todd suggestion but I'd say till then up the
build heap).

St.Ack

> On Tue, Nov 22, 2011 at 2:07 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> The interesting thing is that their counterparts in HBase-0.92 build
>> didn't run out of heap :-)
>>
>>
>> On Tue, Nov 22, 2011 at 2:05 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>
>>> Usually IME this means that some test is generating many GB of logs,
>>> etc... before raising heap let's look and see if some of the tests are
>>> being excessively noisy
>>>
>>> -Todd
>>>
>>> On Tue, Nov 22, 2011 at 2:01 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>> > Hi,
>>> > I found the following in
>>> >
>>> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92-security/lastCompletedBuild/console
>>> > :
>>> >
>>> > Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
>>> > Exception in thread "ThreadedStreamConsumer"
>>> > java.lang.OutOfMemoryError: Java heap space
>>> >        at java.util.Arrays.copyOf(Arrays.java:2882)
>>> >        at
>>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>>> >        at
>>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
>>> >        at java.lang.StringBuffer.append(StringBuffer.java:224)
>>> >        at
>>> org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201)
>>> >        at
>>> org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139)
>>> >        at
>>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:112)
>>> > Running org.apache.hadoop.hbase.master.TestMasterFailover
>>> > Exception in thread "ThreadedStreamConsumer"
>>> > java.lang.OutOfMemoryError: Java heap space
>>> >        at java.util.Arrays.copyOf(Arrays.java:2882)
>>> >        at
>>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>>> >        at
>>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
>>> >        at java.lang.StringBuffer.append(StringBuffer.java:224)
>>> >        at
>>> org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201)
>>> >        at
>>> org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139)
>>> >
>>> > Currently each unit test is given 1.9GB of heap.
>>> > I think we should give more heap to the tests.
>>> >
>>> > Cheers
>>> >
>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>>
>>
>

https://issues.apache.org/jira/browse/HBASE-4856 has been created.

Shall we upgrade hadoop version for hadoop-0.20 to 0.20.205.1-7070-SNAPSHOT
as well ?

On Wed, Nov 23, 2011 at 9:00 AM, Stack <[EMAIL PROTECTED]> wrote:

> On Wed, Nov 23, 2011 at 8:53 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> > The heap space error happened again for build #9
> >
> > TestMasterFailover hung in build 9 too.
> >
>
> Yes.  You mean the pom setting?  Want to make an issue Ted w/ patch?
> (Maybe more crap going on when security in the mix pushing is over
> edge; we should dig in on Todd suggestion but I'd say till then up the
> build heap).
>
> St.Ack
>
> > On Tue, Nov 22, 2011 at 2:07 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >
> >> The interesting thing is that their counterparts in HBase-0.92 build
> >> didn't run out of heap :-)
> >>
> >>
> >> On Tue, Nov 22, 2011 at 2:05 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
> >>
> >>> Usually IME this means that some test is generating many GB of logs,
> >>> etc... before raising heap let's look and see if some of the tests are
> >>> being excessively noisy
> >>>
> >>> -Todd
> >>>
> >>> On Tue, Nov 22, 2011 at 2:01 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
> >>> > Hi,
> >>> > I found the following in
> >>> >
> >>>
> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92-security/lastCompletedBuild/console
> >>> > :
> >>> >
> >>> > Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
> >>> > Exception in thread "ThreadedStreamConsumer"
> >>> > java.lang.OutOfMemoryError: Java heap space
> >>> >        at java.util.Arrays.copyOf(Arrays.java:2882)
> >>> >        at
> >>>
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> >>> >        at
> >>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> >>> >        at java.lang.StringBuffer.append(StringBuffer.java:224)
> >>> >        at
> >>>
> org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201)
> >>> >        at
> >>>
> org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139)
> >>> >        at
> >>>
> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:112)
> >>> > Running org.apache.hadoop.hbase.master.TestMasterFailover
> >>> > Exception in thread "ThreadedStreamConsumer"
> >>> > java.lang.OutOfMemoryError: Java heap space
> >>> >        at java.util.Arrays.copyOf(Arrays.java:2882)
> >>> >        at
> >>>
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> >>> >        at
> >>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> >>> >        at java.lang.StringBuffer.append(StringBuffer.java:224)
> >>> >        at
> >>>
> org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201)
> >>> >        at
> >>>
> org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139)
> >>> >
> >>> > Currently each unit test is given 1.9GB of heap.
> >>> > I think we should give more heap to the tests.
> >>> >
> >>> > Cheers
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Todd Lipcon
> >>> Software Engineer, Cloudera
> >>>
> >>
> >>
> >
>

>
> Shall we upgrade hadoop version for hadoop-0.20 to 0.20.205.1-7070-SNAPSHOT
> as well ?
>

This is an unreleased version that's only necessary for ZooKeeper SASL
authentication using jaas.conf (needs HADOOP-7070).

I think it's better for the core build to remain on a released version
of Hadoop.

I think this might be caused by:
https://issues.apache.org/jira/browse/HBASE-4857

There is a recursive loop on the leader
AuthenticationTokenSecretManger when a KeeperException is encountered.

On Wed, Nov 23, 2011 at 8:53 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
> The heap space error happened again for build #9
>
> TestMasterFailover hung in build 9 too.
>
> On Tue, Nov 22, 2011 at 2:07 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>
>> The interesting thing is that their counterparts in HBase-0.92 build
>> didn't run out of heap :-)
>>
>>
>> On Tue, Nov 22, 2011 at 2:05 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>
>>> Usually IME this means that some test is generating many GB of logs,
>>> etc... before raising heap let's look and see if some of the tests are
>>> being excessively noisy
>>>
>>> -Todd
>>>
>>> On Tue, Nov 22, 2011 at 2:01 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>> > Hi,
>>> > I found the following in
>>> >
>>> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92-security/lastCompletedBuild/console
>>> > :
>>> >
>>> > Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
>>> > Exception in thread "ThreadedStreamConsumer"
>>> > java.lang.OutOfMemoryError: Java heap space
>>> >        at java.util.Arrays.copyOf(Arrays.java:2882)
>>> >        at
>>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>>> >        at
>>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
>>> >        at java.lang.StringBuffer.append(StringBuffer.java:224)
>>> >        at
>>> org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201)
>>> >        at
>>> org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139)
>>> >        at
>>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:112)
>>> > Running org.apache.hadoop.hbase.master.TestMasterFailover
>>> > Exception in thread "ThreadedStreamConsumer"
>>> > java.lang.OutOfMemoryError: Java heap space
>>> >        at java.util.Arrays.copyOf(Arrays.java:2882)
>>> >        at
>>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>>> >        at
>>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
>>> >        at java.lang.StringBuffer.append(StringBuffer.java:224)
>>> >        at
>>> org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201)
>>> >        at
>>> org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139)
>>> >
>>> > Currently each unit test is given 1.9GB of heap.
>>> > I think we should give more heap to the tests.
>>> >
>>> > Cheers
>>> >
>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>>
>>
>

It also looks like TestMasterFailover isn't cleaning up after itself.
Each test method just does TEST_UTIL.shutdownMiniCluster() at the end,
without use of a finally block.  So it wouldn't get called in the case
of an exception.  Testing a patch for that now.

On Wed, Nov 23, 2011 at 9:52 AM, Gary Helmling <[EMAIL PROTECTED]> wrote:
> I think this might be caused by:
> https://issues.apache.org/jira/browse/HBASE-4857
>
> There is a recursive loop on the leader
> AuthenticationTokenSecretManger when a KeeperException is encountered.
>
>
>
> On Wed, Nov 23, 2011 at 8:53 AM, Ted Yu <[EMAIL PROTECTED]> wrote:
>> The heap space error happened again for build #9
>>
>> TestMasterFailover hung in build 9 too.
>>
>> On Tue, Nov 22, 2011 at 2:07 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>
>>> The interesting thing is that their counterparts in HBase-0.92 build
>>> didn't run out of heap :-)
>>>
>>>
>>> On Tue, Nov 22, 2011 at 2:05 PM, Todd Lipcon <[EMAIL PROTECTED]> wrote:
>>>
>>>> Usually IME this means that some test is generating many GB of logs,
>>>> etc... before raising heap let's look and see if some of the tests are
>>>> being excessively noisy
>>>>
>>>> -Todd
>>>>
>>>> On Tue, Nov 22, 2011 at 2:01 PM, Ted Yu <[EMAIL PROTECTED]> wrote:
>>>> > Hi,
>>>> > I found the following in
>>>> >
>>>> https://builds.apache.org/view/G-L/view/HBase/job/HBase-0.92-security/lastCompletedBuild/console
>>>> > :
>>>> >
>>>> > Running org.apache.hadoop.hbase.master.TestDistributedLogSplitting
>>>> > Exception in thread "ThreadedStreamConsumer"
>>>> > java.lang.OutOfMemoryError: Java heap space
>>>> >        at java.util.Arrays.copyOf(Arrays.java:2882)
>>>> >        at
>>>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>>>> >        at
>>>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
>>>> >        at java.lang.StringBuffer.append(StringBuffer.java:224)
>>>> >        at
>>>> org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201)
>>>> >        at
>>>> org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139)
>>>> >        at
>>>> org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:112)
>>>> > Running org.apache.hadoop.hbase.master.TestMasterFailover
>>>> > Exception in thread "ThreadedStreamConsumer"
>>>> > java.lang.OutOfMemoryError: Java heap space
>>>> >        at java.util.Arrays.copyOf(Arrays.java:2882)
>>>> >        at
>>>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
>>>> >        at
>>>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
>>>> >        at java.lang.StringBuffer.append(StringBuffer.java:224)
>>>> >        at
>>>> org.apache.maven.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:201)
>>>> >        at
>>>> org.apache.maven.surefire.report.TestSetRunListener.testError(TestSetRunListener.java:139)
>>>> >
>>>> > Currently each unit test is given 1.9GB of heap.
>>>> > I think we should give more heap to the tests.
>>>> >
>>>> > Cheers
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Todd Lipcon
>>>> Software Engineer, Cloudera
>>>>
>>>
>>>
>>
>